Skip to content

Commit

Permalink
Static Python (#5392)
Browse files Browse the repository at this point in the history
* Move conversion to worker

* Allow .md and .py files in "Add new file"

* Include markdown and python syntax in monaco

* Account for py files

* Add missing markdown/py monaco support

* Add pyprj support

* Pass preferred editor to compiler in sim

* Do a triple toggle for now

* Make the triple-toggle work

* add support for python in experiments

* Pass apiInfo to converter

* Add preferredEditor to pxt.json

* Hook up preferred editor logic

* Hooking up error reporting

* Start on reflecting TS in PY

* More work on importing TS apis

* Build fix

* Start switching to SymbolInfo from py.AST for global symbols

* More friendly type strings in APIs info

* Better TS type parsing

* Add debug target

* Export constructors in APIs info

* Work on functions

* Handle array and module lookups

* Cleanup

* Mark optional arguments with "undefined" as initializer

* Typecheck calls

* starting pydecompiler tests by duping block decompiler tests (for now)

* wip on python decompiler outer infra

* start with ts as baseline template

* can diff baseline with test case

* Add test program

* starting to discrimiate on statement types

* Improve parameter type inference

* Symbol type cleanup

* Remove some data duplication

* Get rid of FieldDesc

* More example

* wip on decompiler

* new game test file

* add a dope file watcher helper script for continuous testing

* progress on decompiler

* progress on classes

* handles most of game.py

* handle blank lines

* handle if statement

* detect enum classes

* remove dead code

* better newline handling

* wip

* remove unported test cases

* make the webapp work again

* Skip apiInfos from main.ts

* Remove VarDesc

* Run conversion until fixpoint of type state

* Test a few more features

* handle anonymous functions

* starting for statement; added (disabled) test case for shadowing

* handle simple "for in range(x, y)" loops; handle unary exp; handle while loop

* handle more expressions and operators

* cleanup

* handle else if

* add 4 more tests; handle constructors

* Support len(); deal with access levels and inheritance

* handle new; handle for (;;); blacklist shadowing tests for now

* handle weird for loops; passing 13 tests

* handle multiline literals

* Move apiInfo logic to service.ts

* add python experiment

* revert experiment

* handle undefined; handle arrow fns

* handle >>, <<

* handle for..of

* handle .length -> len() improperly

* safer handling of ++ --

* unary ops

* determine const expr; more flexible for-in-range handling

* handle if else empty body

* handle !=

* handle comments

* handle some math fns

* runner improvements

* visitExp helper; better ++ and --

* more plubming

* adds many pydecompiler tests; passing 59 failing 61

* basic ts -> py plumbing

* make showConversionFailedDialog aware of python

* add loader dialog

* basic enum support

* fix saving logic

* fixing py->blocks

* re-enable toolbox for python

* basic toolbox support in python

* added python to docs

* Start on diags

* fixing docs issues

* Pipe diagnostics up; introduce error numbers

* Return conversion diagnostics in allDiags

* fix snippet highlighting

* linting

* don't crash on decompilation, log error

* few more toolbox fixes

* handle enum

* handle tagged templates; compiles falling duck!

* add falling duck test!

* fix escaping issues

* added analytics

* disable sig in python

* Make sure to save failed conversion result

* Support for tagged templates

* Tagged template test

* Test b'...' literal

* handle console.log

* emit init types

* use correct icon for blocks

* add loader dialog when switching editors

* adding a quick note about static python

* adding experiment url

* bump to 5.7

* typo

* removed hardcoded python

* 5.7.1

* enable squiggles

* moving interfaces in pxtarget.d.ts (#5391)

* moving interfaces in pxtarget.d.ts

* const enum

* fixing pxtpy.js build dependency

* fix conversion error message

* 5.7.2

* Fix column offset

* Pass up parser diagnostics

* implement shortcut when ts unmodified

* 5.7.3

* Better parser error recovery

* Cleanup

* Friendlier operator names

* decompiler cleanup

* cleanup

* handle empty while loop

* passing program info into pydecompiler

* better state management

* handle string concat

* spelling

* handle return stmt

* fix blocks -> py conversion

* Py field editors (#5394)

* passing filetype in field editor interface

* fix regex

* generate python when needed

* 5.7.4

* Remove space in tagged literals

* Monaco toolbox for python (#5397)

* moving snippet generation in service

* fix py generation

* simplify insertion logic

* liniting

* more linting

* move snippet generation

* dead code

* fix tests

* moving snippet generation to service and on demand

* fixes

* lint

* 5.7.5

* Remove space after hex in translation of b'...'

* track killed message accross restarts. Fix for https://github.com/Microsoft/pxt/issues/5398

* emit lambda fns

* progres on fns

* update template tests

* when a file is empty, use empty line instead of space

* handle switching between empty files

* add a new test for scopes

* rename

* unify block handling

* 5.7.6

* Add snakeifyied names in SymbolInfo

* Fix merge

* Fixing name lookup (and function calls)

* 5.7.7

* remove outdated api

* Snakeify APIs

* snakify toolbox

* 5.7.8

* completion plumbing

* guard

* use blocks decompiler renameMap; improve scope stack; more tests for var scope edge cases

* 5.7.9

* add some dialog when conversion fails

* Make the top-level (non-member) completion kind-of work

* Adding member completion

* fix for debug to start from python (#5406)

* 5.7.10

* more filtering

* fixing 2 crashes in updateToolbox

* 5.7.11

* fix exmaple load issue

* Tutorial tracking (#5409)

* track tutorial state in header

* towards rendering tutorial progress

* hookup

* fix rendering

* color

* reload steps when open script

* 5.7.12

* better sorting + dummy servies

* adding support for snippet info in comments

* Hook up locals conversion

* default image to 16x16

* Implement basic hover provider

* Start on signature helper

* 5.7.13

* add option to skip localization of particular packages

* 5.7.14

* move pydecompiler into pxtpy

* add TODO

* typo

* 5.7.15

* clarifyin comments & TODOs

* move watch .sh script into test folder

* rename

* add fswatch guard

* cleanup TODOs in pydecompiler

* start builtinsMap.ts with some thoughts on mapping between python and ts

* more thoughts on handling builtins

* de-duplicate test cases with blocks decompiler tests (removes ~120 unnecessary files)

* updated image opt script

* convert gifs to mp4

* overwrite mp4

* comment out mp4 generation

* fix double sim on share

* 5.7.16

* updated error message

* 5.7.17

* snaking in toolbox

* add note about function name @darzu

* support for ```spy in docs

* 5.7.18

* added spy macro
  • Loading branch information
pelikhan committed Apr 3, 2019
1 parent 2e2f5f9 commit 7e4dd3e
Show file tree
Hide file tree
Showing 207 changed files with 10,120 additions and 971 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -24,3 +24,4 @@ paper.dvi
paper.fls
paper.fdb_latexmk
paper.aux
tests/**/*.local.*
21 changes: 21 additions & 0 deletions .vscode/launch.json
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,27 @@
"sourceMaps": false,
"outFiles": []
},
{
"name": "pybuild",
"type": "node",
"request": "launch",
"program": "${workspaceRoot}/built/pxt.js",
"stopOnEntry": false,
"args": [
"build"
],
"cwd": "${workspaceRoot}/../pxt-arcade/projects/py",
"runtimeExecutable": null,
"runtimeArgs": [
"--nolazy"
],
"env": {
"NODE_ENV": "development"
},
"console": "integratedTerminal",
"sourceMaps": false,
"outFiles": []
},
{
"name": "console",
"type": "node",
Expand Down
16 changes: 14 additions & 2 deletions Jakefile.js
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ function setupTest(taskName, testFolder, testFile) {
"node_modules/typescript/lib/typescript.js",
"built/pxtlib.js",
"built/pxtcompiler.js",
"built/pxtpy.js",
"built/pxtsim.js",
"built/tests/" + testFolder + "/" + testFile,
],
Expand Down Expand Up @@ -95,6 +96,7 @@ setupTest('testdecompiler', 'decompile-test', 'decompilerunner.js')
setupTest('testlang', 'compile-test', 'compilerunner.js')
setupTest('testerr', 'errors-test', 'errorrunner.js')
setupTest('testfmt', 'format-test', 'formatrunner.js')
setupTest('testpydecompiler', 'pydecompile-test', 'pydecompilerunner.js')


task('testpkgconflicts', ['built/pxt.js'], { async: true }, function () {
Expand All @@ -105,6 +107,7 @@ ju.catFiles('built/pxt.js', [
"node_modules/typescript/lib/typescript.js",
"built/pxtlib.js",
"built/pxtcompiler.js",
"built/pxtpy.js",
"built/pxtsim.js",
"built/cli.js"
],
Expand Down Expand Up @@ -138,13 +141,14 @@ file('built/pxt-common.json', expand(['libs/pxt-common'], ".ts"), function () {

compileDir("pxtlib", "built/typescriptServices.d.ts")
compileDir("pxtcompiler", ["built/pxtlib.js"])
compileDir("pxtpy", ["built/pxtcompiler.js"])
compileDir("pxtwinrt", ["built/pxtlib.js"])
compileDir("pxtblocks", ["built/pxtlib.js", "built/pxtsim.js", "built/pxtcompiler.js"])
ju.catFiles("built/pxtblockly.js", expand(["webapp/public/blockly/blockly_compressed.js", "webapp/public/blockly/blocks_compressed.js", "webapp/public/blockly/msg/js/en.js", "built/pxtblocks.js"]), "")
compileDir("pxtrunner", ["built/pxtlib.js", "built/pxteditor.js", "built/pxtcompiler.js", "built/pxtsim.js", "built/pxtblockly.js"])
compileDir("pxtsim", ["built/pxtlib.js"])
compileDir("pxteditor", ["built/pxtlib.js", "built/pxtblockly.js"])
compileDir("cli", ["built/pxtlib.js", "built/pxtsim.js", "built/pxtcompiler.js"])
compileDir("cli", ["built/pxtlib.js", "built/pxtsim.js", "built/pxtcompiler.js", "built/pxtpy.js"])
compileDir("backendutils", ['pxtlib/commonutil.ts', 'pxtlib/docsrender.ts'])
file("built/web/pxtweb.js", expand(["docfiles/pxtweb"]), { async: true }, function () { tscIn(this, "docfiles/pxtweb", "built") })

Expand Down Expand Up @@ -187,6 +191,7 @@ task("lint", [], { async: true }, function () {
"pxteditor",
"pxtlib",
"pxtcompiler",
"pxtpy",
"pxtrunner",
"pxtsim",
"pxtwinrt",
Expand Down Expand Up @@ -282,6 +287,7 @@ file('built/localization.json', ju.expand1(
task('wapp', [
"built/web/pxtlib.js",
"built/web/pxtcompiler.js",
"built/web/pxtpy.js",
"built/web/pxtsim.js",
"built/web/pxtblockly.js",
"built/web/pxteditor.js",
Expand All @@ -302,6 +308,7 @@ task('wapp', [
file("built/web/pxtlib.js", [
"built/pxtlib.js",
"built/pxtcompiler.js",
"built/pxtpy.js",
"built/pxtblockly.js",
"built/pxtsim.js",
"built/pxtrunner.js",
Expand All @@ -315,6 +322,7 @@ file("built/web/pxtlib.js", [

jake.cpR("built/pxtlib.js", "built/web/")
jake.cpR("built/pxtcompiler.js", "built/web/")
jake.cpR("built/pxtpy.js", "built/web/")
jake.cpR("built/pxtblocks.js", "built/web/")
jake.cpR("built/pxtblockly.js", "built/web/")
jake.cpR("built/pxtsim.js", "built/web/")
Expand Down Expand Up @@ -410,6 +418,8 @@ file('built/web/vs/editor/editor.main.js', ['node_modules/pxt-monaco-typescript/
jake.mkdirP("webapp/public/vs/basic-languages/src")
jake.cpR("node_modules/monaco-editor/min/vs/basic-languages/src/bat.js", "webapp/public/vs/basic-languages/src/")
jake.cpR("node_modules/monaco-editor/min/vs/basic-languages/src/cpp.js", "webapp/public/vs/basic-languages/src/")
jake.cpR("node_modules/monaco-editor/min/vs/basic-languages/src/markdown.js", "webapp/public/vs/basic-languages/src/")
jake.cpR("node_modules/monaco-editor/min/vs/basic-languages/src/python.js", "webapp/public/vs/basic-languages/src/")
jake.mkdirP("webapp/public/vs/language/json")
jake.cpR("node_modules/monaco-editor/min/vs/language/json/", "webapp/public/vs/language/")

Expand Down Expand Up @@ -463,14 +473,16 @@ ju.catFiles('built/web/pxtworker.js', [
"node_modules/fuse.js/dist/fuse.min.js",
"node_modules/lzma/src/lzma_worker-min.js",
"built/web/pxtlib.js",
"built/web/pxtcompiler.js"
"built/web/pxtcompiler.js",
"built/web/pxtpy.js"
], `"use strict";`, ["built/webapp/src/app.js"]);

ju.catFiles('built/web/pxtembed.js', [
"built/web/typescript.js",
"node_modules/lzma/src/lzma_worker-min.js",
"built/web/pxtlib.js",
"built/web/pxtcompiler.js",
"built/web/pxtpy.js",
"built/web/pxtblockly.js",
"built/web/pxteditor.js",
"built/web/pxtsim.js",
Expand Down
33 changes: 30 additions & 3 deletions cli/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

/// <reference path="../built/pxtlib.d.ts"/>
/// <reference path="../built/pxtcompiler.d.ts"/>
/// <reference path="../built/pxtpy.d.ts"/>
/// <reference path="../built/pxtsim.d.ts"/>

(global as any).pxt = pxt;
Expand Down Expand Up @@ -33,6 +34,8 @@ let forceCloudBuild = process.env["KS_FORCE_CLOUD"] !== "no";
let forceLocalBuild = !!process.env["PXT_FORCE_LOCAL"];
let forceBuild = false; // don't use cache

Error.stackTraceLimit = 100;

function parseBuildInfo(parsed?: commandParser.ParsedCommand) {
const cloud = parsed && parsed.flags["cloudbuild"];
const local = parsed && parsed.flags["localbuild"];
Expand Down Expand Up @@ -3675,6 +3678,25 @@ function prepBuildOptionsAsync(mode: BuildOption, quick = false, ignoreTests = f
opts.ast = true
}

// this is suboptimal, but we need apisInfo for the python converter
if (opts.target.preferredEditor == pxt.PYTHON_PROJECT_NAME) {
pxt.log("pre-compiling apisInfo for Python")
const opts2 = U.clone(opts)
opts2.ast = true
opts2.target.preferredEditor = pxt.JAVASCRIPT_PROJECT_NAME
opts2.noEmit = true
// remove previously converted .ts files, so they don't end up in apisinfo
for (let f of opts2.sourceFiles) {
if (U.endsWith(f, ".py"))
opts2.fileSystem[f.slice(0, -3) + ".ts"] = " "
}
const res = pxtc.compile(opts2)
opts.apisInfo = pxtc.getApiInfo(opts2, res.ast)
if (process.env["PXT_SAVE_APISINFO"])
fs.writeFileSync("built/apisinfo.json", JSON.stringify(opts.apisInfo, null, 4))
pxt.log("done pre-compiling apisInfo for Python")
}

return opts;
})
}
Expand Down Expand Up @@ -5481,7 +5503,7 @@ PXT_ASMDEBUG - embed additional information in generated binary.asm file
aliases: ["r"]
},
"githubpages": {
description: "Generate a web site compatiable with GitHub pages",
description: "Generate a web site compatible with GitHub pages",
aliases: ["ghpages", "gh"]
},
"output": {
Expand Down Expand Up @@ -5708,9 +5730,14 @@ PXT_ASMDEBUG - embed additional information in generated binary.asm file
name: "pyconv",
help: "convert from python",
argString: "<package-directory> <support-directory>...",
anyArgs: true,
advanced: true,
}, c => pyconv.convertAsync(c.args))
flags: {
internal: {
description: "use internal Python parser",
aliases: ["i"]
}
}
}, c => pyconv.convertAsync(c.args, !!c.flags["internal"]))

advancedCommand("thirdpartynotices", "refresh third party notices", thirdPartyNoticesAsync);
p.defineCommand({
Expand Down
3 changes: 2 additions & 1 deletion cli/nodeutil.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ export interface SpawnOptions {
input?: string;
silent?: boolean;
envOverrides?: pxt.Map<string>;
allowNonZeroExit?: boolean;
}

//This should be correct at startup when running from command line
Expand Down Expand Up @@ -89,7 +90,7 @@ export function spawnWithPipeAsync(opts: SpawnOptions) {
}
})
ch.on('close', (code: number) => {
if (code != 0)
if (code != 0 && !opts.allowNonZeroExit)
reject(new Error("Exit code: " + code + " from " + info))
resolve(Buffer.concat(bufs))
});
Expand Down
Loading

0 comments on commit 7e4dd3e

Please sign in to comment.