Skip to content

gsoc_2011_interpreter

https://www.google.com/accounts/o8/id?id=AItOawnIDK5ERNYzZQUJALcyMjUQXUyiYA_Tij8 edited this page Apr 23, 2011 · 21 revisions

In-browser PyJS Interpreter

Abstract

The purpose of this project is to re-factor and improve python-to-javascript translator component (pyjs) of Pyjamas in order to reach following goals:

  1. Translator should be able to translate itself, and resulting javascript should be able to correctly translate python code into javascript code while being run in browser environment, turning pyjs into proper python implementation.
  2. Remove deprecated interfaces and provide clean, extensible API for Pyjamas utilities and 3rd party applications/libraries which use pyjs translator, supporting small DSL use-cases in addition to current web applications.
  3. Research and attempt to provide support for translating python 3

In-browser interpreting

Providing ability for PyJS to compile itself is one of long-term goals of Pyjamas project and fulfilling it will significantly ease of debugging of code, developing new widgets and higher-level components, and will be the sign of project maturity. It will allow to further optimize produced code size without sacrificing compatibility, which is one of current critical issues. Immediate benefits would also include ability to implement eval()/exec()/compile()/execfile()/input() functions, and implement importing of raw python code instead of pre-compiled js modules.

API and utilities update

Current Translator class is tailored to module-level translation. This severely limits possible use-cases and should be resolved in order to support eval() and make pyjs more friendly to 3rd party tools. Proposed changes include:

  • Remove all older Translator classes
  • Divide Translator.__init__ method into several, ensuring it can be easily subclassed
  • Include support of statement and expression level translations into the class, with ability to specify global and local context of statements to be translated.
  • Provide easy way to change header and footer code generated for each module/function

Python 3 support

Providing python 3 support in PyJS will be first step to converting Pyjamas to python 3, and would allow developers to start making python 3 widgets for pyjamas. While complete support for whole pyjamas project is major undertaking and is out of scope for GSoC, translator itself may be easiest part, and its good idea to provide as much support as possible while doing rewrite. Some subset of python 3 may be supported with current pyjslib by compiling python 3 AST with python 2 pyjslib, since many changes are purely syntactic. Lib3to2 experience may be used to evaluate extent of this approach.

Timeline and milestones

May 10 - May 23

Conduct research on python 3 support:

  • Evaluate changes needed to pyjs to produce compatible AST for python 3 syntax
  • Evaluate changes needed to pyjs to generate correct JavaScript for python 3 AST
  • Evaluate changes needed to pyjslib to execute produced JavaScript and find best approach to minimize such changes
  • Evaluate possibility of executing pyjs with python 3 interpreters, including subsequent in-browser pyjs execution.
  • Provide report on extent of work on python 3 support feasible in terms of this GSoC project and correct this plan accordingly.

Start of Program (May 24)

There is no specific preparation work needed besides python 3 support research since project goals were already discussed extensively in mailing list prior to GSoC

May 24 - July 11

  • May 24 - May 30: Do basic clean-up and API work for translator/builder/linker
  • May 31 - June 6: First pass on planned python 3 improvements
  • June 7 - June 13: Make python parser to pass pyjs compilation
  • June 14 - June 20: Make pyjs itself pass pyjs compilation
  • June 21 - June 27: Develop utility to test code parsing and compilation by pyjs executed by PyV8
  • June 28 - July 4: Perform testing of subsecuent compilations of pyjs by itself and do other parser/compiler/linker testing.
  • July 5 - July 11: Document changes, prepare to mid-term evaluation

Midterm Evaluation (July 12)

  • At this stage PyJS executed by CPython should compile parser and itself into valid JavaScript
  • Compiled into JavaScript PyJS should be able to compile original python sources of PyJS and parser into identical JavaScript when executed via PyV8.

July 13 - Aug 22

  • July 13 - July 18: Implement statement/expression level compilation and interpretation, implement compile(), eval(), exec(), execfile() APIs to interpreter
  • July 19 - July 25: Implement module importing using original python code without prior compilation
  • July 26 - Aug 1: Update all CLI utilities
  • Aug 2 - Aug 8: Extend testing to include compiling all tests in run-time by in-browser PyJS
  • Aug 9 - Aug 15: Test whole Pyjamas stack according to QA policy
  • Aug 16 - Aug 22: Merge code into master branch, finish documentation, prepare for final evaluation.

Final Evaluation (Aug 23)

  • At this stage PyJS GSoC branch should be merged back into master branch.
  • PyJS should be documented on Pyjamas Wiki.
  • Pyjamas test plan should be extended to include execution of whole test suite by compiled-in-browser PyJS.
  • All changes should be thoroughly tested according to Pyjamas QA policy.

Bugs blocking compilation of pyjs

Special considerations

Decoupling pyjs from pyjamas

  • It tries to import whole pyjamas library even for hello world.
  • Should be possible to install translator into python and depend on it for 3rd party packages

Forking lib2to3/3to2?

  • pgen is already forked into pyjamas repository, but since footprint of compiled js is critical, should be further investigated about cutting out unused parts.

globals()/locals()

(needs verification) globals() is just a reference to $m

locals() is tough. Should be preliminarily tackled before program start as this may be quite hard sub-task

expression/statement-level code compilation

Translator should be able to build all needed namespace-related data from locals/globals provided in the call.

Expression-level translation MUST result with valid JS expression, which could be assigned to variable.

(Part-bonus) Importing

Should support several scenarios:

  1. Compile-time reading of modules for .py.js output. Resolves paths, but does not execute modules.
  2. Modules loading for pyv8/pysm platforms.
  3. Run-time import at browser platforms (either 'import' already loaded module or kick in dynamic loading)
  4. Run-time import at pyv8/pysm platforms.

Dynamic:

  • Start with simple synchronous XHR.
  • Provide some web service for batch loading of modules

Need to check importlib implementation, and pypy import implementation.

Our one should be both compliant to PEPs and reference implementation and sufficiently handle all 4 scenarios.

Most of it can be omitted for GSoC, depends on primary deliverables.

(Part-bonus) Tests for all CLI tools

(Bonus) reload() bltin

(Bonus) semantically compatible to /usr/bin/python CLI tool (pyjsv8?)

Implementation of CLI tool compliant to typical python interpreter (like /usr/bin/python), which could both run shell and run files.

Should support all major command-line switches, including typical -mtimeit invocation.

Should be compatible with python benchmarking suite (usable as python implementation)

Import mess has to be solved first.

Something went wrong with that request. Please try again.