diff --git a/README.rst b/README.rst new file mode 100644 index 0000000..14e9451 --- /dev/null +++ b/README.rst @@ -0,0 +1,7 @@ +================ +RestrictedPython +================ + +RestrictedPython is a defined subset of the Python language which allows to provide a program input into a trusted environment. + +For full documentation please see http://restrictedpython.readthedocs.io/en/python3_update/ or local docs/index. diff --git a/README.txt b/README.txt deleted file mode 100644 index 03610af..0000000 --- a/README.txt +++ /dev/null @@ -1 +0,0 @@ -Please refer to src/RestrictedPython/README.txt. diff --git a/CHANGES.rst b/docs/CHANGES.rst similarity index 100% rename from CHANGES.rst rename to docs/CHANGES.rst diff --git a/docs/conf.py b/docs/conf.py index 2edf26b..3f5aa55 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -29,8 +29,10 @@ # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = [ + 'sphinx.ext.autodoc', 'sphinx.ext.intersphinx', 'sphinx.ext.todo', + 'sphinx.ext.doctest', ] # Add any paths that contain templates here, relative to this directory. @@ -117,6 +119,10 @@ } +# Options for sphinx.ext.todo: +todo_include_todos = True +todo_emit_warnings = True + # -- Options for HTML output ---------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for diff --git a/docs/contributing/index.rst b/docs/contributing/index.rst new file mode 100644 index 0000000..62c3fb3 --- /dev/null +++ b/docs/contributing/index.rst @@ -0,0 +1,11 @@ +Contributing +============ + +Contributing to RestrictedPython 4+ + + + +Todos +----- + +.. todolist:: diff --git a/docs/index.rst b/docs/index.rst index 253d047..2459200 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -25,6 +25,11 @@ Contents upgrade/index upgrade_dependencies/index + roadmap/index + contributing/index + + CHANGES + Indices and tables ================== diff --git a/docs/install/index.rst b/docs/install/index.rst new file mode 100644 index 0000000..0286941 --- /dev/null +++ b/docs/install/index.rst @@ -0,0 +1,10 @@ +Install / Depend on RestrictedPython +==================================== + +RestrictedPython is usually not used stand alone, if you use it in context of your package add it to ``install_requires`` in your ``setup.py`` or a ``requirement.txt`` used by ``pip``. + +For a standalone usage: + +.. code:: bash + + pip install RestrictedPython diff --git a/docs/roadmap/index.rst b/docs/roadmap/index.rst new file mode 100644 index 0000000..3fd98aa --- /dev/null +++ b/docs/roadmap/index.rst @@ -0,0 +1,39 @@ +Roadmap for RestrictedPython +============================ + +RestrictedPython 4.0 +-------------------- + +A feature complete rewrite of RestrictedPython using ``ast`` module instead of ``compile`` package. +RestrictedPython 4.0 should not add any new or remove restrictions. + +A detailed documentation that support usage and further development. + +Full code coverage tests. + +.. todo:: + + Complete documentation of all public API elements with docstyle comments + https://www.python.org/dev/peps/pep-0257/ + http://www.sphinx-doc.org/en/stable/ext/autodoc.html + http://thomas-cokelaer.info/tutorials/sphinx/docstring_python.html + +.. todo:: + + Resolve Discussion in https://github.com/zopefoundation/RestrictedPython/pull/39#issuecomment-283074699 + + compile_restricted optional params flags and dont_inherit will not work as expected with the current implementation. + + stephan-hof did propose a solution, should be discussed and if approved implemented. + + +RestrictedPython 4.1+ +--------------------- + +Enhance RestrictedPython, declare deprecations and possible new restrictions. + +RestrictedPython 5.0+ +--------------------- + +* Python 3+ only, no more support for Python 2.7 +* mypy - Static Code Analysis Annotations diff --git a/docs/update_notes.rst b/docs/update_notes.rst deleted file mode 100644 index 665c089..0000000 --- a/docs/update_notes.rst +++ /dev/null @@ -1,45 +0,0 @@ -Notes on the Update Process to be Python 3 compatible -===================================================== - - -Also RestrictedPython provides a way to define Policies, by redefining restricted versions of ``print``, ``getattr``, ``setattr``, ``import``, etc.. -As shortcutes it offers three stripped down versions of Pythons ``__builtins__``: - -* ``safe_builtins`` (by Guards.py) -* ``limited_builtins`` (by Limits.py), which provides restriced sequence types -* ``utilities_builtins`` (by Utilities.py), which provides access for standard modules math, random, string and for sets. - -There is also a guard function for making attributes immutable --> ``full_write_guard`` (write and delete protected) - - - -Technical foundation of RestrictedPython -........................................ - -RestrictedPython is based on the Python 2 only standard library module ``compiler`` (https://docs.python.org/2.7/library/compiler.html). -RestrictedPython based on the - -* compiler.ast -* compiler.parse -* compiler.pycodegen - -With Python 2.6 the compiler module with all its sub modules has been declared deprecated with no direct upgrade Path or recommendations for a replacement. - - -Approach --------- - -RestrictedPython is a classical approach of compiler construction to create a limited subset of an existing programming language. - -As compiler construction do have basic concepts on how to build a Programming Language and Runtime Environment. - -Defining a Programming Language means to define a regular grammar (Chomsky 3 / EBNF) first. -This grammar will be implemented in an abstract syntax tree (AST), which will be passed on to a code generator to produce a machine understandable version. - -As Python is a plattform independend programming / scripting language, this machine understandable version is a byte code which will be translated on the fly by an interpreter into machine code. -This machine code then gets executed on the specific CPU architecture, with all Operating System restriction. - -Produced byte code has to compatible with the execution environment, the Python Interpreter within this code is called. -So we must not generate the byte code that has to be returned from ``compile_restricted`` and the other ``compile_restricted_*`` methods manually, as this might harm the interpreter. -We actually don't even need that. -The Python ``compile()`` function introduced the capability to compile ``ast.AST`` code into byte code. diff --git a/docs/upgrade/ast/python2_6.ast b/docs/upgrade/ast/python2_6.ast index d4af5b1..3ea12f8 100644 --- a/docs/upgrade/ast/python2_6.ast +++ b/docs/upgrade/ast/python2_6.ast @@ -81,20 +81,49 @@ module Python version "2.6" -- col_offset is the byte offset in the utf8 string the parser uses attributes (int lineno, int col_offset) - expr_context = Load | Store | Del | AugLoad | AugStore | Param - - slice = Ellipsis | Slice(expr? lower, expr? upper, expr? step) + expr_context = Load + | Store + | Del + | AugLoad + | AugStore + | Param + + slice = Ellipsis + | Slice(expr? lower, expr? upper, expr? step) | ExtSlice(slice* dims) | Index(expr value) - boolop = And | Or - - operator = Add | Sub | Mult | Div | Mod | Pow | LShift - | RShift | BitOr | BitXor | BitAnd | FloorDiv - - unaryop = Invert | Not | UAdd | USub - - cmpop = Eq | NotEq | Lt | LtE | Gt | GtE | Is | IsNot | In | NotIn + boolop = And + | Or + + operator = Add + | Sub + | Mult + | Div + | Mod + | Pow + | LShift + | RShift + | BitOr + | BitXor + | BitAnd + | FloorDiv + + unaryop = Invert + | Not + | UAdd + | USub + + cmpop = Eq + | NotEq + | Lt + | LtE + | Gt + | GtE + | Is + | IsNot + | In + | NotIn comprehension = (expr target, expr iter, expr* ifs) @@ -103,11 +132,11 @@ module Python version "2.6" attributes (int lineno, int col_offset) arguments = (expr* args, identifier? vararg, - identifier? kwarg, expr* defaults) + identifier? kwarg, expr* defaults) - -- keyword arguments supplied to call - keyword = (identifier arg, expr value) + -- keyword arguments supplied to call + keyword = (identifier arg, expr value) - -- import name with optional 'as' alias. - alias = (identifier name, identifier? asname) + -- import name with optional 'as' alias. + alias = (identifier name, identifier? asname) } diff --git a/docs/upgrade/ast/python2_7.ast b/docs/upgrade/ast/python2_7.ast index fc3aba1..899cdc5 100644 --- a/docs/upgrade/ast/python2_7.ast +++ b/docs/upgrade/ast/python2_7.ast @@ -11,7 +11,7 @@ module Python version "2.7" | Suite(stmt* body) stmt = FunctionDef(identifier name, arguments args, - stmt* body, expr* decorator_list) + stmt* body, expr* decorator_list) | ClassDef(identifier name, expr* bases, stmt* body, expr* decorator_list) | Return(expr? value) @@ -84,20 +84,48 @@ module Python version "2.7" -- col_offset is the byte offset in the utf8 string the parser uses attributes (int lineno, int col_offset) - expr_context = Load | Store | Del | AugLoad | AugStore | Param + expr_context = Load + | Store + | Del + | AugLoad + | AugStore + | Param - slice = Ellipsis | Slice(expr? lower, expr? upper, expr? step) + slice = Ellipsis + | Slice(expr? lower, expr? upper, expr? step) | ExtSlice(slice* dims) | Index(expr value) boolop = And | Or - operator = Add | Sub | Mult | Div | Mod | Pow | LShift - | RShift | BitOr | BitXor | BitAnd | FloorDiv - - unaryop = Invert | Not | UAdd | USub - - cmpop = Eq | NotEq | Lt | LtE | Gt | GtE | Is | IsNot | In | NotIn + operator = Add + | Sub + | Mult + | Div + | Mod + | Pow + | LShift + | RShift + | BitOr + | BitXor + | BitAnd + | FloorDiv + + unaryop = Invert + | Not + | UAdd + | USub + + cmpop = Eq + | NotEq + | Lt + | LtE + | Gt + | GtE + | Is + | IsNot + | In + | NotIn comprehension = (expr target, expr iter, expr* ifs) @@ -106,11 +134,11 @@ module Python version "2.7" attributes (int lineno, int col_offset) arguments = (expr* args, identifier? vararg, - identifier? kwarg, expr* defaults) + identifier? kwarg, expr* defaults) - -- keyword arguments supplied to call - keyword = (identifier arg, expr value) + -- keyword arguments supplied to call + keyword = (identifier arg, expr value) - -- import name with optional 'as' alias. - alias = (identifier name, identifier? asname) + -- import name with optional 'as' alias. + alias = (identifier name, identifier? asname) } diff --git a/docs/upgrade/ast/python3_0.ast b/docs/upgrade/ast/python3_0.ast index 2241781..44d6e18 100644 --- a/docs/upgrade/ast/python3_0.ast +++ b/docs/upgrade/ast/python3_0.ast @@ -3,117 +3,145 @@ module Python version "3.0" { - mod = Module(stmt* body) - | Interactive(stmt* body) - | Expression(expr body) - - -- not really an actual node but useful in Jython's typesystem. - | Suite(stmt* body) - - stmt = FunctionDef(identifier name, arguments args, - stmt* body, expr* decorator_list, expr? returns) - | ClassDef(identifier name, - expr* bases, - keyword* keywords, - expr? starargs, - expr? kwargs, - stmt* body, - expr *decorator_list) - | Return(expr? value) - - | Delete(expr* targets) - | Assign(expr* targets, expr value) - | AugAssign(expr target, operator op, expr value) - - -- use 'orelse' because else is a keyword in target languages - | For(expr target, expr iter, stmt* body, stmt* orelse) - | While(expr test, stmt* body, stmt* orelse) - | If(expr test, stmt* body, stmt* orelse) - | With(expr context_expr, expr? optional_vars, stmt* body) - - | Raise(expr? exc, expr? cause) - | TryExcept(stmt* body, excepthandler* handlers, stmt* orelse) - | TryFinally(stmt* body, stmt* finalbody) - | Assert(expr test, expr? msg) - - | Import(alias* names) - | ImportFrom(identifier module, alias* names, int? level) - - | Global(identifier* names) - | Nonlocal(identifier* names) - | Expr(expr value) - | Pass | Break | Continue - - -- XXX Jython will be different - -- col_offset is the byte offset in the utf8 string the parser uses - attributes (int lineno, int col_offset) - - -- BoolOp() can use left & right? - expr = BoolOp(boolop op, expr* values) - | BinOp(expr left, operator op, expr right) - | UnaryOp(unaryop op, expr operand) - | Lambda(arguments args, expr body) - | IfExp(expr test, expr body, expr orelse) - | Dict(expr* keys, expr* values) - | Set(expr* elts) - | ListComp(expr elt, comprehension* generators) - | SetComp(expr elt, comprehension* generators) - | DictComp(expr key, expr value, comprehension* generators) - | GeneratorExp(expr elt, comprehension* generators) - -- the grammar constrains where yield expressions can occur - | Yield(expr? value) - -- need sequences for compare to distinguish between - -- x < 4 < 3 and (x < 4) < 3 - | Compare(expr left, cmpop* ops, expr* comparators) - | Call(expr func, expr* args, keyword* keywords, - expr? starargs, expr? kwargs) - | Num(object n) -- a number as a PyObject. - | Str(string s) -- need to specify raw, unicode, etc? - | Bytes(string s) - | Ellipsis - -- other literals? bools? - - -- the following expression can appear in assignment context - | Attribute(expr value, identifier attr, expr_context ctx) - | Subscript(expr value, slice slice, expr_context ctx) - | Starred(expr value, expr_context ctx) - | Name(identifier id, expr_context ctx) - | List(expr* elts, expr_context ctx) - | Tuple(expr* elts, expr_context ctx) - - -- col_offset is the byte offset in the utf8 string the parser uses - attributes (int lineno, int col_offset) - - expr_context = Load | Store | Del | AugLoad | AugStore | Param - - slice = Slice(expr? lower, expr? upper, expr? step) - | ExtSlice(slice* dims) - | Index(expr value) - - boolop = And | Or - - operator = Add | Sub | Mult | Div | Mod | Pow | LShift - | RShift | BitOr | BitXor | BitAnd | FloorDiv - - unaryop = Invert | Not | UAdd | USub - - cmpop = Eq | NotEq | Lt | LtE | Gt | GtE | Is | IsNot | In | NotIn - - comprehension = (expr target, expr iter, expr* ifs) - - -- not sure what to call the first argument for raise and except - excepthandler = ExceptHandler(expr? type, identifier? name, stmt* body) - attributes (int lineno, int col_offset) - - arguments = (arg* args, identifier? vararg, expr? varargannotation, - arg* kwonlyargs, identifier? kwarg, - expr? kwargannotation, expr* defaults, - expr* kw_defaults) - arg = (identifier arg, expr? annotation) - - -- keyword arguments supplied to call - keyword = (identifier arg, expr value) - - -- import name with optional 'as' alias. - alias = (identifier name, identifier? asname) + mod = Module(stmt* body) + | Interactive(stmt* body) + | Expression(expr body) + + -- not really an actual node but useful in Jython's typesystem. + | Suite(stmt* body) + + stmt = FunctionDef(identifier name, arguments args, + stmt* body, expr* decorator_list, expr? returns) + | ClassDef(identifier name, + expr* bases, + keyword* keywords, + expr? starargs, + expr? kwargs, + stmt* body, + expr *decorator_list) + | Return(expr? value) + + | Delete(expr* targets) + | Assign(expr* targets, expr value) + | AugAssign(expr target, operator op, expr value) + + -- use 'orelse' because else is a keyword in target languages + | For(expr target, expr iter, stmt* body, stmt* orelse) + | While(expr test, stmt* body, stmt* orelse) + | If(expr test, stmt* body, stmt* orelse) + | With(expr context_expr, expr? optional_vars, stmt* body) + + | Raise(expr? exc, expr? cause) + | TryExcept(stmt* body, excepthandler* handlers, stmt* orelse) + | TryFinally(stmt* body, stmt* finalbody) + | Assert(expr test, expr? msg) + + | Import(alias* names) + | ImportFrom(identifier module, alias* names, int? level) + + | Global(identifier* names) + | Nonlocal(identifier* names) + | Expr(expr value) + | Pass | Break | Continue + + -- XXX Jython will be different + -- col_offset is the byte offset in the utf8 string the parser uses + attributes (int lineno, int col_offset) + + -- BoolOp() can use left & right? + expr = BoolOp(boolop op, expr* values) + | BinOp(expr left, operator op, expr right) + | UnaryOp(unaryop op, expr operand) + | Lambda(arguments args, expr body) + | IfExp(expr test, expr body, expr orelse) + | Dict(expr* keys, expr* values) + | Set(expr* elts) + | ListComp(expr elt, comprehension* generators) + | SetComp(expr elt, comprehension* generators) + | DictComp(expr key, expr value, comprehension* generators) + | GeneratorExp(expr elt, comprehension* generators) + -- the grammar constrains where yield expressions can occur + | Yield(expr? value) + -- need sequences for compare to distinguish between + -- x < 4 < 3 and (x < 4) < 3 + | Compare(expr left, cmpop* ops, expr* comparators) + | Call(expr func, expr* args, keyword* keywords, + expr? starargs, expr? kwargs) + | Num(object n) -- a number as a PyObject. + | Str(string s) -- need to specify raw, unicode, etc? + | Bytes(string s) + | Ellipsis + -- other literals? bools? + + -- the following expression can appear in assignment context + | Attribute(expr value, identifier attr, expr_context ctx) + | Subscript(expr value, slice slice, expr_context ctx) + | Starred(expr value, expr_context ctx) + | Name(identifier id, expr_context ctx) + | List(expr* elts, expr_context ctx) + | Tuple(expr* elts, expr_context ctx) + + -- col_offset is the byte offset in the utf8 string the parser uses + attributes (int lineno, int col_offset) + + expr_context = Load + | Store + | Del + | AugLoad + | AugStore + | Param + + slice = Slice(expr? lower, expr? upper, expr? step) + | ExtSlice(slice* dims) + | Index(expr value) + + boolop = And + | Or + + operator = Add + | Sub + | Mult + | Div + | Mod + | Pow + | LShift + | RShift + | BitOr + | BitXor + | BitAnd + | FloorDiv + + unaryop = Invert + | Not + | UAdd + | USub + + cmpop = Eq + | NotEq + | Lt + | LtE + | Gt + | GtE + | Is + | IsNot + | In + | NotIn + + comprehension = (expr target, expr iter, expr* ifs) + + -- not sure what to call the first argument for raise and except + excepthandler = ExceptHandler(expr? type, identifier? name, stmt* body) + attributes (int lineno, int col_offset) + + arguments = (arg* args, identifier? vararg, expr? varargannotation, + arg* kwonlyargs, identifier? kwarg, + expr? kwargannotation, expr* defaults, + expr* kw_defaults) + arg = (identifier arg, expr? annotation) + + -- keyword arguments supplied to call + keyword = (identifier arg, expr value) + + -- import name with optional 'as' alias. + alias = (identifier name, identifier? asname) } diff --git a/docs/upgrade/ast/python3_1.ast b/docs/upgrade/ast/python3_1.ast index 8b9dd80..a0bbfb6 100644 --- a/docs/upgrade/ast/python3_1.ast +++ b/docs/upgrade/ast/python3_1.ast @@ -3,117 +3,147 @@ module Python version "3.1" { - mod = Module(stmt* body) - | Interactive(stmt* body) - | Expression(expr body) - - -- not really an actual node but useful in Jython's typesystem. - | Suite(stmt* body) - - stmt = FunctionDef(identifier name, arguments args, - stmt* body, expr* decorator_list, expr? returns) - | ClassDef(identifier name, - expr* bases, - keyword* keywords, - expr? starargs, - expr? kwargs, - stmt* body, - expr *decorator_list) - | Return(expr? value) - - | Delete(expr* targets) - | Assign(expr* targets, expr value) - | AugAssign(expr target, operator op, expr value) - - -- use 'orelse' because else is a keyword in target languages - | For(expr target, expr iter, stmt* body, stmt* orelse) - | While(expr test, stmt* body, stmt* orelse) - | If(expr test, stmt* body, stmt* orelse) - | With(expr context_expr, expr? optional_vars, stmt* body) - - | Raise(expr? exc, expr? cause) - | TryExcept(stmt* body, excepthandler* handlers, stmt* orelse) - | TryFinally(stmt* body, stmt* finalbody) - | Assert(expr test, expr? msg) - - | Import(alias* names) - | ImportFrom(identifier module, alias* names, int? level) - - | Global(identifier* names) - | Nonlocal(identifier* names) - | Expr(expr value) - | Pass | Break | Continue - - -- XXX Jython will be different - -- col_offset is the byte offset in the utf8 string the parser uses - attributes (int lineno, int col_offset) - - -- BoolOp() can use left & right? - expr = BoolOp(boolop op, expr* values) - | BinOp(expr left, operator op, expr right) - | UnaryOp(unaryop op, expr operand) - | Lambda(arguments args, expr body) - | IfExp(expr test, expr body, expr orelse) - | Dict(expr* keys, expr* values) - | Set(expr* elts) - | ListComp(expr elt, comprehension* generators) - | SetComp(expr elt, comprehension* generators) - | DictComp(expr key, expr value, comprehension* generators) - | GeneratorExp(expr elt, comprehension* generators) - -- the grammar constrains where yield expressions can occur - | Yield(expr? value) - -- need sequences for compare to distinguish between - -- x < 4 < 3 and (x < 4) < 3 - | Compare(expr left, cmpop* ops, expr* comparators) - | Call(expr func, expr* args, keyword* keywords, - expr? starargs, expr? kwargs) - | Num(object n) -- a number as a PyObject. - | Str(string s) -- need to specify raw, unicode, etc? - | Bytes(string s) - | Ellipsis - -- other literals? bools? - - -- the following expression can appear in assignment context - | Attribute(expr value, identifier attr, expr_context ctx) - | Subscript(expr value, slice slice, expr_context ctx) - | Starred(expr value, expr_context ctx) - | Name(identifier id, expr_context ctx) - | List(expr* elts, expr_context ctx) - | Tuple(expr* elts, expr_context ctx) - - -- col_offset is the byte offset in the utf8 string the parser uses - attributes (int lineno, int col_offset) - - expr_context = Load | Store | Del | AugLoad | AugStore | Param - - slice = Slice(expr? lower, expr? upper, expr? step) - | ExtSlice(slice* dims) - | Index(expr value) - - boolop = And | Or - - operator = Add | Sub | Mult | Div | Mod | Pow | LShift - | RShift | BitOr | BitXor | BitAnd | FloorDiv - - unaryop = Invert | Not | UAdd | USub - - cmpop = Eq | NotEq | Lt | LtE | Gt | GtE | Is | IsNot | In | NotIn - - comprehension = (expr target, expr iter, expr* ifs) - - -- not sure what to call the first argument for raise and except - excepthandler = ExceptHandler(expr? type, identifier? name, stmt* body) - attributes (int lineno, int col_offset) - - arguments = (arg* args, identifier? vararg, expr? varargannotation, - arg* kwonlyargs, identifier? kwarg, - expr? kwargannotation, expr* defaults, - expr* kw_defaults) - arg = (identifier arg, expr? annotation) - - -- keyword arguments supplied to call - keyword = (identifier arg, expr value) - - -- import name with optional 'as' alias. - alias = (identifier name, identifier? asname) + mod = Module(stmt* body) + | Interactive(stmt* body) + | Expression(expr body) + + -- not really an actual node but useful in Jython's typesystem. + | Suite(stmt* body) + + stmt = FunctionDef(identifier name, arguments args, + stmt* body, expr* decorator_list, expr? returns) + | ClassDef(identifier name, + expr* bases, + keyword* keywords, + expr? starargs, + expr? kwargs, + stmt* body, + expr *decorator_list) + | Return(expr? value) + + | Delete(expr* targets) + | Assign(expr* targets, expr value) + | AugAssign(expr target, operator op, expr value) + + -- use 'orelse' because else is a keyword in target languages + | For(expr target, expr iter, stmt* body, stmt* orelse) + | While(expr test, stmt* body, stmt* orelse) + | If(expr test, stmt* body, stmt* orelse) + | With(expr context_expr, expr? optional_vars, stmt* body) + + | Raise(expr? exc, expr? cause) + | TryExcept(stmt* body, excepthandler* handlers, stmt* orelse) + | TryFinally(stmt* body, stmt* finalbody) + | Assert(expr test, expr? msg) + + | Import(alias* names) + | ImportFrom(identifier module, alias* names, int? level) + + | Global(identifier* names) + | Nonlocal(identifier* names) + | Expr(expr value) + | Pass + | Break + | Continue + + -- XXX Jython will be different + -- col_offset is the byte offset in the utf8 string the parser uses + attributes (int lineno, int col_offset) + + -- BoolOp() can use left & right? + expr = BoolOp(boolop op, expr* values) + | BinOp(expr left, operator op, expr right) + | UnaryOp(unaryop op, expr operand) + | Lambda(arguments args, expr body) + | IfExp(expr test, expr body, expr orelse) + | Dict(expr* keys, expr* values) + | Set(expr* elts) + | ListComp(expr elt, comprehension* generators) + | SetComp(expr elt, comprehension* generators) + | DictComp(expr key, expr value, comprehension* generators) + | GeneratorExp(expr elt, comprehension* generators) + -- the grammar constrains where yield expressions can occur + | Yield(expr? value) + -- need sequences for compare to distinguish between + -- x < 4 < 3 and (x < 4) < 3 + | Compare(expr left, cmpop* ops, expr* comparators) + | Call(expr func, expr* args, keyword* keywords, + expr? starargs, expr? kwargs) + | Num(object n) -- a number as a PyObject. + | Str(string s) -- need to specify raw, unicode, etc? + | Bytes(string s) + | Ellipsis + -- other literals? bools? + + -- the following expression can appear in assignment context + | Attribute(expr value, identifier attr, expr_context ctx) + | Subscript(expr value, slice slice, expr_context ctx) + | Starred(expr value, expr_context ctx) + | Name(identifier id, expr_context ctx) + | List(expr* elts, expr_context ctx) + | Tuple(expr* elts, expr_context ctx) + + -- col_offset is the byte offset in the utf8 string the parser uses + attributes (int lineno, int col_offset) + + expr_context = Load + | Store + | Del + | AugLoad + | AugStore + | Param + + slice = Slice(expr? lower, expr? upper, expr? step) + | ExtSlice(slice* dims) + | Index(expr value) + + boolop = And + | Or + + operator = Add + | Sub + | Mult + | Div + | Mod + | Pow + | LShift + | RShift + | BitOr + | BitXor + | BitAnd + | FloorDiv + + unaryop = Invert + | Not + | UAdd + | USub + + cmpop = Eq + | NotEq + | Lt + | LtE + | Gt + | GtE + | Is + | IsNot + | In + | NotIn + + comprehension = (expr target, expr iter, expr* ifs) + + -- not sure what to call the first argument for raise and except + excepthandler = ExceptHandler(expr? type, identifier? name, stmt* body) + attributes (int lineno, int col_offset) + + arguments = (arg* args, identifier? vararg, expr? varargannotation, + arg* kwonlyargs, identifier? kwarg, + expr? kwargannotation, expr* defaults, + expr* kw_defaults) + arg = (identifier arg, expr? annotation) + + -- keyword arguments supplied to call + keyword = (identifier arg, expr value) + + -- import name with optional 'as' alias. + alias = (identifier name, identifier? asname) } diff --git a/docs/upgrade/ast/python3_2.ast b/docs/upgrade/ast/python3_2.ast index ab325d9..054d63e 100644 --- a/docs/upgrade/ast/python3_2.ast +++ b/docs/upgrade/ast/python3_2.ast @@ -3,142 +3,144 @@ module Python version "3.2" { - mod = Module(stmt* body) - | Interactive(stmt* body) - | Expression(expr body) - - -- not really an actual node but useful in Jython's typesystem. - | Suite(stmt* body) - - stmt = FunctionDef(identifier name, - arguments args, - stmt* body, - expr* decorator_list, - expr? returns) - | ClassDef(identifier name, - expr* bases, - keyword* keywords, - expr? starargs, - expr? kwargs, - stmt* body, - expr* decorator_list) - | Return(expr? value) - - | Delete(expr* targets) - | Assign(expr* targets, expr value) - | AugAssign(expr target, operator op, expr value) - - -- use 'orelse' because else is a keyword in target languages - | For(expr target, expr iter, stmt* body, stmt* orelse) - | While(expr test, stmt* body, stmt* orelse) - | If(expr test, stmt* body, stmt* orelse) - | With(expr context_expr, expr? optional_vars, stmt* body) - - | Raise(expr? exc, expr? cause) - | TryExcept(stmt* body, excepthandler* handlers, stmt* orelse) - | TryFinally(stmt* body, stmt* finalbody) - | Assert(expr test, expr? msg) - - | Import(alias* names) - | ImportFrom(identifier? module, alias* names, int? level) - - | Global(identifier* names) - | Nonlocal(identifier* names) - | Expr(expr value) - | Pass | Break | Continue - - -- XXX Jython will be different - -- col_offset is the byte offset in the utf8 string the parser uses - attributes (int lineno, int col_offset) - - -- BoolOp() can use left & right? - expr = BoolOp(boolop op, expr* values) - | BinOp(expr left, operator op, expr right) - | UnaryOp(unaryop op, expr operand) - | Lambda(arguments args, expr body) - | IfExp(expr test, expr body, expr orelse) - | Dict(expr* keys, expr* values) - | Set(expr* elts) - | ListComp(expr elt, comprehension* generators) - | SetComp(expr elt, comprehension* generators) - | DictComp(expr key, expr value, comprehension* generators) - | GeneratorExp(expr elt, comprehension* generators) - -- the grammar constrains where yield expressions can occur - | Yield(expr? value) - -- need sequences for compare to distinguish between - -- x < 4 < 3 and (x < 4) < 3 - | Compare(expr left, cmpop* ops, expr* comparators) - | Call(expr func, expr* args, keyword* keywords, - expr? starargs, expr? kwargs) - | Num(object n) -- a number as a PyObject. - | Str(string s) -- need to specify raw, unicode, etc? - | Bytes(string s) - | Ellipsis - -- other literals? bools? - - -- the following expression can appear in assignment context - | Attribute(expr value, identifier attr, expr_context ctx) - | Subscript(expr value, slice slice, expr_context ctx) - | Starred(expr value, expr_context ctx) - | Name(identifier id, expr_context ctx) - | List(expr* elts, expr_context ctx) - | Tuple(expr* elts, expr_context ctx) - - -- col_offset is the byte offset in the utf8 string the parser uses - attributes (int lineno, int col_offset) - - expr_context = Load | Store | Del | AugLoad | AugStore | Param - - slice = Slice(expr? lower, expr? upper, expr? step) - | ExtSlice(slice* dims) - | Index(expr value) - - boolop = And | Or - - operator = Add - | Sub - | Mult - | Div - | Mod - | Pow - | LShift - | RShift - | BitOr - | BitXor - | BitAnd - | FloorDiv - - unaryop = Invert - | Not - | UAdd - | USub - - cmpop = Eq - | NotEq - | Lt - | LtE - | Gt - | GtE - | Is - | IsNot - | In - | NotIn - - comprehension = (expr target, expr iter, expr* ifs) - - -- not sure what to call the first argument for raise and except - excepthandler = ExceptHandler(expr? type, identifier? name, stmt* body) - attributes (int lineno, int col_offset) - - arguments = (arg* args, identifier? vararg, expr? varargannotation, - arg* kwonlyargs, identifier? kwarg, - expr? kwargannotation, expr* defaults, - expr* kw_defaults) - arg = (identifier arg, expr? annotation) - - -- keyword arguments supplied to call - keyword = (identifier arg, expr value) - - -- import name with optional 'as' alias. - alias = (identifier name, identifier? asname) + mod = Module(stmt* body) + | Interactive(stmt* body) + | Expression(expr body) + + -- not really an actual node but useful in Jython's typesystem. + | Suite(stmt* body) + + stmt = FunctionDef(identifier name, + arguments args, + stmt* body, + expr* decorator_list, + expr? returns) + | ClassDef(identifier name, + expr* bases, + keyword* keywords, + expr? starargs, + expr? kwargs, + stmt* body, + expr* decorator_list) + | Return(expr? value) + + | Delete(expr* targets) + | Assign(expr* targets, expr value) + | AugAssign(expr target, operator op, expr value) + + -- use 'orelse' because else is a keyword in target languages + | For(expr target, expr iter, stmt* body, stmt* orelse) + | While(expr test, stmt* body, stmt* orelse) + | If(expr test, stmt* body, stmt* orelse) + | With(expr context_expr, expr? optional_vars, stmt* body) + + | Raise(expr? exc, expr? cause) + | TryExcept(stmt* body, excepthandler* handlers, stmt* orelse) + | TryFinally(stmt* body, stmt* finalbody) + | Assert(expr test, expr? msg) + + | Import(alias* names) + | ImportFrom(identifier? module, alias* names, int? level) + + | Global(identifier* names) + | Nonlocal(identifier* names) + | Expr(expr value) + | Pass + | Break + | Continue + + -- XXX Jython will be different + -- col_offset is the byte offset in the utf8 string the parser uses + attributes (int lineno, int col_offset) + + -- BoolOp() can use left & right? + expr = BoolOp(boolop op, expr* values) + | BinOp(expr left, operator op, expr right) + | UnaryOp(unaryop op, expr operand) + | Lambda(arguments args, expr body) + | IfExp(expr test, expr body, expr orelse) + | Dict(expr* keys, expr* values) + | Set(expr* elts) + | ListComp(expr elt, comprehension* generators) + | SetComp(expr elt, comprehension* generators) + | DictComp(expr key, expr value, comprehension* generators) + | GeneratorExp(expr elt, comprehension* generators) + -- the grammar constrains where yield expressions can occur + | Yield(expr? value) + -- need sequences for compare to distinguish between + -- x < 4 < 3 and (x < 4) < 3 + | Compare(expr left, cmpop* ops, expr* comparators) + | Call(expr func, expr* args, keyword* keywords, + expr? starargs, expr? kwargs) + | Num(object n) -- a number as a PyObject. + | Str(string s) -- need to specify raw, unicode, etc? + | Bytes(string s) + | Ellipsis + -- other literals? bools? + + -- the following expression can appear in assignment context + | Attribute(expr value, identifier attr, expr_context ctx) + | Subscript(expr value, slice slice, expr_context ctx) + | Starred(expr value, expr_context ctx) + | Name(identifier id, expr_context ctx) + | List(expr* elts, expr_context ctx) + | Tuple(expr* elts, expr_context ctx) + + -- col_offset is the byte offset in the utf8 string the parser uses + attributes (int lineno, int col_offset) + + expr_context = Load | Store | Del | AugLoad | AugStore | Param + + slice = Slice(expr? lower, expr? upper, expr? step) + | ExtSlice(slice* dims) + | Index(expr value) + + boolop = And | Or + + operator = Add + | Sub + | Mult + | Div + | Mod + | Pow + | LShift + | RShift + | BitOr + | BitXor + | BitAnd + | FloorDiv + + unaryop = Invert + | Not + | UAdd + | USub + + cmpop = Eq + | NotEq + | Lt + | LtE + | Gt + | GtE + | Is + | IsNot + | In + | NotIn + + comprehension = (expr target, expr iter, expr* ifs) + + -- not sure what to call the first argument for raise and except + excepthandler = ExceptHandler(expr? type, identifier? name, stmt* body) + attributes (int lineno, int col_offset) + + arguments = (arg* args, identifier? vararg, expr? varargannotation, + arg* kwonlyargs, identifier? kwarg, + expr? kwargannotation, expr* defaults, + expr* kw_defaults) + arg = (identifier arg, expr? annotation) + + -- keyword arguments supplied to call + keyword = (identifier arg, expr value) + + -- import name with optional 'as' alias. + alias = (identifier name, identifier? asname) } diff --git a/docs/upgrade/ast/python3_3.ast b/docs/upgrade/ast/python3_3.ast index 30c19b5..ad9e258 100644 --- a/docs/upgrade/ast/python3_3.ast +++ b/docs/upgrade/ast/python3_3.ast @@ -15,42 +15,42 @@ module Python version "3.3" stmt* body, expr* decorator_list, expr? returns) - | ClassDef(identifier name, - expr* bases, - keyword* keywords, - expr? starargs, - expr? kwargs, - stmt* body, - expr* decorator_list) - | Return(expr? value) - - | Delete(expr* targets) - | Assign(expr* targets, expr value) - | AugAssign(expr target, operator op, expr value) - - -- use 'orelse' because else is a keyword in target languages - | For(expr target, expr iter, stmt* body, stmt* orelse) - | While(expr test, stmt* body, stmt* orelse) - | If(expr test, stmt* body, stmt* orelse) - | With(withitem* items, stmt* body) - - | Raise(expr? exc, expr? cause) - | Try(stmt* body, excepthandler* handlers, stmt* orelse, stmt* finalbody) - | Assert(expr test, expr? msg) - - | Import(alias* names) - | ImportFrom(identifier? module, alias* names, int? level) - - | Global(identifier* names) - | Nonlocal(identifier* names) - | Expr(expr value) - | Pass | Break | Continue - - -- XXX Jython will be different - -- col_offset is the byte offset in the utf8 string the parser uses - attributes (int lineno, int col_offset) - - -- BoolOp() can use left & right? + | ClassDef(identifier name, + expr* bases, + keyword* keywords, + expr? starargs, + expr? kwargs, + stmt* body, + expr* decorator_list) + | Return(expr? value) + + | Delete(expr* targets) + | Assign(expr* targets, expr value) + | AugAssign(expr target, operator op, expr value) + + -- use 'orelse' because else is a keyword in target languages + | For(expr target, expr iter, stmt* body, stmt* orelse) + | While(expr test, stmt* body, stmt* orelse) + | If(expr test, stmt* body, stmt* orelse) + | With(withitem* items, stmt* body) + + | Raise(expr? exc, expr? cause) + | Try(stmt* body, excepthandler* handlers, stmt* orelse, stmt* finalbody) + | Assert(expr test, expr? msg) + + | Import(alias* names) + | ImportFrom(identifier? module, alias* names, int? level) + + | Global(identifier* names) + | Nonlocal(identifier* names) + | Expr(expr value) + | Pass | Break | Continue + + -- XXX Jython will be different + -- col_offset is the byte offset in the utf8 string the parser uses + attributes (int lineno, int col_offset) + + -- BoolOp() can use left & right? expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) | UnaryOp(unaryop op, expr operand) @@ -87,18 +87,37 @@ module Python version "3.3" -- col_offset is the byte offset in the utf8 string the parser uses attributes (int lineno, int col_offset) - expr_context = Load | Store | Del | AugLoad | AugStore | Param + expr_context = Load + | Store + | Del + | AugLoad + | AugStore + | Param slice = Slice(expr? lower, expr? upper, expr? step) | ExtSlice(slice* dims) | Index(expr value) - boolop = And | Or - - operator = Add | Sub | Mult | Div | Mod | Pow | LShift - | RShift | BitOr | BitXor | BitAnd | FloorDiv - - unaryop = Invert | Not | UAdd | USub + boolop = And + | Or + + operator = Add + | Sub + | Mult + | Div + | Mod + | Pow + | LShift + | RShift + | BitOr + | BitXor + | BitAnd + | FloorDiv + + unaryop = Invert + | Not + | UAdd + | USub cmpop = Eq | NotEq diff --git a/docs/upgrade/ast/python3_4.ast b/docs/upgrade/ast/python3_4.ast index e2bc06f..edfb756 100644 --- a/docs/upgrade/ast/python3_4.ast +++ b/docs/upgrade/ast/python3_4.ast @@ -15,36 +15,38 @@ module Python version "3.4" stmt* body, expr* decorator_list, expr? returns) - | ClassDef(identifier name, - expr* bases, - keyword* keywords, - expr? starargs, - expr? kwargs, - stmt* body, - expr* decorator_list) - | Return(expr? value) - - | Delete(expr* targets) - | Assign(expr* targets, expr value) - | AugAssign(expr target, operator op, expr value) - - -- use 'orelse' because else is a keyword in target languages - | For(expr target, expr iter, stmt* body, stmt* orelse) - | While(expr test, stmt* body, stmt* orelse) - | If(expr test, stmt* body, stmt* orelse) - | With(withitem* items, stmt* body) - - | Raise(expr? exc, expr? cause) - | Try(stmt* body, excepthandler* handlers, stmt* orelse, stmt* finalbody) - | Assert(expr test, expr? msg) - - | Import(alias* names) - | ImportFrom(identifier? module, alias* names, int? level) - - | Global(identifier* names) - | Nonlocal(identifier* names) - | Expr(expr value) - | Pass | Break | Continue + | ClassDef(identifier name, + expr* bases, + keyword* keywords, + expr? starargs, + expr? kwargs, + stmt* body, + expr* decorator_list) + | Return(expr? value) + + | Delete(expr* targets) + | Assign(expr* targets, expr value) + | AugAssign(expr target, operator op, expr value) + + -- use 'orelse' because else is a keyword in target languages + | For(expr target, expr iter, stmt* body, stmt* orelse) + | While(expr test, stmt* body, stmt* orelse) + | If(expr test, stmt* body, stmt* orelse) + | With(withitem* items, stmt* body) + + | Raise(expr? exc, expr? cause) + | Try(stmt* body, excepthandler* handlers, stmt* orelse, stmt* finalbody) + | Assert(expr test, expr? msg) + + | Import(alias* names) + | ImportFrom(identifier? module, alias* names, int? level) + + | Global(identifier* names) + | Nonlocal(identifier* names) + | Expr(expr value) + | Pass + | Break + | Continue -- XXX Jython will be different -- col_offset is the byte offset in the utf8 string the parser uses @@ -69,7 +71,7 @@ module Python version "3.4" -- x < 4 < 3 and (x < 4) < 3 | Compare(expr left, cmpop* ops, expr* comparators) | Call(expr func, expr* args, keyword* keywords, - expr? starargs, expr? kwargs) + expr? starargs, expr? kwargs) | Num(object n) -- a number as a PyObject. | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) @@ -98,7 +100,8 @@ module Python version "3.4" | ExtSlice(slice* dims) | Index(expr value) - boolop = And | Or + boolop = And + | Or operator = Add | Sub diff --git a/docs/upgrade/ast/python3_5.ast b/docs/upgrade/ast/python3_5.ast index d43061d..dfe5bf1 100644 --- a/docs/upgrade/ast/python3_5.ast +++ b/docs/upgrade/ast/python3_5.ast @@ -12,45 +12,45 @@ module Python version "3.5" stmt = FunctionDef(identifier name, arguments args, stmt* body, expr* decorator_list, expr? returns) - | AsyncFunctionDef(identifier name, arguments args, - stmt* body, expr* decorator_list, expr? returns) - - | ClassDef(identifier name, - expr* bases, - keyword* keywords, - stmt* body, - expr* decorator_list) - | Return(expr? value) - - | Delete(expr* targets) - | Assign(expr* targets, expr value) - | AugAssign(expr target, operator op, expr value) - - -- use 'orelse' because else is a keyword in target languages - | For(expr target, expr iter, stmt* body, stmt* orelse) - | AsyncFor(expr target, expr iter, stmt* body, stmt* orelse) - | While(expr test, stmt* body, stmt* orelse) - | If(expr test, stmt* body, stmt* orelse) - | With(withitem* items, stmt* body) - | AsyncWith(withitem* items, stmt* body) - - | Raise(expr? exc, expr? cause) - | Try(stmt* body, excepthandler* handlers, stmt* orelse, stmt* finalbody) - | Assert(expr test, expr? msg) - - | Import(alias* names) - | ImportFrom(identifier? module, alias* names, int? level) - - | Global(identifier* names) - | Nonlocal(identifier* names) - | Expr(expr value) - | Pass | Break | Continue - - -- XXX Jython will be different - -- col_offset is the byte offset in the utf8 string the parser uses - attributes (int lineno, int col_offset) - - -- BoolOp() can use left & right? + | AsyncFunctionDef(identifier name, arguments args, + stmt* body, expr* decorator_list, expr? returns) + + | ClassDef(identifier name, + expr* bases, + keyword* keywords, + stmt* body, + expr* decorator_list) + | Return(expr? value) + + | Delete(expr* targets) + | Assign(expr* targets, expr value) + | AugAssign(expr target, operator op, expr value) + + -- use 'orelse' because else is a keyword in target languages + | For(expr target, expr iter, stmt* body, stmt* orelse) + | AsyncFor(expr target, expr iter, stmt* body, stmt* orelse) + | While(expr test, stmt* body, stmt* orelse) + | If(expr test, stmt* body, stmt* orelse) + | With(withitem* items, stmt* body) + | AsyncWith(withitem* items, stmt* body) + + | Raise(expr? exc, expr? cause) + | Try(stmt* body, excepthandler* handlers, stmt* orelse, stmt* finalbody) + | Assert(expr test, expr? msg) + + | Import(alias* names) + | ImportFrom(identifier? module, alias* names, int? level) + + | Global(identifier* names) + | Nonlocal(identifier* names) + | Expr(expr value) + | Pass | Break | Continue + + -- XXX Jython will be different + -- col_offset is the byte offset in the utf8 string the parser uses + attributes (int lineno, int col_offset) + + -- BoolOp() can use left & right? expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) | UnaryOp(unaryop op, expr operand) @@ -87,20 +87,49 @@ module Python version "3.5" -- col_offset is the byte offset in the utf8 string the parser uses attributes (int lineno, int col_offset) - expr_context = Load | Store | Del | AugLoad | AugStore | Param + expr_context = Load + | Store + | Del + | AugLoad + | AugStore + | Param slice = Slice(expr? lower, expr? upper, expr? step) | ExtSlice(slice* dims) | Index(expr value) - boolop = And | Or - - operator = Add | Sub | Mult | MatMult | Div | Mod | Pow | LShift - | RShift | BitOr | BitXor | BitAnd | FloorDiv - - unaryop = Invert | Not | UAdd | USub - - cmpop = Eq | NotEq | Lt | LtE | Gt | GtE | Is | IsNot | In | NotIn + boolop = And + | Or + + operator = Add + | Sub + | Mult + | MatMult + | Div + | Mod + | Pow + | LShift + | RShift + | BitOr + | BitXor + | BitAnd + | FloorDiv + + unaryop = Invert + | Not + | UAdd + | USub + + cmpop = Eq + | NotEq + | Lt + | LtE + | Gt + | GtE + | Is + | IsNot + | In + | NotIn comprehension = (expr target, expr iter, expr* ifs) diff --git a/docs/upgrade/ast/python3_6.ast b/docs/upgrade/ast/python3_6.ast index d829426..6cf106a 100644 --- a/docs/upgrade/ast/python3_6.ast +++ b/docs/upgrade/ast/python3_6.ast @@ -19,50 +19,52 @@ module Python version "3.6" stmt* body, expr* decorator_list, expr? returns) - | AsyncFunctionDef(identifier name, - arguments args, - stmt* body, - expr* decorator_list, - expr? returns) - - | ClassDef(identifier name, - expr* bases, - keyword* keywords, - stmt* body, - expr* decorator_list) - | Return(expr? value) - - | Delete(expr* targets) - | Assign(expr* targets, expr value) - | AugAssign(expr target, operator op, expr value) - -- 'simple' indicates that we annotate simple name without parens - | AnnAssign(expr target, expr annotation, expr? value, int simple) - - -- use 'orelse' because else is a keyword in target languages - | For(expr target, expr iter, stmt* body, stmt* orelse) - | AsyncFor(expr target, expr iter, stmt* body, stmt* orelse) - | While(expr test, stmt* body, stmt* orelse) - | If(expr test, stmt* body, stmt* orelse) - | With(withitem* items, stmt* body) - | AsyncWith(withitem* items, stmt* body) - - | Raise(expr? exc, expr? cause) - | Try(stmt* body, excepthandler* handlers, stmt* orelse, stmt* finalbody) - | Assert(expr test, expr? msg) - - | Import(alias* names) - | ImportFrom(identifier? module, alias* names, int? level) - - | Global(identifier* names) - | Nonlocal(identifier* names) - | Expr(expr value) - | Pass | Break | Continue - - -- XXX Jython will be different - -- col_offset is the byte offset in the utf8 string the parser uses - attributes (int lineno, int col_offset) - - -- BoolOp() can use left & right? + | AsyncFunctionDef(identifier name, + arguments args, + stmt* body, + expr* decorator_list, + expr? returns) + + | ClassDef(identifier name, + expr* bases, + keyword* keywords, + stmt* body, + expr* decorator_list) + | Return(expr? value) + + | Delete(expr* targets) + | Assign(expr* targets, expr value) + | AugAssign(expr target, operator op, expr value) + -- 'simple' indicates that we annotate simple name without parens + | AnnAssign(expr target, expr annotation, expr? value, int simple) + + -- use 'orelse' because else is a keyword in target languages + | For(expr target, expr iter, stmt* body, stmt* orelse) + | AsyncFor(expr target, expr iter, stmt* body, stmt* orelse) + | While(expr test, stmt* body, stmt* orelse) + | If(expr test, stmt* body, stmt* orelse) + | With(withitem* items, stmt* body) + | AsyncWith(withitem* items, stmt* body) + + | Raise(expr? exc, expr? cause) + | Try(stmt* body, excepthandler* handlers, stmt* orelse, stmt* finalbody) + | Assert(expr test, expr? msg) + + | Import(alias* names) + | ImportFrom(identifier? module, alias* names, int? level) + + | Global(identifier* names) + | Nonlocal(identifier* names) + | Expr(expr value) + | Pass + | Break + | Continue + + -- XXX Jython will be different + -- col_offset is the byte offset in the utf8 string the parser uses + attributes (int lineno, int col_offset) + + -- BoolOp() can use left & right? expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) | UnaryOp(unaryop op, expr operand) @@ -113,7 +115,8 @@ module Python version "3.6" | ExtSlice(slice* dims) | Index(expr value) - boolop = And | Or + boolop = And + | Or operator = Add | Sub diff --git a/docs/usage/api.rst b/docs/usage/api.rst new file mode 100644 index 0000000..086efab --- /dev/null +++ b/docs/usage/api.rst @@ -0,0 +1,108 @@ +API overview +------------ + +RestrictedPython has tree major scopes: + +1. ``compile_restricted`` methods: + +.. py:method:: compile_restricted(source, filename, mode, flags, dont_inherit, policy) + :module: RestrictedPython + + Compiles source code into interpretable byte code. + + :param source: (required). The source code that should be compiled + :param filename: (optional). + :param mode: (optional). + :param flags: (optional). + :param dont_inherit: (optional). + :param policy: (optional). + :type source: str or unicode text + :type filename: str or unicode text + :type mode: str or unicode text + :type flags: int + :type dont_inherit: int + :type policy: RestrictingNodeTransformer class + :return: Byte Code + +.. py:method:: compile_restricted_exec(source, filename, flags, dont_inherit, policy) + :module: RestrictedPython + + Compiles source code into interpretable byte code. + + :param source: (required). The source code that should be compiled + :param filename: (optional). + :param flags: (optional). + :param dont_inherit: (optional). + :param policy: (optional). + :type source: str or unicode text + :type filename: str or unicode text + :type mode: str or unicode text + :type flags: int + :type dont_inherit: int + :type policy: RestrictingNodeTransformer class + :return: CompileResult (a namedtuple with code, errors, warnings, used_names) + +.. py:method:: compile_restricted_eval(source, filename, flags, dont_inherit, policy) + :module: RestrictedPython + + Compiles source code into interpretable byte code. + + :param source: (required). The source code that should be compiled + :param filename: (optional). + :param flags: (optional). + :param dont_inherit: (optional). + :param policy: (optional). + :type source: str or unicode text + :type filename: str or unicode text + :type mode: str or unicode text + :type flags: int + :type dont_inherit: int + :type policy: RestrictingNodeTransformer class + :return: CompileResult (a namedtuple with code, errors, warnings, used_names) + +.. py:method:: compile_restricted_single(source, filename, flags, dont_inherit, policy) + :module: RestrictedPython + + Compiles source code into interpretable byte code. + + :param source: (required). The source code that should be compiled + :param filename: (optional). + :param flags: (optional). + :param dont_inherit: (optional). + :param policy: (optional). + :type source: str or unicode text + :type filename: str or unicode text + :type mode: str or unicode text + :type flags: int + :type dont_inherit: int + :type policy: RestrictingNodeTransformer class + :return: CompileResult (a namedtuple with code, errors, warnings, used_names) + +.. py:method:: compile_restricted_function(p, body, name, filename, globalize=None) + :module: RestrictedPython + + Compiles source code into interpretable byte code. + + :param p: (required). + :param body: (required). + :param name: (required). + :param filename: (required). + :param globalize: (optional). + :type p: + :type body: + :type name: str or unicode text + :type filename: str or unicode text + :type globalize: + :return: byte code + + + +2. restricted builtins + + * ``safe_builtins`` + * ``limited_builtins`` + * ``utility_builtins`` + +3. helper modules + + * ``PrintCollector`` diff --git a/docs/usage/basic_usage.rst b/docs/usage/basic_usage.rst new file mode 100644 index 0000000..5fa89e2 --- /dev/null +++ b/docs/usage/basic_usage.rst @@ -0,0 +1,90 @@ +Basic usage +----------- + +The general workflow to execute Python code that is loaded within a Python program is: + +.. testcode:: + + source_code = """ + def do_something(): + pass + """ + + byte_code = compile(source_code, filename='', mode='exec') + exec(byte_code) + do_something() + +With RestrictedPython that workflow should be as straight forward as possible: + +.. testcode:: + + from RestrictedPython import compile_restricted + + source_code = """ + def do_something(): + pass + """ + + byte_code = compile_restricted(source_code, + filename='', + mode='exec') + exec(byte_code) + do_something() + +You might also use the replacement import: + +.. testcode:: + + from RestrictedPython import compile_restricted as compile + +``compile_restricted`` uses a predefined policy that checks and modify the source code and checks against a restricted subset of the Python language. +The compiled source code is still executed against the full available set of library modules and methods. + +The Python :py:func:`exec` takes three parameters: + +* ``code`` which is the compiled byte code +* ``globals`` which is global dictionary +* ``locals`` which is the local dictionary + +By limiting the entries in the ``globals`` and ``locals`` dictionaries you +restrict the access to the available library modules and methods. + +Providing defined dictionaries for ``exec()`` should be used in context of RestrictedPython. + +.. code:: Python + + byte_code = + exec(byte_code, { ... }, { ... }) + +Typically there is a defined set of allowed modules, methods and constants used in that context. +RestrictedPython provides three predefined built-ins for that (see :ref:`predefined_builtins` for details): + +* ``safe_builtins`` +* ``limited_builtins`` +* ``utility_builtins`` + +So you normally end up using: + +.. testcode:: + + from RestrictedPython import compile_restricted + + from RestrictedPython import safe_builtins + from RestrictedPython import limited_builtins + from RestrictedPython import utility_builtins + + source_code = """ + def do_something(): + pass + """ + + try: + byte_code = compile_restricted(source_code, + filename='', + mode='exec') + + exec(byte_code, safe_builtins, None) + except SyntaxError as e: + pass + +One common advanced usage would be to define an own restricted builtin dictionary. diff --git a/docs/usage/framework_usage.rst b/docs/usage/framework_usage.rst new file mode 100644 index 0000000..0711d8a --- /dev/null +++ b/docs/usage/framework_usage.rst @@ -0,0 +1,85 @@ +.. _sec_usage_frameworks: + +Usage in frameworks and Zope +---------------------------- + +One major issue with using ``compile_restricted`` directly in a framework is, that you have to use try-except statements to handle problems and it might be a bit harder to provide useful information to the user. +RestrictedPython provides four specialized compile_restricted methods: + +* ``compile_restricted_exec`` +* ``compile_restricted_eval`` +* ``compile_restricted_single`` +* ``compile_restricted_function`` + +Those four methods return a named tuple (``CompileResult``) with four elements: + +* ``code`` ```` object or ``None`` if ``errors`` is not empty +* ``errors`` a tuple with error messages +* ``warnings`` a list with warnings +* ``used_names`` a set / dictionary with collected used names of library calls + +Those three information "lists" could be used to provide the user with informations about the compiled source code. + +Typical uses cases for the four specialized methods: + +* ``compile_restricted_exec`` --> Python Modules or Scripts that should be used or called by the framework itself or from user calls +* ``compile_restricted_eval`` --> Templates +* ``compile_restricted_single`` +* ``compile_restricted_function`` + +Modifying the builtins is straight forward, it is just a dictionary containing access pointers to available library elements. +Modification is normally removing elements from existing builtins or adding allowed elements by copying from globals. + +For frameworks it could possibly also be useful to change handling of specific Python language elements. +For that use case RestrictedPython provides the possibility to pass an own policy. + +A policy is basically a special ``NodeTransformer`` that could be instantiated with three params for ``errors``, ``warnings`` and ``used_names``, it should be a subclass of RestrictedPython.RestrictingNodeTransformer. + +.. testcode:: own_policy + + from RestrictedPython import compile_restricted + from RestrictedPython import RestrictingNodeTransformer + + class OwnRestrictingNodeTransformer(RestrictingNodeTransformer): + pass + + policy_instance = OwnRestrictingNodeTransformer(errors=[], + warnings=[], + used_names=[]) + +All ``compile_restricted*`` methods do have a optional parameter ``policy``, where a specific policy could be provided. + +.. testcode:: own_policy + + source_code = """ + def do_something(): + pass + """ + + policy = OwnRestrictingNodeTransformer + + byte_code = compile_restricted(source_code, + filename='', + mode='exec', + policy=policy # Policy Class + ) + exec(byte_code, globals(), None) + +One special case "unrestricted RestrictedPython" (defined to unblock ports of Zope Packages to Python 3) is to actually use RestrictedPython in an unrestricted mode, by providing a Null-Policy (aka ``None``). +That special case would be written as: + +.. testcode:: + + from RestrictedPython import compile_restricted + + source_code = """ + def do_something(): + pass + """ + + byte_code = compile_restricted(source_code, + filename='', + mode='exec', + policy=None # Null-Policy -> unrestricted + ) + exec(byte_code, globals(), None) diff --git a/docs/usage/index.rst b/docs/usage/index.rst index 5079e0d..88b25c6 100644 --- a/docs/usage/index.rst +++ b/docs/usage/index.rst @@ -1,169 +1,7 @@ Usage of RestrictedPython ========================= -API overview ------------- - -RestrictedPython has tree major scopes: - -1. ``compile_restricted`` methods: - - * ``compile_restricted`` - * ``compile_restricted_exec`` - * ``compile_restricted_eval`` - * ``compile_restricted_single`` - * ``compile_restricted_function`` - -2. restricted builtins - - * ``safe_builtins`` - * ``limited_builtins`` - * ``utility_builtins`` - -3. helper modules - - * ``PrintCollector`` - -Basic usage ------------ - -The general workflow to execute Python code that is loaded within a Python program is: - -.. code:: Python - - source_code = """ - def do_something(): - pass - """ - - byte_code = compile(source_code, filename='', mode='exec') - exec(byte_code) - do_something() - -With RestrictedPython that workflow should be as straight forward as possible: - -.. code:: Python - - from RestrictedPython import compile_restricted as compile - - source_code = """ - def do_something(): - pass - """ - - byte_code = compile(source_code, filename='', mode='exec') - exec(byte_code) - do_something() - -With that simple addition: - -.. code:: Python - - from RestrictedPython import compile_restricted as compile - -it uses a predefined policy that checks and modify the source code and checks against a restricted subset of the Python language. -The compiled source code is still executed against the full available set of library modules and methods. - -The Python :py:func:`exec` takes three parameters: - -* ``code`` which is the compiled byte code -* ``globals`` which is global dictionary -* ``locals`` which is the local dictionary - -By limiting the entries in the ``globals`` and ``locals`` dictionaries you -restrict the access to the available library modules and methods. - -Providing defined dictionaries for ``exec()`` should be used in context of RestrictedPython. - -.. code:: Python - - byte_code = - exec(byte_code, { ... }, { ... }) - -Typically there is a defined set of allowed modules, methods and constants used in that context. -RestrictedPython provides three predefined built-ins for that: - -* ``safe_builtins`` -* ``limited_builtins`` -* ``utilities_builtins`` - -So you normally end up using: - -.. code:: Python - - from RestrictedPython import ..._builtins - from RestrictedPython import compile_restricted as compile - - source_code = """""" - - try: - byte_code = compile(source_code, filename='', mode='exec') - - used_builtins = ..._builtins + { } - exec(byte_code, used_buildins, None) - except SyntaxError as e: - ... - -One common advanced usage would be to define an own restricted builtin dictionary. - -.. _sec_usage_frameworks: - -Usage in frameworks and Zope ----------------------------- - -One major issue with using ``compile_restricted`` directly in a framework is, that you have to use try-except statements to handle problems and it might be a bit harder to provide useful information to the user. -RestrictedPython provides four specialized compile_restricted methods: - -* ``compile_restricted_exec`` -* ``compile_restricted_eval`` -* ``compile_restricted_single`` -* ``compile_restricted_function`` - -Those four methods return a tuple with four elements: - -* ``byte_code`` object or ``None`` if ``errors`` is not empty -* ``errors`` a tuple with error messages -* ``warnings`` a list with warnings -* ``used_names`` a set / dictionary with collected used names of library calls - -Those three information "lists" could be used to provide the user with informations about the compiled source code. - -Typical uses cases for the four specialized methods: - -* ``compile_restricted_exec`` --> Python Modules or Scripts that should be used or called by the framework itself or from user calls -* ``compile_restricted_eval`` --> Templates -* ``compile_restricted_single`` -* ``compile_restricted_function`` - -Modifying the builtins is straight forward, it is just a dictionary containing access pointers to available library elements. -Modification is normally removing elements from existing builtins or adding allowed elements by copying from globals. - -For frameworks it could possibly also be useful to change handling of specific Python language elements. -For that use case RestrictedPython provide the possibility to pass an own policy. -A policy is basically a special ``NodeTransformer`` that could be instantiated with three params for ``errors``, ``warnings`` and ``used_names``, it should be a subclass of RestrictingNodeTransformer (that subclassing will maybe later be enforced). - -.. code:: Python - - OwnRestrictingNodeTransformer(errors=[], warnings=[], used_names=[]) - -One special case (defined to unblock ports of Zope Packages to Python 3) is to actually use RestrictedPython in an unrestricted mode, by providing a Null-Policy (aka ``None``). - -All ``compile_restricted*`` methods do have a optional parameter ``policy``, where a specific policy could be provided. - -.. code:: Python - - source_code = """""" - - policy = OwnRestrictingNodeTransformer - - byte_code = compile(source_code, filename='', mode='exec', policy=policy) - exec(byte_code, { ... }, { ... }) - -The Special case "unrestricted RestrictedPython" would be: - -.. code:: Python - - source_code = """""" - - byte_code = compile(source_code, filename='', mode='exec', policy=None) - exec(byte_code, globals(), None) +.. include:: basic_usage.rst +.. include:: framework_usage.rst +.. include:: policy.rst +.. include:: api.rst diff --git a/docs/usage/policy.rst b/docs/usage/policy.rst new file mode 100644 index 0000000..245ebc7 --- /dev/null +++ b/docs/usage/policy.rst @@ -0,0 +1,46 @@ +.. _policy_builtins: + +Policies & builtins +------------------- + +.. todo:: + + Should be described in detail. + Especially the difference between builtins and a policy which is a NodeTransformer. + + +RestrictedPython provides a way to define Policies, by redefining restricted versions of ``print``, ``getattr``, ``setattr``, ``import``, etc.. +As shortcuts it offers three stripped down versions of Pythons ``__builtins__``: + +.. _predefined_builtins: + +Predefined builtins +................... + +.. todo:: + + Describe more in details + +* ``safe_builtins`` a safe set of builtin modules and functions, +* ``limited_builtins`` which provides restricted sequence types, +* ``utility_builtins`` which provides access for standard modules math, random, string and for sets. + +Guards +...... + +.. todo:: + + Describe Guards and predefined guard methods in details + +RestrictedPython predefines several guarded access and manipulation methods: + +* ``guarded_setattr`` +* ``guarded_delattr`` +* ``guarded_iter_unpack_sequence`` +* ``guarded_unpack_sequence`` + +Those and additional methods rely on a helper construct ``full_write_guard``, which is intended to help implement immutable and semi mutable objects and attributes. + +.. todo:: + + Describe full_write_guard more in detail and how it works. diff --git a/setup.py b/setup.py index 1fc6ddb..78d0bc2 100644 --- a/setup.py +++ b/setup.py @@ -29,8 +29,8 @@ def read(*rnames): license='ZPL 2.1', description='RestrictedPython provides a restricted execution ' 'environment for Python, e.g. for running untrusted code.', - long_description=(read('src', 'RestrictedPython', 'README.rst') + '\n' + - read('CHANGES.rst')), + long_description=(read('README.rst') + '\n' + + read('docs', 'CHANGES.rst')), classifiers=[ 'License :: OSI Approved :: Zope Public License', 'Programming Language :: Python', diff --git a/tox.ini b/tox.ini index 64ac2b5..68eed01 100644 --- a/tox.ini +++ b/tox.ini @@ -8,8 +8,8 @@ envlist = py35, py36, pypy, - isort, docs, + isort, coverage-report, skip_missing_interpreters = False @@ -75,6 +75,7 @@ commands = flake8 --doctests src tests setup.py basepython = python2.7 commands = sphinx-build -b html -d build/docs/doctrees docs build/docs/html + sphinx-build -b doctest docs build/docs/doctrees deps = .[docs] Sphinx