extend doc

docs/idea.rst

@@ -46,7 +46,7 @@ For RestrictedPython this ``compile()`` method is replaced by:
 
     compile_restricted(source, filename, mode [, flags [, dont_inherit]])
 
-The primary parameter ``source`` has to be a ASCII or ``unicode`` string (With Python 2.6 an additional option for source was added: ``ast.AST`` for :ref:`Code generation <code generation>`).
+The primary parameter ``source`` has to be a ASCII or ``unicode`` string (With Python 2.6 an additional option for source was added: ``ast.AST`` for :ref:`Code generation <_sec_code_generation>`).
 Both methods either returns compiled byte code that the interpreter could execute or raise exceptions if the provided source code is invalid.
 
 As ``compile`` and ``compile_restricted`` just compile the provided source code to byte code it is not sufficient to sandbox the environment, as all calls to libraries are still available.

docs/upgrade/index.rst

@@ -6,6 +6,8 @@ RestrictedPython is a classic approach of compiler construction to create a limi
 Defining a programming language requires a regular grammar (`Chomsky 3`_ / `EBNF`_) definition.
 This grammar will be implemented in an abstract syntax tree (AST), which will be passed on to a code generator to produce a machine-readable version.
 
+.. _`_sec_code_generation`
+
 Code generation
 ---------------
 
@@ -14,15 +16,16 @@ This machine code then gets executed on the specific CPU architecture, with the
 
 The byte code produced must be compatible with the execution environment that the Python interpreter is running in, so we do not generate the byte code directly from ``compile_restricted`` and the other ``compile_restricted_*`` methods manually, it may not match what the interpreter expects.
 
-Thankfully, the Python ``compile()`` function introduced the capability to compile ``ast.AST`` code into byte code in Python 2.6, so we can return the platform-independent AST and keep bytecode generation delegated to the interpreter.
-
-As the ``compiler`` module was deprecated in Python 2.6 and was removed before Python 3.0 was released it has never been avaliable for any Python 3 version.
-So we need to move from ``compiler.ast`` to ``ast`` to support newer Python versions.
+Thankfully, the Python ``compile()`` function introduced the capability to compile ``ast.AST`` code into byte code in Python 2.6, so we can return the platform-independent AST and keep byte code generation delegated to the interpreter.
 
 ``compiler.ast`` --> ``ast``
 ----------------------------
 
-From the point of view of compiler design, the concepts of the compiler module and the ast module are similar.
+As the ``compiler`` module was deprecated in Python 2.6 and was removed before Python 3.0 was released it has never been available for any Python 3 version.
+Instead Python 2.6 / Python 3 introduced the new ``ast`` module, that is more widly supported.
+So we need to move from ``compiler.ast`` to ``ast`` to support newer Python versions.
+
+From the point of view of compiler design, the concepts of the ``compiler`` module and the ``ast`` module are similar.
 The ``compiler`` module predates several major improvements of the Python development like a generally applied style guide.
 While ``compiler`` still uses the old `CamelCase`_ Syntax (``visitNode(self, node, walker)``) the ``ast.AST`` did now use the Python common ``visit_Node(self, node)`` syntax.
 Also the names of classes have been changed, where ``compiler`` uses ``Walker`` and ``Mutator`` the corresponding elements in ``ast.AST`` are ``NodeVisitor`` and ``NodeTransformator``.
@@ -47,11 +50,20 @@ The ``ast`` module consists of four areas:
 
 
 ``NodeVisitor`` & ``NodeTransformer``
--------------------------------------
+.....................................
 
 A ``NodeVisitor`` is a class of a node / AST consumer, it reads the data by stepping through the tree without modifying it.
 In contrast, a ``NodeTransformer`` (which inherits from a ``NodeVisitor``) is allowed to modify the tree and nodes.
 
+Modifying the AST
+-----------------
+
+
+
+
+
+
+
 
 Technical Backgrounds - Links to External Documentation
 ---------------------------------------------------------

docs/usage/index.rst

@@ -1,31 +1,31 @@
 Usage of RestrictedPython
 =========================
 
-Basics
-------
+API overview
+------------
 
 RestrictedPython do have tree major scopes:
 
-* compile_restricted methods
+* ``compile_restricted`` methods
 
-  * compile_restricted
-  * compile_restricted_exec
-  * compile_restricted_eval
-  * compile_restricted_single
-  * compile_restricted_function
+  * ``compile_restricted``
+  * ``compile_restricted_exec``
+  * ``compile_restricted_eval``
+  * ``compile_restricted_single``
+  * ``compile_restricted_function``
 
 * restricted builtins
 
-  * safe_builtins
-  * limited_builtins
-  * utility_builtins
+  * ``safe_builtins``
+  * ``limited_builtins``
+  * ``utility_builtins``
 
 * Helper Moduls
 
-  * PrintCollector
+  * ``PrintCollector``
 
-heading
--------
+Basic usage
+-----------
 
 The general workflow to execute Python code that is loaded within a Python program is:
 
@@ -55,5 +55,72 @@ With RestrictedPython that workflow should be as straight forward as possible:
     exec(byte_code)
     do_something()
 
-With that simple addition ``from RestrictedPython import compile_restricted as compile`` it uses a predefined policy that checks and modify the source code and checks against restricted subset of the Python language.
-Execution of the compiled source code is still against the full avaliable set of library modules and methods.
+With that simple addition:
+
+.. code:: Python
+
+    from RestrictedPython import compile_restricted as compile
+
+it uses a predefined policy that checks and modify the source code and checks against a restricted subset of the Python language.
+Execution of the compiled source code is still against the full available set of library modules and methods.
+
+The ``exec()`` :ref:`Python ``exec()`` method <python3:meth:exec>` did take three params:
+
+* ``code`` which is the compiled byte code
+* ``globals`` which is global dictionary
+* ``locals`` which is the local dictionary
+
+By limiting the entries in globals and locals dictionary you restrict access to available library modules and methods.
+
+So providing defined dictionaries for the ``exec()`` method should be used in context of RestrictedPython.
+
+.. code:: Python
+
+    byte_code = <code>
+    exec(byte_code, { ... }, { ... })
+
+Typically there is a defined set of allowed modules, methods and constants used in that context.
+RestrictedPython did provide three predefined builtins for that:
+
+* ``safe_builtins``
+* ``limited_builtins``
+* ``utilities_builtins``
+
+So you normally end up using:
+
+.. code:: Python
+
+    from RestrictedPython import ..._builtins
+    from RestrictedPython import compile_restricted as compile
+
+    source_code = """<demo code>"""
+
+    try:
+        byte_code = compile(source_code, filename='<name>', mode='exec')
+
+        used_builtins = ..._builtins + { <additionl elems> }
+        exec(byte_code, used_buildins, None)
+    except SyntaxError as e:
+        ...
+
+One common advanced usage would be to define an own restricted builtin dictionary.
+
+
+
+Usage on frameworks and Zope
+----------------------------
+
+One major issue with using ``compile_restricted`` directly in a framework is, that you have to use try except statements to handle problems and it might be a bit harder to provide useful information to the user.
+RestrictedPython did provide four specialized compile_restricted methods:
+
+* ``compile_restricted_exec``
+* ``compile_restricted_eval``
+* ``compile_restricted_single``
+* ``compile_restricted_function``
+
+Those four methods return a tuple with four elements:
+
+* ``byte_code`` <code> object or ``None`` if ``errors`` is not empty
+* ``errors`` a tuple with error messages
+* ``warnings`` a list with warnings
+* ``used_names`` a set / dictionary with collected used names of library calls