Merge pull request #27 from zopefoundation/usage-doc

Python3 usage documentation

.gitignore

@@ -22,6 +22,7 @@ pyvenv.cfg
 /htmlcov
 /include
 /lib
+/share
 /local.cfg
 /parts
 /src/*.egg-info

.travis.yml

@@ -8,13 +8,13 @@ python:
     - pypy-5.4
 env:
     - ENVIRON=py
-    - ENVIRON=isort,flake8
+    - ENVIRON=isort,flake8,docs
 matrix:
     exclude:
-        - env: ENVIRON=isort,flake8
+        - env: ENVIRON=isort,flake8,docs
     include:
         - python: "3.6"
-          env: ENVIRON=isort,flake8
+          env: ENVIRON=isort,flake8,docs
 install:
     - pip install tox coveralls coverage
 script:

CHANGES.txt → CHANGES.rst

@@ -5,7 +5,7 @@ Changes
 ------------------
 
 - Mostly complete rewrite based on Python AST module.
-  [loechel (Alexander Loechel), icemac (Michael Howitz), stephan-hof (Stephan Hof), tlotze (Thomas Lotze)]
+  [loechel (Alexander Loechel), icemac (Michael Howitz), stephan-hof (Stephan Hofmockel), tlotze (Thomas Lotze)]
 
 - switch to pytest
 

docs/RestrictedPython3/index.rst

@@ -8,9 +8,9 @@ Technical foundation of RestrictedPython
 RestrictedPython is based on the Python 2 only standard library module ``compiler`` (https://docs.python.org/2.7/library/compiler.html).
 RestrictedPython based on the
 
-* compiler.ast
-* compiler.parse
-* compiler.pycodegen
+* ``compiler.ast``
+* ``compiler.parse``
+* ``compiler.pycodegen``
 
 With Python 2.6 the compiler module with all its sub modules has been declared deprecated with no direct upgrade Path or recommendations for a replacement.
 
@@ -29,11 +29,11 @@ RestrictedPython 3.6.x aims on supporting Python versions:
 * 2.6
 * 2.7
 
-Even if the README claims that Compatibility Support is form Python 2.3 - 2.7 I found some Code in RestricedPython and related Packages which test if Python 1 is used.
+Even if the README claims that Compatibility Support is form Python 2.3 - 2.7 I found some Code in RestrictedPython and related Packages which test if Python 1 is used.
 
 Due to this approach to support all Python 2 Versions the code uses only statements that are compatible with all of those versions.
 
-So oldstyle classes and newstyle classes are mixed,
+So old style classes and new style classes are mixed,
 
 The following language elements are statements and not functions:
 

docs/RestrictedPython4/index.rst

@@ -12,6 +12,9 @@ Zope and Plone should become Python 3 compatible.
 
 One of the core features of Zope 2 and therefore Plone is the possibility to implement and modify Python scripts and templates through the web (TTW) without harming the application or server itself.
 
+As Python is a `Turing complete`_ programming language programmers don't have any limitation and could potentially harm the Application and Server itself.
+
+RestrictedPython and AccessControl aims on this topic to provide a reduced subset of the Python Programming language, where all functions that could harm the system are permitted by default.
 
 Targeted Versions to support
 ----------------------------
@@ -27,6 +30,7 @@ will be completed):
 * PyPy2.7
 
 .. _`security support` : https://docs.python.org/devguide/index.html#branchstatus
+.. _`Turing complete`: https://en.wikipedia.org/wiki/Turing_completeness
 
 We explicitly excluded Python 3.3 and PyPy3 (which is based on the Python 3.3 specification) as the changes in Python 3.4 are significant and the Python 3.3 is nearing the end of its supported lifetime.
 
@@ -43,6 +47,46 @@ The following packages / modules have hard dependencies on RestrictedPython:
 * Products.PluginIndexes -->
 * five.pt (wrapping some functions and protection for Chameleon) -->
 
-Additionally the folowing add ons have dependencies on RestrictedPython
+Additionally the following add ons have dependencies on RestrictedPython
 
 * None
+
+How RestrictedPython 4+ works internally
+----------------------------------------
+
+RestrictedPython's core functions are split over several files:
+
+* __init__.py --> It exports the API directly in the ``RestrictedPython`` namespace. It should be not necessary to import from any other module inside the package.
+* compile.py --> It contains the ``compile_restricted`` functions where internally ``_compile_restricted_mode`` is the important one
+* transformer.py --> Home of the ``RestrictingNodeTransformer``
+
+``RestrictingNodeTransformer``
+..............................
+
+The ``RestrictingNodeTransformer`` is one of the core elements of RestrictedPython, it provides the base policy used by itself.
+
+``RestrictingNodeTransformer`` is a subclass of a ``NodeTransformer`` which has as set of ``visit_<AST_Elem>`` methods and a ``generic_visit`` method.
+
+``generic_visit`` is a predefined method of any ``NodeVisitor`` which sequential visit all sub nodes, in RestrictedPython this behavior is overwritten to always call a new internal method ``not_allowed(node)``.
+This results in a implicit whitelisting of all allowed AST elements.
+Any possible new introduced AST element in Python (new language element) will implicit be blocked and not allowed in RestrictedPython.
+
+So if new elements should be introduced an explicit ``visit_<new AST elem>`` is necessary.
+
+
+``_compile_restricted_mode``
+............................
+
+``_compile_restricted_mode`` is an internal method that does the whole mapping against the used policy and compiles provided source code, with respecting the mode.
+It is wrapped by the explicit functions:
+
+* ``compile_restricted_exec``
+* ``compile_restricted_eval``
+* ``compile_restricted_single``
+* ``compile_restricted_function``
+
+They are still exposed as those are the nominal used API.
+
+For advanced usage this function is interesting as it is the point where the policy came into play.
+If ``policy`` is ``None`` it just call the Python builtin ``compile`` method.
+Else it parse the provided Python source code into an ``ast.AST`` and let it check and transform by the provided policy.

docs/api/index.rst

@@ -1,8 +1,24 @@
-API von RestrictedPython 4.0
-============================
+API of RestrictedPython 4.0
+===========================
 
+.. code:: Python
 
+    compile_restricted(source, filename, mode [, flags [, dont_inherit]])
 
 .. code:: Python
 
-    compile_restricted(source, filename, mode [, flags [, dont_inherit]])
+    compile_restricted_exec(source, filename, mode [, flags [, dont_inherit [, policy]]])
+
+.. code:: Python
+
+    compile_restricted_eval(source, filename, mode [, flags [, dont_inherit [, policy]]])
+
+
+.. code:: Python
+
+    compile_restricted_single(source, filename, mode [, flags [, dont_inherit [, policy]]])
+
+
+.. code:: Python
+
+    compile_restricted_function(source, filename, mode [, flags [, dont_inherit [, policy]]])

docs/conf.py

@@ -150,7 +150,7 @@
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['_static']
+# html_static_path = ['_static']
 
 # Add any extra paths that contain custom files (such as robots.txt or
 # .htaccess) here, relative to this directory. These files are copied

docs/idea.rst

@@ -1,10 +1,10 @@
 The Idea behind RestrictedPython
 ================================
 
-Python is a `Turing-complete`_ programming language.
+Python is a `Turing complete`_ programming language.
 To offer a Python interface for users in web context is a potential security risk.
 Web frameworks and Content Management Systems (CMS) want to offer their users as much extensibility as possible through the web (TTW).
-This also means to have permissions to add functionallity via a Python Script.
+This also means to have permissions to add functionality via a Python Script.
 
 There should be additional preventive measures taken to ensure integrity of the application and the server itself, according to information security best practice and unrelated to Restricted Python.
 
@@ -14,18 +14,18 @@ The `Ada Ravenscar profile`_ is another example of such an approach.
 
 Defining a secure subset of the language involves restricting the `EBNF`_ elements and explicitly allowing or disallowing language features.
 Much of the power of a programming language derives from its standard and contributed libraries, so any calling of these methods must also be checked and potentially restricted.
-RestricedPython generally disallows calls to any library that is not explicit whitelisted.
+RestrictedPython generally disallows calls to any library that is not explicit whitelisted.
 
 As Python is a scripting language that is executed by an interpreter.
-Any Python code that should be executed have to be explict checked before executing a generated byte code by the interpreter.
+Any Python code that should be executed have to be explicit checked before executing a generated byte code by the interpreter.
 
 Python itself offers three methods that provide such a workflow:
 
 * ``compile()`` which compiles source code to byte code
 * ``exec`` / ``exec()`` which executes the byte code in the interpreter
 * ``eval`` / ``eval()`` which executes a byte code expression
 
-Therefore RestrictedPython offers a repacement for the python builtin function ``compile()`` (Python 2: https://docs.python.org/2/library/functions.html#compile / Python 3 https://docs.python.org/3/library/functions.html#compile).
+Therefore RestrictedPython offers a replacement for the python builtin function ``compile()`` (Python 2: https://docs.python.org/2/library/functions.html#compile / Python 3 https://docs.python.org/3/library/functions.html#compile).
 This method is defined as following:
 
 .. code:: Python
@@ -40,16 +40,16 @@ There are three valid string values for ``mode``:
 * ``'eval'``
 * ``'single'``
 
-For RestricedPython this ``compile()`` method is replaced by:
+For RestrictedPython this ``compile()`` method is replaced by:
 
 .. code:: Python
 
-    compile_restriced(source, filename, mode [, flags [, dont_inherit]])
+    compile_restricted(source, filename, mode [, flags [, dont_inherit]])
 
-The primary parameter ``source`` has to be a ASCII or ``unicode`` string.
+The primary parameter ``source`` has to be a ASCII or ``unicode`` string (With Python 2.6 an additional option for source was added: ``ast.AST`` for :ref:`Code generation <_sec_code_generation>`).
 Both methods either returns compiled byte code that the interpreter could execute or raise exceptions if the provided source code is invalid.
 
-As ``compile`` and ``compile_restricted`` just compile the provided source code to bytecode it is not sufficient to sandbox the environment, as all calls to libraries are still available.
+As ``compile`` and ``compile_restricted`` just compile the provided source code to byte code it is not sufficient to sandbox the environment, as all calls to libraries are still available.
 
 The two methods / Statements:
 
@@ -58,12 +58,12 @@ The two methods / Statements:
 
 have two parameters:
 
-* globals
-* locals
+* ``globals``
+* ``locals``
 
 which are a reference to the Python builtins.
 
-By modifing and restricting the avaliable moules, methods and constants from globals and locals we could limit the possible calls.
+By modifying and restricting the available modules, methods and constants from globals and locals we could limit the possible calls.
 
 Additionally RestrictedPython offers a way to define a policy which allows developers to protect access to attributes.
 This works by defining a restricted version of:
@@ -76,11 +76,11 @@ This works by defining a restricted version of:
 Also RestrictedPython provides three predefined, limited versions of Python's own ``__builtins__``:
 
 * ``safe_builtins`` (by Guards.py)
-* ``limited_builtins`` (by Limits.py), which provides restriced sequence types
+* ``limited_builtins`` (by Limits.py), which provides restricted sequence types
 * ``utilities_builtins`` (by Utilities.py), which provides access for standard modules math, random, string and for sets.
 
 Additional there exist guard functions to make attributes of Python objects immutable --> ``full_write_guard`` (write and delete protected)
 
-.. _Turing-complete: https://en.wikipedia.org/wiki/Turing_completeness
+.. _`Turing complete`: https://en.wikipedia.org/wiki/Turing_completeness
 .. _Ada Ravenscar Profile: https://en.wikipedia.org/wiki/Ravenscar_profile
 .. _EBNF: https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form

docs/index.rst

@@ -17,11 +17,13 @@ Contents
 
    idea
    basics/index
-   RestrictedPython3/index
-   RestrictedPython4/index
-   update/index
+   usage/index
    api/index
 
+   RestrictedPython3/index
+   RestrictedPython4/index
+   upgrade/index
+   upgrade_dependencies/index
 
 Indices and tables
 ==================

docs/notes.rst

@@ -1,6 +1,9 @@
 How it works
 ============
 
+*Caution:* This is old documentation from RestrictedPython 3 and before.
+Information should be transferred and this file should be removed.
+
 Every time I see this code, I have to relearn it.  These notes will
 hopefully make this a little easier. :)