move parts of update_notes into specific documentation

docs/RestrictedPython3/index.rst

@@ -8,9 +8,9 @@ Technical foundation of RestrictedPython
 RestrictedPython is based on the Python 2 only standard library module ``compiler`` (https://docs.python.org/2.7/library/compiler.html).
 RestrictedPython based on the
 
-* compiler.ast
-* compiler.parse
-* compiler.pycodegen
+* ``compiler.ast``
+* ``compiler.parse``
+* ``compiler.pycodegen``
 
 With Python 2.6 the compiler module with all its sub modules has been declared deprecated with no direct upgrade Path or recommendations for a replacement.
 
@@ -29,11 +29,11 @@ RestrictedPython 3.6.x aims on supporting Python versions:
 * 2.6
 * 2.7
 
-Even if the README claims that Compatibility Support is form Python 2.3 - 2.7 I found some Code in RestricedPython and related Packages which test if Python 1 is used.
+Even if the README claims that Compatibility Support is form Python 2.3 - 2.7 I found some Code in RestrictedPython and related Packages which test if Python 1 is used.
 
 Due to this approach to support all Python 2 Versions the code uses only statements that are compatible with all of those versions.
 
-So oldstyle classes and newstyle classes are mixed,
+So old style classes and new style classes are mixed,
 
 The following language elements are statements and not functions:
 

docs/RestrictedPython4/index.rst

@@ -12,6 +12,9 @@ Zope and Plone should become Python 3 compatible.
 
 One of the core features of Zope 2 and therefore Plone is the possibility to implement and modify Python scripts and templates through the web (TTW) without harming the application or server itself.
 
+As Python is a `Turing Complete`_ programming language programmers don't have any limitation and could potentially harm the Application and Server itself.
+
+RestrictedPython and AccessControl aims on this topic to provide a reduced subset of the Python Programming language, where all functions that could harm the system are permitted by default.
 
 Targeted Versions to support
 ----------------------------

docs/idea.rst

@@ -4,7 +4,7 @@ The Idea behind RestrictedPython
 Python is a `Turing-complete`_ programming language.
 To offer a Python interface for users in web context is a potential security risk.
 Web frameworks and Content Management Systems (CMS) want to offer their users as much extensibility as possible through the web (TTW).
-This also means to have permissions to add functionallity via a Python Script.
+This also means to have permissions to add functionality via a Python Script.
 
 There should be additional preventive measures taken to ensure integrity of the application and the server itself, according to information security best practice and unrelated to Restricted Python.
 
@@ -14,18 +14,18 @@ The `Ada Ravenscar profile`_ is another example of such an approach.
 
 Defining a secure subset of the language involves restricting the `EBNF`_ elements and explicitly allowing or disallowing language features.
 Much of the power of a programming language derives from its standard and contributed libraries, so any calling of these methods must also be checked and potentially restricted.
-RestricedPython generally disallows calls to any library that is not explicit whitelisted.
+RestrictedPython generally disallows calls to any library that is not explicit whitelisted.
 
 As Python is a scripting language that is executed by an interpreter.
-Any Python code that should be executed have to be explict checked before executing a generated byte code by the interpreter.
+Any Python code that should be executed have to be explicit checked before executing a generated byte code by the interpreter.
 
 Python itself offers three methods that provide such a workflow:
 
 * ``compile()`` which compiles source code to byte code
 * ``exec`` / ``exec()`` which executes the byte code in the interpreter
 * ``eval`` / ``eval()`` which executes a byte code expression
 
-Therefore RestrictedPython offers a repacement for the python builtin function ``compile()`` (Python 2: https://docs.python.org/2/library/functions.html#compile / Python 3 https://docs.python.org/3/library/functions.html#compile).
+Therefore RestrictedPython offers a replacement for the python builtin function ``compile()`` (Python 2: https://docs.python.org/2/library/functions.html#compile / Python 3 https://docs.python.org/3/library/functions.html#compile).
 This method is defined as following:
 
 .. code:: Python
@@ -40,16 +40,16 @@ There are three valid string values for ``mode``:
 * ``'eval'``
 * ``'single'``
 
-For RestricedPython this ``compile()`` method is replaced by:
+For RestrictedPython this ``compile()`` method is replaced by:
 
 .. code:: Python
 
-    compile_restriced(source, filename, mode [, flags [, dont_inherit]])
+    compile_restricted(source, filename, mode [, flags [, dont_inherit]])
 
-The primary parameter ``source`` has to be a ASCII or ``unicode`` string.
+The primary parameter ``source`` has to be a ASCII or ``unicode`` string (With Python 2.6 an additional option for source was added: ``ast.AST`` for :ref:`Code generation <code generation>`).
 Both methods either returns compiled byte code that the interpreter could execute or raise exceptions if the provided source code is invalid.
 
-As ``compile`` and ``compile_restricted`` just compile the provided source code to bytecode it is not sufficient to sandbox the environment, as all calls to libraries are still available.
+As ``compile`` and ``compile_restricted`` just compile the provided source code to byte code it is not sufficient to sandbox the environment, as all calls to libraries are still available.
 
 The two methods / Statements:
 
@@ -58,12 +58,12 @@ The two methods / Statements:
 
 have two parameters:
 
-* globals
-* locals
+* ``globals``
+* ``locals``
 
 which are a reference to the Python builtins.
 
-By modifing and restricting the avaliable moules, methods and constants from globals and locals we could limit the possible calls.
+By modifying and restricting the available modules, methods and constants from globals and locals we could limit the possible calls.
 
 Additionally RestrictedPython offers a way to define a policy which allows developers to protect access to attributes.
 This works by defining a restricted version of:
@@ -76,7 +76,7 @@ This works by defining a restricted version of:
 Also RestrictedPython provides three predefined, limited versions of Python's own ``__builtins__``:
 
 * ``safe_builtins`` (by Guards.py)
-* ``limited_builtins`` (by Limits.py), which provides restriced sequence types
+* ``limited_builtins`` (by Limits.py), which provides restricted sequence types
 * ``utilities_builtins`` (by Utilities.py), which provides access for standard modules math, random, string and for sets.
 
 Additional there exist guard functions to make attributes of Python objects immutable --> ``full_write_guard`` (write and delete protected)

docs/update_notes.rst

@@ -1,39 +1,6 @@
 Notes on the Update Process to be Python 3 compatible
 =====================================================
 
-*Note, due to my English, I sometimes fall back to German on describing my ideas and learnings.*
-
-Current state
--------------
-
-Idea of RestrictedPython
-........................
-
-RestrictedPython offers a replacement for the Python builtin function ``compile()`` (https://docs.python.org/2/library/functions.html#compile) which is defined as:
-
-.. code:: Python
-
-    compile(source, filename, mode [, flags [, dont_inherit]])
-
-The definition of compile has changed over the time, but the important element is the param ``mode``, there are three allowed values for this string param:
-
-* ``'exec'``
-* ``'eval'``
-* ``'single'``
-
-RestrictedPython has it origin in the time of Python 1 and early Python 2.
-The optional params ``flags`` and ``dont_inherit`` has been added to Python's ``compile()`` function with Version Python 2.3.
-RestrictedPython never added those new parameters to implementation.
-The definition of
-
-.. code:: Python
-
-    compile_restricted(source, filename, mode)
-
-
-The primary param ``source`` has been restriced to be an ASCII string or ``unicode`` string.
-
-
 
 Also RestrictedPython provides a way to define Policies, by redefining restricted versions of ``print``, ``getattr``, ``setattr``, ``import``, etc..
 As shortcutes it offers three stripped down versions of Pythons ``__builtins__``:
@@ -59,83 +26,6 @@ RestrictedPython based on the
 With Python 2.6 the compiler module with all its sub modules has been declared deprecated with no direct upgrade Path or recommendations for a replacement.
 
 
-Version Support of RestrictedPython 3.6.x
-.........................................
-
-RestrictedPython 3.6.x aims on supporting Python versions:
-
-* 2.0
-* 2.1
-* 2.2
-* 2.3
-* 2.4
-* 2.5
-* 2.6
-* 2.7
-
-Even if the README claims that Compatibility Support is form Python 2.3 - 2.7 I found some Code in RestricedPython and related Packages which test if Python 1 is used.
-
-Due to this approach to support all Python 2 Versions the code uses only statements that are compatible with all of those versions.
-
-So oldstyle classes and newstyle classes are mixed,
-
-The following language elements are statements and not functions:
-
-* exec
-* print
-
-
-
-Goals for Rewrite
------------------
-
-We want to rewrite RestrictedPython as it is one of the core dependencies for the Zope2 Application Server which is the base for the CMS Plone.
-Zope2 should become Python3 compatible.
-
-One of the core features of Zope2 and therefore Plone is the capability to write and modify Code and Templates TTW (through the web).
-As Python is a Turing Complete programming language programmers don't have any limitation and could potentially harm the Application and Server itself.
-
-RestrictedPython and AccessControl aims on this topic to provide a reduced subset of the Python Programming language, where all functions that could harm the system are permitted by default.
-
-Therefore RestrictedPython provide a way to define Policies and
-
-
-
-
-
-Zope2 Core Packages that has RestrictedPython as dependencies
-.............................................................
-
-The following Packages used in Zope2 for Plone depend on RestricedPython:
-
-* AccessControl
-* zope.untrustedpython
-* DocumentTemplate
-* Products.PageTemplates
-* Products.PythonScripts
-* Products.PluginIndexes
-* five.pt (wrapping some functions and protection for Chameleon)
-
-Targeted Versions to support
-............................
-
-For a RestrictedPython 4.0.0+ Update we aim to support only current Python Versions (under active Security Support):
-
-* 2.7
-* 3.4
-* 3.5
-* 3.6
-
-Targeted API
-............
-
-
-.. code:: Python
-
-    compile(source, filename, mode [, flags [, dont_inherit]])
-    compile_restricted(source, filename, mode [, flags [, dont_inherit]])
-
-
 Approach
 --------
 
@@ -153,27 +43,3 @@ Produced byte code has to compatible with the execution environment, the Python
 So we must not generate the byte code that has to be returned from ``compile_restricted`` and the other ``compile_restricted_*`` methods manually, as this might harm the interpreter.
 We actually don't even need that.
 The Python ``compile()`` function introduced the capability to compile ``ast.AST`` code into byte code.
-
-Technical Backgrounds
-.....................
-
-https://docs.python.org/3.5/library/ast.html#abstract-grammar
-
-NodeVistiors (https://docs.python.org/3.5/library/ast.html#ast.NodeVisitor)
-
-NodeTransformer (https://docs.python.org/3.5/library/ast.html#ast.NodeTransformer)
-
-dump (https://docs.python.org/3.5/library/ast.html#ast.dump)
-
-
-
-
-
-
-Links
------
-
-* Concept of Immutable Types and Python Example (https://en.wikipedia.org/wiki/Immutable_object#Python)
-* Python 3 Standard Library Documentation on AST module ``ast`` (https://docs.python.org/3/library/ast.html)
-* Indetail Documentation on the Python AST module ``ast`` (https://greentreesnakes.readthedocs.org/en/latest/)
-* Example how to Instrumenting the Python AST (``ast.AST``) (http://www.dalkescientific.com/writings/diary/archive/2010/02/22/instrumenting_the_ast.html)

docs/upgrade/index.rst

@@ -3,13 +3,16 @@ Concept for a upgrade to Python 3
 
 RestrictedPython is a classic approach of compiler construction to create a limited subset of an existing programming language.
 
-Defining a programming language requires a regular grammar (Chomsky 3 / EBNF) definition.
+Defining a programming language requires a regular grammar (`Chomsky 3`_ / `EBNF`_) definition.
 This grammar will be implemented in an abstract syntax tree (AST), which will be passed on to a code generator to produce a machine-readable version.
 
+Code generation
+---------------
+
 As Python is a platform independent programming language, this machine readable version is a byte code which will be translated on the fly by an interpreter into machine code.
 This machine code then gets executed on the specific CPU architecture, with the standard operating system restrictions.
 
-The bytecode produced must be compatible with the execution environment that the Python interpreter is running in, so we do not generate the byte code directly from ``compile_restricted`` and the other ``compile_restricted_*`` methods manually, it may not match what the interpreter expects.
+The byte code produced must be compatible with the execution environment that the Python interpreter is running in, so we do not generate the byte code directly from ``compile_restricted`` and the other ``compile_restricted_*`` methods manually, it may not match what the interpreter expects.
 
 Thankfully, the Python ``compile()`` function introduced the capability to compile ``ast.AST`` code into byte code in Python 2.6, so we can return the platform-independent AST and keep bytecode generation delegated to the interpreter.
 
@@ -25,41 +28,76 @@ While ``compiler`` still uses the old `CamelCase`_ Syntax (``visitNode(self, nod
 Also the names of classes have been changed, where ``compiler`` uses ``Walker`` and ``Mutator`` the corresponding elements in ``ast.AST`` are ``NodeVisitor`` and ``NodeTransformator``.
 
 
-ast module (Abstract Syntax Trees)
-----------------------------------
+``ast`` module (Abstract Syntax Trees)
+--------------------------------------
 
-The ast module consists of four areas:
+The ``ast`` module consists of four areas:
 
-* AST (Basis of all Nodes) + all node class implementations
-* NodeVisitor and NodeTransformer (tool to consume and modify the AST)
+* ``AST`` (Basis of all Nodes) + all node class implementations
+* ``NodeVisitor`` and ``NodeTransformer`` (tool to consume and modify the AST)
 * Helper methods
 
-  * parse
-  * walk
-  * dump
+  * ``parse``
+  * ``walk``
+  * ``dump``
 
 * Constants
 
-  * PyCF_ONLY_AST
+  * ``PyCF_ONLY_AST``
 
 
-NodeVisitor & NodeTransformer
------------------------------
+``NodeVisitor`` & ``NodeTransformer``
+-------------------------------------
 
-A NodeVisitor is a class of a node / AST consumer, it reads the data by stepping through the tree without modifying it.
-In contrast, a NodeTransformer (which inherits from a NodeVisitor) is allowed to modify the tree and nodes.
+A ``NodeVisitor`` is a class of a node / AST consumer, it reads the data by stepping through the tree without modifying it.
+In contrast, a ``NodeTransformer`` (which inherits from a ``NodeVisitor``) is allowed to modify the tree and nodes.
 
 
-Links
------
+Technical Backgrounds - Links to External Documentation
+---------------------------------------------------------
 
 * Concept of Immutable Types and Python Example (https://en.wikipedia.org/wiki/Immutable_object#Python)
 * Python 3 Standard Library Documentation on AST module ``ast`` (https://docs.python.org/3/library/ast.html)
 
-  * AST Gramar of Python https://docs.python.org/3.5/library/ast.html#abstract-grammar https://docs.python.org/2.7/library/ast.html#abstract-grammar
-  * NodeVistiors (https://docs.python.org/3.5/library/ast.html#ast.NodeVisitor)
-  * NodeTransformer (https://docs.python.org/3.5/library/ast.html#ast.NodeTransformer)
-  * dump (https://docs.python.org/3.5/library/ast.html#ast.dump)
+  * AST Grammar of Python
+
+    * `Python 3.6 AST`_
+    * `Python 3.5 AST`_
+    * `Python 3.4 AST`_
+    * `Python 3.3 AST`_
+    * `Python 3.2 AST`_
+    * `Python 3.1 AST`_
+    * `Python 3.0 AST`_
+    * `Python 2.7 AST`_
+    * `Python 2.6 AST`_
 
-* Indetail Documentation on the Python AST module ``ast`` (https://greentreesnakes.readthedocs.org/en/latest/)
+  * ``NodeVistiors``  (https://docs.python.org/3.5/library/ast.html#ast.NodeVisitor)
+  * ``NodeTransformer``  (https://docs.python.org/3.5/library/ast.html#ast.NodeTransformer)
+  * ``dump`` (https://docs.python.org/3.5/library/ast.html#ast.dump)
+
+* In detail Documentation on the Python AST module ``ast`` (Green Tree Snakes) (https://greentreesnakes.readthedocs.org/en/latest/)
 * Example how to Instrumenting the Python AST (``ast.AST``) (http://www.dalkescientific.com/writings/diary/archive/2010/02/22/instrumenting_the_ast.html)
+
+.. _`CamelCase`: https://en.wikipedia.org/wiki/Camel_case
+
+.. _`EBNF`: https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form
+
+.. _`Chomsky 3`: https://en.wikipedia.org/wiki/Chomsky_hierarchy#Type-3_grammars
+
+.. _`Python 3.6 AST`: https://docs.python.org/3.6/library/ast.html#abstract-grammar
+
+.. _`Python 3.5 AST`: https://docs.python.org/3.5/library/ast.html#abstract-grammar
+
+.. _`Python 3.4 AST`: https://docs.python.org/3.4/library/ast.html#abstract-grammar
+
+.. _`Python 3.3 AST`: https://docs.python.org/3.3/library/ast.html#abstract-grammar
+
+.. _`Python 3.2 AST`: https://docs.python.org/3.2/library/ast.html#abstract-grammar
+
+.. _`Python 3.1 AST`: https://docs.python.org/3.1/library/ast.html#abstract-grammar
+
+.. _`Python 3.0 AST`: https://docs.python.org/3.0/library/ast.html#abstract-grammar
+
+.. _`Python 2.7 AST`: https://docs.python.org/2.7/library/ast.html#abstract-grammar
+
+.. _`Python 2.6 AST`: https://docs.python.org/2.6/library/ast.html#abstract-grammar

docs/upgrade_dependencies/index.rst

@@ -0,0 +1,15 @@
+Upgrade dependencies
+====================
+
+Zope2 Core Packages that has RestrictedPython as dependencies
+-------------------------------------------------------------
+
+The following Packages used in Zope2 for Plone depend on RestricedPython:
+
+* AccessControl
+* zope.untrustedpython
+* DocumentTemplate
+* Products.PageTemplates
+* Products.PythonScripts
+* Products.PluginIndexes
+* five.pt (wrapping some functions and protection for Chameleon)