Merge branch 'Python3_update' of https://github.com/zopefoundation/Re…

…strictedPython into Python3_update

docs_de/RestrictedPython4/index.rst

@@ -26,3 +26,19 @@ For a RestrictedPython 4.0.0+ Update we aim to support only current Python Versi
 * 3.3
 * 3.4
 * 3.5
+
+Abhängigkeiten
+--------------
+
+Folgende Packete haben Abhägigkeiten zu RestrictedPython:
+
+* AccessControl -->
+* zope.untrustedpython --> SelectCompiler
+* DocumentTemplate -->
+* Products.PageTemplates -->
+* Products.PythonScripts -->
+* Products.PluginIndexes -->
+* five.pt (wrapping some functions and procetion for Chameleon) -->
+
+Zusätzlich sind in folgenden Add'ons Abhängigkeiten zu RestrictedPython
+*

docs_de/api/index.rst

@@ -0,0 +1,8 @@
+API von RestrictedPython 4.0
+============================
+
+
+
+.. code:: Python
+
+    restricted_compile(source, filename, mode [, flags [, dont_inherit]])

docs_de/grundlagen/index.rst

@@ -1,4 +1,4 @@
-u.Grundlagen von RestrictedPython und der Sicherheitskonzepte von Zope2
+Grundlagen von RestrictedPython und der Sicherheitskonzepte von Zope2
 =====================================================================
 
 

docs_de/idee.rst

@@ -24,20 +24,25 @@ Die Definition von ``comile()`` hat sich mit der Zeit verändert, aber die relev
 * ``'single'``
 
 Diese ist für RestrictedPython durch folgende Funktion ersetzt:
+
 .. code:: Python
 
     compile_restriced(source, filename, mode [, flags [, dont_inherit]])
 
+Der primäre Parameter ``source`` musste ein ASCII oder ``unicode`` String sein, heute nimmt es auch einen ast.AST auf.
 
-The primary param ``source`` has been restriced to be an ASCII string or ``unicode`` string.
-
+Zusätzlich bietet RestrictedPython einen Weg Policies zu definieren.
+Dies funktioniert über das redefinieren von eingeschränkten (restricted) Versionen von:
 
+* ``print``
+* ``getattr``
+* ``setattr``
+* ``import``
 
-Also RestrictedPython provides a way to define Policies, by redefining restricted versions of ``print``, ``getattr``, ``setattr``, ``import``, etc..
-As shortcutes it offers three stripped down versions of Pythons ``__builtins__``:
+Als Abkürzungen bietet es drei vordefinierte, runtergekürzte Versionen der Python ``__builtins__`` an:
 
 * ``safe_builtins`` (by Guards.py)
 * ``limited_builtins`` (by Limits.py), which provides restriced sequence types
 * ``utilities_builtins`` (by Utilities.py), which provides access for standard modules math, random, string and for sets.
 
-There is also a guard function for making attributes immutable --> ``full_write_guard`` (write and delete protected)
+Zusätzlich git es eine Guard-Function (Schutzfunktion) um Attribute von Python Objekten unveränderbar (immutable) zu machen --> ``full_write_guard`` (Schreib und Lösch-Schutz / write and delete protected)

docs_de/index.rst

@@ -16,15 +16,11 @@ Contents
    :maxdepth: 2
 
    idee
-
-
    grundlagen/index
-   api/index
-
    RestrictedPython3/index
    RestrictedPython4/index
-
    update/index
+   api/index
 
 
 Indices and tables

docs_de/update/index.rst

@@ -0,0 +1,69 @@
+Konzept für das Update auf Python 3
+===================================
+
+
+
+RestrictedPython is a classical approach of compiler construction to create a limited subset of an existing programming language.
+
+As compiler construction do have basic concepts on how to build a Programming Language and Runtime Environment.
+
+Defining a Programming Language means to define a regular grammar (Chomsky 3 / EBNF) first.
+This grammar will be implemented in an abstract syntax tree (AST), which will be passed on to a code generator to produce a machine understandable version.
+
+As Python is a plattform independend programming / scripting language, this machine understandable version is a byte code which will be translated on the fly by an interpreter into machine code.
+This machine code then gets executed on the specific CPU architecture, with all Operating System restriction.
+
+Produced byte code has to compatible with the execution environment, the Python Interpreter within this code is called.
+So we must not generate the byte code that has to be returned from ``compile_restricted`` and the other ``compile_restricted_*`` methods manually, as this might harm the interpreter.
+We actually don't even need that.
+The Python ``compile()`` function introduced the capability to compile ``ast.AST`` code into byte code.
+
+
+compiler.ast --> ast.AST
+------------------------
+
+Aus Sicht des Compilerbaus sind die Konzepte der Module compiler und ast vergleichbar, bzw. ähnlich.
+Primär hat sich mit der Entwicklung von Python ein Styleguide gebildet wie bestimmte Sachen ausgezeichnet werden sollen.
+Während compiler eine alte CamelCase Syntax (``visitNode(self, node, walker)``) nutzt ist in AST die Python übliche ``visit_Node(self, node)`` Syntax heute üblich.
+Auch habe sich die Namen genändert, wärend compiler von ``Walker`` und ``Mutator`` redet heissen die im AST kontext ``NodeVisitor`` und ``NodeTransformator``
+
+
+ast Modul (Abstract Syntax Trees)
+---------------------------------
+
+Das ast Modul besteht aus drei(/vier) Bereichen:
+
+* AST (Basis aller Nodes) + aller Node Classen Implementierungen
+* NodeVisitor und NodeTransformer (Tools zu Ver-/Bearbeiten des AST)
+* Helper-Methoden
+
+  * parse
+  * walk
+  * dump
+
+* Constanten
+
+  * PyCF_ONLY_AST
+
+
+NodeVisitor & NodeTransformer
+-----------------------------
+
+Ein NodeVisitor ist eine Klasse eines Node / AST Konsumenten beim durchlaufen des AST-Baums.
+Ein Visitor liest Daten aus dem Baum verändert aber nichts am Baum, ein Transformer - vom Visitor abgeleitet - erlaubt modifikationen am Baum, bzw. den einzelnen Knoten.
+
+
+
+Links
+-----
+
+* Concept of Immutable Types and Python Example (https://en.wikipedia.org/wiki/Immutable_object#Python)
+* Python 3 Standard Library Documentation on AST module ``ast`` (https://docs.python.org/3/library/ast.html)
+
+  * AST Gramar of Python https://docs.python.org/3.5/library/ast.html#abstract-grammar https://docs.python.org/2.7/library/ast.html#abstract-grammar
+  * NodeVistiors (https://docs.python.org/3.5/library/ast.html#ast.NodeVisitor)
+  * NodeTransformer (https://docs.python.org/3.5/library/ast.html#ast.NodeTransformer)
+  * dump (https://docs.python.org/3.5/library/ast.html#ast.dump)
+
+* Indetail Documentation on the Python AST module ``ast`` (https://greentreesnakes.readthedocs.org/en/latest/)
+* Example how to Instrumenting the Python AST (``ast.AST``) (http://www.dalkescientific.com/writings/diary/archive/2010/02/22/instrumenting_the_ast.html)

src/RestrictedPython/MutatingVisitor.py

@@ -0,0 +1,76 @@
+##############################################################################
+#
+# Copyright (c) 2002 Zope Foundation and Contributors.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE
+#
+##############################################################################
+
+from ast import NodeVisitor
+
+import ast
+
+ListType = type([])
+TupleType = type(())
+SequenceTypes = (ListType, TupleType)
+
+
+class MutatingVisitor(NodeVisitor):
+
+    def __init__(self, visitor):
+        self.visitor = visitor
+        self._cache = {}
+
+    def defaultVisitNode(self, node, walker=None, exclude=None):
+        for name, child in node.__dict__.items():
+            if exclude is not None and name in exclude:
+                continue
+            v = self.dispatchObject(child)
+            if v is not child:
+                # Replace the node.
+                node.__dict__[name] = v
+        return node
+
+    def visit_Sequence(self, seq):
+        res = seq
+        for idx in range(len(seq)):
+            child = seq[idx]
+            v = self.dispatchObject(child)
+            if v is not child:
+                # Change the sequence.
+                if type(res) is ListType:
+                    res[idx: idx + 1] = [v]
+                else:
+                    res = res[:idx] + (v,) + res[idx + 1:]
+        return res
+
+    def dispatchObject(self, ob):
+        '''
+        Expected to return either ob or something that will take
+        its place.
+        '''
+        if isinstance(ob, ast.Node):
+            return self.dispatchNode(ob)
+        elif type(ob) in SequenceTypes:
+            return self.visitSequence(ob)
+        else:
+            return ob
+
+    def dispatchNode(self, node):
+        klass = node.__class__
+        meth = self._cache.get(klass, None)
+        if meth is None:
+            className = klass.__name__
+            meth = getattr(self.visitor, 'visit' + className,
+                           self.defaultVisitNode)
+            self._cache[klass] = meth
+        return meth(node, self)
+
+
+def walk(tree, visitor):
+    return MutatingWalker(visitor).dispatchNode(tree)

src/RestrictedPython/RCompile.py

@@ -34,6 +34,7 @@
 
 
 def niceParse(source, filename, mode):
+    import ipdb; ipdb.set_trace()
     if isinstance(source, unicode):
         # Use the utf-8-sig BOM so the compiler
         # detects this as a UTF-8 encoded string.
@@ -68,8 +69,8 @@ def parse(self):
         return code
 
     def _get_tree(self):
-        tree = self.parse()
-        MutatingWalker.walk(tree, self.rm)
+        c_tree = self.parse()
+        MutatingWalker.walk(c_tree, self.rm)
         if self.rm.errors:
             raise SyntaxError(self.rm.errors[0])
         c_misc.set_filename(self.filename, tree)