Merge branch 'master' into py38

.gitignore

@@ -60,6 +60,7 @@ secret.cfg
 /.ropeproject/
 /.project
 /.pydevproject
+.mypy_cache/
 
 # Import Data
 /*.csv

README.rst

@@ -25,7 +25,7 @@ This would not harm any system.
 .. code-block:: pycon
 
     >>> from RestrictedPython import compile_restricted
-    >>> from RestrictedPython import safe_builtins
+    >>> from RestrictedPython import safe_globals
     >>>
     >>> source_code = """
     ... def example():
@@ -34,7 +34,7 @@ This would not harm any system.
     >>>
     >>> loc = {}
     >>> byte_code = compile_restricted(source_code, '<inline>', 'exec')
-    >>> exec(byte_code, safe_builtins, loc)
+    >>> exec(byte_code, safe_globals, loc)
     >>>
     >>> loc['example']()
     'Hello World!'
@@ -47,14 +47,14 @@ This example directly executed in Python could harm your system.
 .. code-block:: pycon
 
     >>> from RestrictedPython import compile_restricted
-    >>> from RestrictedPython import safe_builtins
+    >>> from RestrictedPython import safe_globals
     >>>
     >>> source_code = """
     ... import os
     ...
     ... os.listdir('/')
     ... """
     >>> byte_code = compile_restricted(source_code, '<inline>', 'exec')
-    >>> exec(byte_code, {'__builtins__': safe_builtins}, {})
+    >>> exec(byte_code, safe_globals, {})
     Traceback (most recent call last):
     ImportError: __import__ not found

appveyor.yml

@@ -1,39 +1,22 @@
 # AppVeyor CI settings (Windows Machine CI Tests)
 
-matrix:
-  allow_failures:
-  - PROFILE: py38
-
 environment:
   matrix:
     - PROFILE: py27-conventions
-      PYTHON_VERSION: 2.7"
-      TOXENV: "lint,docs"
-    - PROFILE: py35-conventions
-      PYTHON_VERSION: 3.6"
-      TOXENV: "lint,docs"
-    - PROFILE: py36-conventions
-      PYTHON_VERSION: 3.6"
-      TOXENV: "lint,docs"
-    - PROFILE: py37-conventions
-      PYTHON_VERSION: 3.6"
+      PYTHON_VERSION: "2.7"
       TOXENV: "lint,docs"
     - PROFILE: py27
-      PYTHON_VERSION: 2.7"
+      PYTHON_VERSION: "2.7"
       TOXENV: "py27,py27-datetime"
     - PROFILE: py35
-      PYTHON_VERSION: 3.5"
+      PYTHON_VERSION: "3.5"
       TOXENV: "py35,py35-datetime"
     - PROFILE: py36
-      PYTHON_VERSION: 3.6"
+      PYTHON_VERSION: "3.6"
       TOXENV: "py36,py36-datetime"
     - PROFILE: py37
-      PYTHON_VERSION: 3.7"
+      PYTHON_VERSION: "3.7"
       TOXENV: "py37,py37-datetime"
-    - PROFILE: py38
-      PYTHON_VERSION: 3.8"
-      TOXENV: "py38,py38-datetime"
-
 
 
 cache:

docs/CHANGES.rst

@@ -1,12 +1,27 @@
 Changes
 =======
 
-4.0b7 (unreleased)
+4.0b8 (unreleased)
 ------------------
 
 - Nothing changed yet.
 
 
+4.0b7 (2018-10-30)
+------------------
+
+- Fix documentation issue which leads to unnecessary unsafe code: Never use
+  ``safe_builtins`` as argument of ``compile_resticted`` but use
+  ``dict(__builtins__=safe_builtins)`` or the newly introduced
+  ``safe_globals``. Otherwise the default Python built-ins which are not in
+  ``safe_builtins`` can still be accessed.
+  (`#142 <https://github.com/zopefoundation/RestrictedPython/issues/142>`_)
+
+- Improve `.Guards.safer_getattr` to prevent accessing names starting with
+  underscore.
+  (`#142 <https://github.com/zopefoundation/RestrictedPython/issues/142>`_)
+
+
 4.0b6 (2018-10-05)
 ------------------
 

docs/RestrictedPython4/index.rst

@@ -12,26 +12,23 @@ Zope and Plone should become Python 3 compatible.
 
 One of the core features of Zope 2 and therefore Plone is the possibility to implement and modify Python scripts and templates through the web (TTW) without harming the application or server itself.
 
-As Python is a `Turing complete`_ programming language programmers don't have any limitation and could potentially harm the Application and Server itself.
+As Python is a `Turing complete <https://en.wikipedia.org/wiki/Turing_completeness>`_ programming language programmers don't have any limitation and could potentially harm the Application and Server itself.
 
 RestrictedPython and AccessControl aims on this topic to provide a reduced subset of the Python Programming language, where all functions that could harm the system are permitted by default.
 
 Targeted Versions to support
 ----------------------------
 
 For the RestrictedPython 4 update we aim to support only current Python
-versions (the ones that will have active `security support`_ after this update
+versions (the ones that will have active `security support <https://docs.python.org/devguide/index.html#branchstatus>`_ after this update
 will be completed):
 
 * 2.7
-* 3.4
 * 3.5
 * 3.6
+* 3.7
 * PyPy2.7
 
-.. _`security support` : https://docs.python.org/devguide/index.html#branchstatus
-.. _`Turing complete`: https://en.wikipedia.org/wiki/Turing_completeness
-
 We explicitly excluded Python 3.3 and PyPy3 (which is based on the Python 3.3 specification) as the changes in Python 3.4 are significant and the Python 3.3 is nearing the end of its supported lifetime.
 
 Dependencies

docs/idea.rst

@@ -1,7 +1,7 @@
 The Idea behind RestrictedPython
 ================================
 
-Python is a `Turing complete`_ programming language.
+Python is a `Turing complete <https://en.wikipedia.org/wiki/Turing_completeness>`_ programming language.
 To offer a Python interface for users in web context is a potential security risk.
 Web frameworks and Content Management Systems (CMS) want to offer their users as much extensibility as possible through the web (TTW).
 This also means to have permissions to add functionality via a Python Script.
@@ -10,9 +10,9 @@ There should be additional preventive measures taken to ensure integrity of the
 
 RestrictedPython defines a safe subset of the Python programming language.
 This is a common approach for securing a programming language.
-The `Ada Ravenscar profile`_ is another example of such an approach.
+The `Ada Ravenscar profile <https://en.wikipedia.org/wiki/Ravenscar_profile>`_ is another example of such an approach.
 
-Defining a secure subset of the language involves restricting the `EBNF`_ elements and explicitly allowing or disallowing language features.
+Defining a secure subset of the language involves restricting the `EBNF <https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form>`_ elements and explicitly allowing or disallowing language features.
 Much of the power of a programming language derives from its standard and contributed libraries, so any calling of these methods must also be checked and potentially restricted.
 RestrictedPython generally disallows calls to any library that is not explicit whitelisted.
 
@@ -25,7 +25,7 @@ Python itself offers three methods that provide such a workflow:
 * ``exec`` / ``exec()`` which executes the byte code in the interpreter
 * ``eval`` / ``eval()`` which executes a byte code expression
 
-Therefore RestrictedPython offers a replacement for the python builtin function ``compile()`` (Python 2: https://docs.python.org/2/library/functions.html#compile / Python 3 https://docs.python.org/3/library/functions.html#compile).
+Therefore RestrictedPython offers a replacement for the python builtin function ``compile()`` (`Python 2 <https://docs.python.org/2/library/functions.html#compile>`_ / `Python 3 <https://docs.python.org/3/library/functions.html#compile>`_).
 This method is defined as following:
 
 .. code-block:: python
@@ -79,8 +79,8 @@ Also RestrictedPython provides three predefined, limited versions of Python's ow
 * ``limited_builtins`` (by Limits.py), which provides restricted sequence types
 * ``utilities_builtins`` (by Utilities.py), which provides access for standard modules math, random, string and for sets.
 
-Additional there exist guard functions to make attributes of Python objects immutable --> ``full_write_guard`` (write and delete protected)
+One special shortcut:
+
+* ``safe_globals`` for ``{'__builtins__': safe_builtins}`` (by Guards.py)
 
-.. _`Turing complete`: https://en.wikipedia.org/wiki/Turing_completeness
-.. _Ada Ravenscar Profile: https://en.wikipedia.org/wiki/Ravenscar_profile
-.. _EBNF: https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form
+Additional there exist guard functions to make attributes of Python objects immutable --> ``full_write_guard`` (write and delete protected)

docs/index.rst

@@ -12,7 +12,7 @@ Welcome to RestrictedPython's documentation!
 Supported Python versions
 =========================
 
-RestrictedPython support CPython 2.7, 3.4, 3.5 and 3.6.
+RestrictedPython support CPython 2.7, 3.5, 3.6 and 3.7.
 It does _not_ support PyPy or other alternative Python implementations.
 
 Contents

docs/old_README.rst

@@ -61,9 +61,12 @@ Specifically:
    takes two arguments.  The first is the base object to be accessed,
    while the second is the attribute name or item index that will be
    read.  The guard function should return the attribute or subitem,
-   or raise an exception. RestrictedPython ships with a default implementation
-   for ``_getattr_`` which prevents from using the format method of
-   strings as it is considered harmful.
+   or raise an exception.
+   RestrictedPython ships with a default implementation
+   for ``_getattr_`` which prevents the following actions:
+
+   * accessing an attribute those name start with an underscore
+   * accessing the format method of strings as it is considered harmful.
 
 4. ``__import__`` is the normal Python import hook, and should be used
    to control access to Python packages and modules.

docs/upgrade/index.rst

@@ -3,7 +3,7 @@ Concept for a upgrade to Python 3
 
 RestrictedPython is a classic approach of compiler construction to create a limited subset of an existing programming language.
 
-Defining a programming language requires a regular grammar (`Chomsky 3`_ / `EBNF`_) definition.
+Defining a programming language requires a regular grammar (`Chomsky 3 <https://en.wikipedia.org/wiki/Chomsky_hierarchy#Type-3_grammars>`_ / `EBNF <https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form>`_) definition.
 This grammar will be implemented in an abstract syntax tree (AST), which will be passed on to a code generator to produce a machine-readable version.
 
 .. _`_sec_code_generation`:
@@ -27,7 +27,7 @@ So we need to move from ``compiler.ast`` to ``ast`` to support newer Python vers
 
 From the point of view of compiler design, the concepts of the ``compiler`` module and the ``ast`` module are similar.
 The ``compiler`` module predates several major improvements of the Python development like a generally applied style guide.
-While ``compiler`` still uses the old `CamelCase`_ Syntax (``visitNode(self, node, walker)``) the ``ast.AST`` did now use the Python common ``visit_Node(self, node)`` syntax.
+While ``compiler`` still uses the old `CamelCase <https://en.wikipedia.org/wiki/Camel_case>`_ Syntax (``visitNode(self, node, walker)``) the ``ast.AST`` did now use the Python common ``visit_Node(self, node)`` syntax.
 Also the names of classes have been changed, where ``compiler`` uses ``Walker`` and ``Mutator`` the corresponding elements in ``ast.AST`` are ``NodeVisitor`` and ``NodeTransformator``.
 
 ``ast`` module (Abstract Syntax Trees)
@@ -64,43 +64,20 @@ Technical Backgrounds - Links to External Documentation
 
   * AST Grammar of Python
 
-    * `Python 3.6 AST`_
-    * `Python 3.5 AST`_
-    * `Python 3.4 AST`_
-    * `Python 3.3 AST`_
-    * `Python 3.2 AST`_
-    * `Python 3.1 AST`_
-    * `Python 3.0 AST`_
-    * `Python 2.7 AST`_
-    * `Python 2.6 AST`_
+    * `Python 3.7 AST <https://docs.python.org/3.7/library/ast.html#abstract-grammar>`_
+    * `Python 3.6 AST <https://docs.python.org/3.6/library/ast.html#abstract-grammar>`_
+    * `Python 3.5 AST <https://docs.python.org/3.5/library/ast.html#abstract-grammar>`_
+    * `Python 3.4 AST <https://docs.python.org/3.4/library/ast.html#abstract-grammar>`_
+    * `Python 3.3 AST <https://docs.python.org/3.3/library/ast.html#abstract-grammar>`_
+    * `Python 3.2 AST <https://docs.python.org/3.2/library/ast.html#abstract-grammar>`_
+    * `Python 3.1 AST <https://docs.python.org/3.1/library/ast.html#abstract-grammar>`_
+    * `Python 3.0 AST <https://docs.python.org/3.0/library/ast.html#abstract-grammar>`_
+    * `Python 2.7 AST <https://docs.python.org/2.7/library/ast.html#abstract-grammar>`_
+    * `Python 2.6 AST <https://docs.python.org/2.6/library/ast.html#abstract-grammar>`_
 
   * ``NodeVistiors``  (https://docs.python.org/3.5/library/ast.html#ast.NodeVisitor)
   * ``NodeTransformer``  (https://docs.python.org/3.5/library/ast.html#ast.NodeTransformer)
   * ``dump`` (https://docs.python.org/3.5/library/ast.html#ast.dump)
 
 * In detail Documentation on the Python AST module ``ast`` (Green Tree Snakes) (https://greentreesnakes.readthedocs.org/en/latest/)
 * Example how to Instrumenting the Python AST (``ast.AST``) (http://www.dalkescientific.com/writings/diary/archive/2010/02/22/instrumenting_the_ast.html)
-
-.. _`CamelCase`: https://en.wikipedia.org/wiki/Camel_case
-
-.. _`EBNF`: https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form
-
-.. _`Chomsky 3`: https://en.wikipedia.org/wiki/Chomsky_hierarchy#Type-3_grammars
-
-.. _`Python 3.6 AST`: https://docs.python.org/3.6/library/ast.html#abstract-grammar
-
-.. _`Python 3.5 AST`: https://docs.python.org/3.5/library/ast.html#abstract-grammar
-
-.. _`Python 3.4 AST`: https://docs.python.org/3.4/library/ast.html#abstract-grammar
-
-.. _`Python 3.3 AST`: https://docs.python.org/3.3/library/ast.html#abstract-grammar
-
-.. _`Python 3.2 AST`: https://docs.python.org/3.2/library/ast.html#abstract-grammar
-
-.. _`Python 3.1 AST`: https://docs.python.org/3.1/library/ast.html#abstract-grammar
-
-.. _`Python 3.0 AST`: https://docs.python.org/3.0/library/ast.html#abstract-grammar
-
-.. _`Python 2.7 AST`: https://docs.python.org/2.7/library/ast.html#abstract-grammar
-
-.. _`Python 2.6 AST`: https://docs.python.org/2.6/library/ast.html#abstract-grammar

docs/upgrade_dependencies/index.rst

@@ -28,6 +28,7 @@ RestrictedPython did move some of the imports to the base namespace, so you shou
 
 * predefined built-ins:
 
+  * ``from RestrictedPython import safe_globals``
   * ``from RestrictedPython import safe_builtins``
   * ``from RestrictedPython import limited_builtins``
   * ``from RestrictedPython import utility_builtins``

docs/usage/api.rst

@@ -130,6 +130,7 @@ RestrictedPython has tree major scopes:
 
 2. restricted builtins
 
+  * ``safe_globals``
   * ``safe_builtins``
   * ``limited_builtins``
   * ``utility_builtins``

docs/usage/basic_usage.rst

@@ -86,12 +86,14 @@ So you normally end up using:
             filename='<inline code>',
             mode='exec'
         )
-        exec(byte_code, safe_builtins, None)
+        exec(byte_code, {'__builtins__': safe_builtins}, None)
     except SyntaxError as e:
         pass
 
 One common advanced usage would be to define an own restricted builtin dictionary.
 
+There is a shortcut for ``{'__builtins__': safe_builtins}`` named ``safe_globals`` which can be imported from ``RestrictedPython``.
+
 Necessary setup
 ---------------
 
@@ -104,4 +106,8 @@ To use classes in Python 3
 To use ``for`` statements and comprehensions
     ``_iter_unpack_sequence_`` must point to :func:`RestrictedPython.Guards.guarded_iter_unpack_sequence`.
 
+To use ``getattr``
+    you have to provide an implementation for it.
+    :func:``RestrictedPython.Guards.safer_getattr`` can be a starting point.
+
 The usage of `RestrictedPython` in :mod:`AccessControl.ZopeGuards` can serve as example.

docs/usage/policy.rst

@@ -24,6 +24,8 @@ Predefined builtins
 * ``limited_builtins`` which provides restricted sequence types,
 * ``utility_builtins`` which provides access for standard modules math, random, string and for sets.
 
+* ``safe_globals`` a shortcut for ``glb = {'__builtins__': safe_builtins}``
+
 Guards
 ......
 
@@ -33,6 +35,7 @@ Guards
 
 RestrictedPython predefines several guarded access and manipulation methods:
 
+* ``safer_getattr``
 * ``guarded_setattr``
 * ``guarded_delattr``
 * ``guarded_iter_unpack_sequence``

setup.py

@@ -32,8 +32,8 @@ def read(*rnames):
 
 setup(
     name='RestrictedPython',
-    version='4.0b7.dev0',
-    url='http://pypi.python.org/pypi/RestrictedPython',
+    version='4.0b8.dev0',
+    url='https://github.com/zopefoundation/RestrictedPython',
     license='ZPL 2.1',
     description=(
         'RestrictedPython is a defined subset of the Python language which '

src/RestrictedPython/Guards.py

@@ -256,7 +256,7 @@ def guarded_delattr(object, name):
 safe_builtins['delattr'] = guarded_delattr
 
 
-def safer_getattr(object, name, getattr=getattr):
+def safer_getattr(object, name, default=None, getattr=getattr):
     """Getattr implementation which prevents using format on string objects.
 
     format() is considered harmful:
@@ -266,7 +266,12 @@ def safer_getattr(object, name, getattr=getattr):
     if isinstance(object, _compat.basestring) and name == 'format':
         raise NotImplementedError(
             'Using format() on a %s is not safe.' % object.__class__.__name__)
-    return getattr(object, name)
+    if name.startswith('_'):
+        raise AttributeError(
+            '"{name}" is an invalid attribute name because it '
+            'starts with "_"'.format(name=name)
+        )
+    return getattr(object, name, default)
 
 
 safe_builtins['_getattr_'] = safer_getattr
@@ -308,3 +313,6 @@ def guarded_unpack_sequence(it, spec, _getiter_):
         ret[idx] = guarded_unpack_sequence(ret[idx], child_spec, _getiter_)
 
     return ret
+
+
+safe_globals = {'__builtins__': safe_builtins}

src/RestrictedPython/__init__.py

@@ -27,6 +27,7 @@
 
 # predefined builtins
 from RestrictedPython.Guards import safe_builtins  # isort:skip
+from RestrictedPython.Guards import safe_globals  # isort:skip
 from RestrictedPython.Limits import limited_builtins  # isort:skip
 from RestrictedPython.Utilities import utility_builtins  # isort:skip