Merge 8ddc10a into ce1ecab

docs/collapse_literals.rst

@@ -70,3 +70,4 @@ If the branch is constant, and thus known at decoration time, then this flaw won
 .. todo:: Support sets?
 .. todo:: Always commit changes within a block, and only mark values as non-deterministic outside of conditional blocks
 .. todo:: Support list/set/dict comprehensions
+.. todo:: Support known elements of format strings (JoinedStr) in python 3.6+

docs/inline.rst

@@ -6,12 +6,12 @@ Inlining Functions
 Inline specified functions into the decorated function. Unlike in C, this directive is placed not on the function getting inlined, but rather the function into which it's getting inlined (since that's the one whose code needs to be modified and hence decorated). Currently, this is implemented in the following way:
 
 - When a function is called, its call code is placed within the current code block immediately before the line where its value is needed
-- The code is wrapped in a one-iteration ``for`` loop (effectively a ``do {} while(0)``), and the ``return`` statement is replaced by a ``break``
+- The code is wrapped in a ``try/except`` block, and the return value is passed back out using a special exception type
 - Arguments are stored into a dictionary, and variadic keyword arguments are passed as ``dict_name.update(kwargs)``; this dictionary has the name ``_[funcname]`` where ``funcname`` is the name of the function being inlined, so other variables of this name should not be used or relied upon
 - The return value is assigned to the function name as well, deleting the argument dictionary, freeing its memory, and making the return value usable when the function's code is exited by the ``break``
 - The call to the function is replaced by the variable holding the return value
 
-As a result, ``pragma.inline`` cannot currently handle functions which contain a ``return`` statement within a loop. Since Python doesn't support anything like ``goto`` besides wrapping the code in a function (which this function implicitly shouldn't do), I don't know how to surmount this problem. Without much effort, it can be overcome by tailoring the function to be inlined.
+As a result, ``pragma.inline`` cannot currently handle functions which contain a ``return`` statement within a bare ``try/except`` or ``except BaseException``. Since Python doesn't support anything like ``goto`` besides wrapping the code in a function (which this function implicitly shouldn't do), I don't know how to surmount this problem. Without much effort, it can be overcome by tailoring the function to be inlined. In general, it's bad practice to use a bare ``except:`` or ``except BaseException:``, and such calls should generally be replaced with ``except Exception:``, which would this issue.
 
 To inline a function ``f`` into the code of another function ``g``, use ``pragma.inline(g)(f)``, or, as a decorator::
 
@@ -23,49 +23,35 @@ To inline a function ``f`` into the code of another function ``g``, use ``pragma
         z = y + 3
         return f(z * 4)
 
-    # ... g Becomes something like ...
+    # ... g Becomes...
 
     def g(y):
         z = y + 3
-        _f = dict(x=z * 4)  # Store arguments
-        for ____ in [None]:  # Function body
-            _f['return'] = _f['x'] ** 2  # Store the "return"ed value
-            break  # Return, terminate the function body
-        _f_return = _f.get('return', None)  # Retrieve the returned value
-        del _f  # Discard everything else
-        return _f_return
-
-This loop can be removed, if it's not necessary, using :func:``pragma.unroll``. This can be accomplished if there are no returns within a conditional or loop block. In this case::
-
-    def f(x):
-        return x**2
-
-    @pragma.unroll
-    @pragma.inline(f)
-    def g(y):
-        z = y + 3
-        return f(z * 4)
-
-    # ... g Becomes ...
-
-    def g(y):
-        z = y + 3
-        _f = {}
-        _f['x'] = z * 4
-        _f = _f['x'] ** 2
-        return _f
-
-It needs to be noted that, besides arguments getting stored into a dictionary, other variable names remain unaltered when inlined. Thus, if there are shared variable names in the two functions, they might overwrite each other in the resulting inlined function.
+        _f_0 = dict(x=z * 4)
+        try:  # Function body
+            raise _PRAGMA_INLINE_RETURN(_f_0['x'] ** 2)
+        except _PRAGMA_INLINE_RETURN as _f_return_0_exc:
+            _f_return_0 = _f_return_0_exc.return_val
+        else:
+            _f_return_0 = None
+        finally:  # Discard artificial stack frame
+            del _f_0
+        return _f_return_0
 
 .. todo:: Fix name collision by name-mangling non-free variables
 
-Eventually, this could be collapsed using :func:``pragma.collapse_literals``, to produce simply ``return ((y + 3) * 4) ** 2``, but dictionaries aren't yet supported for collapsing.
+Eventually, this could be collapsed using :func:``pragma.collapse_literals``, to produce simply ``return ((y + 3) * 4) ** 2``, but there are numerous hurtles in the way toward making this happen.
 
-When inlining a generator function, the function's results are collapsed into a list, which is then returned. This will break in two main scenarios:
+When inlining a generator function, the function's results are collapsed into a list, which is then returned. This is equivalent to calling ``list(generator_func(*args, **kwargs))``. This will break in two main scenarios:
 
 - The generator never ends, or consumes excessive amounts of resources.
-- The calling code relies on the resulting generator being more than just iterable.
+- The calling code relies on the resulting generator being more than just iterable, e.g. if data is passed back in using calls to ``next``.
+
+.. todo:: Fix generators to return something more like ``iter(list(f(*args, **kwargs))``, since ``list`` itself is not an iterator, but the return of a generator is.
 
 In general, either this won't be an issue, or you should know better than to try to inline the infinite generator.
 
-.. todo:: Support inlining a generator into another generator by merging the functions together. E.g., ``for x in my_range(5): yield x + 2`` becomes ``i = 0; while i < 5: yield i + 2; i += 1`` (or something vaguely like that).
+.. todo:: Support inlining a generator into another generator by merging the functions together. E.g., ``for x in my_range(5): yield x + 2`` becomes ``i = 0; while i < 5: yield i + 2; i += 1`` (or something vaguely like that).
+.. todo:: Support inlining closures; if the inlined function refers to global or nonlocal variables, import them into the closure of the final function.
+
+Recursive calls are handled by keeping a counter of the inlined recursion depth, and changing the suffix number of the local variables dictionary (e.g., ``_f_0``). These dictionaries serve as stack frames: their unique naming permits multiple, even stacked, inlined function calls, and their deletion enforces the usual life span of function-local variables.

pragma/collapse_literals.py

@@ -8,9 +8,10 @@
 # noinspection PyPep8Naming
 class CollapseTransformer(TrackedContextTransformer):
     def visit_Name(self, node):
-        res = self.resolve_literal(node)
-        if isinstance(res, primitive_ast_types):
-            return res
+        if isinstance(node.ctx, ast.Load):
+            res = self.resolve_literal(node)
+            if isinstance(res, primitive_ast_types):
+                return res
         return node
 
     def visit_BinOp(self, node):

pragma/core/transformer.py

@@ -36,21 +36,27 @@ def function_ast(f):
 
 class DebugTransformerMixin:  # pragma: nocover
     def visit(self, node):
-        orig_node_code = astor.to_source(node).strip()
-        log.debug("Starting to visit >> {} << ({})".format(orig_node_code, type(node)))
+        cls = type(self).__name__
+
+        try:
+            orig_node_code = astor.to_source(node).strip()
+        except Exception as ex:
+            log.error("{} ({})".format(type(node), astor.dump_tree(node)), exc_info=ex)
+            raise ex
+        log.debug("{} Starting to visit >> {} << ({})".format(cls, orig_node_code, type(node)))
 
         new_node = super().visit(node)
 
         try:
             if new_node is None:
-                log.debug("Deleted >>> {} <<<".format(orig_node_code))
+                log.debug("{} Deleted >>> {} <<<".format(cls, orig_node_code))
             elif isinstance(new_node, ast.AST):
-                log.debug("Converted >>> {} <<< to >>> {} <<<".format(orig_node_code, astor.to_source(new_node).strip()))
+                log.debug("{} Converted >>> {} <<< to >>> {} <<<".format(cls, orig_node_code, astor.to_source(new_node).strip()))
             elif isinstance(new_node, list):
-                log.debug("Converted >>> {} <<< to [[[ {} ]]]".format(orig_node_code, ", ".join(
+                log.debug("{} Converted >>> {} <<< to [[[ {} ]]]".format(cls, orig_node_code, ", ".join(
                     astor.to_source(n).strip() for n in new_node)))
         except Exception as ex:
-            log.error("Failed on {} >>> {}".format(orig_node_code, astor.dump_tree(new_node)), exc_info=ex)
+            log.error("{} Failed on {} >>> {}".format(cls, orig_node_code, astor.dump_tree(new_node)), exc_info=ex)
             raise ex
 
         return new_node
@@ -237,11 +243,12 @@ def assign(self, name, val):
             log.debug("Failed to assign {}={}, rvalue cannot be converted to AST".format(name, val))
 
     def visit_Assign(self, node):
-        node.value = self.visit(node.value)
+        node = self.generic_visit(node)
         self.assign(node.targets, node.value)
         return node
 
     def visit_AugAssign(self, node):
+        node.target = self.visit(node.target)
         node = copy.deepcopy(node)
         node.value = self.visit(node.value)
         new_val = self.resolve_literal(ast.BinOp(op=node.op, left=node.target, right=node.value))
@@ -292,40 +299,46 @@ def visit_ClassDef(self, node):
 
     def visit_For(self, node):
         node.iter = self.visit(node.iter)
+        node.target = self.visit(node.target)
         node.body = self.nested_visit(node.body)
         node.orelse = self.nested_visit(node.orelse)
-        return self.generic_visit_less(node, 'body', 'orelse', 'iter')
+        return node
 
     def visit_AsyncFor(self, node):
         node.iter = self.visit(node.iter)
+        node.target = self.visit(node.target)
         node.body = self.nested_visit(node.body)
         node.orelse = self.nested_visit(node.orelse)
-        return self.generic_visit_less(node, 'body', 'orelse', 'iter')
+        return node
 
     def visit_While(self, node):
+        node.test = self.visit(node.test)
         node.body = self.nested_visit(node.body)
         node.orelse = self.nested_visit(node.orelse)
-        return self.generic_visit_less(node, 'body', 'orelse')
+        return node
 
     def visit_If(self, node):
         node.test = self.visit(node.test)
         node.body = self.nested_visit(node.body)
         node.orelse = self.nested_visit(node.orelse)
-        return self.generic_visit_less(node, 'body', 'orelse', 'test')
+        return node
 
     def visit_With(self, node):
+        node.items = self.nested_visit(node.items, set_conditional_exec=False)
         node.body = self.nested_visit(node.body, set_conditional_exec=False)
-        return self.generic_visit_less(node, 'body')
+        return node
 
     def visit_AsyncWith(self, node):
+        node.items = self.nested_visit(node.items, set_conditional_exec=False)
         node.body = self.nested_visit(node.body, set_conditional_exec=False)
-        return self.generic_visit_less(node, 'body')
+        return node
 
     def visit_Try(self, node):
         node.body = self.nested_visit(node.body)
+        node.handlers = self.nested_visit(node.handlers)
         node.orelse = self.nested_visit(node.orelse)
         node.finalbody = self.nested_visit(node.finalbody, set_conditional_exec=False)
-        return self.generic_visit_less(node, 'body', 'orelse', 'finalbody')
+        return node
 
     def visit_Module(self, node):
         node.body = self.nested_visit(node.body, set_conditional_exec=False)