Python resultant block evaluation

This Python library provides the block_eval function which is similar to the the standard eval function, but

• can handle multiple statements like the exec call
• returns the result of the last statement if it is an expression

This library uses only modules available in the standard library, namely

• ast for analysis of the abstract syntax tree of the block to be evaluated
• inspect for capturing variable scope in the calling frame to fully emulate eval behavior

In some ways it is similar how IPython evaluates blocks of code. It can be also thought of as an equivalent of the progn function in Lisp.

Example in interactive mode

In this example the factorial of 3 is calculated in a for loop. This shows how to execute blocks of code in interactive mode.

>>> block_eval("""
... f = 1
... for i in range(1, 3+1):
...     f *= i
... f
... """)
6
>>> f * 2
12

Note that

• in normal interactive mode an empty line would have to follow after f *= 1 to close the for loop
• the value of f is returned since it is an expression as is the last statement in the block
• the defined variables are later accessible as they are are evaluated in the current scope.

Caveat and workaround/alternative approach in non-interactive mode

Because Python uses optimized fast access to local variables, they may not be updated during evaluation, see http://bugs.python.org/issue4831 for details. Therefore, the example above may not work in non-top-level scopes (e.g. functions calling other functions), the f variable will not be defined after evaluation. Nevertheless, it should work in interactive interpreted mode as in the example.

A workaround is to use the split_block function to split the code block into expr and eval parts of the block as code objects, which can then be executed in the given order via eval in the given scope which will properly handle the local scope internally. For an example see the test suite.

In some situations this approach of explicitly evaluating first the exec part and then the eval part may be even preferable, because it makes it possible to separate the side effects (e.g. printing output) of the expr part from the eval part.

Motivation

The Python support in Org-mode Babel provided by ob-python.el is somewhat lacking, because Python returns the result of evaluating something only if it is an expression. Furthermore, only in interactive mode (which has limitations regarding continuation lines) is the result of the last printed expression available in the _ variable. For this reason ob-python.el uses hacks like wrapping code in a function and requires return statements outside of seemingly function code. That leads to code blocks that cannot be simply tangled into a python file and various other issues. ob-ipython.el solves some of these issues by working with IPython which can evaluate blocks of code, but that requires extra libraries.

Essentially, for better Python support in Org-mode Babel something like the Lisp progn function is needed in Python. This library tries to provide such a progn equivalent.

Implementation

The library uses the ast library for analyzing and manipulating the abstract syntax tree and the inspect library to capture the variable scope of the calling frame.

import ast
import inspect

For reasons that will become apparent later, it is useful to pre-compile an AST representing an Expression that evaluates to None.

_NONE_EXPRESSION = ast.Expression(ast.parse('None').body[0].value)

The main idea is to parse a code block string int an abstract syntax tree and determine if the last statement is an expression. If it is, the code preceding it is compiled in exec mode and then this last expression is compiled in eval mode so that its result can be returned. If either of these code parts are empty (e.g. if the last statement is not an expression or there are no preceding statements), they are compiled into code objects that just evaluate to None and won’t do anything. block_name is the filename argument for compile calls. These two code objects are then returned and can be evaluated in the given order with eval.

def split_block(code_str, block_name='<string>'):
    """Split a code (block) string into an exec and eval mode code objects

    They are returned as a tuple (exec_part, eval_part). If the last statement
    in the block is an expression, it is compiled in eval mode into a code
    object eval_part. Remaining preceding statements are compiled in exec code
    into a code object, If none remain, exec_part will be a void code object
    (can be evaluated, but won't do anything). If the last statement is not an
    expression, eval_part will a code object represetning an expression which
    returns None (so effectively it is ignored as the exec_part would retunr
    None anyways) and exec_part will be compiled in exec mode.
    """
    code_ast = ast.parse(code_str)
    if len(code_ast.body) > 0 and isinstance(code_ast.body[-1], ast.Expr):
        expr = ast.Expression(code_ast.body[-1].value)
        del code_ast.body[-1]   # may become empty (void) code
    else:
        expr = _NONE_EXPRESSION # just evaluates to None
    eval_part = compile(expr, block_name, 'eval')
    exec_part = compile(code_ast, block_name, 'exec')
    return exec_part, eval_part

For convenience the block_eval function with a similar signature to the standard eval function is provided. However, it works reliably only in a top-elvel scope, e.g. in the interactive interpreter. The block_name argument is passed on to split_block.

def block_eval(code_str, globals_=None, locals_=None, block_name='<string>'):
    """Evaluate a code (block) string and possibly return its result

    The result is the result of the last statement if it is an expression. This
    function is a compromise between exec and eval: It evaluates all the
    statements like exec, but uses eval for the last statement if it is an
    expression and returns its value. If it is not, None is returned (exec mode)

    To emaulate eval behavior, the variable scope of the parent frame is
    captured and modified, which is know to work reliably only in top-level
    scope (e.g. interactive intepreter mode). It may not update local variable
    scope when used in a lower level scope (functions calling other functions).
    """

The parent frame is searched for global and local variable scope to fully emulate eval and falls back to the current frame using the standard functions globals and locals which should always work. This is perhaps the hackiest part of the code, but is needed.

# get scope in calling frame to truly emulate eval
current_frame = inspect.currentframe()
try:
    parent_frame = current_frame.f_back
    p_globals, p_locals = parent_frame.f_globals, parent_frame.f_locals
except AttributeError:      # cannot get that frame or its vars
    p_globals, p_locals = locals(), globals() # these should always work
finally:
    del current_frame       # otherwise might create reference cycle
if globals_ is not None:
    p_globals = globals_
    # this is documented eval behavior
    p_locals = p_globals if locals_ is None else locals_

Finally, the AST of the code block is parsed and split into an exec and eval mode part using split_block. These parts are then evaluated in the captured scope and the result of the eval part is returned.

# parse and split block, then evaluate
exec_part, eval_part = split_block(code_str, block_name)
exec(exec_part, p_globals, p_locals)
return eval(eval_part, p_globals, p_locals)

Test suite

import unittest
from textwrap import dedent

from block_eval import split_block, block_eval

class TestBlockEval(unittest.TestCase):

The block_eval function is first tested on a simple expression.

def test_simple_expr(self):
    ret = block_eval("6 * 7")
    self.assertEqual(ret, 42)

Then the referencing of a local variable in a simple expression is tested.

def test_simple_expr_with_var(self):
    a = 6
    ret = block_eval("a * 7")
    self.assertEqual(ret, 42)

Then a more complicated expression is tested.

def test_complicated_expr(self):
    alpha = 1.0 / 137
    ret = block_eval("alpha.is_integer() is False")
    self.assertIs(ret, True)

A for loop block is tested, it should not return anything as the last statement is not an expression.

def test_non_returning_block(self):
    ret = block_eval(dedent("""
    for i in range(3):
        i * 3
    """))
    self.assertIs(ret, None)

This more complicated block returns a result as the last statement is an expression.

def test_returning_block(self):
    ret = block_eval(dedent("""
    f = 1
    for i in range(1, 3+1):
        f *= i
    f
    """))
    self.assertEqual(ret, 6)

Due to local scope caveats the local variables likely won’t be updated (oddly enough when the error is inspected with nosetests --pdb they will suddenly appear, probably due to interactive mode).

def test_locals_update_fails(self):
    block_eval('b = 42')
    with self.assertRaises(NameError):
        self.assertEqual(b, 42)

Nevertheless, the proposed workaround works reliably.

def test_eval_in_current_scope_workaround(self):
    a = 1
    exec_part, eval_part = split_block(dedent("""
    f = a
    for i in range(1, 3+1):
        f *= i
    f / a
    """))
    exec(exec_part)
    ret = eval(eval_part)

    self.assertEqual(ret, 6)
    self.assertEqual(f, ret)