Introspection generator and function closure state

[ncoghlan]

BPO	13062
Nosy	@ncoghlan, @meadori, @durban, @ericsnowcurrently
Files	issue13062.patch: v1 patch against tip (3.3.0a0) issue13062-2.patch: v2 patch against tip (3.3.0a0) issue13062-3.patch: v3 patch against tip (3.3.0a0) issue13062-combined.diff: Combined patch for getclosurevars and getgeneratorlocals issue13062-getclosurevars.diff: Just the getclosurevars changes

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ncoghlan'
closed_at = <Date 2012-06-23.09:40:22.753>
created_at = <Date 2011-09-29.17:43:09.969>
labels = ['type-feature', 'library']
title = 'Introspection generator and function closure state'
updated_at = <Date 2012-06-23.09:40:22.685>
user = 'https://github.com/ncoghlan'

bugs.python.org fields:

activity = <Date 2012-06-23.09:40:22.685>
actor = 'python-dev'
assignee = 'ncoghlan'
closed = True
closed_date = <Date 2012-06-23.09:40:22.753>
closer = 'python-dev'
components = ['Library (Lib)']
creation = <Date 2011-09-29.17:43:09.969>
creator = 'ncoghlan'
dependencies = []
files = ['23303', '23356', '23368', '26103', '26104']
hgrepos = []
issue_num = 13062
keywords = ['patch']
message_count = 18.0
messages = ['144606', '144636', '144638', '144640', '144641', '144798', '144799', '144801', '145264', '145296', '145300', '145308', '146277', '162671', '162707', '163558', '163562', '163564']
nosy_count = 7.0
nosy_names = ['ncoghlan', 'ron_adam', 'meador.inge', 'daniel.urban', 'Yury.Selivanov', 'python-dev', 'eric.snow']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue13062'
versions = ['Python 3.3']

[ncoghlan]

Based on the python-ideas thread about closures, I realised there are two features the inspect module could offer to greatly simplify some aspects of testing closure and generator behaviour:

  inspect.getclosure(func)
    Returns a dictionary mapping closure references from the supplied function to their current values.

  inspect.getgeneratorlocals(generator)
    Returns the same result as would be reported by calling locals() in the generator's frame of execution

The former would just involve syncing up the names on the code object with the cell references on the function object, while the latter would be equivalent to doing generator.gi_frame.f_locals with some nice error checking for when the generator's frame is already gone (or the supplied object isn't a generator iterator).

[meadori]

I'll take a shot and writing a patch for this one. Nick, are the
elements in 'co_freevars' and '__closures__' always expected to match
up? In other words, is the 'closure' function below always expected to
work (simplified; no error checking):

>>> def make_adder(x):
...     def add(y):
...         return x + y
...     return add
... 
>>> def curry(func, arg1):
...     return lambda arg2: func(arg1, arg2)
... 
>>> def less_than(a, b):
...     return a < b
... 
>>> greater_than_five = curry(less_than, 5)
>>> def closure(func):
...     vars = [var for var in func.__code__.co_freevars]
...     values = [cell.cell_contents for cell in func.__closure__]
...     return dict(zip(vars, values))
...
>>> inc = make_adder(1)
>>> print(closure(inc))
{'x': 1}
>>> print(closure(greater_than_five))
{'arg1': 5, 'func': <function less_than at 0xb74c6924>}

?

[ericsnowcurrently]

See:
http://hg.python.org/cpython/file/default/Include/funcobject.h#l34
http://hg.python.org/cpython/file/default/Objects/funcobject.c#l454

454 /* func_new() maintains the following invariants for closures. The
455 closure must correspond to the free variables of the code object.
456
457 if len(code.co_freevars) == 0:
458 closure = NULL
459 else:
460 len(closure) == len(code.co_freevars)
461 for every elt in closure, type(elt) == cell
462 */

[ncoghlan]

Yep, that looks right to me. The eval loop then references those cells from the frame object during execution.

[ncoghlan]

Huh, I didn't actually realise getclosure() could be written as a one liner until seeing Meador's version above:

{var : cell.cell_contents for var, cell in zip(func.__code__.co_freevars, func.__closure__)}

[meadori]

Here is a first cut at a patch. There is one slight deviation from the original spec:

some nice error checking for when the generator's frame is already gone > (or the supplied object isn't a generator iterator).

The attached patch returns empty mappings for these cases. I can easily
add the error checks, but in what cases is it useful to know *exactly*
why a mapping could not be created? Having an empty mapping for all
invalid cases is simpler and seems more robust.

[ncoghlan]

Because a generator can legitimately have no locals:

>>> def gen():
...     yield 1
... 
>>> g = gen()
>>> g.gi_frame.f_locals
{}

Errors should be reported as exceptions - AttributeError or TypeError if there's no gi_frame and then ValueError or RuntimeError if gi_frame is None.

[ncoghlan]

The function case is simpler - AttributeError or TypeError if there's no __closure__ attribute, empty mapping if there's no closure.

I've also changed my mind on the "no frame" generator case - since that mapping will evolve over time as the generator executes anyway, the empty mapping accurately reflects the "no locals currently defined" that applies when the generator either hasn't been started yet or has finished. People can use getgeneratorstate() to find that information if they need to know.

[meadori]

Here is an updated patch with error handling. One other thought is that
'getclosure' should be called something like 'getclosureenv' since
technically a closure is a function plus its environment and our
implementation only returns the environment. But that may be converging
on pedantic.

[ncoghlan]

No, the naming problem had occurred to me as well. Given the 'vars' builtin, perhaps 'getclosurevars' would do as the name?

[meadori]

perhaps 'getclosurevars' would do as the name?

I like vars. Updated patch attached.

[ncoghlan]

In reviewing Meador's patch (which otherwise looks pretty good), I had a thought about the functionality and signature of getclosurevars().

Currently, it equates "closure" to "nonlocal scope", which isn't really true - the function's closure is really the current binding of *all* of its free variables, and that includes globals and builtins in addition to the lexically scoped variables from outer scopes.

So what do people think about this signature:

  ClosureVars = namedtuple("ClosureVars", "nonlocals globals builtins unbound")
  def getclosurevars(func):
    """Returns a named tuple of dictionaries of the current nonlocal, global and builtin references as seen by the body of the function. A final set of unbound names is also provided."""
    # figure out nonlocal_vars (current impl)
    # figure out global_vars (try looking up names in f_globals)
    # figure out builtin_vars (try looking up names in builtins)
    # any leftover names go in unbound_vars
    return ClosureVars(nonlocal_vars, global_vars, builtin_vars, unbound_vars)

Also, something that just occurred to me is that getclosurevars() should work for already instantiated generator iterators as well as generator functions, so the current typecheck may need to be made a bit more flexible.

[meadori]

Nick, the revised definition of 'getclosurevars' seems reasonable to me.
I will cut a new patch this week.