Make generator state easier to introspect

[ncoghlan]

BPO	10220
Nosy	@gvanrossum, @ncoghlan, @pitrou, @merwok
Files	getgeneratorstate.patch getgeneratorstate_v2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ncoghlan'
closed_at = <Date 2010-11-30.06:36:50.777>
created_at = <Date 2010-10-28.12:42:52.397>
labels = ['easy', 'type-feature', 'library']
title = 'Make generator state easier to introspect'
updated_at = <Date 2010-11-30.06:36:50.776>
user = 'https://github.com/ncoghlan'

bugs.python.org fields:

activity = <Date 2010-11-30.06:36:50.776>
actor = 'ncoghlan'
assignee = 'ncoghlan'
closed = True
closed_date = <Date 2010-11-30.06:36:50.777>
closer = 'ncoghlan'
components = ['Library (Lib)']
creation = <Date 2010-10-28.12:42:52.397>
creator = 'ncoghlan'
dependencies = []
files = ['19705', '19712']
hgrepos = []
issue_num = 10220
keywords = ['patch', 'easy']
message_count = 16.0
messages = ['119774', '119775', '119776', '119795', '119830', '119834', '119836', '121749', '121771', '121780', '121781', '121844', '121856', '122138', '122146', '122889']
nosy_count = 7.0
nosy_names = ['gvanrossum', 'ncoghlan', 'pitrou', 'rbp', 'eric.araujo', 'zbysz', 'Rodolpho.Eckhardt']
pr_nums = []
priority = 'normal'
resolution = 'accepted'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue10220'
versions = ['Python 3.2']

[ncoghlan]

Generators can be in four different states that may be relevant to framework code making use of them (especially as coroutines). This state is all currently available from Python code, but is rather obscure and could be made readable.

The four states are:

Created:
"g.gi_frame is not None and g.gi_frame.f_lasti == -1"
(Frame exists, but no instructions have been executed yet)

Currently executing:
"g.gi_running"
(This being true implies other things about the state as well, but this is all you need to check)

Suspended at a yield point:
"g.gi_frame is not None and g.gi_frame.f_lasti != -1 and not g.gi_running"

Exhausted/closed:
"g.gi_frame is None"

My API proposal is to add the following to the inspect module:

GEN_CREATED, GEN_RUNNING, GEN_SUSPENDED, GEN_CLOSED = range(4)

def getgeneratorstate(g):
  if g.gi_running:
    return GEN_RUNNING
  if g.gi_frame is None:
    return GEN_CLOSED
  if g.gi_frame.f_lasti == -1:
    return GEN_CREATED
  return GEN_SUSPENDED

(the lack of underscores in the function name follows the general style of the inspect module)

[pitrou]

Is it CPython-specific?
Does "currently executing" also include "currently closing"?

[ncoghlan]

On Thu, Oct 28, 2010 at 10:55 PM, Antoine Pitrou <report@bugs.python.org> wrote:

Antoine Pitrou <pitrou@free.fr> added the comment:

Is it CPython-specific?

The states are not CPython-specific (they're logical states of the
underlying generator), but I don't know if other implementations
expose generator and frame details in the same way (all the more
reason to put this in inspect - other implementations can provide the
information without needing to exactly mimic gi_frame and f_lasti).

Does "currently executing" also include "currently closing"?

"Currently executing" means the frame is being evaluated in a Python
thread (the thread running it may be suspended in a multi-threaded
environment, but the frame itself is in the middle of doing something,
which may include processing a thrown in GeneratorExit)

[gvanrossum]

I could imagine separating the state into two parts:

• a three-valued enum distinguishing created, active, or exhausted

• a bool (only relevant in the active state) whether it is currently running or suspended

The latter is just g.gi_running so we don't need a new API for this. :-)

[ncoghlan]

So something like:

GEN_CREATED, GEN_ACTIVE, GEN_CLOSED = range(3)

def getgeneratorstate(g):
  """Get current state of a generator-iterator.
  
  Possible states are:
    GEN_CREATED: Created, waiting to start execution
    GEN_ACTIVE: Currently being executed (or suspended at yield)
    GEN_CLOSED: Execution has completed

Use g.gi_running to determine if an active generator is running or
is suspended at a yield expression.
"""
if g.gi_frame is None:
return GEN_CLOSED
if g.gi_frame.f_lasti == -1:
return GEN_CREATED
return GEN_ACTIVE

Having 4 separate states actually makes the documentation a little easier to write:

def getgeneratorstate(g):
  """Get current state of a generator-iterator.
  
  Possible states are:
    GEN_CREATED: Waiting to start execution
    GEN_RUNNING: Currently being executed by the interpreter
    GEN_SUSPENDED: Currently suspended at a yield expression
    GEN_CLOSED: Execution has completed
  """

Checking if the generator is active is then just a matter of checking "gen_state in (GEN_RUNNING, GEN_SUSPENDED)".

[gvanrossum]

I take it back. The 4-value state looks better.

My initial hesitance was that if you ever see GEN_RUNNING you are
probably already in trouble, since you can't call send, next, throw or
even close on a running generator (they all throw ValueError), so why
are you looking at its state at all? But most reasons for looking at
the state are obscure anyway, and from a different perspective it's a
nice state machine. (Note that there's no transition from SUSPENDED to
CLOSED -- you have to go through RUNNING to possibly handle
GeneratorExit.)

[ncoghlan]

Indeed, the minimal lifecycles are:

GEN_CREATED->GEN_CLOSED (exception thrown in before the generator was even started)
GEN_CREATED->GEN_RUNNING->GEN_CLOSED (initial next() with internal logic that skips all yields)
GEN_CREATED->GEN_RUNNING->GEN_SUSPENDED->GEN_RUNNING->GEN_CLOSED (initial next() with a throw, next or send to close it)

Other cases following the same basic pattern as the last one, they just bounce back and forth between suspended and running more times.

It occurred to me that threads really use the same state machine, it's just that almost nobody writes their own Python thread schedulers, so only _thread and threading care about the suspended/running distinction. There are quite a few different generator schedulers though, so the distinctions matters to more 3rd party code than it does for threads.

[RodolphoEckhardt]

This patch adds the getgeneratorstate function and related tests.

[merwok]

Looks good, modulo two nitpicks:

1. Using literals for constants would avoid a lookup and call of range.

2. Please remove four leading spaces in your docstring.

[RodolphoEckhardt]

Done, thank you!

Second version attached.

[merwok]

Nice. Now you can sit back, relax and wait for Nick to commit the patch or make further comments.

[ncoghlan]

I'll actually go with version 1 of the patch as far as the variable initialisation goes. Yes, it is fractionally slower, but you get a maintenance gain from the fact that the enum values are guaranteed to be orthogonal, and this is clearly obvious to the reader.

When you write the assignments out explicitly, the reader has to actually look at the assigned values to determine whether or not the same value is ever assigned twice.

(No need to post a modified patch - I'll fix it before I check it in)

[ncoghlan]

Committed in r86633.

I added the missing docs changes, and tweaked the tests a little bit:

• added a helper method to retrieve the generator state in the test case
• this allowed test_running to be simplified a bit
• added an explicit test for the CREATED->CLOSED case
• renamed the test functions to match the existing idiom in test_inspect

[ncoghlan]

Temporarily reopening to remind me to switch from using integer constants to strings (which are much friendlier for debugging purposes).

[gvanrossum]

Yes please.

On Mon, Nov 22, 2010 at 7:44 AM, Nick Coghlan <report@bugs.python.org> wrote:

Nick Coghlan <ncoghlan@gmail.com> added the comment:

Temporarily reopening to remind me to switch from using integer constants to strings (which are much friendlier for debugging purposes).

----------
status: closed -> open

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue10220\>

---

[ncoghlan]

Switched to strings in r86879.