Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make generator state easier to introspect #54429

Closed
ncoghlan opened this issue Oct 28, 2010 · 16 comments
Closed

Make generator state easier to introspect #54429

ncoghlan opened this issue Oct 28, 2010 · 16 comments
Assignees
Labels
easy stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@ncoghlan
Copy link
Contributor

ncoghlan commented Oct 28, 2010

BPO 10220
Nosy @gvanrossum, @ncoghlan, @pitrou, @merwok
Files
  • getgeneratorstate.patch
  • getgeneratorstate_v2.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/ncoghlan'
    closed_at = <Date 2010-11-30.06:36:50.777>
    created_at = <Date 2010-10-28.12:42:52.397>
    labels = ['easy', 'type-feature', 'library']
    title = 'Make generator state easier to introspect'
    updated_at = <Date 2010-11-30.06:36:50.776>
    user = 'https://github.com/ncoghlan'

    bugs.python.org fields:

    activity = <Date 2010-11-30.06:36:50.776>
    actor = 'ncoghlan'
    assignee = 'ncoghlan'
    closed = True
    closed_date = <Date 2010-11-30.06:36:50.777>
    closer = 'ncoghlan'
    components = ['Library (Lib)']
    creation = <Date 2010-10-28.12:42:52.397>
    creator = 'ncoghlan'
    dependencies = []
    files = ['19705', '19712']
    hgrepos = []
    issue_num = 10220
    keywords = ['patch', 'easy']
    message_count = 16.0
    messages = ['119774', '119775', '119776', '119795', '119830', '119834', '119836', '121749', '121771', '121780', '121781', '121844', '121856', '122138', '122146', '122889']
    nosy_count = 7.0
    nosy_names = ['gvanrossum', 'ncoghlan', 'pitrou', 'rbp', 'eric.araujo', 'zbysz', 'Rodolpho.Eckhardt']
    pr_nums = []
    priority = 'normal'
    resolution = 'accepted'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue10220'
    versions = ['Python 3.2']

    @ncoghlan
    Copy link
    Contributor Author

    ncoghlan commented Oct 28, 2010

    Generators can be in four different states that may be relevant to framework code making use of them (especially as coroutines). This state is all currently available from Python code, but is rather obscure and could be made readable.

    The four states are:

    Created:
    "g.gi_frame is not None and g.gi_frame.f_lasti == -1"
    (Frame exists, but no instructions have been executed yet)

    Currently executing:
    "g.gi_running"
    (This being true implies other things about the state as well, but this is all you need to check)

    Suspended at a yield point:
    "g.gi_frame is not None and g.gi_frame.f_lasti != -1 and not g.gi_running"

    Exhausted/closed:
    "g.gi_frame is None"

    My API proposal is to add the following to the inspect module:

    GEN_CREATED, GEN_RUNNING, GEN_SUSPENDED, GEN_CLOSED = range(4)

    def getgeneratorstate(g):
      if g.gi_running:
        return GEN_RUNNING
      if g.gi_frame is None:
        return GEN_CLOSED
      if g.gi_frame.f_lasti == -1:
        return GEN_CREATED
      return GEN_SUSPENDED

    (the lack of underscores in the function name follows the general style of the inspect module)

    @ncoghlan ncoghlan added stdlib Python modules in the Lib dir easy type-feature A feature request or enhancement labels Oct 28, 2010
    @pitrou
    Copy link
    Member

    pitrou commented Oct 28, 2010

    Is it CPython-specific?
    Does "currently executing" also include "currently closing"?

    @ncoghlan
    Copy link
    Contributor Author

    ncoghlan commented Oct 28, 2010

    On Thu, Oct 28, 2010 at 10:55 PM, Antoine Pitrou <report@bugs.python.org> wrote:

    Antoine Pitrou <pitrou@free.fr> added the comment:

    Is it CPython-specific?

    The states are not CPython-specific (they're logical states of the
    underlying generator), but I don't know if other implementations
    expose generator and frame details in the same way (all the more
    reason to put this in inspect - other implementations can provide the
    information without needing to exactly mimic gi_frame and f_lasti).

    Does "currently executing" also include "currently closing"?

    "Currently executing" means the frame is being evaluated in a Python
    thread (the thread running it may be suspended in a multi-threaded
    environment, but the frame itself is in the middle of doing something,
    which may include processing a thrown in GeneratorExit)

    @gvanrossum
    Copy link
    Member

    gvanrossum commented Oct 28, 2010

    I could imagine separating the state into two parts:

    • a three-valued enum distinguishing created, active, or exhausted

    • a bool (only relevant in the active state) whether it is currently running or suspended

    The latter is just g.gi_running so we don't need a new API for this. :-)

    @ncoghlan
    Copy link
    Contributor Author

    ncoghlan commented Oct 29, 2010

    So something like:

    GEN_CREATED, GEN_ACTIVE, GEN_CLOSED = range(3)

    def getgeneratorstate(g):
      """Get current state of a generator-iterator.
      
      Possible states are:
        GEN_CREATED: Created, waiting to start execution
        GEN_ACTIVE: Currently being executed (or suspended at yield)
        GEN_CLOSED: Execution has completed

    Use g.gi_running to determine if an active generator is running or
    is suspended at a yield expression.
    """
    if g.gi_frame is None:
    return GEN_CLOSED
    if g.gi_frame.f_lasti == -1:
    return GEN_CREATED
    return GEN_ACTIVE

    Having 4 separate states actually makes the documentation a little easier to write:

    def getgeneratorstate(g):
      """Get current state of a generator-iterator.
      
      Possible states are:
        GEN_CREATED: Waiting to start execution
        GEN_RUNNING: Currently being executed by the interpreter
        GEN_SUSPENDED: Currently suspended at a yield expression
        GEN_CLOSED: Execution has completed
      """

    Checking if the generator is active is then just a matter of checking "gen_state in (GEN_RUNNING, GEN_SUSPENDED)".

    @gvanrossum
    Copy link
    Member

    gvanrossum commented Oct 29, 2010

    I take it back. The 4-value state looks better.

    My initial hesitance was that if you ever see GEN_RUNNING you are
    probably already in trouble, since you can't call send, next, throw or
    even close on a running generator (they all throw ValueError), so why
    are you looking at its state at all? But most reasons for looking at
    the state are obscure anyway, and from a different perspective it's a
    nice state machine. (Note that there's no transition from SUSPENDED to
    CLOSED -- you have to go through RUNNING to possibly handle
    GeneratorExit.)

    @ncoghlan
    Copy link
    Contributor Author

    ncoghlan commented Oct 29, 2010

    Indeed, the minimal lifecycles are:

    GEN_CREATED->GEN_CLOSED (exception thrown in before the generator was even started)
    GEN_CREATED->GEN_RUNNING->GEN_CLOSED (initial next() with internal logic that skips all yields)
    GEN_CREATED->GEN_RUNNING->GEN_SUSPENDED->GEN_RUNNING->GEN_CLOSED (initial next() with a throw, next or send to close it)

    Other cases following the same basic pattern as the last one, they just bounce back and forth between suspended and running more times.

    It occurred to me that threads really use the same state machine, it's just that almost nobody writes their own Python thread schedulers, so only _thread and threading care about the suspended/running distinction. There are quite a few different generator schedulers though, so the distinctions matters to more 3rd party code than it does for threads.

    @ncoghlan ncoghlan self-assigned this Oct 29, 2010
    @RodolphoEckhardt
    Copy link
    Mannequin

    RodolphoEckhardt mannequin commented Nov 20, 2010

    This patch adds the getgeneratorstate function and related tests.

    @merwok
    Copy link
    Member

    merwok commented Nov 20, 2010

    Looks good, modulo two nitpicks:

    1. Using literals for constants would avoid a lookup and call of range.

    2. Please remove four leading spaces in your docstring.

    @RodolphoEckhardt
    Copy link
    Mannequin

    RodolphoEckhardt mannequin commented Nov 20, 2010

    Done, thank you!

    Second version attached.

    @merwok
    Copy link
    Member

    merwok commented Nov 20, 2010

    Nice. Now you can sit back, relax and wait for Nick to commit the patch or make further comments.

    @ncoghlan
    Copy link
    Contributor Author

    ncoghlan commented Nov 21, 2010

    I'll actually go with version 1 of the patch as far as the variable initialisation goes. Yes, it is fractionally slower, but you get a maintenance gain from the fact that the enum values are guaranteed to be orthogonal, and this is clearly obvious to the reader.

    When you write the assignments out explicitly, the reader has to actually look at the assigned values to determine whether or not the same value is ever assigned twice.

    (No need to post a modified patch - I'll fix it before I check it in)

    @ncoghlan
    Copy link
    Contributor Author

    ncoghlan commented Nov 21, 2010

    Committed in r86633.

    I added the missing docs changes, and tweaked the tests a little bit:

    • added a helper method to retrieve the generator state in the test case
    • this allowed test_running to be simplified a bit
    • added an explicit test for the CREATED->CLOSED case
    • renamed the test functions to match the existing idiom in test_inspect

    @ncoghlan
    Copy link
    Contributor Author

    ncoghlan commented Nov 22, 2010

    Temporarily reopening to remind me to switch from using integer constants to strings (which are much friendlier for debugging purposes).

    @ncoghlan ncoghlan reopened this Nov 22, 2010
    @gvanrossum
    Copy link
    Member

    gvanrossum commented Nov 22, 2010

    Yes please.

    On Mon, Nov 22, 2010 at 7:44 AM, Nick Coghlan <report@bugs.python.org> wrote:

    Nick Coghlan <ncoghlan@gmail.com> added the comment:

    Temporarily reopening to remind me to switch from using integer constants to strings (which are much friendlier for debugging purposes).

    ----------
    status: closed -> open


    Python tracker <report@bugs.python.org>
    <http://bugs.python.org/issue10220\>


    @ncoghlan
    Copy link
    Contributor Author

    ncoghlan commented Nov 30, 2010

    Switched to strings in r86879.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    easy stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants