Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement PEP 654: Exception Groups #89455

Closed
iritkatriel opened this issue Sep 26, 2021 · 12 comments
Closed

Implement PEP 654: Exception Groups #89455

iritkatriel opened this issue Sep 26, 2021 · 12 comments
Assignees
Labels
3.11 docs Documentation in the Doc dir interpreter-core Interpreter core (Objects, Python, Grammar, and Parser dirs) stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@iritkatriel
Copy link
Member

iritkatriel commented Sep 26, 2021

BPO 45292
Nosy @gvanrossum, @markshannon, @tirkarthi, @erlend-aasland, @iritkatriel
PRs
  • bpo-45292: [PEP 654] add the ExceptionGroup and BaseExceptionGroup classes #28569
  • bpo-45292: [PEP 654] Update traceback display code to work with exception groups #29207
  • bpo-45292: Use raw strings for regex in tests #29545
  • bpo-45292: [PEP-654] add except* #29581
  • bpo-45292: [PEP-654] exception groups and except* documentation #30158
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/iritkatriel'
    closed_at = <Date 2022-01-06.19:06:15.397>
    created_at = <Date 2021-09-26.12:07:11.663>
    labels = ['interpreter-core', '3.11', 'type-feature', 'library', 'docs']
    title = 'Implement PEP 654: Exception Groups'
    updated_at = <Date 2022-01-06.19:06:15.396>
    user = 'https://github.com/iritkatriel'

    bugs.python.org fields:

    activity = <Date 2022-01-06.19:06:15.396>
    actor = 'iritkatriel'
    assignee = 'iritkatriel'
    closed = True
    closed_date = <Date 2022-01-06.19:06:15.397>
    closer = 'iritkatriel'
    components = ['Documentation', 'Interpreter Core', 'Library (Lib)']
    creation = <Date 2021-09-26.12:07:11.663>
    creator = 'iritkatriel'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 45292
    keywords = ['patch']
    message_count = 12.0
    messages = ['402661', '404837', '405773', '406266', '406400', '408549', '408557', '408559', '408560', '408589', '408590', '409898']
    nosy_count = 5.0
    nosy_names = ['gvanrossum', 'Mark.Shannon', 'xtreak', 'erlendaasland', 'iritkatriel']
    pr_nums = ['28569', '29207', '29545', '29581', '30158']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue45292'
    versions = ['Python 3.11']

    @iritkatriel iritkatriel self-assigned this Sep 26, 2021
    @iritkatriel iritkatriel added docs Documentation in the Doc dir interpreter-core Interpreter core (Objects, Python, Grammar, and Parser dirs) stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels Sep 26, 2021
    @iritkatriel
    Copy link
    Member Author

    iritkatriel commented Sep 26, 2021

    We will implement Exception Groups and except* in a series of PRs:

    1. Add the ExceptionGroup and BaseExceptionGroup classes
    2. Update traceback rendering code (python and C) for exception groups.
    3. Implement except*
    4. Write documentation sections.

    @vstinner vstinner changed the title Implement PEP 654 Implement PEP 654: Exception Groups Sep 26, 2021
    @iritkatriel
    Copy link
    Member Author

    iritkatriel commented Oct 22, 2021

    New changeset f30ad65 by Irit Katriel in branch 'main':
    bpo-45292: [PEP-654] add the ExceptionGroup and BaseExceptionGroup classes (GH-28569)
    f30ad65

    @iritkatriel
    Copy link
    Member Author

    iritkatriel commented Nov 5, 2021

    New changeset 3509b26 by Irit Katriel in branch 'main':
    bpo-45292: [PEP-654] Update traceback display code to work with exception groups (GH-29207)
    3509b26

    @tirkarthi
    Copy link
    Member

    tirkarthi commented Nov 13, 2021

    The tests emit some deprecation warnings :

    PYTHONWARNINGS=always ./python -Wall -m test test_exception_group
    0:00:00 load avg: 0.39 Run tests sequentially
    0:00:00 load avg: 0.39 [1/1] test_exception_group
    /home/karthikeyan/stuff/python/cpython/Lib/test/test_exception_group.py:41: DeprecationWarning: invalid escape sequence '\('
      MSG = 'second argument \(exceptions\) must be a sequence'
    /home/karthikeyan/stuff/python/cpython/Lib/test/test_exception_group.py:47: DeprecationWarning: invalid escape sequence '\('
      MSG = 'second argument \(exceptions\) must be a non-empty sequence'
    /home/karthikeyan/stuff/python/cpython/Lib/test/test_exception_group.py:52: DeprecationWarning: invalid escape sequence '\('
      MSG = ('Item [0-9]+ of second argument \(exceptions\)'

    == Tests result: SUCCESS ==

    1 test OK.

    Total duration: 51 ms
    Tests result: SUCCESS

    @tirkarthi
    Copy link
    Member

    tirkarthi commented Nov 16, 2021

    New changeset 8b06d01 by Irit Katriel in branch 'main':
    bpo-45292: Use raw strings for regex in tests (GH-29545)
    8b06d01

    @iritkatriel
    Copy link
    Member Author

    iritkatriel commented Dec 14, 2021

    New changeset d60457a by Irit Katriel in branch 'main':
    bpo-45292: [PEP-654] add except* (GH-29581)
    d60457a

    @gvanrossum
    Copy link
    Member

    gvanrossum commented Dec 14, 2021

    We should have a discussion here about improvements that Mark Shannon would like to see, in particular to do more work in the compiler instead of the interpreter.

    @markshannon
    Copy link
    Member

    markshannon commented Dec 14, 2021

    PR 29581 resulted in a 1% slowdown, which is not terrible, but code not using except* should not be slowed down at all.

    IMO, the way to avoid the slowdown is to implement except* using the existing instruction set (perhaps with a few minor additions)

    We already implement try-finally, named except blocks and with statements without any complex bytecodes (except perhaps WITH_EXCEPT_START).

    These used to involve a lot of state and more complex bytecodes. So it is possible to make these simplifications,
    but it does take work.

    There are a number of techniques we can use:

    If any state is needed, push it to the stack as we do with ctx.__exit__ in the with statement, and when pushing f_lasti in exception handlers.
    Duplicate code paths when the semantics differ in different cases, as we do for finally blocks.
    If anything is too complex to handle on the stack, put it in a temporary variable.
    Be liberal in your use of virtual try-excepts (SETUP_FINALLY, POP_FINALLY pairs), as zero-cost exception handling should keep the cost down.

    It may be too late for this advice, but if I were writing the except* implementation from scratch, I would:

    1. Sketch out the pseudo Python that a try-except* would map to. This is a good opportunity to discover any design bugs that might result in undesirable behavior for corner cases.

    2. Implement the translation in the compiler, not worrying about any redundancy or inefficiency, just correctness.

    3. Look to improve the above, either in the compiler front-end, or by replacing inefficient code patterns in the back-end. Handling the optimization in the backend has the advantage that other code might benefit as well.

    @iritkatriel
    Copy link
    Member Author

    iritkatriel commented Dec 14, 2021

    The PR adds two new opcodes. Let's start with the simpler of the two - JUMP_IF_NOT_EG_MATCH. This is the exception-group variation on JUMP_IF_NOT_EXC_MATCH.

    JUMP_IF_NOT_EXC_MATCH checks for a match by checking if the exception is of the given type. The result is boolean.

    JUMP_IF_NOT_EG_MATCH checks for a matching by calling .split() on the exception group. The result is two exception groups (the matching part and the non-matching part).

    Can we do this without a new opcode?

    @iritkatriel
    Copy link
    Member Author

    iritkatriel commented Dec 15, 2021

    The second opcode that the PR adds is PREP_RERAISE_STAR.

    This opcode takes a list that contains:

    1. all the exceptions that were raised in the executed except* clauses
    2. the unmatched part of the exception group

    It constructs the exception group that needs to be raised at the end. This is done through a fairly complex operation on the BaseExceptionGroup, which merges the re-raised exceptions into the same nesting structure they had in the original exception group, so that

    try:
    raise eg
    except* ValueError:
    raise
    except* TypeError:
    raise

    is equivalent to just 'raise eg'.

    Is there any overlap with existing opcodes?

    @iritkatriel
    Copy link
    Member Author

    iritkatriel commented Dec 15, 2021

    The way these two opcodes are combined by the compiler to implement except* is described in the pseudo code here:

    #29581 (comment)

    except* uses JUMP_IF_NOT_EG_MATCH. The excepts (not-) that collect exceptions raised in the except clauses are virtual.

    The do_reraise_star at the end is PREP_RERAISE_STAR followed by POP_EXCEPT_AND_RERAISE.

    @iritkatriel
    Copy link
    Member Author

    iritkatriel commented Jan 6, 2022

    New changeset 9925e70 by Irit Katriel in branch 'main':
    bpo-45292: [PEP-654] exception groups and except* documentation (GH-30158)
    9925e70

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    gpshead added a commit that referenced this issue Apr 14, 2022
    Fix an uninitialized bool in exception print context.
        
    `struct exception_print_context.need_close` was uninitialized.
        
    Found by oss-fuzz in a test case running under the undefined behavior sanitizer.
        
    https://oss-fuzz.com/testcase-detail/6217746058182656
        
    ```
    Python/pythonrun.c:1241:28: runtime error: load of value 253, which is not a valid value for type 'bool'
        #0 0xbf2203 in print_chained cpython3/Python/pythonrun.c:1241:28
        #1 0xbea4bb in print_exception_cause_and_context cpython3/Python/pythonrun.c:1320:19
        #2 0xbea4bb in print_exception_recursive cpython3/Python/pythonrun.c:1470:13
        #3 0xbe9e39 in _PyErr_Display cpython3/Python/pythonrun.c:1517:9
    ```
        
    Pretty obvious what the ommission was upon code inspection.
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.11 docs Documentation in the Doc dir interpreter-core Interpreter core (Objects, Python, Grammar, and Parser dirs) stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants