Provide a more convenient way to set an exception as "active", from Python code

[njsmith]

BPO	45361
Nosy	@gvanrossum, @ncoghlan, @njsmith, @markshannon, @1st1, @iritkatriel

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-10-04.06:42:56.085>
labels = ['type-feature']
title = 'Provide a more convenient way to set an exception as "active", from Python code'
updated_at = <Date 2021-10-10.05:47:09.812>
user = 'https://github.com/njsmith'

bugs.python.org fields:

activity = <Date 2021-10-10.05:47:09.812>
actor = 'gvanrossum'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2021-10-04.06:42:56.085>
creator = 'njs'
dependencies = []
files = []
hgrepos = []
issue_num = 45361
keywords = []
message_count = 4.0
messages = ['403121', '403122', '403287', '403572']
nosy_count = 6.0
nosy_names = ['gvanrossum', 'ncoghlan', 'njs', 'Mark.Shannon', 'yselivanov', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue45361'
versions = []

[njsmith]

Inside 'except' and 'finally' blocks, the interpreter keeps track of the 'active exception in the thread-state. It can be introspected via sys.exc_info(), it enables bare raise, and it triggers implicit context propagation.

In an odd bit of synchronicity, I recently bumped into two completely separate cases where Python code would benefit from being able to control this state directly:

• in the flat exception groups discussion [1], I noted that it would be useful if concurrency libraries could propagate this state into child tasks

• in bpo-27089, regarding some gnarly code in contextlib.ExitStack that simulates a set of nested try/finally blocks by running callback with the wrong exception context, and then introspecting the result and trying to fix it up after the fact. It turns out this code is difficult to understand and subtly buggy -- the PR at I think this is a small bug in urlparse.py #71276 fixes one of the bugs here, but there are others [2]. It would be much simpler if it could just set the correct exc_info before calling each cleanup function, and then no fixups would be needed things would be set up correctly in the first place.

[1] https://discuss.python.org/t/flat-exception-groups-alternative-to-pep-654/10433

[2] #27089 (comment)

But, currently, the *only* way to make an exception 'active' like this is to raise it and then catch it again. And this has some significant limitations:

• It unconditionally mutates the exception -- in particular both __context__ and __traceback__ are modified

• The "active exception" has type Optional[BaseException], since try/raise/except is block syntax, and you can't raise None. So if you want to propagate an arbitrary sys.exc_info() into a child task or into a simulated finally block, then you need two separate code paths depending on whether sys.exc_info()[1] is None or not.

• technically I think you can work around both of these issues with enough effort... but since try/raise/except is block syntax, the workarounds can't be hidden inside a function; you have to inline them into each usage site.

So... I'm thinking maybe we should have some stupid-simple, sharp-edged API to set sys.exc_info() from Python. Like, idk, sys.set_exc_info(...), as a trivial wrapper around PyErr_Restore.

My main uncertainty is that I know the code for handling the exception state is quite complex, between the "exception stack", the tricky optimizations around partially-initialized exc_info tuples, pushing/popping the exc state in ceval.c, the tricks generators/coroutines use to save/restore exc_info across yields, etc. There might be some hidden dragons in here that require careful handling, and maybe it'll turn out we need to push/pop an exception instead of just setting it, or something like that.

But does the core idea make sense? Anyone aware of any dragons off the top of your head?

[njsmith]

Ugh, the gnarly ExitStack bug is bpo-44594, not whatever I wrote above

[gvanrossum]

Do you think the API you're looking for is available at the C level? Like PyErr_SetExcInfo()?

[gvanrossum]

In a brief off-line chat we concluded that the desired API would be closer to PyErr_Restore().

[iritkatriel]

In a brief off-line chat we concluded that the desired API would be closer to PyErr_Restore().

It doesn't make sense to me to call PyErr_Restore() from python without raising an exception. What does that mean?

[gvanrossum]

I suppose you pass it an exception you caught and saved earlier?

[iritkatriel]

I can see you calling PyErr_SetExcInfo to set the "handled/active" exception. But what becomes the status of this exception after the call to PyErr_Restore?

[gvanrossum]

I don't think the proposal was to reuse PyErr_Restore(). I think the point was that it should look similar to that.

But honestly unless @njsmith responds this issue isn't going to be resolved for 3.11, it just doesn't seem to have priority (I can't find the time to really read Nathaniel's message with use cases, sorry).

[iritkatriel]

The original post is talking about the exc_info (caught exception). Later (offline) it became ‘something like’ pyerr_restore (in-flight exception). It’s totally confused, and we’re not getting and clarifications from the OP, so I am closing this.