IDLE ignores sys.excepthook in normal, subprocess mode

[Kenny2github]

BPO	43008
Nosy	@terryjreedy, @aroberge, @miss-islington, @Kenny2github
PRs	bpo-43008: Make IDLE respect sys.excepthook #24302 [3.9] bpo-43008: Make IDLE respect sys.excepthook (GH-24302) #24346 [3.8] bpo-43008: Make IDLE respect sys.excepthook (GH-24302) #24347 bpo-43008: Add 'Patch by Ken Hilton' #24370 [3.9] bpo-43008: Add 'Patch by Ken Hilton' (GH-24370) #24374 [3.8] bpo-43008: Add 'Patch by Ken Hilton' (GH-24370) #24375
Files	bpo-43008.patch: Fix for this issue, patch file from GitHub PR bpo-43008.patch.2: New patch with requested changes

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = <Date 2021-01-27.00:27:44.633>
created_at = <Date 2021-01-23.11:30:46.092>
labels = ['expert-IDLE', 'type-bug', '3.8', '3.9', '3.10']
title = 'IDLE ignores sys.excepthook in normal, subprocess mode'
updated_at = <Date 2021-02-04.17:48:45.168>
user = 'https://github.com/Kenny2github'

bugs.python.org fields:

activity = <Date 2021-02-04.17:48:45.168>
actor = 'terry.reedy'
assignee = 'terry.reedy'
closed = True
closed_date = <Date 2021-01-27.00:27:44.633>
closer = 'terry.reedy'
components = ['IDLE']
creation = <Date 2021-01-23.11:30:46.092>
creator = 'AbyxDev'
dependencies = []
files = ['49757', '49758']
hgrepos = []
issue_num = 43008
keywords = ['patch']
message_count = 20.0
messages = ['385529', '385540', '385542', '385547', '385548', '385550', '385560', '385561', '385719', '385741', '385742', '385743', '385744', '385745', '385748', '385924', '385925', '385926', '386475', '386487']
nosy_count = 4.0
nosy_names = ['terry.reedy', 'aroberge', 'miss-islington', 'AbyxDev']
pr_nums = ['24302', '24346', '24347', '24370', '24374', '24375']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue43008'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[Kenny2github]

Similar to https://bugs.python.org/issue12643 - IDLE ignores sys.excepthook and always prints exceptions.

[terryjreedy]

I looked at all 4 hooks.

sys.breakpointhook is invoked by breakpoint() in user code, so is not IDLE's concern at present.

sys.unraiseablehook is invoked by the interpreter when user code messes up badly, so is not IDLE's condern. (And it is an expert feature that beginners should ignore.)

sys.displayhook is set to rpc.displayhook in run.py or, if started with -n no subprocess, pyshell.py. IDLE executes user code with exec(). Shell entries are compiled in 'single' mode. Exec invokes displayhook for expression statements so compiled. In both normal and -n mode, displayhook exceptions are displayed and processed as user code exceptions, with the last traceback line giving the bad line.

sys.excepthook is invoked by the interpreter for uncaught exceptions.
In IDLE's -n mode, sys.excepthook *is* invoked. If it raises, both tracebacks are printed, separated by "\nDuring handling of the above exception, another exception occurred:\n". This issue is about having the same effect in normal 2-process mode.

Since, in the subprocess, IDLE catches all user exceptions for its custom handling, there are no uncaught user exceptions. I believe your patch calls the user hook in the proper place, but an uncaught exception in sys.excepthook would be nasty. Replace the call with something like

    try:
        sys.e...k(...)
    except BaseException as e:
        print_exception()
        print("\nDuring handling of the above exception, another exception occurred:\n", file=sys.stderr)
        self.usr_exc_info = sys.exc_info()
        print_exception()

If possible, I want a unittest added in test_run.py. (I am not sure exactly how as I don't think we have yet directly faked receipt of compiled user code.)

class Executive(unittest.TextCase):

    def test_good_excepthook(self):

    def test_bad_excepthook(self):

A blurb is needed (and always is for anything non-trivial).

In Doc/library/idle.rst, I want to insert " and reset display and exception handling" before the period ending the current Shell entry.

Restart Shell
Restart the shell to clean the environment.

[Kenny2github]

Made requested changes, besides the tests. I didn't manage to figure out how to go about testing an excepthook in the first place, let alone a good or bad one. Hope that's not an issue.

[terryjreedy]

Andre: while I would like friendlier tracebacks available in IDLE without 3rd party installation, I doubt this will happen soon. In the meanwhile, I see that you are working to polish Friendly Traceback with a friendly contest. I presume FT uses sys.excepthook. If so, this issue is opportune for supporting your package. Feel free to test the minimal patch as is, or when fleshed out, and let us know how it works.

[terryjreedy]

Ken: I will probably work on tests, perhaps as a followup or parallel issue. I will likely do a bit or refactoring to make testing easier.

I just looked at print_exception and discovered that the double traceback can likely by handled by properly setting exc.context for the 2nd exception. But getting that right will be aided by having something that already works.

Possible manual test protocol in Shell

>> import sys
>> def egood(a,b,c): print('Exception traceback', file=sys.stderr)

>>> sys.excepthook = egood
>>> 1/0
# "Exception traceback"
>>> def ebad(a,b,c): z

>>> sys.excepthook = ebad
>>> 1/0
# Double traceback for ZeroDivisionError and NameError

[aroberge]

Terry: I will put it on my todo list.

Friendly-traceback can currently work (indirectly) with IDLE; see https://aroberge.github.io/friendly-traceback-docs/docs/html/editor.html

I am not sure what benefit this patch will yield when it comes to using Friendly-traceback with IDLE compared with what is currently possible.

[terryjreedy]

Andre: with this patch, you could potentially add a function or submodule that would point sys.excepthook to a function that would print the exception and then interact before returning. If this worked, one could add, for instance, 'import friendly_traceback.idle' at the top of a file or at a Shell prompt and have friendly-traceback only activate when there is an exception. So it would work in Shell.

I hope by 3.10 to have IDLE '>>>' prompts moved into a sidebar (instead of line numbers. F-T prompts would be distinctly different.

[terryjreedy]

Ken, I read on another issue that an exception in an except suite is automatically linked to the caught exception, though the printed result may not be exactly the same. Once your CLA is 'activated', I will experiment with the options.

[Kenny2github]

CLA registers as signed now.

[terryjreedy]

New changeset 7a34380 by Ken in branch 'master':
bpo-43008: Make IDLE respect sys.excepthook (GH-24302)
7a34380

[miss-islington]

New changeset 68102fb by Miss Islington (bot) in branch '3.9':
bpo-43008: Make IDLE respect sys.excepthook (GH-24302)
68102fb

[miss-islington]

New changeset 6f0346d by Miss Islington (bot) in branch '3.8':
bpo-43008: Make IDLE respect sys.excepthook (GH-24302)
6f0346d

[terryjreedy]

Ken, for next time -- there is a separate News category for IDLE.

[Kenny2github]

Oops, gotcha. Thanks for doing the tests. I'm a Python contributor! :D

[terryjreedy]

I hope this is not your last contribution. Another hint: if one uses bpo-12643 or bpo-12643, then the tracker makes the link -- and strikes it out if closed.

[terryjreedy]

New changeset 11d75ec by Terry Jan Reedy in branch 'master':
bpo-43008: Add 'Patch by Ken Hilton' (GH-24370)
11d75ec

[miss-islington]

New changeset cf88382 by Miss Islington (bot) in branch '3.8':
bpo-43008: Add 'Patch by Ken Hilton' (GH-24370)
cf88382

[terryjreedy]

New changeset bf782b2 by Miss Islington (bot) in branch '3.9':
bpo-43008: Add 'Patch by Ken Hilton' (GH-24370) (bpo-24374)
bf782b2

[aroberge]

Terry: as per your earlier comment, I have tried to use Friendly-traceback with the latest version. While it does set the exception hook correctly, it is not useful.

Friendly-traceback (FT) needs to have access to the code that was executed. FT's REPL caches the code (similarly to what IPython does, but not CPython) prior to execution so that it can retrieve it correctly, have access to the relevant execution frame, etc. This is not possible when using IDLE's REPL. In fact, for some types of exceptions, when FT tries to get access to the code, it ends up triggering secondary exceptions.

However, using the method describe in FT's documentation (with relies on FT's own console) still works correctly.

[terryjreedy]

IDLE also caches shell input. Off topic here, I sent Andre an email about how FT could access it.