Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add threading.excepthook() to handle uncaught exceptions raised by Thread.run() #42148

Closed
ellisj mannequin opened this issue Jun 30, 2005 · 42 comments
Closed

Add threading.excepthook() to handle uncaught exceptions raised by Thread.run() #42148

ellisj mannequin opened this issue Jun 30, 2005 · 42 comments
Labels
3.8 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error

Comments

@ellisj
Copy link
Mannequin

ellisj mannequin commented Jun 30, 2005

BPO 1230540
Nosy @mwhudson, @tim-one, @ncoghlan, @pitrou, @vstinner, @merwok, @vlasovskikh, @serhiy-storchaka, @lazka, @AraHaan
PRs
  • bpo-1230540: Invoke custom sys.excepthook for threads in threading #8610
  • bpo-1230540: Add threading.excepthook() #13515
  • Files
  • fork_thread.py
  • sys_threading_excepthook.py
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2019-05-27.22:53:59.194>
    created_at = <Date 2005-06-30.19:06:10.000>
    labels = ['interpreter-core', 'type-bug', '3.8']
    title = 'Add threading.excepthook() to handle uncaught exceptions raised by Thread.run()'
    updated_at = <Date 2019-05-28.10:20:57.279>
    user = 'https://bugs.python.org/ellisj'

    bugs.python.org fields:

    activity = <Date 2019-05-28.10:20:57.279>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2019-05-27.22:53:59.194>
    closer = 'vstinner'
    components = ['Interpreter Core']
    creation = <Date 2005-06-30.19:06:10.000>
    creator = 'ellisj'
    dependencies = []
    files = ['48351', '48354']
    hgrepos = []
    issue_num = 1230540
    keywords = ['patch']
    message_count = 42.0
    messages = ['25694', '25695', '25696', '62933', '91243', '91244', '272002', '272003', '272006', '288513', '288514', '297221', '302320', '302645', '322892', '322895', '322932', '323111', '324422', '324436', '343252', '343258', '343260', '343261', '343276', '343280', '343293', '343443', '343444', '343474', '343494', '343510', '343512', '343515', '343516', '343522', '343523', '343524', '343692', '343694', '343748', '343758']
    nosy_count = 15.0
    nosy_names = ['mwh', 'tim.peters', 'ncoghlan', 'ellisj', 'pitrou', 'vstinner', 'tiagoaoa', 'eric.araujo', 'undercoveridiot', 'vlasovskikh', 'serhiy.storchaka', 'lazka', 'Decorater', 'CyberJacob', 'Matt Groth']
    pr_nums = ['8610', '13515']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue1230540'
    versions = ['Python 3.8']

    @ellisj
    Copy link
    Mannequin Author

    ellisj mannequin commented Jun 30, 2005

    simple script to reproduce:

    import sys, threading
    
    def log_exception(*args):
        print 'got exception %s' % (args,)
    sys.excepthook = log_exception
    
    def foo():
        a = 1 / 0
    threading.Thread(target=foo).start()

    Note that a normal traceback is printed instead of "got
    exception."

    @ellisj ellisj mannequin added interpreter-core (Objects, Python, Grammar, and Parser dirs) labels Jun 30, 2005
    @mwhudson
    Copy link

    mwhudson commented Jun 6, 2007

    I've just run into this, and it's very annoying. The stupid part is that threads started with the thread module don't have this problem, it's just a problem with threading.Thread()s trying to be too clever.

    I would vote for deleting all the code to do with exception printing in threading.py and letting the C machinery take care of it.

    @ellisj
    Copy link
    Mannequin Author

    ellisj mannequin commented Jun 15, 2007

    Here is a workaround in the meantime:

    def install_thread_excepthook():
        """
        Workaround for sys.excepthook thread bug
        (https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1230540&group_id=5470).
        Call once from __main__ before creating any threads.
        If using psyco, call psycho.cannotcompile(threading.Thread.run)
        since this replaces a new-style class method.
        """
        import sys
        run_old = threading.Thread.run
        def run(*args, **kwargs):
            try:
                run_old(*args, **kwargs)
            except (KeyboardInterrupt, SystemExit):
                raise
            except:
                sys.excepthook(*sys.exc_info())
        threading.Thread.run = run

    @tiagoaoa
    Copy link
    Mannequin

    tiagoaoa mannequin commented Feb 24, 2008

    I don't see it as a problem or as the threading module trying to be
    "clever". It clearly was a design choice to make the module have its own
    exception treatment in order to make it clear in which thread the
    exception occurred.
    IMHO the decision here should be to implement per thread excepthook's.

    @undercoveridiot
    Copy link
    Mannequin

    undercoveridiot mannequin commented Aug 3, 2009

    I found that the workaround suggested doesn't work when you have a
    subclass of threading.Thread and you want to catch everything in the
    module that contains the class to a common log.

    Say you have a module with a socket server that spawns a thread on
    accept and you want to log anything that tracebacks in the module. This
    seems to work:

    import sys
    import logging
    from functools import wraps
    
    def myExceptHook(type, value, tb):
        """ Redirect tracebacks to error log """
        import traceback
        rawreport = traceback.format_exception(type, value, tb)
        report = '\n'.join(rawreport)
        log.error(report)
    
    sys.excepthook = myExceptHook
        
    def use_my_excepthook(view):
        """ Redirect any unexpected tracebacks """ 
        @wraps(view)
        def run(*args, **kwargs):
            try:
                return view(*args, **kwargs)
            except:
                sys.excepthook(*sys.exc_info())
        return run

    Then in your thread subclass:

    class MyThread(threading.Thread):
        def __init__(self, socket_conn):
            threading.Thread.__init__(self)
            self.my_conn = socket_conn
    
        @use_my_excepthook
        def run(self):
            """ Do stuff """

    @undercoveridiot
    Copy link
    Mannequin

    undercoveridiot mannequin commented Aug 3, 2009

    Instead of using decorators, this is a slightly simpler modification to
    the proposed workaround that allows for any subclassed run method to
    also be caught.

    def installThreadExcepthook():
        """
        Workaround for sys.excepthook thread bug
        From
    http://spyced.blogspot.com/2007/06/workaround-for-sysexcepthook-bug.html
       
    (https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1230540&group_id=5470).
        Call once from __main__ before creating any threads.
        If using psyco, call psyco.cannotcompile(threading.Thread.run)
        since this replaces a new-style class method.
        """
        init_old = threading.Thread.__init__
        def init(self, *args, **kwargs):
            init_old(self, *args, **kwargs)
            run_old = self.run
            def run_with_except_hook(*args, **kw):
                try:
                    run_old(*args, **kw)
                except (KeyboardInterrupt, SystemExit):
                    raise
                except:
                    sys.excepthook(*sys.exc_info())
            self.run = run_with_except_hook
        threading.Thread.__init__ = init

    @AraHaan
    Copy link
    Mannequin

    AraHaan mannequin commented Aug 5, 2016

    I too agree that I hate the thread exceptions being printed in the console I would suggest python was to error log it all to a file instead (so it does not spam up the console). I get it a lot with websocket / opus errors and it is annoying because it does not cleanup ffmpeg for me.

    @AraHaan
    Copy link
    Mannequin

    AraHaan mannequin commented Aug 5, 2016

    personally these exceptions in console can be annoying to me as I hate seeing them.

    Exception in thread Thread-11:
    Traceback (most recent call last):
      File "threading.py", line 914, in _bootstrap_inner
      File "E:\Users\Elsword\Desktop\DecoraterBot\Async\test\resources\Dependencies\discord\voice_client.py", line 150, in run
        super().run()
      File "E:\Users\Elsword\Desktop\DecoraterBot\Async\test\resources\Dependencies\discord\voice_client.py", line 108, in run
        self.player(data)
      File "E:\Users\Elsword\Desktop\DecoraterBot\Async\test\resources\Dependencies\discord\voice_client.py", line 669, in play_audio
        sent = self.socket.sendto(packet, (self.endpoint_ip, self.voice_port))
    OSError: [WinError 10055] An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full
    
    Exception in thread Thread-21:
    Traceback (most recent call last):
      File "threading.py", line 914, in _bootstrap_inner
      File "E:\Users\Elsword\Desktop\DecoraterBot\Async\test\resources\Dependencies\discord\voice_client.py", line 150, in run
        super().run()
      File "E:\Users\Elsword\Desktop\DecoraterBot\Async\test\resources\Dependencies\discord\voice_client.py", line 108, in run
        self.player(data)
      File "E:\Users\Elsword\Desktop\DecoraterBot\Async\test\resources\Dependencies\discord\voice_client.py", line 669, in play_audio
        sent = self.socket.sendto(packet, (self.endpoint_ip, self.voice_port))
    OSError: [WinError 10038] An operation was attempted on something that is not a socket

    @AraHaan
    Copy link
    Mannequin

    AraHaan mannequin commented Aug 5, 2016

    Ok, so I just found out you can bypass thread exceptions by wraping the line that actually runs the threads in a try/except block and then using the logging module to log it to a file instead of the console.

    @CyberJacob
    Copy link
    Mannequin

    CyberJacob mannequin commented Feb 24, 2017

    Does this affect threads started with the multiprocessing library as well?

    @CyberJacob
    Copy link
    Mannequin

    CyberJacob mannequin commented Feb 24, 2017

    Just confirmed, this does affect multiprocessing. script to reproduce:

    import sys, multiprocessing
    
    def log_exception(*args):
        print 'got exception %s' % (args,)
    
    sys.excepthook = log_exception
    
    def foo():
        a = 1 / 0

    multiprocessing.Process(target=foo).start()

    again, just a normal traceback instead of "got exception"

    @pitrou
    Copy link
    Member

    pitrou commented Jun 28, 2017

    Apparently the traceback module has been used from the start (since changset 7f5013a) to print the exception and not sys.excepthook. I presume this is an oversight, as I don't see any reason to prefer traceback here. Tim, any insight?

    @pitrou pitrou added 3.7 (EOL) end of life type-bug An unexpected behavior, bug, or error labels Jun 28, 2017
    @MattGroth
    Copy link
    Mannequin

    MattGroth mannequin commented Sep 16, 2017

    Thank you Antoine Pitrou, I was able to disable this behavior by commenting out some lines of code in 'traceback' and replacing them with the appropriate call to 'sys.excepthook'. Note you also have to comment out a few lines in "Modules/_threadmodule.c" to correspond to this change or else that file complains that there wasn't the expected output printed to stderr.

    @pitrou
    Copy link
    Member

    pitrou commented Sep 20, 2017

    Matt, do you want to post your patch here?

    @vlasovskikh
    Copy link
    Member

    I've added a PR with a patch I developed during the EuroPython 2018 sprint.

    I've fixed this issue in a way that is more or less consistent with how '_thread' threads interact with sys.excepthook, but I haven't added the log line they print to sys.stderr about an unhandled exception in a thread:

    Unhandled exception in thread started by <function <lambda> at 0x108c840d0>
    

    I can add these messages if they are needed.

    I'd like to mention the issue I discussed with Victor Stinner at the sprints (adding Victor to the nosy list as he requested). The fix could possibly break the de facto contract that has been there for ages that 'theading' threads don't invoke sys.excepthook on uncaught exceptions.

    If the fact that sys.execepthook doesn't work with threading is considered a feature, then an alternative solution could be a new threading-specific hook: threading.excepthook.

    @pitrou
    Copy link
    Member

    pitrou commented Aug 1, 2018

    No, I think this is a bug that deserves fixing, at least in 3.8.

    @pitrou pitrou added 3.8 only security fixes and removed 3.7 (EOL) end of life labels Aug 1, 2018
    @vstinner
    Copy link
    Member

    vstinner commented Aug 2, 2018

    Would it be possible to modify the default implementation of sys.excepthook to have a different output when it's not called from the main thread? Mimick the current traceback from threads.

    Would it be possible to call threading.current_thread() from the default sys.excepthook? The problem is to get the threading module and to decide how to handle error on getting the current thread.

    An alternative would be to introduce a new hook to log exceptions in threads, ex: sys.threadexcepthook. That hoook would take a 4th parameter. But Andrey Vlasovskikh didn't like this idea.

    @vlasovskikh
    Copy link
    Member

    Would it be possible to modify the default implementation of sys.excepthook to have a different output when it's not called from the main thread? Mimick the current traceback from threads.

    I agree it's a good idea to mimic what _thread does in case of both handled and unhandled exceptions. Why would you want to do it in sys.excepthook? The code for handling unhandled exceptions in _thread threads is located in _threadmodule.c itself, so there is no dependency from sys to _thread. By following the same logic, the code for handling unhandled exceptions in threading should be located in threading.py, like it does now. So the only thing that's missing in the pull request compared to _thread is the output:

    Unhandled exception in thread started by <function <lambda> at 0x7f6769ca8e18>

    for all the exceptions: not caught by the sys.excepthook **and** caught by the sys.exceptook.

    An alternative would be to introduce a new hook to log exceptions in threads, ex: sys.threadexcepthook. That hoook would take a 4th parameter. But Andrey Vlasovskikh didn't like this idea.

    I'm overall OK with this idea, but I would prefer to make the existing sys.excepthook API applicable to all the exceptions: for the main thread (works well), for _thread threads (works well) and for threading threads (doesn't work).

    @vstinner
    Copy link
    Member

    My concern is more about backward compatibility. Documentation is one thing, but usually users rely on the actual implementation, not on the documentation, to define what is the Python behaviour.

    Would you mind to open a thread on python-dev about this change?

    I don't want to take the responsibility alone of such change :-) Moreover, I don't really have the bandwidth to work on this specific issue :-(

    @lazka
    Copy link
    Mannequin

    lazka mannequin commented Aug 31, 2018

    To add one more use case for this:

    concurrent.futures.Future.add_done_callback() currently ignores exceptions for the callback and just logs them which is easy to miss. I assume because it's not clear when and in which thread it gets called, and there is no "right" way to handle it.

    It would be nice if it would call something like sys.excepthook instead.

    @vstinner
    Copy link
    Member

    See also bpo-36829: I just added sys.unraisablehook().

    @vstinner
    Copy link
    Member

    I wrote PR 13515 which adds threading.excepthook(). I chose to call threading.excepthook() even when run() raises SystemExit. In this case, threading.excepthook() simply does nothing. The idea is to really give the full control when threading.excepthook() is overriden. For example, log a warning when run() raises SystemExit. By the way, is it really a good idea to call sys.exit() from a thread? It sounds like a bug that should be reported, and not silently ignored, no?

    Differences between sys.excepthook() and threading.excepthook():

    • API: sys.excepthook(exctype, value, traceback, /) vs threading.excepthook(exc_type, exc_value, exc_tb, thread, /) -- addition thread parameter to display the name of the thread which raises an exception

    • For SystemExit corner case, sys.excepthook() displays the exception, whereas threading.excepthook() silently ignores it

    • When sys.stderr is None, sys.excepthook() does nothing, whereas threading.excepthook() tries harder: use its own copy of sys.stderr (saved when the thread has been created) from thread._stderr.

    Thread._stderr was added by bpo-754449:

    commit cc4e935
    Author: Brett Cannon <bcannon@gmail.com>
    Date: Sat Jul 3 03:52:35 2004 +0000

    threading.Thread objects will now print a traceback for an exception raised
    during interpreter shutdown instead of masking it with another traceback about
    accessing a NoneType when trying to print the exception out in the first place.
    
    Closes bug bpo-754449 (using patch bpo-954922).
    

    Note: When sys.stderr is None, threading.excepthook() avoids the traceback module and renders the exception itself. Maybe threading.excepthook() should be reimplemented in C to make it even more reliable and more correct, especially during Python shutdown. Only daemon threads are impacted: Python finalization (Py_Finalize() C function) starts by calling threading._shutdown() which joins all non-daemon threads.

    IMHO the threading.Thread semantics is too different than sys.excepthook() to reuse sys.excepthook() to handle threading.Thread.run() exception.

    Another explanation is that sadly sys.excepthook() API uses exactly 3 positional-only arguments, and so the API cannot be easily extended to get a thread argument. When I designed sys.unraisablehook(), I chose to pass only one argument which has attributes, to prevent this issue.

    I'm not comfortable to attempt to modify sys.excepthook() to make it behave differently if it's called from the main thread or from a different thread. It would have to call threading.current_thread().name to get the name of the current thread.

    Let's say that in Python 3.8 threading.Thread now calls sys.execpthook() to handle uncaught run() exception. All applications which override sys.excepthook() on purpose will behave differently: start to log exceptions from threads. But existing code is unlikely to be prepared to implement black magic to check if we are a "thread" or the main thread, to decide if we should display a thread name, and also the "black magic" to get the current thread name.

    One of my concern of reusing sys.excepthook to display threading exceptions is that adding more code to handle threads is a risk of raising a new exception while logging a threading exception :-(

    IMHO threading.excepthook() is safer since it already has access to the thread.

    @vstinner
    Copy link
    Member

    I dislike PR 8610: threading.Thread doesn't call sys.excepthook to handle run() exception by default, it only calls sys.excepthook if it's overridden. Moreover, when sys.excepthook is called, the hook doesn't get access to the thread object :-(

    @vstinner
    Copy link
    Member

    About threading.excepthook() API, maybe we should not reproduce sys.excepthook() API but instead reuse something closer to sys.unraisablehook() API: use a single parameter which has attributes. It would allow to pass more parameters as new attributes in the future, maybe some new "optional" parameters (None by default).

    For example, we can imagine calling threading.excepthook() to handle threading.excepthook() failure. We would need an argument to prevent an infine loop :-)

    @lazka
    Copy link
    Mannequin

    lazka mannequin commented May 23, 2019

    Let's say that in Python 3.8 threading.Thread now calls sys.execpthook() to handle uncaught run() exception. All applications which override sys.excepthook() on purpose will behave differently: start to log exceptions from threads. But existing code is unlikely to be prepared to implement black magic to check if we are a "thread" or the main thread, to decide if we should display a thread name, and also the "black magic" to get the current thread name.

    Note that PyErr_Print() and PyErr_PrintEx() can be called in threads, and CPython itself uses it in some places which can be called in threads and I also use it in thread callbacks in C extensions I work on (PyGObject and pycairo for example). Nothing states currently that it's not allowed to call it in such cases :(

    @pitrou
    Copy link
    Member

    pitrou commented May 23, 2019

    Moreover, when sys.excepthook is called, the hook doesn't get access to the thread object

    You can get it with threading.current_thread(), no?

    @vstinner
    Copy link
    Member

    There is a special case. If a thread calls os.fork() and Thread.run() raises an exception, the thread name is still logged even if there is only 1 thread after fork. Try attached fork_thread.py.

    Output on Python 3.7:
    ---

    main thread: spawn fork thread
    thread: before fork [<_MainThread(MainThread, started 140623217481536)>, <ForkThread(Thread-1, started 140622996952832)>]
    thread: after fork [<ForkThread(Thread-1, started 140622996952832)>]
    thread: after fork: raise
    Exception in thread Thread-1:
    Traceback (most recent call last):
      File "/usr/lib64/python3.7/threading.py", line 917, in _bootstrap_inner
        self.run()
      File "fork_thread.py", line 17, in run
        raise Exception("what happens here?")
    Exception: what happens here?

    main thread: done
    ---

    Moreover, threading.Thread silently ignores SystemExit in run() even if it's the only remaining thread (main thread is gone after fork).

    I don't think that we *have to* change the current behavior. It's just that we have to take it in account if we modify how exceptions are handled.

    @vstinner
    Copy link
    Member

    I rewrote my PR 13515:

    • threading.excepthook() now gets a single argument which has multiple attributes: (exc_type, exc_value, exc_traceback, thread)

    • The default threading.excepthook() implementation in written in C which reduces the risk of missing symbol during Python shutdown. There is also a simple implementation in Python, for other Python implementations which don't want to implement _thread._excepthook() in C.

    • New _make_invoke_excepthook() function which handles the gory details for daemon threads. It creates a "local namespace" with references to all required functons and creates a _invoke_excepthook() function.

    • _invoke_excepthook() packs arguments as a C structseq / Python namedtuple (depending on the implementation), calls threading.excepthook(). If threading.excepthook() raises an exception, sys.excepthook() is called to handle it.

    --

    First I also added a "stderr" argument to the arguments passed to threading.excepthook(): sys.stderr, or an old copy of sys.stderr if sys.stderr no longer exists or is set to None. But I decided to keep this as an implementation detail instead.

    @vstinner
    Copy link
    Member

    If you want to reuse sys.excepthook to handle uncaught Thread.run() exception, you can now write:
    ---

    def hook(args):
        if args.exc_type == SystemExit:
            return
        sys.excepthook(args.exc_type, args.exc_value, args.exc_traceback)
    
    threading.excepthook = hook

    Try attached sys_threading_excepthook.py for a full example.

    @serhiy-storchaka
    Copy link
    Member

    I propose to add the Thead.excepthook() method with the signature compatible with sys.excepthook(). This will allow to set easily per-thread hooks and a global hook.

    @pitrou
    Copy link
    Member

    pitrou commented May 25, 2019

    A Thread.excepthook() method does not allow to notify exceptions raised in C-created threads ("dummy threads").

    Also, as I said already, you can get the current thread by calling threading.current_thread() in the except hook. There is no need to pass it explicitly to the except hook.

    @vstinner
    Copy link
    Member

    A Thread.excepthook() method does not allow to notify exceptions raised in C-created threads ("dummy threads").

    The main difference between sys.excepthook and threading.excepthook is that the threading hook displays the thread name. Which output do you expect if you don't pass a threading.Thread object (which has a 'name' attribute)? The thread identifier from _thread.get_ident()?

    I can easily modify the default hook implementation to support args.thread=None and displays _thread.get_ident() as the name instead.

    Also, as I said already, you can get the current thread by calling threading.current_thread() in the except hook. There is no need to pass it explicitly to the except hook.

    I understand that your plan is to use sys.excepthook to handle threading.Thread uncaught exception.

    My main concern is that sys.excepthook can fail on importing the threading module. One solution would be to import threading when the sys module is created and keep a strong reference to threading.current_thread(). But I dislike this idea.

    I already raised my concern, but so far, you didn't explain how you plan to fix this practical issue.

    Moreover, the current threading code is quite complex. My guess is that this complexity is needed to display exception very late during Python shutdown, when module attributes are set to None and import no longer works.

    I would prefer to not move this complexity into sys.excepthook which has currently a simple implementation.

    --

    Daemon threads are evil. We should drop this nasty feature... but I would prefer to not block this issue by the removal of daemon threads, since I'm sure that a lot of code rely on them and so it will be really hard to really remove them.

    @vstinner
    Copy link
    Member

    Serhiy Storchaka:

    I propose to add the Thead.excepthook() method with the signature compatible with sys.excepthook(). This will allow to set easily per-thread hooks and a global hook.

    I don't see the relationship between the API (signature) and the ability to set a per-thread hook.

    I'm not convinced that a per-thread hook is needed. My proposed global threading.excepthook gets a thread parameter which allows a custom hook to implement a different behavior depending on the thread. You can use a different behavior depending on the thread name, depending on a custom Thread attribute, etc.

    Would it be acceptable for first add a global hook and see later if a per-thread hook is needed?

    @pitrou
    Copy link
    Member

    pitrou commented May 25, 2019

    Sorry, I had overlooked the issue with global variables at shutdown. Though that issue only occurs with daemonic threads, since non-daemonic threads are joined before global variables are cleared.

    In any case, I think the namedtuple / structseq solution is elegant, because we can add additional information later (the user must only be careful not to use tuple unpacking).

    @pitrou
    Copy link
    Member

    pitrou commented May 25, 2019

    Le 25/05/2019 à 23:09, STINNER Victor a écrit :

    I don't see the relationship between the API (signature) and the ability to set a per-thread hook.

    I'm not convinced that a per-thread hook is needed.

    Indeed, if you write your own Thread class, you can add a try...except
    in the Thread.run() method. You don't need a dedicated
    Thread.excepthook() method.

    The only way a per-thread hook could be useful is if you could set it
    *outside* of the Thread class (so not as a method), so that one can e.g.
    catch / report exceptions raised in threads launches by third-party
    libraries.

    @vstinner
    Copy link
    Member

    Indeed, if you write your own Thread class, you can add a try...except
    in the Thread.run() method. You don't need a dedicated
    Thread.excepthook() method.

    Exactly. You can already do you best in your run() method to handle exceptions.

    threading.excepthook is only there is everything else already failed.

    FYI in my implementation, if threading.excepthook raises a new exception, it's also handled... by sys.excepthook this time ;-)

    The only way a per-thread hook could be useful is if you could set it
    *outside* of the Thread class (so not as a method), so that one can e.g.
    catch / report exceptions raised in threads launches by third-party
    libraries.

    I discuss threading excepthook with Pablo and he asked me if it would be possible to have a different behavior depending if the thread is spawn by my application or by "third party code". Using threading.excepthook, you can mark your threads that you spawn directly using a specific name, a special attribute, or you may even track them in a list (maybe using weak references).

    If sys.excepthook is used to handle threading exceptions, you call threading.current_thread(), but then we come back to the issue to "dying" Python: exception which occurs late in Python finalization, when most modules are already cleared and import no longer works.

    @vstinner
    Copy link
    Member

    Antoine Pitrou:

    A Thread.excepthook() method does not allow to notify exceptions raised in C-created threads ("dummy threads").

    I modified my PR to use the threading identitifer (threading.get_ident()) as the thread name if args.thread is None.

    @vstinner
    Copy link
    Member

    In any case, I think the namedtuple / structseq solution is elegant, because we can add additional information later

    I used the same design for the new sys.unraisablehook in bpo-36829 and I'm already working on adding a new 'err_msg' field to the argument passed to this took: PR 13488 :-)

    I was trapped in the past when I had to modify warnings "hooks" (warnings.showwarning and warnings.formatwarning) when I had to add a new 'source' parameter. I had to write wrappers which are fragile, to keep the backward compatibility.

    (the user must only be careful not to use tuple unpacking)

    threading.excepthook doesn't mention the compatibility with tuple on purpose. It only documents attributes with their names.

    @vstinner
    Copy link
    Member

    New changeset cd590a7 by Victor Stinner in branch 'master':
    bpo-1230540: Add threading.excepthook() (GH-13515)
    cd590a7

    @vstinner
    Copy link
    Member

    It took 14 years, but this issue is now fixed ;-) I close it.

    Python 3.8 beta 1 will be released in a few days with threading.excepthook(): please test it to ensure that it works as you expected.

    As a follow-up, I created bpo-37069: "regrtest: log unraisable exceptions and uncaught thread exceptions".

    @serhiy-storchaka
    Copy link
    Member

    If we want to support low-level threads created by start_new_thread() we should call excepthook() from t_bootstrap instead of Thread._bootstrap_inner. Otherwise implementing excepthook() as a Thread method would be more convenient.

    @vstinner vstinner changed the title sys.excepthook doesn't work in threads Add threading.excepthook() to handle uncaught exceptions raised by Thread.run() May 28, 2019
    @vstinner vstinner changed the title sys.excepthook doesn't work in threads Add threading.excepthook() to handle uncaught exceptions raised by Thread.run() May 28, 2019
    @vstinner
    Copy link
    Member

    If we want to support low-level threads created by start_new_thread() we should call excepthook() from t_bootstrap instead of Thread._bootstrap_inner. Otherwise implementing excepthook() as a Thread method would be more convenient.

    I created bpo-37076: "_thread.start_new_thread(): call sys.unraisablehook() to handle uncaught exceptions".

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 9, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    5 participants