Thread shutdown exception in Thread.notify()

[yangzhang]

BPO	1722344
Nosy	@loewis, @amauryfa, @pitrou, @bitdancer, @ssbarnea
Files	1722344_squelch_exception.patch: Patch to hide these exceptions (and further ones caused by it) for the purposes of releasing a production application. nondaemon_thread_shutdown.diff: Move WaitForThreadShutdown call into Py_Finalize, so all exit paths use it

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/pitrou'
closed_at = <Date 2009-10-27.13:10:46.882>
created_at = <Date 2007-05-20.22:24:37.000>
labels = ['interpreter-core', 'type-bug']
title = 'Thread shutdown exception in Thread.notify()'
updated_at = <Date 2010-04-13.18:41:19.256>
user = 'https://bugs.python.org/yangzhang'

bugs.python.org fields:

activity = <Date 2010-04-13.18:41:19.256>
actor = 'brett.cannon'
assignee = 'pitrou'
closed = True
closed_date = <Date 2009-10-27.13:10:46.882>
closer = 'pitrou'
components = ['Interpreter Core']
creation = <Date 2007-05-20.22:24:37.000>
creator = 'yangzhang'
dependencies = []
files = ['9356', '10154']
hgrepos = []
issue_num = 1722344
keywords = ['patch']
message_count = 25.0
messages = ['32092', '32093', '32094', '32095', '32096', '62073', '62078', '62079', '62080', '62081', '62083', '62084', '62091', '62108', '66054', '66055', '94283', '94290', '94291', '94303', '94558', '99671', '99686', '99694', '103025']
nosy_count = 15.0
nosy_names = ['loewis', 'gpk', 'amaury.forgeotdarc', 'ggenellina', 'Rhamphoryncus', 'pitrou', 'yangzhang', 'thomasda', 'lobais', 'jamescooper', 'hongqn', 'reacocard', 'r.david.murray', 'ssbarnea', 'thijs']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue1722344'
versions = ['Python 2.6']

[yangzhang]

Hi, I sometimes see the following exceptions when shutting down my app (using Python 2.5.1):

  Unhandled exception in thread started by
  Error in sys.excepthook:
  
  Original exception was:
  Exception in thread Thread-3 (most likely raised during interpreter shutdown):
  Traceback (most recent call last):
    File "/usr/local/lib/python2.5/threading.py", line 460, in __bootstrap
    File "/usr/local/lib/python2.5/threading.py", line 440, in run
    File "/home/yang/local/armed/lib/python2.5/site-packages/afx/threads.py", line 71, in worker
    File "/usr/local/lib/python2.5/Queue.py", line 176, in get
    File "/usr/local/lib/python2.5/threading.py", line 248, in notify
  <type 'exceptions.TypeError'>: exceptions must be classes, instances, or strings (deprecated), not NoneType
  Unhandled exception in thread started by
  Error in sys.excepthook:
  
  Original exception was:
  Exception in thread Thread-6 (most likely raised during interpreter shutdown):
  Traceback (most recent call last):
    File "/usr/local/lib/python2.5/threading.py", line 460, in __bootstrap
    File "/usr/local/lib/python2.5/threading.py", line 440, in run
    File "/home/yang/local/armed/lib/python2.5/site-packages/afx/threads.py", line 71, in worker
    File "/usr/local/lib/python2.5/Queue.py", line 176, in get
    File "/usr/local/lib/python2.5/threading.py", line 248, in notify
  <type 'exceptions.TypeError'>: exceptions must be classes, instances, or strings (deprecated), not NoneType
  Unhandled exception in thread started by
  Error in sys.excepthook:
  
  Original exception was:

Here is the code from my application:

        def worker():
            debug( 'starting worker' )
            while True:
                msg = i.get() # <-- THIS IS LINE 71
                if msg is stop_msg: break
                resultbuf, func, args, kwargs = msg
                result, exc = None, None
                try:
                    result = func( *args, **kwargs )
                except:
                    t, v, tb = exc_info()
                    exc = t, v, tb.tb_next
                o.put( ( resultbuf, result, exc ) )
                s.send( 'x' ) # assuming socket.send is thread-safe
            debug( 'stopping worker' )

Here is the origin of the exception (in threading.py):

    def notify(self, n=1):
        assert self._is_owned(), "notify() of un-acquire()d lock" # <-- THIS IS LINE 248
        __waiters = self.__waiters
        waiters = __waiters[:n]
        if not waiters:
            if __debug__:
                self._note("%s.notify(): no waiters", self)
            return
        self._note("%s.notify(): notifying %d waiter%s", self, n,
                   n!=1 and "s" or "")
        for waiter in waiters:
            waiter.release()
            try:
                __waiters.remove(waiter)
            except ValueError:
                pass

I'm not sure why this is happening. The threads are not daemon threads; I terminate them cleanly. When I get a SIGINT (I usu. shut down my app with ctrl-C), I enqueue n stop_msg's to the 'i' Queue so that the n workers can all exit.

Note I usually launch 5 workers, so I'm not consistently getting an exception per worker. Also, I've been unable to reproduce this at will.

[ggenellina]

Do you join() the worker threads, waiting until they finish, before exiting the main thread?

[yangzhang]

No, as they are not daemon threads.

[thomasda]

I'm getting the same kind of errors.
I'm using a lot of threads, and one of them throws this thread nearly everytime I close my program (by gtk.main_quit)

It seems that python sets every variable to None, and then wakeup sleeping threads, which then crash, as they try to work with the None variables

Exception in thread Thread-3 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib/python2.4/threading.py", line 442, in __bootstrap
  File "/home/thomas/Programmering/python/skak/0.7/lib/pychess/System/ThreadPool.py", line 49, in run
  File "/usr/lib/python2.4/Queue.py", line 89, in put
  File "/usr/lib/python2.4/threading.py", line 237, in notify
exceptions.TypeError: exceptions must be classes, instances, or strings (deprecated), not NoneType
Unhandled exception in thread started by 
Error in sys.excepthook

[gpk]

I see the same problem. I'm not sure my code is clean,
but I'll be darned if I can find the problem.

I've traced my code with print statements,
and I see some threads reach their return statement,
but the process hangs when you try to join() that thread.
It seems that there is something in the clean-up code
in threading.py (presumably) that can hang under some
obscure circumstances.

As a result, these threads don't terminate on time,
and various exceptions happen as python is being
dismantled and all the variables de-allocated.

[jamescooper]

Though these exceptions while shutting down are mostly harmless, they
are very noisy and must be squelched in a production application. Here
is the patch which we at Solido Design (www.solidodesign.com) are using
to hide the exceptions. Note that this doesn't fix the problem, but it
does hide the exception for the purposes of releasing an application to
the public.

[brettcannon]

I think the general idea of the problem has been stated, but I figured I
would state the official issue. When Python begins to shutdown it takes
each module and sets each variable in the global namespace to None. If a
thread has not terminated before the interpreter terminates then the
thread tries to use a global variable which has been set to None.

This is not about to change since this occurs because of coding
"errors". You must make sure that either your thread is as safe as a
__del__ method (which means no global namespace access) or you can't let
the app exit until you are positive all of your threads have terminated,
not just asked them to shutdown since this is all asynchronous.

[Rhamphoryncus]

Py_Main calls WaitForThreadShutdown before calling Py_Finalize, which
should wait for all these threads to finish shutting down before it
starts wiping their globals.

However, if SystemExit is raised (such as via sys.exit()), Py_Exit is
called, and it directly calls Py_Finalize, bypassing the
WaitForThreadShutdown.

Can someone who's experienced this bug check if they're using
SystemExit/sys.exit?

[Rhamphoryncus]

To put it another way: SystemExit turns non-daemon threads into daemon
threads. This is clearly wrong. Brent, could you reopen the bug?

[brettcannon]

Hold on, why is that wrong? What if the threads block forever,
preventing shutdown? sys.exit() is not exactly some namby-pamby function
but a forced shutdown of the interpreter that should guarantee that the
interpreter quits. Changing its semantics now would take that away.

[Rhamphoryncus]

I disagree. sys.exit() attempts to gracefully shutdown the interpreter,
invoking try/finally blocks and the like. If you want to truly force
shutdown you should use os.abort() or os._exit().

Note that, as python doesn't call a main function, you have to use
sys.exit() to have an exit status.

[brettcannon]

OK, I will re-open to see if some other core developer wants to take
this on, but personally I am passing.

[lobais]

which means no global namespace access
Does that mean that you cannot use len and range in a Thread?

[brettcannon]

> which means no global namespace access
Does that mean that you cannot use len and range in a Thread?

No, it means you have to be careful if you do. Shutting down properly
will take care of things. Otherwise you need to save a reference
locally (either on an object or as a local variable) and use that
reference instead of relying on the one defined in the global
namespace.

[Rhamphoryncus]

This bug was introduced by r53249, which was fixing bug bpo-1566280.

Fixed by moving the WaitForThreadShutdown call into Py_Finalize, so all
shutdown paths use it. I also tweaked the name to follow local helper
function conventions.

Martin, since you did the previous fix, can you review this one?

[Rhamphoryncus]

Oh, and the patch includes a testcase. The current test_threading.py
doesn't work with older versions, but a freestanding version of this
testcase passes in 2.1 to 2.4, fails in 2.5 and trunk, and passes with
the patch.

[pitrou]

The patch looks good to me.
And since Py_Finalize() claims to "destroy all sub-interpreters" and
"free all memory allocated by the Python interpreter", I guess your
approach makes sense. Can you commit?

[Rhamphoryncus]

Nope, no access.

[pitrou]

Ok, I'll do it then!

[pitrou]

Patch was committed in trunk, py3k and 3.1. Waiting for 2.6 to be
unfrozen before I commit it there too.

[pitrou]

Backported to 2.6 in r75749.

[ssbarnea]

Any idea if there is a nightly build for Python 2.6? The latest release was 2.6.4 and was 2 days before submitting the patch.

Or the only alternative is to build it myself? Any ideas on when we could see 2.6.5? - I tried to look for a release timeline but I wasn't able to locate one.

[amauryfa]

I have seen somewhere (ask google), that python 2.6.5 would be released mid-march.
But except for a few platforms, python.org does not provide compiled binaries.

[bitdancer]

According to Barry's latest email on the subject, the dates are:

2009-03-01 Python 2.6.5 rc 1
2009-03-15 Python 2.6.5 final

And no, there are no nightly builds, you have to build it yourself, I'm afraid.

[thijs]

Looks like this influenced mod_wsgi as well: http://groups.google.com/group/modwsgi/browse_thread/thread/ba82b2643564d2dd