multiprocessing 'NoneType' object is not callable

[mcdonc]

BPO	15881
Nosy	@birkenfeld, @abalkin, @larryhastings, @pjenvey, @benjaminp, @mitar, @cjerdonek, @hynek
Files	shutdown_typeerror-tip.patch shutdown_typeerror-27.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/abalkin'
closed_at = <Date 2013-02-02.16:18:57.947>
created_at = <Date 2012-09-07.18:56:39.581>
labels = ['type-bug', 'library', 'release-blocker']
title = "multiprocessing 'NoneType' object is not callable"
updated_at = <Date 2013-02-02.16:18:57.947>
user = 'https://bugs.python.org/mcdonc'

bugs.python.org fields:

activity = <Date 2013-02-02.16:18:57.947>
actor = 'benjamin.peterson'
assignee = 'belopolsky'
closed = True
closed_date = <Date 2013-02-02.16:18:57.947>
closer = 'benjamin.peterson'
components = ['Library (Lib)']
creation = <Date 2012-09-07.18:56:39.581>
creator = 'mcdonc'
dependencies = []
files = ['27142', '27143']
hgrepos = []
issue_num = 15881
keywords = ['patch']
message_count = 18.0
messages = ['170003', '170008', '170009', '170022', '170023', '170119', '170120', '170121', '170122', '170187', '170215', '170220', '170230', '170446', '180305', '180438', '181024', '181173']
nosy_count = 13.0
nosy_names = ['georg.brandl', 'tseaver', 'belopolsky', 'larry', 'pjenvey', 'benjamin.peterson', 'Arfrever', 'mcdonc', 'mitar', 'chris.jerdonek', 'python-dev', 'sbt', 'hynek']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue15881'
versions = ['Python 3.2', 'Python 3.3', 'Python 3.4']

[mcdonc]

The symptom is an exact duplicate of the symptom reported in the following (closed) issue:

http://bugs.python.org/issue9775

The issue is also related to the following other issues:

http://bugs.python.org/issue4106
http://bugs.python.org/issue9205
http://bugs.python.org/issue9207

To reproduce the symptom driving the patches that will be attached to this issue:

git clone git://github.com/pypa/pip.git
cd pip
/any/python setup.py dev
/any/python setup.py test

You can either wait for the entire test suite to finish or you can press ctrl-C at any time (the tests take a long time). In either case, a traceback like the following will be printed to the console.

  Error in atexit._run_exitfuncs:
  Traceback (most recent call last):
    File "/home/chrism/opt/Python-2.7.3/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
    File "/home/chrism/opt/Python-2.7.3/lib/python2.7/multiprocessing/util.py", line 284, in _exit_function
      info('process shutting down')
  TypeError: 'NoneType' object is not callable
  Error in sys.exitfunc:
  Traceback (most recent call last):
    File "/home/chrism/opt/Python-2.7.3/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
      func(*targs, **kargs)
    File "/home/chrism/opt/Python-2.7.3/lib/python2.7/multiprocessing/util.py", line 284, in _exit_function
      info('process shutting down')
  TypeError: 'NoneType' object is not callable

From what I understand in other issues, multiprocessing.util._exit_function shouldn't actually be called *after* the containing module's globals are destroyed (it should be called before), but this does indeed happen.

Patches will be attached that anticipate the symptom and prevent a shutdown error. One will be attached for Python 2.7 branch, the other for the Python tip. Each causes functions that are called at shutdown time to keep a reference around to other functions and globals used within the function, and each does some checks for the insane state and prevents an error from happening in this insane state.

[mcdonc]

Patch for tip.

[mcdonc]

2.7 branch patch.

[abalkin]

The patch makes sense. I'll take another look over the weekend, but it seems to be ready to be applied.

[cjerdonek]

+ # NB: we hold on to references to functions in the arglist due to the

This is a nit, but I think adding "NB:", "Note:", etc. to the beginning of a comment is redundant because by being a comment it is already implicit that it should be noted.

[python-dev]

New changeset 27d410dd5431 by Alexander Belopolsky in branch '3.2':
Issue bpo-15881: Fixed atexit hook in multiprocessing.
http://hg.python.org/cpython/rev/27d410dd5431

New changeset 08c680918ff8 by Alexander Belopolsky in branch 'default':
Issue bpo-15881: Fixed atexit hook in multiprocessing.
http://hg.python.org/cpython/rev/08c680918ff8

[python-dev]

New changeset db67b848ddc3 by Alexander Belopolsky in branch '3.2':
Issue bpo-15881: Fixed 3.2 backport.
http://hg.python.org/cpython/rev/db67b848ddc3

[abalkin]

Applied to 3.2 and 3.3. Thanks for the patch!

Leaving it open pending 2.7 commit.

[python-dev]

New changeset b05547e8ff92 by Alexander Belopolsky in branch '3.2':
Issue bpo-15881: Added NEWS entry and proper credit.
http://hg.python.org/cpython/rev/b05547e8ff92

[sbt]

I see the same error on Windows (when pressing ^C), but on Linux I get

Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 28, in _run_exitfuncs
    import traceback
  File "/usr/lib/python2.7/traceback.py", line 3, in <module>
    import linecache
  File "/usr/lib/python2.7/linecache.py", line 9, in <module>
    import os
  File "/usr/lib/python2.7/os.py", line 119, in <module>
    sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'

This also suggests that module teardown has begun before/while sys.exitfunc is running.

[sbt]

I suspect the problem is caused by nose's isolate plugin.

With this enabled, a copy of sys.modules is saved before each test and then restored after the test. This causes garbage collection of newly imported modules. The destructor for the module type causes all globals to be replaced by None.

This will break the atexit function registered by multiprocessing since it depends on globals.

PS. A simple work-around (which does not require people to upgrade to a bugfixed version of Python) is to put

try:
    import multiprocessing
except ImportError:
    pass

near the beginning of setup.py. After this change I don't get the error when running "python setup.py test".

[sbt]

Actually, I am not so sure it is the isolate plugin. But I do think that sys.modules is being manipulated somewhere before shutdown.