multiprocessing not compatible with functools.partial

[ndbecker]

BPO	5228
Nosy	@amauryfa, @pitrou, @jackdied, @tiran, @jwilk

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/jackdied'
closed_at = <Date 2009-03-31.23:48:06.735>
created_at = <Date 2009-02-12.14:26:28.378>
labels = ['type-bug', 'library']
title = 'multiprocessing not compatible with functools.partial'
updated_at = <Date 2011-04-03.21:39:03.325>
user = 'https://bugs.python.org/ndbecker'

bugs.python.org fields:

activity = <Date 2011-04-03.21:39:03.325>
actor = 'jwilk'
assignee = 'jackdied'
closed = True
closed_date = <Date 2009-03-31.23:48:06.735>
closer = 'jackdied'
components = ['Library (Lib)']
creation = <Date 2009-02-12.14:26:28.378>
creator = 'ndbecker'
dependencies = []
files = []
hgrepos = []
issue_num = 5228
keywords = []
message_count = 11.0
messages = ['81766', '81767', '81788', '81789', '84871', '84872', '84949', '100669', '130511', '130513', '130548']
nosy_count = 9.0
nosy_names = ['amaury.forgeotdarc', 'pitrou', 'jackdied', 'christian.heimes', 'jwilk', 'jnoller', 'ndbecker', 'joseph.h.garvin', 'uptimebox']
pr_nums = []
priority = 'low'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue5228'
versions = ['Python 2.6']

[ndbecker]

from multiprocessing import Pool

def power (x, pwr=2):
    return x**pwr

import functools
run_test = functools.partial (power, pwr=3)

if __name__ == "__main__":

    pool = Pool()
    cases = [3,4,5]
    results = pool.map (run_test, cases)

TypeError: type 'partial' takes at least one argument
Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-
    self.run()
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-
    task = get()
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-
    return recv()
TypeError: type 'partial' takes at least one argument

[jnoller]

See this mail thread:
http://mail.python.org/pipermail/python-dev/2009-February/086034.html

[pitrou]

Rather than implement a multiprocessing-specific fix, it would certainly
be better to make functools.partial picklable, as pointed out in the ML
thread Jesse linked to.

[jnoller]

Antoine Pitrou <pitrou@free.fr> added the comment:

Rather than implement a multiprocessing-specific fix, it would certainly
be better to make functools.partial picklable, as pointed out in the ML
thread Jesse linked to.

I would tend to agree

[jnoller]

I agree that this is a nice feature, however it requires adding a
getstate/setstate within the functools C code.

I would need a patch which does this, and adds tests to the
functools/pickle test suite for this. Also, the feature will only be
accepted for py3k (3.1) currently. Features such as this will not be
backported to the 2.x branch

[jnoller]

Jack offered to take a peek

[jackdied]

Fixed rev 70931. Happy pickling!

[josephhgarvin]

I think this bug still exists in Python 2.6.4, and I haven't tested 2.6.5, but since 2.6.4 was released 6 months after this bug was closed I assume it's still an issue. Running ndbecker's test program as is produces the following output on Solaris 10:

Process PoolWorker-1:
Process PoolWorker-2:
Traceback (most recent call last):
Traceback (most recent call last):
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
Process PoolWorker-3:
Traceback (most recent call last):
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
    self.run()
    self.run()
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/process.py", line 88, in run
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
    self._target(*self._args, **self._kwargs)
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/pool.py", line 57, in worker
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/pool.py", line 57, in worker
    self.run()
    task = get()
    task = get()
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/queues.py", line 352, in get
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/process.py", line 88, in run
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/queues.py", line 352, in get
    self._target(*self._args, **self._kwargs)
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/pool.py", line 57, in worker
    return recv()
    return recv()
TypeError: type 'partial' takes at least one argument
TypeError: type 'partial' takes at least one argument
    task = get()
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/queues.py", line 352, in get
    return recv()
TypeError: type 'partial' takes at least one argument

[uptimebox]

The bug still exists in 2.6.6

Process PoolWorker-2:
Traceback (most recent call last):
Process PoolWorker-1:
Traceback (most recent call last):
  File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
  File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
    self.run()
    self.run()
  File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
  File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.6/multiprocessing/pool.py", line 57, in worker
  File "/usr/lib/python2.6/multiprocessing/pool.py", line 57, in worker
    task = get()
    task = get()
  File "/usr/lib/python2.6/multiprocessing/queues.py", line 352, in get
  File "/usr/lib/python2.6/multiprocessing/queues.py", line 352, in get
    return recv()
    return recv()
TypeError: type 'partial' takes at least one argument
TypeError: type 'partial' takes at least one argument

[amauryfa]

The bug was fixed in 2.7, and not backported to 2.6.
2.6 is now in security-fix-only mode, if possible you should upgrade to 2.7.

[uptimebox]

Unfortunately, upgrading to 2.7 is not an option for me, since my application works on Debian stable which only includes 2.6. For now I'm settled with the following workaround: http://paste.pocoo.org/show/351774/