Note: originally this message was send to firstname.lastname@example.org however for some reason 24 H later is still wasn't on the mailing list, therefore I assume it was dropped for some reason. Anyway I as think about it this is probably a more appropriate place to put it.
I'm currently trying to make a basic parallel map using IPython's
kernel.client and I run into two issues.
-- Issue 1 --
When the function that is to be called has multiple
arguments and is mapped using MultiEngineClient for instance as:
mec.map(lambda a: f(a,**c_args),np.arange(100)) (1)
where c_args = dict(b=10,c=3). This is equivalent to:
mec.map(lambda a: f(a,10,3),np.arange(100)) (2)
The point is the formulation (1) works in the main but fails when
called from a class method, while formulation (2) works all the time.
Just in case I have pushed c_args with MultiEngineClient
I get the following error:
ValueError: Sorry, cannot pickle code objects with closures
I can not use formulation (2) as in real application c_args depends on
the current object properties.
You can find a minimal working example here:
as well as the traceback:
As for functions that I managed to map with MultiEngineClient the
execution was quite slow, so I made a short benchmark using this file on
and the ouput shows that for me the normal map is always faster than
mec.map while TaskClient's map just hungs using all available CPU power.
Additional logs can be found here:
Well I'm new to Ipython parallel processing capabilities, so am I
missing something here ? Or are there some issues with the software
versions I'm using ?
I use Gentoo with
Python 2.7.1 (r271:86832, Feb 28 2011, 13:20:38)
IPython 0.10.1 USE flags: (examples gnuplot readline -doc -emacs -smp
Thank you in advance for your help,
The closure is actually straightforward:
The lambda function is referring to the c_args in its parent scope via a closure. To fix this, simply rename the one declared in __init__ to something else (b_args), then when the lambda function is called, it will look in globals, and should be fine (with this change, your script works for me).
As for general and performance issues: we are in the process of completely replacing IPython.kernel (with IPython.zmq.parallel, see the 'newparallel' branch on GitHub), so I am afraid that we aren't likely to fix much there.
Thank you for your fast reply.
Indeed, the closure exception is quite self-explanatory: It's just that I wasn't aware of the closure concept before....
I'll have a look at the 'newparallel' branch.
IPython.kernel has been removed and replaced by IPython.zmq.parallel.