Skip to content

Loading…

IPython.kernel: parallel map issues #317

Closed
rth opened this Issue · 3 comments

3 participants

@rth

Note: originally this message was send to ipython-user@scipy.org however for some reason 24 H later is still wasn't on the mailing list, therefore I assume it was dropped for some reason. Anyway I as think about it this is probably a more appropriate place to put it.

Hello,

I'm currently trying to make a basic parallel map using IPython's
kernel.client and I run into two issues.

-- Issue 1 --

When the function that is to be called has multiple
arguments and is mapped using MultiEngineClient for instance as:
mec.map(lambda a: f(a,**c_args),np.arange(100)) (1)
where c_args = dict(b=10,c=3). This is equivalent to:
mec.map(lambda a: f(a,10,3),np.arange(100)) (2)

The point is the formulation (1) works in the main but fails when
called from a class method, while formulation (2) works all the time.
Just in case I have pushed c_args with MultiEngineClient

I get the following error:
ValueError: Sorry, cannot pickle code objects with closures

I can not use formulation (2) as in real application c_args depends on
the current object properties.

You can find a minimal working example here:
http://perso.crans.org/~yurchak/ipython/mwe.py

as well as the traceback:

http://perso.crans.org/~yurchak/ipython/mwe.log

--Issue 2--

As for functions that I managed to map with MultiEngineClient the
execution was quite slow, so I made a short benchmark using this file on
3 processes:
http://perso.crans.org/~yurchak/ipython/benchmark.py

and the ouput shows that for me the normal map is always faster than
mec.map while TaskClient's map just hungs using all available CPU power.
see http://perso.crans.org/~yurchak/ipython/ipython_latency.png

Additional logs can be found here:
http://perso.crans.org/~yurchak/ipython/ipcontroller.log

http://perso.crans.org/~yurchak/ipython/log/

Well I'm new to Ipython parallel processing capabilities, so am I
missing something here ? Or are there some issues with the software
versions I'm using ?

I use Gentoo with
Python 2.7.1 (r271:86832, Feb 28 2011, 13:20:38)
IPython 0.10.1 USE flags: (examples gnuplot readline -doc -emacs -smp
-test -wxwidgets)
dev-python/twisted 10.2.0
net-zope/zope-interface 3.6.1

Thank you in advance for your help,
best regards,
-- rth

@minrk
IPython member

The closure is actually straightforward:

The lambda function is referring to the c_args in its parent scope via a closure. To fix this, simply rename the one declared in __init__ to something else (b_args), then when the lambda function is called, it will look in globals, and should be fine (with this change, your script works for me).

As for general and performance issues: we are in the process of completely replacing IPython.kernel (with IPython.zmq.parallel, see the 'newparallel' branch on GitHub), so I am afraid that we aren't likely to fix much there.

@rth

Thank you for your fast reply.

Indeed, the closure exception is quite self-explanatory: It's just that I wasn't aware of the closure concept before....

I'll have a look at the 'newparallel' branch.

@rth rth closed this
@ellisonbg
IPython member

IPython.kernel has been removed and replaced by IPython.zmq.parallel.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.