Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Make IPython work with multiprocessing #86

Closed
ipython opened this Issue · 3 comments

3 participants

@ipython

Original Launchpad bug 315251: https://bugs.launchpad.net/ipython/+bug/315251
Reported by: gael-varoquaux (Gael Varoquaux).

Because the '.module' attribute of functions defined interactively in IPython is set to a virtual module non existant on the disk, distributing these functions with multiprocessing does not work:

resting ~ $ ipython
Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52) 
Type "copyright", "credits" or "license" for more information.

IPython 0.9.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.

In [1]: import multiprocessing as mp

In [2]: p = mp.Pool(2)

In [3]: def foo(x):
   ...:     return 2*x
   ...: 

In [4]: p.map(foo, range(4))
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.5/threading.py", line 486, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.5/threading.py", line 446, in run
    self.__target(*self.__args, **self.__kwargs)
  File "build/bdist.linux-i686/egg/multiprocessing/pool.py", line 225, in _handle_tasks
PicklingError: Can't pickle : attribute lookup __builtin__.function failed
@fperez
Owner

Darren Dale also provided this example that shows what I believe is the same problem. We need to ensure that both do work before closing the ticket.

from multiprocessing import Pool

import IPython.core.ipapi as ip
ip.get()

def update(i):
   print i

def f(i):
   return i*i

if __name__ == '__main__':
   pool = Pool()
   for i in range(10):
       pool.apply_async(f, [i], callback=update)
   pool.close()
   pool.join()

Upon running Darren's, I see this:

dreamweaver[test]> python darren_bug.py 
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.6/multiprocessing/pool.py", line 225, in _handle_tasks
    put(task)
PicklingError: Can't pickle : attribute lookup __builtin__.function failed
@minrk
Owner

This is the same bug as #29

@minrk
Owner

Closing to merge with #29

@minrk minrk closed this
@damianavila damianavila referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.