Multiprocessing hangs in iPython console

[spyder-bot]

From oscarbra...@gmail.com on 2014-07-20T01:09:49Z

Spyder Version: 2.3.0
Python Version: 2.7.8
Qt Version : 4.8.6, PyQt4 (API v2) 4.10.4 on Darwin
pyflakes >=0.5.0: None (NOK)
pep8 >=0.6 : None (NOK)
IPython >=0.13 : 2.1.0 (OK)
pygments >=1.6 : 1.6 (OK)
sphinx >=0.6.6 : None (NOK)
psutil >=0.3 : None (NOK)
rope >=0.9.2 : None (NOK)
matplotlib >=1.0: 1.3.1 (OK)
sympy >=0.7.0 : 0.7.5 (OK)
pylint >=0.25 : None (NOK)

What steps will reproduce the problem?

1. Run the following:

from multiprocessing import Pool

eg = [1,2,3,4,5,6,7]

def multi_fn(x):
return x**2

pool = Pool()
out = pool.map(multi_fn,eg)
pool.close()
pool.join()

In the normal python console, this will complete with no problems.

In the ipython console, this will hand on 'pool.join()'

The code works in iPython notebook, iPython run in terminal and normal python run in terminal.

The hang appears to be unique to the ipython console within Spyder - any ideas what is causing this?

Original issue: http://code.google.com/p/spyderlib/issues/detail?id=1900

[spyder-bot]

From ccordoba12 on 2014-08-17T10:19:05Z

Labels: Cat-IPythonConsole MS-v2.3.2

[spyder-bot]

From ccordoba12 on 2014-09-20T14:58:18Z

This is working fine for me on Linux. Could you check if this is still an issue with 2.3.1?

[spyder-bot]

From ccordoba12 on 2014-09-21T10:51:30Z

We need to verify if this is a problem in Mac and/or Windows, before closing the issue.

Labels: Easy

[spyder-bot]

From ccordoba12 on 2014-10-31T15:07:53Z

Labels: -MS-v2.3.2 MS-v2.3.3

[spyder-bot]

From ccordoba12 on 2015-01-08T15:10:53Z

Labels: -MS-v2.3.3 MS-v2.3.4

[mikofski]

this is apparently an issue for me as well.

python 2.7.9
spyder 2.3.4

[ccordoba12]

@mikofski, what's your IPython version?

[mikofski]

ipython 3.1.0

background

I have a python extension that runs the MATLAB Common Runtime in a child process using multiprocessing. This is convenient because it lets me run the MCR multiple times, whereas from the main process I can only run the MCR once - because the MCR must be initialized and can only be initialized and terminated once per process. Leaving it running for a long time is risky because the MATLAB memory manager is unpredictable.

methods that work in spyder

• a normal python console

• running a script in a dedicated python console with commands wrapped in

  if __name__ == "__main__":
      do child proc stuff
  

• run in current python console and selecting python console, also with protection for __main__

• running a script with __main__ protection in current IPython console

methods that fail in spyder

• using the Ipython console hangs, no visible traceback. I must terminate the console and restart kernel.

[rhjmoore]

This is definitely a problem on my Windows machines. I haven't had the same problem on Linux.

[ccordoba12]

Multiprocessing is know to not work well with the IPython console, but I think that's beyond Spyder.

[yanwu26]

Hello, I know this is an older thread but I have this problem as well. What is a suggested alternative?

[mikofski]

@yanwu26 use the basic Python console instead of ipython kennel and it should work fine

[goanpeca]

This works fine on latest Master and Mac

@dalthviz or @jnsebgosselin, could you guys check on Win ?

from multiprocessing import Pool


def multi_fn(x):
    return x**2

eg = [1,2,3,4,5,6,7]

pool = Pool()
out = pool.map(multi_fn,eg)
pool.close()
pool.join()

[dalthviz]

@goanpeca it hangs for me on Windows

[jnsebgosselin]

This is a problem specific to Windows OS.

[goanpeca]

Indeed.

Some reference material: https://medium.com/@grvsinghal/speed-up-your-python-code-using-multiprocessing-on-windows-and-jupyter-or-ipython-2714b49d6fac

Any ideas on how we could detect this and warn the user?

[goanpeca]

Do you think we could do something here for windows as well @impact27 ?

Warn the users that they need to do something special at least?

[impact27]

The easiest way would be to subclass Pool() in spydercustomize to print a warning when created on windows. And add an option to silence the warning. Like:

Warning: `multiprocessing.Pool` is known to hang in the console on windows. Please call this in a `if __name__ == "__main__":` statement. You can disable this warning by ...

[ccordoba12]

@impact27, that's a good compromise, but how can we detect if Pool() is being called inside __main__ or not?

[impact27]

I don’t think we can, unless we do code analysis, so I would just print the warning in any case. (which is why it should be disabled by an option)

[impact27]

Apparently multiprocessing should display a warning itself:

Could this issue be assigned to someone who works on windows?