Spyder runs into infinite loop when using multiprocessing

[michi1510]

Spyder freezes when following code is run with big arrays (>100k entries each).

Code:

import multiprocessing.Pool as Pool
from scipy.interpolate import LinearNDInterpolator
if __name__=="__main__":
    p=Pool(4)
    ...
    lndi = LinearNDInterpolator(pointsA, valuesA)
    xi= list(np.split(pointsB, 4))
    valuesB = p.map(lndi.__call__, valuesB)

Running the above code brings the described problem.
If the map-function is commented and called independetly, I get the expected result, if run in Spyder.
As the script is supposed to work automatically, I found out that it can be executed correctly from the command line " python ..script.py" and from IPython.

Versions and main components

*scipy Version: 0.17.1

• Spyder Version: 2.3.8
• Python Version: 3.5.1
• Operating system: Windows 7 Enterprise, SP1, 64 Bit, Intel

Dependencies

IPython >=1.0 : 4.0.3 (OK)
jedi >=0.8.1;<0.9.0: 0.9.0 (NOK)
matplotlib >=1.0 : 1.5.1 (OK)
pandas >=0.13.1 : 0.17.1 (OK)
pep8 >=0.6 : 1.7.0 (OK)
pyflakes >=0.6.0 : 1.0.0 (OK)
pygments >=1.6 : 2.1 (OK)
pylint >=0.25 : None (NOK)
qtconsole >=4.0 : 4.1.1 (OK)
rope >=0.9.2 : 0.9.4-1 (OK)
sphinx >=0.6.6 : 1.3.1 (OK)
sympy >=0.7.3 : 0.7.6.1 (OK)
zmq >=2.1.11 : 15.2.0 (OK)

[Nodd]

We can't try your example until you provide some values for pointsA, valuesA (and same for B).

Also, how are you running you code in spyder ? in a dedicated console ? Python or IPython ?

[ccordoba12]

Yes, please provide a reproducible example :-)

[ccordoba12]

Closing because we can't run this code.

[darinhitchings]

I'm having the same problem. And I also am able to run a multi-processing session from command-line but not from Spyder. I have a Win 10 machine. I have Anaconda installed. And one other point is that I recently installed the tifffile package within Anaconda.

"Python 2.7.14 |Anaconda, Inc.| (default, Nov 8 2017, 13:40:45) [MSC v.1500 64 bit (AMD64)]
Type "copyright", "credits" or "license" for more information.

IPython 5.4.1 -- An enhanced Interactive Python."

I am able to generate an infinite loop using the code:

import multiprocessing as mp
import logging
import time

log=mp.log_to_stderr()
log.setLevel(mp.SUBDEBUG)

def foo_pool(x):
time.sleep(2)
return x*x

if name == "main":

print("Pool opened \n");

pool = mp.Pool(processes=4);
            

results = [pool.apply_async(foo_pool, args=(x,)) for x in range(20)];

output = [p.get() for p in results];

#https://stackoverflow.com/questions/38271547/when-should-we-call-multiprocessing-pool-join
pool.close();
print("Pool closed \n");
pool.join();
print("Pool joined \n");
  
print(output);