multiprocessing failing when function contains a class on the same file

[andreamoro-git]

Problem Description

The following code executes properly from the command line but not within spyder. The program hangs and the command window show the traceback pasted below. Placing the class code (everything before if __name__ == "__main__") in a separate file and importing the class solves the problem - but it is not clear why this code does not work as it does when executing from the command line.

What steps reproduce the problem?

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
from multiprocessing import Pool
class myClass():
    
    def __init__(self,q_seed=0,*args,**kwargs,) :
        self.q_seed = q_seed
        print(self.q_seed)

def simulateLocPool(kwargs) : 

    m = myClass(**kwargs)  
    return m   

#%%
if __name__ == "__main__":
    
    nboots = 10
    nprocs = 10
    
    # construct list to pass to processes
    iterlist = []
    kwargs = {'q_seed' : 0}
    
    # change seeds
    for bootn in range(nboots):
        newkw = kwargs.copy()
        newkw['q_seed'] = kwargs['q_seed'] + bootn
        iterlist.append((newkw)) 
    
    #spawn nboots processes, nprocs at once    
    pool = Pool(processes=nprocs)
    allmodels = pool.map(simulateLocPool,(iterlist))
    pool.close()
    pool.join()

Output

The code hangs with the following traceback

Exception in thread Thread-23:
Traceback (most recent call last):
  File "/Users/moroa/opt/miniconda3/lib/python3.9/threading.py", line 954, in _bootstrap_inner
    self.run()
  File "/Users/moroa/opt/miniconda3/lib/python3.9/threading.py", line 892, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/moroa/opt/miniconda3/lib/python3.9/multiprocessing/pool.py", line 576, in _handle_results
    task = get()
  File "/Users/moroa/opt/miniconda3/lib/python3.9/multiprocessing/connection.py", line 256, in recv
    return _ForkingPickler.loads(buf.getbuffer())
AttributeError: Can't get attribute 'myClass' on <module 'spyder_kernels.console.__main__' from '/Users/moroa/opt/miniconda3/lib/python3.9/site-packages/spyder_kernels/console/__main__.py'>

Expected output

A printout of digits from 0 to 9

Versions

• Spyder 5.1.5|Python 3.9.5 64-bit | Qt 5.12.10 | PyQt5 5.12.3 | Darwin 20.6.0
• Operating System name/version: MacOS 11.5.2

[steff456]

Hi @andreamoro-git,

Thanks for opening this issue, I could reproduce the problem with the snippet you provided. @jitseniesen, @ccordoba12 do you know why this is happening?

[athompson673]

fwiw, I have always directed people towards using "run in external console" when using multiprocessing due to a number of issues. Chiefly among them is that stdout is not always correctly handled (no child print statements in ipython console). The way to debug things like this is then to start spyder like so: python -m spyder.app.start, and all child output will go to the terminal, while main process output goes to the ipython console.

Doing this shows that the code is being run in the child, so __main__ is clearly importable in the child, and confirms what the error message suggests (go figure....) that the error occurs when un-pickling themyClass instances sent back from the children. I also took a look at sys.modules in various scenarios, and while I don't necessarily understand the difference... __main__ shows up in two different ways:

'__main__': <module '__main__' (built-in)>
or
'__main__': <module '__main__' from 'C:\\Users\\aaron\\Documents\\scripts\\python\\untitled0.py'>

[steff456]

So, @andreamoro-git and @athompson673, if you try to run the snippet of code in an ipython console, the error appears?

[andreamoro-git]

I think @athompson673 meant "Execute in an external system terminal".

[ccordoba12]

I don't know why this is not working, but

• If you're running your code in an unsaved file (as @athompson673 seems to be doing), it could be related to this because multiprocessing could have problems pickling code in that situation.
• The work @impact27 is doing in PR PR: Fix handling of kernel stderr, and capture stdout and segfaults too (IPython console) #14025 could help at least to debug these problems.

[andreamoro-git]

The file is saved

[impact27]

So I checked this issue. I can reproduce with iPython. If you:

• import multiprocessing.spawn
• then run the file
  You will get: AttributeError: Can't get attribute 'myClass' on <module '__main__'>

This might be a python error?

Desktop % ipython3
Python 3.8.6 (default, Oct  8 2020, 14:06:32) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.29.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import multiprocessing.spawn

In [2]: %run untitled1.py
0
1
Exception in thread Thread-3:
2
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", line 932, in _bootstrap_inner
3
4
5
6
7
8
9
    self.run()
  File "/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/pool.py", line 576, in _handle_results
    task = get()
  File "/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
AttributeError: Can't get attribute 'myClass' on <module '__main__'>

[impact27]

The problem here is that spyder-kernels imports multiprocessing.spawn. But this should not fail just because a module was imported!

[ccordoba12]

Thanks @impact27 for your help with this!

This will be fixed in our next version (5.2.2), to be released next week.