Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
coverage run and multiprocessing problem #745
I'm facing some issue with coverage and multiprocessing module.
I have a simple class wrapping multiprocessing.Pool:
from multiprocessing import Pool def job(a): return a class MyPool: def __init__(self): self.pool = Pool(processes=1) def run_job_in_worker(self): return self.pool.apply(job, ("hello",))
and a simple unittest:
import unittest from matlab_interop.my_pool import MyPool class TestMyPool(unittest.TestCase): def test_simple(self): pool = MyPool() print(pool.run_job_in_worker())
coverage halts and I get errors that I don't get if running nose directly without coverage.
====================================================================== ERROR: test_simple (tests.test_my_pool.TestMyPool) ---------------------------------------------------------------------- Traceback (most recent call last): File "F:\views\g\qprism\QPrism\MatlabInterop\py\src\tests\test_my_pool.py", line 9, in test_simple pool = MyPool() File "F:\views\g\qprism\QPrism\MatlabInterop\py\src\matlab_interop\my_pool.py", line 10, in __init__ self.pool = Pool(processes=1) File "F:\views\g\qprism\Qrelease\3rd\Python3.6.7\lib\multiprocessing\context.py", line 119, in Pool context=self.get_context()) File "F:\views\g\qprism\Qrelease\3rd\Python3.6.7\lib\multiprocessing\pool.py", line 175, in __init__ self._repopulate_pool() File "F:\views\g\qprism\Qrelease\3rd\Python3.6.7\lib\multiprocessing\pool.py", line 236, in _repopulate_pool self._wrap_exception) File "F:\views\g\qprism\Qrelease\3rd\Python3.6.7\lib\multiprocessing\pool.py", line 255, in _repopulate_pool_static w.start() File "F:\views\g\qprism\Qrelease\3rd\Python3.6.7\lib\multiprocessing\process.py", line 105, in start self._popen = self._Popen(self) File "F:\views\g\qprism\Qrelease\3rd\Python3.6.7\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj) File "F:\views\g\qprism\Qrelease\3rd\Python3.6.7\lib\multiprocessing\popen_spawn_win32.py", line 33, in __init__ prep_data = spawn.get_preparation_data(process_obj._name) File "F:\views\g\qprism\Qrelease\3rd\Python3.6.7\lib\multiprocessing\spawn.py", line 143, in get_preparation_data _check_not_importing_main() File "F:\views\g\qprism\Qrelease\3rd\Python3.6.7\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main is not going to be frozen to produce an executable.''') RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase. This probably means that you are not using fork to start your child processes and you have forgotten to use the proper idiom in the main module: if __name__ == '__main__': freeze_support() ... The "freeze_support()" line can be omitted if the program is not going to be frozen to produce an executable.
I would appreciate your help with figuring out what could be the problem. I don't understand why all works when running nose directly (without coverage).
Sorry for missing required information.
Attached 2 files: my_pool.py and test_my_pool.py. please download the files to 'src' folder and within 'src' run:
This should reproduce the problem.
I am experiencing the same issue but with
It appears to work on the linux CI servers though so I'm not sure if it's platform specific or related to the environment.
I am seeing the same as OP with python 3.6.8 nose 1.3.7 and coverage 4.4.1, on linux when multiprocessing method is spawn or forkserver (not fork). here is my repro:
Same problem here.
Simple way to reproduce it:
Works fine with unittest:
When running in coverage:
I tested both with and without "concurrency = multiprocessing" in .coveragerc
I'm seeing this on Mac, and it looks like ali1234 was using Linux. Can we get the 'Windows' label removed?
This appeared for me when changing from a Python 3.7 venv to a Python 3.8 one.
I can work-around by using
Spawn was made the default start method for MacOS in 3.8 so this also points to that being the cause. (It has always been the default on Windows as it doesn't have
Spawn requires multiprocessing to pickle functions and send them to the subprocesses to run. Pickling a function just sends the canonical name - that name must exist/be importable in the subprocess.
I tried to fix this in https://github.com/nedbat/coveragepy/pull/836/files. This fixed the exception but I think it broke all coverage checking in subprocesses. The ModuleSpec was certainly something to do with it. (I later gave up with this because I found that spawn was not a suitable solution for my application due to re-importing all modules taking too long; I changed to run a new interpreter manually using the subprocess module.)