-
-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Handle cythonize(..., nthreads=1) for "spawn" start method #3262
Comments
6 tasks
j-luo93
added a commit
to djwyen/sound-law-benchmark
that referenced
this issue
Nov 6, 2020
This was referenced Dec 2, 2020
samster25
added a commit
to Eventual-Inc/Daft
that referenced
this issue
Sep 22, 2022
* fixes issue when multithreaded cython build is set. the following error appears: ```RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase. This probably means that you are not using fork to start your child processes and you have forgotten to use the proper idiom ``` * Related to cython/cython#3262
this is still an issue with python 3.10.4 and 3.11.1 and 3.11.4 (tested as had virtual envs ready) with Mac on 13.5.1 (22G90) |
10 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Configuration
multiprocessing.get_start_method() == 'spawn'
:multiprocessing
's new default behavior (ref: bpo-33725: multiprocessing uses spawn by default on macOS python/cpython#13603).Observed behavior
When
Cython.Build.cythonize(..., nthreads=1)
is calledsetup.py
scriptif __name__ == '__main__'
idiommultiprocessing
's start method is set to"spawn"
,then
cythonize
/multiprocessing.Pool
call,result.get(99999)
.Small equivalent reproducer
Possible solutions/workarounds:
multiprocessing.get_start_method() == 'spawn'
and display a warning.nthreads and multiprocessing.get_start_method() == 'spawn'
.multiprocessing.set_start_method('fork', force=True)
.Caveats:
re. 1.: If a script/module properly guards execution with
if __name__ == '__main__':
, there wouldn't be a need to disable multiprocessing. Many (/most?)setup.py
don't, though.re. 2.: Cython already only warns and continues with serial processing if
nthreads
is non-zero butmultiprocessing
isn't available. So raising a error seems inconsequential.re. 3.: If a script, e.g.,
setup.py
, already does some persistent changes (e.g., adding release information to files), those would be executed again, which is most likely undesired behavior.re. 4.: Won't work on Windows. And it might be unexpected/undesired by the user to change this.
I favor approach 1 as it is the "least breaking" change. "Least breaking" in the sense that scipts with
nthreads
that work with macOS and Python up 3.7 would continue to work, albeit serially instead of parallely processed.The text was updated successfully, but these errors were encountered: