-
Notifications
You must be signed in to change notification settings - Fork 796
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Numba workqueue threading error #665
Comments
i think i might have a related problem, so I'm commenting here to stay in the loop. I'm trying to serve a small dash app using Flask/wsgi/apache2, and while it runs fine locally, my app gets stuck with fit_transform()... and more specifically, fit() There aren't any error messages... it just times out. |
This is not a trivial issue to solve unfortunately -- the problems are ultimately caused by other libraries (here flask and numba) both wanting to do their own threading and not playing well. My best guess for a solution would be to look into getting numba to use a different threading layer. The suggested TBB threading is definitely going to be better. To get numba to use that it requires that you have |
Following up on this thread, using the omp threading layer (similar to tbb) worked for me. Glad to know that's also the suggested approach, so I'll close this out. Thanks for the response! |
Hopefully one of omp or tbb works for @celinesin . If it doesn't work for you please feel free get the issue re-opened Celine. |
interim update: omp didn't work, but as promised getting numba to use tbb is indeed not so straight forward ^^; Thanks for the hints so far! |
The most common cause of this issue is nesting parallel kernels, here's an example: from numba import njit, prange
import numpy as np
@njit(parallel=True)
def nested(x):
for i in prange(len(x)):
x[i] += 1
@njit(parallel=True)
def main():
Z = np.zeros((5, 10))
for i in prange(Z.shape[0]):
nested(Z[i]) # A parallel region is calling another function with a parallel region!
return Z
main() The My first question is.... do you have something like the case above or do you have a parallel kernel being called from two python threads? I think the effect would manifest in the same error message as the |
(copypasta from the numba thread) After much help from @stuartarchibald, we've figured out a few things:
The search continues! My next hint is: https://modwsgi.readthedocs.io/en/develop/user-guides/processes-and-threading.html# Thanks for all your help! |
@celinesin : thanks for reporting back -- it sounds like a difficult issue and may not be related to threading, but instead some other aspect of things. You are welcome to open a separate issue for this if you want. |
And, in the end my problem was incorrect wsgi configuration... what a treasure hunt! |
This causes issues on Mac OS X. Also the fix results in a 2x speedup on Windows in some cases. Also know as fix for "Terminating: Nested parallel kernel launch detected, the workqueue threading layer does not supported nested parallelism. Try the TBB threading layer." Ref insightful comment on GitHub: lmcinnes/umap#665 (comment)
This causes issues on Mac OS X. Also the fix results in a 2x speedup on Windows in some cases. Also know as fix for "Terminating: Nested parallel kernel launch detected, the workqueue threading layer does not supported nested parallelism. Try the TBB threading layer." Ref insightful comment on GitHub: lmcinnes/umap#665 (comment)
Thanks for the great library!
I'm getting the following error:
Terminating: Nested parallel kernel launch detected, the workqueue threading layer does not supported nested parallelism. Try the TBB threading layer.
The context is that I'm calling the umap library from flask and two clients are hitting a umap call at the same time. I'm guessing this is what is causing nested threading -- flask is already using some kind of threading, and umap seems to be doing the same under the hood. I think this is related to numba threading, but I'm setting a random seed so as per the docs I don't think this should be an issue.
Is there a proposed/preferred solution to this issue?
The text was updated successfully, but these errors were encountered: