New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Numba installation scripts exhibit a race condition during parallel builds #7063
Comments
@leycec thank you for bringing this to our attention. Fortunately Numba doesn't take long to compile (compared to e.g. LLVM) which is probably why nobody has ever tried to compile this in parallel and which is why nobody has ever hit this issue. In fact, I am even unable to reproduce this, but I am on Python 3.8 and OSX, so that may have something to do with it? Thus, it will probably make sense to take a closer look and figure out exactly which platforms are affected. For the time being, I don't think this should be a high priority to fix since it isn't an impediment or blocker at the moment. I will also update the issue title to more closely reflect the submitted issue. |
No worries whatsoever. Thanks for the fast acknowledgement, @esc. Gentooers like myself tend to be particularly hard-hit by these sorts of build-time race conditions, as we tend to do much more (re)compilation on an ongoing basis than under competing platforms. Gentoo: we're on the front lines and all I got was this penguin T-shirt. In theory, increasing the thread count with Fortunately, we've already resolved this on our end by forcing Since no one is vociferously shouting into your gitter, I reluctantly agree this is really low-priority. As Numba's suite of C extensions continues to metastasise into a labyrinthine juggernaut of interdependent code, this issue will probably begin to materialize a bit more frequently. Thus spake @leycec. Thanks again, @esc. The whole data science community's excited to see where Numba goes next – and so am I. Keep up the fantastic work. 👍 |
@leycec thank you for the feedback and your kind words about the project, we really appreciate it! I am glad you have managed to find a workaround for the Gentoo system for the time being. Additionally, I have now attached label |
I can't confirm whether prior versions of Numba suffer similar issues, ...they almost certainly do but Numba 0.53.1 non-deterministically blows up at compilation time when enabling parallel make jobs – that is, when passing anything other than
-j 1
to thesetup.py build
phase.When passing
-j {n}
forn > 1
, installation frequently fails with a wall of text resembling:The tell-tale error line is actually embedded halfway up that log output, which is utterly ridiculous but what can you do:
That means that parallel compilation has failed, which (of course) parallel compilation almost always does when building anything scientific under Python 3. Pretty much all of Python's standard scientific stack suffer similar issues – including:
-j
numpy/numpy#15957.Ideally, upstream
setuptools
would resolve this for everybody and thus us. Pragmatically,setuptools
developers have no interest or intention of resolving this within the expected lifetime of our Universe. So, a bit of a wait time there.The recommended resolution adopted by SciPy is to disable parallel building entirely by forcing
-j 1
insetup.py
. Yes, that's demonstrably awful – but there are no working alternatives on the table. Forcing-j 1
at C extension build time is exactly what everyone does. That's the MostlyRightThing™ to do, because even ifsetuptools
were to eventually resolve their decades-old unresolved race conditions (which we should all have profound doubts about) that would do little to assist everyone stuck on oldersetuptools
versions.Fortunately, this is trivial. Just override the
finalize_options()
method of your customNumbaBuildExt
class insetup.py
to resemble:Boom. That's it. Then we can all get back to solving actual computational problems that matter, like racing each other to
@numba.njit
random Rosetta Stone tasks.The text was updated successfully, but these errors were encountered: