Override the number of parallel compilation processes #1857
Comments
|
After some more digging, I figured out that this happens in the extension building step due to Python's diff --git a/setup.py b/setup.py
--- a/setup.py 2021-02-19 12:31:44.000000000 +0000
+++ b/setup.py 2021-03-26 18:40:33.171643349 +0000
@@ -352,7 +352,7 @@
self.missing = []
self.disabled_configure = []
if '-j' in os.environ.get('MAKEFLAGS', ''):
- self.parallel = True
+ self.parallel = 4
def add(self, ext):
self.extensions.append(ext)Upstream issue: https://bugs.python.org/issue43634 |
|
Since this is a limitation of |
|
I'm aware of that, which is why I filed an issue on bpo. But pyenv already fixes a number of bugs in CPython, and I figured this may be worth such a patch as well. If I'm the only person who'd ever need this though, I guess it doesn't warrant the effort. And seeing that nobody reacted to this in the past almost 6 weeks, I suppose that may well be the case. |
|
@JustAnotherArchivist You can add a patch for the Python version(s) you're experiencing problems with. See the patches in Python-build on how it's done. |
|
That is in fact what I did to get it to work on that system. It's a rather unusual case I guess, and my patch was very hacky and is definitely not suitable for inclusion in pyenv. A proper patch would have to parse |
native-api
added
third-party: python
|
Just for the record, I'm having a problem in the same family regarding parallelism. I am running pyenv to install python on a very large machine (perhaps 500 CPUs). At a point in the installation, the python installation is calling "compileall.py" with the "-j0" option. That is then spawning 500+ compileall.py processes, and the machine is halting the installation with "Too many open files", presumably because 500 processes are all trying to do file manipulation. So yes, it would be nice to be able to turn off the "-j0" flag. Here is the relevant part of the installation: Thank you, |
|
As specified earlier, we'll accept a patch from any interested parties. There are only a few ways to pass |
|
FWIW, the issue got fixed upstream in CPython 3.12 since extensions get built via a Makefile now. Backporting that sounds painful though. |
I'm trying to figure out how to set the number of parallel processes used for compilation in a
pyenv install 3.9.2. The background is that I have a small VM on a bigger host machine; the VM sees all host CPU cores but only has a limited amount of RAM, and compiling with the default parallelism causes the loadavg to shoot into the dozens and the machine to virtually freeze within seconds. As a result, I cannot actually run a build to completion, and the partial build log contains no information whatsoever on this parallelisation. (Let me know if it's needed anyway.)I tried to set
MAKE_OPTSand/orPYTHON_MAKE_OPTSto-j 4, but this appears to be ignored for the actual CPython build; specifically, the./python -E ./setup.py buildprocess (as it shows up in htop) still spawns as manygccprocesses as there are CPUs regardless of those environment variables.Digging in CPython a bit, it looks like this parallel compilation was only introduced relatively recently with 3.8.0 (bpo-36786). The patch there hardcodes
-j0in CPython'sMakefile.pre.in, and I don't see any obvious way to override that through an environment variable or similar. If this is correct, the proper fix would likely require an upstream change, but even if that lands immediately, it won't help with existing 3.8.x and 3.9.x versions, so a workaround in pyenv may still be necessary. However, I attempted to patch eitherMakefile.pre.inorLib/compileall.pywithout success, so my understanding here may well be incorrect.The text was updated successfully, but these errors were encountered: