-
Notifications
You must be signed in to change notification settings - Fork 4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Explain java.lang.OutOfMemoryError during bazel bootstrapping #2177
Comments
IMO we should just use a lower number of threads if we cannot use that much thread. |
We can't automatically detect this situation. However, for non-network file systems and no remote execution, we don't need nearly this many threads, so maybe we should lower the default for Bazel (but keeping it higher internally). |
@ulfjack I agree that we should lower the default value of --jobs. Maybe to the number of CPU threads available on the machine, WDYT? |
I think that's a reasonable default, but we need to make sure that our internal default is higher (using the invocation policy?). Also, note that we have a separate flag for loading phase threads. |
On a host with limited resources; the number of threads can sometimes overwhelm the JVM. I suffered symptoms similar to those reported here.
bazel/src/main/java/com/google/devtools/build/lib/skyframe/SkyframeExecutor.java Line 207 in 8cc5dcf
and seemingly also the BuildView phase
The
|
Would I be better off taking this to bazel-dev rather than trying to revive this old issue? |
We fixed |
I've started to run into issues when running multiple bazel builds (for different workspaces) against remote execution at the same time. I've had to lower |
I was trying to bootstrap bazel 0.4.1 (and 0.4.0) and hit multiple errors, like this:
,
and
bazel ran out of memory and crashed.
After all, I was able to successfully build it, and now I wish to share some knowledge about what I've tried and what actually worked:
I'm not the only one, who ran into this issue (see e.g. Blaze ran out of memory and crashed. #885 and compiling from source gives java.lang.OutOfMemoryError: unable to create new native thread #1341). Some users even gave up trying to make this work. Reading these threads definitely gave me some clues about what's going on, but they are still not giving all the necessary information. The official user manual is not useful at all here.
Although all these error messages may look like "there's not enough free RAM", the actual issue is not related to the amount of free RAM at all. At least in my case - I've got ~190GB of free system RAM, and no user-specific limit (see ulimit info below)
... neither it is related to java heap size. At least in my case, creating the following "MaxMemory.java" file:
and running it gives me a rather large value.
export _JAVA_OPTIONS="-XX:ParallelGCThreads=2"
are ignored.--jobs=10
, at least, the "bazelrc" file is not picked up.ulimit -u
(threads and processes are just the same term here). My limit on this system is 1024 (see ulimit info below).watch -n 0.5 "ps -f -T -u myusername |grep -v grep|wc -l"
.ulimit -u
minus your current number of processes is ~750 - just try ten more times, there's a nice chance that you would make it :)taskset -c 0 ./compile.sh
, to send all processes to a single CPU. Assuming that the affinity of all child processes would be the same, as the affinity of the parent process, one may assume, that if there's some wise dynamic parallelism inside - the number of processes would be significantly decreased. However, this seem to give no effect.ulimit -u 100000
. It's a pity that I've noticed his post only after I solved the issue myself. Surprisingly, user might be able to significantly increase number of processes allowed without root privileges.ulimit:
So, as a feature request:
The text was updated successfully, but these errors were encountered: