-
Notifications
You must be signed in to change notification settings - Fork 87
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bazel is crashing while uploading BES events to BuildBuddy #992
Comments
Hey @amit-mittal - thanks for reporting! This type of error message most commonly occurs when your BuildBuddy instance restarts for whatever reason. Are you able to see if your BuildBuddy app has any restarts? Could it be running out of memory? |
Hey @siggisim - thanks for looking into it! No, I don't see any crashes or restarts on the BuildBuddy server side. The node on which BuildBuddy is running has 60 GB memory, while only 16 GB is being used. So far, we are able to repro this issue only while running tests of one of our Go services (if that is relevant). |
Interesting, is there any chance this is a CI run that runs multiple Bazel invocations back-to-back? By default, Bazel will try to re-use the build event stream connection across invocations - which could be leading to the issues we're seeing here. You can disable this behavior with the bazel flag |
I added
I don't think it should matter, but the run is happening on the developer machine (MacOS). Regarding the multiple invocations, we are NOT running bazel commands in parallel, but as part of the usual developer workflow, we do run bazel commands one after another. That is one of the reasons, that we have |
There are (unfortunately) lots of Bazel bugs with the
There is some work being done to improve the BES artifact uploader:
Do you see the same error without that Do you see the same error if you add the flags |
That's true! 😞 I don't think we would be able to change the We will also prioritize upgrading bazel to |
We also see this crash only on our Linux CI (macOS CI is fine). We're on Bazel 4.2.0 and we don't even set |
This is likely fixed by bazelbuild/bazel#13959 which hasn't made it into any Bazel releases yet. Are either of you able to share your grpc log for one of these invocations captured with Bazel's |
Going to close this issue now that bazelbuild/bazel@e855a26 seems to have made it into Bazel 5.0 release candidates bazelbuild/bazel#14013 Please re-open this issue if you're able to reproduce with this with Bazel 5.0 (in which case either a grpc log or a BuildBuddy Cloud invocation would be super helpful). |
We have hosted BuildBuddy OnPrem version on Kubernetes, but while running some of the bazel targets, we are seeing that bazel is crashing even after the build/tests succeeded.
Bazel:
v4.1.0
BuildBuddy:
v2.5.3
(andv2.3.3
)Stack trace:
And in the next bazel command, we see the below message which means the bazel crashed.
Related error I am seeing in the BuildBuddy server logs.
Relevant settings in our
.bazelrc
Please let me know if you need any other details to debug the issue.
The text was updated successfully, but these errors were encountered: