Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: collect core dumps from builders #49165

Open
prattmic opened this issue Oct 26, 2021 · 1 comment
Open

x/build: collect core dumps from builders #49165

prattmic opened this issue Oct 26, 2021 · 1 comment

Comments

@prattmic
Copy link
Member

@prattmic prattmic commented Oct 26, 2021

Sometimes tests dump core, perhaps with no extra crash information. Investigating these issues would be much improved if we could get the core file from the builder.

This came up recently in https://golang.org/cl/358675, where https://storage.googleapis.com/go-build-log/2fb3aaf5/linux-amd64_26aedb56.log failed with little more than:

signal: segmentation fault (core dumped)
FAIL	runtime	5.189s
FAIL

#35248 is another case where post-submit failures could likely have been fixed without a reproducer if the core dump was available.

I believe there is an issue regarding collecting multiple files from builders generally which is likely a prerequisite for this, though I can't find it...

cc @mknyszek @golang/release

@gopherbot gopherbot added this to the Unreleased milestone Oct 26, 2021
@mknyszek
Copy link
Contributor

@mknyszek mknyszek commented Oct 26, 2021

Looks like this needs:

  1. The ability to collect multiple files from a builder,
  2. setting a unique core pattern in run.bash,
  3. enabling core dumps in run.bash (currently disabled),
  4. running builders with GOTRACEBACK=crash instead of GOTRACEBACK=system.

Then the coordinator could, say, blindly search the work tree and /tmp for anything resembling the core dump pattern and then upload it to some GCS bucket with an expiry (so it doesn't grow infinitely large).

For most use-cases, it's probably enough to stop there. A stretch goal would be surfacing it in the UI somehow.

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants