x/build: collect core dumps from builders

[prattmic]

Sometimes tests dump core, perhaps with no extra crash information. Investigating these issues would be much improved if we could get the core file from the builder.

This came up recently in https://golang.org/cl/358675, where https://storage.googleapis.com/go-build-log/2fb3aaf5/linux-amd64_26aedb56.log failed with little more than:

signal: segmentation fault (core dumped)
FAIL	runtime	5.189s
FAIL

#35248 is another case where post-submit failures could likely have been fixed without a reproducer if the core dump was available.

I believe there is an issue regarding collecting multiple files from builders generally which is likely a prerequisite for this, though I can't find it...

cc @mknyszek @golang/release

[mknyszek]

Looks like this needs:

1. The ability to collect multiple files from a builder,
2. setting a unique core pattern in run.bash,
3. enabling core dumps in run.bash (currently disabled),
4. running builders with GOTRACEBACK=crash instead of GOTRACEBACK=system.

Then the coordinator could, say, blindly search the work tree and /tmp for anything resembling the core dump pattern and then upload it to some GCS bucket with an expiry (so it doesn't grow infinitely large).

For most use-cases, it's probably enough to stop there. A stretch goal would be surfacing it in the UI somehow.

[thanm]

Another similar use case would be copying back profiles (coverage profiles, for example, or CPU profiles for PGO purposes).