cmd/compile: unexplained compiler panic build flakes

[DanielMorsing]

Over the last couple of days, I've seen some mysterious failures show up on the builders

#78074 (build, log) Unrelated issue
#74470 (build, log)
#71170 (build, log)

and probably some more that I'm missing. They only seem to happen on amd64 boxes and they're spread over a variety of operating systems and backing builders.

Either we have a memory corruption bug (which I think is unlikely, some of the failures are dereferencing statically initialized memory), or there is some infrastructure issue with the swarm.

[DanielMorsing]

Should have mentioned, I tried reproducing some of these with gomote, but having no luck so far.

[gabyhelp]

Related Issues

• cmd/compile: racecompile builder failures on multiple packages #28170 (closed)
• cmd/compile: sporadic memory corruption on 386 (32-bit) builders #37881 (closed)
• x/build: memory corruption crashes on darwin builders #77515
• cmd/compile: failures with `signal: killed` on ppc64le builders #64096 (closed)
• cmd/link/internal/ld: unrecognized failures #78033
• build: linux/mipsle failure -- memory corruption? #19396 (closed)
• cmd/compile/internal/ir: unrecognized failures #76899
• runtime: "signal: segmentation fault (core dumped)" on several builders #35248 (closed)
• runtime,cmd/compile: apparent memory corruption in compress/flate #54596 (closed)
• x/tools/cmd/godex: unrecognized failures #77517 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[mknyszek]

A shot in the dark: we don't pin a specific version of COS (https://docs.cloud.google.com/container-optimized-os/docs) but rather pick up whatever stable is. This has caused similar problems in the past. CC @dmitshur

[mknyszek]

Ah, just kidding. That was sanitizer-specific. (#59418)

[dmitshur]

Issue #78074 is due to CL 751260 adding an unconditional write to ml.data[2], and the ml.data slice being given a size of len(load.cmd.data) which can have fewer than 3 elements on darwin/amd64. It's probably not related to others.

[DanielMorsing]

While I don't have the full build logs, I suspect this is happening on the perf builders as well (build 1, build 2)

[prattmic]

cc @golang/compiler

[DanielMorsing]

Unsure if related, but here's a compiler panic flake on an arm64 box