cmd/cgo/internal/test: seg fault on linux-arm-aws builder

[cherrymui]

The dashboard shows cmd/cgo/internal/test seg faults on the linux-arm-aws builder since https://golang.org/cl/508135 . Specifically, it is cmd/cgo/internal/test:static failing. The failure is quite consistent, every commit on the same test. Also the same failure on trybot https://go-review.googlesource.com/c/go/+/320910/3#message-34862db7fe6eb712e5699ce7ab0a7f53aed5a2e5

However, I tried to run all.bash or go tool dist test cmd/cgo/internal/test:static on the linux-arm-aws gomote and it doesn't fail. I double checked that the environment looks the same for gomote and the builder.

It is strange as CL 508135 shouldn't change anything semantically (even the order of sorted items). It is double strange that the failure is not reproducible locally.

Filing an issue to investigate.

[eliben]

Thanks for filing the issue, @cherrymui -- this is very odd, indeed.

Could the test be failing sporadically after all (and we get unlucky), maybe running it locally with a high count can hit a failure?

[ianlancetaylor]

Anything is possible, but the build dashboard shows the test failing consistently since that CL and never before that CL.

[cherrymui]

I tried running go tool dist test cmd/cgo/internal/test:static in a loop for 100 times and it doesn't fail. I'm now trying to do some experiments with trybots in CL 511978

[cherrymui]

Did some experiments on CL 511978. Essentially if I change any function(s) to use old code, like https://go-review.googlesource.com/c/go/+/511978/8/src/sort/sort_impl_go121.go , or https://go-review.googlesource.com/c/go/+/511978/9/src/sort/sort_impl_go121.go , trybot passes. It fails if all functions using new implementation. (They all pass on gomote.)

On the other hand, on the build dashboard it passes again since CL https://go.dev/cl/511837 .

Maybe there is something weird on the builder for that time, or how the test are run on the builder/trybot (bad caching?).

Anyway, closing for now. We can reopen if it occurs again.