runtime: Go 1.22.0 fails to build from source on armv7 Alpine Linux

[nmeum]

Go version

N/A

Output of go env in your module/workspace:

N/A

What did you do?

I maintain the Go package for Alpine Linux, this package builds Go from source using ./make.bash.

While upgrading from Go 1.21.6 to Go 1.22.0 we noticed that Go itself wouldn't build on our armv7 CI.

What did you see happen?

Go 1.22.0 builds successful on Alpine x86_64, x86, armhf (GOARM=6), ppc64le, aarch64 and riscv64. It only fails to build on armv7 (GOARM=7). Furthermore, Go 1.21.6 builds fine on our armv7 CI. Therefore, I believe this to be a regression in Go 1.22.0. The build failure message is not consistent, there is probably a race condition somewhere. However, it always fails either after Building Go toolchain2 using go_bootstrap and Go toolchain1 or Building Go toolchain3 using go_bootstrap and Go toolchain2.

Build logs of some of the failures we have seen so far (as I said, the exact error message differs):

• SIGSEGV: segmentation violation: alpine-edge-armv7-go-failure-sigsev.txt
• fatal error: checkdead: runnable g: alpine-edge-armv7-go-failure-checkdead.txt
• SIGILL: illegal instruction: alpine-edge-armv7-go-failure-sigill.txt
• Sometimes it just deadlocks after Building Go toolchain2 using go_bootstrap and Go toolchain1.

Any idea what might be causing this? Any hints on debugging this further?

What did you expect to see?

A successful build of Go 1.22.0 on all architectures.

[dmitshur]

CC @golang/arm.

[seankhliao]

Are these bare metal or cross arch virtualized machines?

[cherrymui]

Thank for the report. Would it be okay for you to do a bisection? Thanks.

[nmeum]

Are these bare metal or cross arch virtualized machines?

These are armv7 Docker containers running on an aarch64 host.

Thank for the report. Would it be okay for you to do a bisection? Thanks.

I am currently somewhat short-pressed on time, but I will try to look into it.

[nmeum]

The commit which introduced the regression is 66b8107 (CC: @randall77).

Reverting this commit fixes the build.

[cherrymui]

It might be an interesting interaction between CL 525637 and CL 514907. Specifically, this code https://cs.opensource.google/go/go/+/master:src/cmd/go/internal/work/gc.go;l=369-381 is defining assembler macros GOARM_x based on cfg.GOARM and expecting a value like "5", "6", or "7", but with CL 514907, it can also be "7,hardfloat" (which is probably the case if you set GOARM at make.bash), which caused the wrong macro to be defined, which in turn makes it possible to drop some memory barrier instructions. There is also run-time CPU feature detection based on the HWCAP, but for a Docker container running on ARM64, that could go off.

Could you try if CL https://go.dev/cl/562995 helps? Thanks.

[gopherbot]

Change https://go.dev/cl/562995 mentions this issue: cmd/go: handle GOARM value with soft/hardfloat in defining assembly macros

[nmeum]

Could you try if CL https://go.dev/cl/562995 helps? Thanks.

Still deadlocks with this patch.

[cherrymui]

Thanks. I updated the CL. Could you try the new version of the CL? It seems we also need to define GOARM_x in cmd/dist.

[nmeum]

Thanks for the patch. Unfortunately, it still deadlocks for me with the updated version of the CL.

[gopherbot]

Change https://go.dev/cl/564855 mentions this issue: runtime/internal/atomic: correct GOARM=7 guard at a DMB instruction

[cherrymui]

Thanks. Could you try https://go.dev/cl/564855 ? That should fix the build.
(CL 562995 is probably good to have but is not required for a successful build.)

[cherrymui]

@gopherbot please backport this to Go 1.22 release. This is a regression, causing runtime crashes with GOARM=7.

[gopherbot]

Backport issue(s) opened: #65760 (for 1.22).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

[gopherbot]

Change https://go.dev/cl/567555 mentions this issue: [release-branch.go1.22] runtime/internal/atomic: correct GOARM=7 guard at a DMB instruction

[clausecker]

@cherrymui It seems that your backport was incomplete. Commit e4ebd13 of CL562995 has not been backported, but seems to be required to fix the build. As is, Go 1.22.3 doesn't build on armv7 FreeBSD. Could you backport the missing commit, too?

[cherrymui]

@clausecker thanks for reporting. I thought that CL 562995 is nice to have to but not a requirement: if the macro is not defined correctly it just builds a binary that is not optimized for GOARM=7. It is possible I missed something. What is the error you saw? Thanks.

[clausecker]

@cherrymui I get crashes and hangs that look very similar to what @nmeum saw. A bisection led me to 6ecadb4 for the first commit where this happens (may be incorrect) and to e4ebd13 for where this behaviour ceases.

See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=278953 for where I track the build failure in our port.