Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/cgo/internal/test: consistently failing with segmentation fault on linux-arm-aws since CL 545277 #64447

Open
bcmills opened this issue Nov 29, 2023 · 15 comments
Assignees
Labels
arch-arm Issues solely affecting the 32-bit arm architecture. compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Linux
Milestone

Comments

@bcmills
Copy link
Member

bcmills commented Nov 29, 2023

Go version

b2efd1d

What operating system and processor architecture are you using (go env)?

linux-arm-aws at b2efd1de97402ec4b8fb4e9e0ec29c8e49e8e200


##### Test execution environment.
# GOARCH: arm
# CPU: ARMv8 Processor rev 1 (v8l)
# GOOS: linux
# OS Version: Linux 4.19.0-12-arm64 #1 SMP Debian 4.19.152-1 (2020-10-18) armv8l

What did you do?

https://build.golang.org/log/4023a6ee5a18667fe161b4b917a1a0c4579527d8

What did you expect to see?

Passing tests.

What did you see instead?

##### Testing cgo
ok  	cmd/cgo/internal/test	0.916s
ok  	cmd/cgo/internal/test	0.880s
ok  	cmd/cgo/internal/testtls	0.005s
ok  	cmd/cgo/internal/testtls	0.003s
ok  	cmd/cgo/internal/testnocgo	0.004s
ok  	cmd/cgo/internal/testnocgo	0.003s
signal: segmentation fault
FAIL	cmd/cgo/internal/test	0.521s
@bcmills bcmills added compiler/runtime Issues related to the Go compiler and/or runtime. arch-arm Issues solely affecting the 32-bit arm architecture. OS-Linux release-blocker labels Nov 29, 2023
@dmitshur dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 29, 2023
@dmitshur
Copy link
Contributor

dmitshur commented Nov 29, 2023

CC @mknyszek, @golang/compiler.

@dmitshur dmitshur added this to the Go1.22 milestone Nov 29, 2023
@mknyszek
Copy link
Contributor

This doesn't make any sense to me... That CL should be totally unrelated to this test. I'll investigate.

@mknyszek
Copy link
Contributor

mknyszek commented Nov 29, 2023

I can't reproduce by running the tests cmd/cgo/internal/test tests directly via go tool dist test on a gomote. I tried all variants, including the ones that fall under the ##### Testing cgo heading. Trying run.bash.

@mknyszek
Copy link
Contributor

Yep, I can't reproduce at all on a linux-arm-aws gomote. I do not know how to proceed here. :(

@cherrymui
Copy link
Member

This reminds me #61514, which I also don't understand and it magically stopped...

@mknyszek
Copy link
Contributor

Yeah, that seems like exactly the same issue to me.

@bcmills
Copy link
Member Author

bcmills commented Nov 29, 2023

Possibly something relating to code or data alignment changing in the compiled binary?
(I'm thinking of something along the general lines of #60825.)

@cherrymui
Copy link
Member

It is possible that this triggers some bug that is very sensitive to how the binary is laid out. For #61514, I couldn't reproduce on gomote or anything other than pre-/post-submit builds, and the test doesn't print anything besides segmentation fault, which makes debugging very hard, though.

@bcmills
Copy link
Member Author

bcmills commented Nov 29, 2023

@mknyszek, looking back at successful runs (https://build.golang.org/log/4cad61d2d6f9b3b16aa53609780318e36b984a8c) to work around #51050, I see some environment variables that might be relevant:
"GOARM=6" "GOHOSTARCH=arm" "CGO_CFLAGS=-march=armv6" "CGO_LDFLAGS=-march=armv6" (set from here).

Maybe the difficultly reproducing the problem with go tool dist test comes down to an environment setting?

@mknyszek mknyszek self-assigned this Nov 29, 2023
@mknyszek
Copy link
Contributor

mknyszek commented Nov 30, 2023

Explicitly setting the environment variables didn't help. IIRC, the buildlet does set them correctly for the gomote anyway.

Here was what I ran:

gomote run -e GOARM=6 -e GOHOSTARCH=arm -e CGO_CFLAGS=-march=armv6 -e CGO_LDFLAGS=-march=armv6 go/bin/go tool dist test cmd/cgo/internal/test:static

I also tried the other cmd/cgo/internal/test test variants and run.bash. I then made sure make.bash was run with these flags set and then tried the tests again. No dice. :(

@mknyszek
Copy link
Contributor

Aaaaaand it magically stopped happening again...

@bcmills
Copy link
Member Author

bcmills commented Nov 30, 2023

Ah! The failing invocation was the third invocation of cmd/cgo/internal/test, which I think makes it this one (it's also the first one after testnocgo):
https://cs.opensource.google/go/go/+/master:src/cmd/dist/test.go;l=1228;drc=d7fcb5cf80953f1d63246f1ae9defa60c5ce2d76

That would be -buildmode=static -ldflags='-linkmode=external -extldflags "-static -pthread"', I think?

@bcmills
Copy link
Member Author

bcmills commented Nov 30, 2023

Or go tool dist test cmd/cgo/internal/test:static.

@bcmills
Copy link
Member Author

bcmills commented Nov 30, 2023

Ah, which I see is what you already tried. 😞

@cherrymui
Copy link
Member

As it magically passes now, we still want to investigate it but it is not currently a release blocker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arch-arm Issues solely affecting the 32-bit arm architecture. compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Linux
Projects
Status: In Progress
Status: No status
Development

No branches or pull requests

5 participants