Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: tests failing to terminate on ios-arm64-corellium #57962

Closed
bcmills opened this issue Jan 23, 2023 · 12 comments
Closed

runtime: tests failing to terminate on ios-arm64-corellium #57962

bcmills opened this issue Jan 23, 2023 · 12 comments
Assignees
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge mobile Android, iOS, and x/mobile NeedsFix The path to resolution is known, but the work has not been done. OS-Darwin
Milestone

Comments

@bcmills
Copy link
Contributor

bcmills commented Jan 23, 2023

https://build.golang.org/log/5fd56793a86a5e6392ab13c645ab0f9b24fa3268:

Installed Go for ios/arm64 in /tmp/workdir-host-ios-arm64-corellium-ios/go
Installed commands in /tmp/workdir-host-ios-arm64-corellium-ios/go/bin
Test "go_test:archive/tar" ran over 20m0s limit (20m0.000566363s); saw output:

XXXBANNERXXX:Test execution environment.
# GOARCH: arm64
# CPU: 
# GOOS: ios
# OS Version: Darwin 20.6.0 Darwin Kernel Version 20.6.0: Tue Aug 24 21:10:43 PDT 2021; root:xnu-7195.140.44~1/RELEASE_ARM64_T8010 iPhone9,1

XXXBANNERXXX:Testing packages.

The hanging test command is for archive/tar, but I don't think the specific package is relevant — that just happens to be the very first test in the run.

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jan 23, 2023
@bcmills bcmills added OS-Darwin NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. mobile Android, iOS, and x/mobile labels Jan 23, 2023
@bcmills
Copy link
Contributor Author

bcmills commented Jan 23, 2023

This may have started on or around CL 454836, but since this builder is slow and heavily backlogged we don't have a clean bisection in the dashboard runs.

It appears that something in either the toolchain or the test process itself is failing to terminate properly. However, due to #53201 I can't even tell which process is hung.

@golang/release, can we prioritize a fix for #53201 to unblock investigation of this issue?

(CC @golang/runtime @golang/ios)

@bcmills bcmills added this to the Go1.21 milestone Jan 23, 2023
@heschi
Copy link
Contributor

heschi commented Jan 23, 2023

Since ios-arm64 is not a first class port, I'd like the iOS maintainers to spend at least some time investigating before we discuss prioritizing #53201.

@changkun
Copy link
Member

Could you maybe clarify what needs to be investigated? It is not entirely clear after a large number of CLs are landed.

@heschi
Copy link
Contributor

heschi commented Jan 23, 2023

Essentially, see why the tests are hanging. archive/tar is the first test to run, so I suspect the port is completely broken.

@cherrymui
Copy link
Member

It is not always the first test timed out (although it seems the case in a few most recent runs). https://build.golang.org/log/fe7dc38047bccedcf67640847ff602864fabc40b includes a few successful tests before hanging.

@cherrymui
Copy link
Member

For ones who have access to the builder, would it be possible to see which process(es) hang, and would it be possible to send it a SIGQUIT and get a stack dump? Thanks.

@changkun
Copy link
Member

Based on what I saw from the Corellium dashboard, the Go team has admin access to these builders (although I don't know how it is access from the Go team side)

@cherrymui, could you help me understand exactly what you expect me to do in which steps? Do you mean to kill the builtlet process and share the core dump file?

@cherrymui
Copy link
Member

I'm not familiar with the builder infrastructure, so I don't know exactly. If you can log in to the builder machine/VM that is running the build, maybe run command like ps or top to see which processes are running, and send a SIGQUIT to a long-running one?

But before we spend more time on this, I'd like to hear some answer about the meta-question: if this is really due to cgo not enabled, do we really want to support non-cgo programs on iOS? If the answer is no, cmd/dist shouldn't disable cgo. If the answer is yes, maybe we want to support internal linking, so it is actually pure Go.

@bcmills
Copy link
Contributor Author

bcmills commented Jan 23, 2023

would it be possible to see which process(es) hang, and would it be possible to send it a SIGQUIT and get a stack dump?

That is exactly why #53201 is relevant. The buildlet should be sending SIGQUIT itself in case of timeouts (on all platforms that support it, not just ios), but today does not seem to be doing so.

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/463739 mentions this issue: cmd/dist: leave cgo enabled if external linking is required

@dr2chase
Copy link
Contributor

dr2chase commented Feb 1, 2023

Is this fixed?

@bcmills bcmills self-assigned this Feb 1, 2023
@bcmills
Copy link
Contributor Author

bcmills commented Feb 1, 2023

This is indeed fixed by CL 463739, which re-enables cgo when building cmd on this platform.

I suspect that it was related to some iffy behavior in runtime/cgo on the ios/arm64 platform (filed as #58225), although there may be other issues involved as well.

@bcmills bcmills closed this as completed Feb 1, 2023
@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Feb 2, 2023
@golang golang locked and limited conversation to collaborators Feb 2, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge mobile Android, iOS, and x/mobile NeedsFix The path to resolution is known, but the work has not been done. OS-Darwin
Projects
None yet
Development

No branches or pull requests

7 participants