Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: tests failing to terminate on ios-arm64-corellium #57962

Open
bcmills opened this issue Jan 23, 2023 · 9 comments
Open

runtime: tests failing to terminate on ios-arm64-corellium #57962

bcmills opened this issue Jan 23, 2023 · 9 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. mobile Android, iOS, and x/mobile NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin
Milestone

Comments

@bcmills
Copy link
Member

bcmills commented Jan 23, 2023

https://build.golang.org/log/5fd56793a86a5e6392ab13c645ab0f9b24fa3268:

Installed Go for ios/arm64 in /tmp/workdir-host-ios-arm64-corellium-ios/go
Installed commands in /tmp/workdir-host-ios-arm64-corellium-ios/go/bin
Test "go_test:archive/tar" ran over 20m0s limit (20m0.000566363s); saw output:

XXXBANNERXXX:Test execution environment.
# GOARCH: arm64
# CPU: 
# GOOS: ios
# OS Version: Darwin 20.6.0 Darwin Kernel Version 20.6.0: Tue Aug 24 21:10:43 PDT 2021; root:xnu-7195.140.44~1/RELEASE_ARM64_T8010 iPhone9,1

XXXBANNERXXX:Testing packages.

The hanging test command is for archive/tar, but I don't think the specific package is relevant — that just happens to be the very first test in the run.

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jan 23, 2023
@bcmills bcmills added OS-Darwin NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. mobile Android, iOS, and x/mobile labels Jan 23, 2023
@bcmills
Copy link
Member Author

bcmills commented Jan 23, 2023

This may have started on or around CL 454836, but since this builder is slow and heavily backlogged we don't have a clean bisection in the dashboard runs.

It appears that something in either the toolchain or the test process itself is failing to terminate properly. However, due to #53201 I can't even tell which process is hung.

@golang/release, can we prioritize a fix for #53201 to unblock investigation of this issue?

(CC @golang/runtime @golang/ios)

@bcmills bcmills added this to the Go1.21 milestone Jan 23, 2023
@heschi
Copy link
Contributor

heschi commented Jan 23, 2023

Since ios-arm64 is not a first class port, I'd like the iOS maintainers to spend at least some time investigating before we discuss prioritizing #53201.

@changkun
Copy link
Member

Could you maybe clarify what needs to be investigated? It is not entirely clear after a large number of CLs are landed.

@heschi
Copy link
Contributor

heschi commented Jan 23, 2023

Essentially, see why the tests are hanging. archive/tar is the first test to run, so I suspect the port is completely broken.

@cherrymui
Copy link
Member

It is not always the first test timed out (although it seems the case in a few most recent runs). https://build.golang.org/log/fe7dc38047bccedcf67640847ff602864fabc40b includes a few successful tests before hanging.

@cherrymui
Copy link
Member

For ones who have access to the builder, would it be possible to see which process(es) hang, and would it be possible to send it a SIGQUIT and get a stack dump? Thanks.

@changkun
Copy link
Member

Based on what I saw from the Corellium dashboard, the Go team has admin access to these builders (although I don't know how it is access from the Go team side)

@cherrymui, could you help me understand exactly what you expect me to do in which steps? Do you mean to kill the builtlet process and share the core dump file?

@cherrymui
Copy link
Member

I'm not familiar with the builder infrastructure, so I don't know exactly. If you can log in to the builder machine/VM that is running the build, maybe run command like ps or top to see which processes are running, and send a SIGQUIT to a long-running one?

But before we spend more time on this, I'd like to hear some answer about the meta-question: if this is really due to cgo not enabled, do we really want to support non-cgo programs on iOS? If the answer is no, cmd/dist shouldn't disable cgo. If the answer is yes, maybe we want to support internal linking, so it is actually pure Go.

@bcmills
Copy link
Member Author

bcmills commented Jan 23, 2023

would it be possible to see which process(es) hang, and would it be possible to send it a SIGQUIT and get a stack dump?

That is exactly why #53201 is relevant. The buildlet should be sending SIGQUIT itself in case of timeouts (on all platforms that support it, not just ios), but today does not seem to be doing so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. mobile Android, iOS, and x/mobile NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin
Projects
Status: No status
Development

No branches or pull requests

5 participants