Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/cmd/coordinator: misc/cgo/testplugin.TestIssue25756pie fails on darwin-arm64 builder deterministically but only in sharded mode #46239

Open
cherrymui opened this issue May 18, 2021 · 6 comments

Comments

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented May 18, 2021

CL https://go-review.googlesource.com/c/go/+/319489 added a test to misc/cgo/testplugin. The test is failing in the darwin-arm64 builder.

Interestingly, I cannot reproduce the failure locally on a darwin-arm64 machine, by running cd GOROOT/misc/cgo/testplugin; go test. I also cannot reproduce the failure on gomote, by running gomote run $VM go/bin/go tool dist test testplugin or gomote run $VM go/src/all.bash.

But it does fail consistently on the build dashboard and with trybot ( https://storage.googleapis.com/go-build-log/d4df70bf/darwin-arm64-11_0-toothrot_5af7c061.log )

Maybe there is some unusual setting on the builder?

cc @toothrot @cagedmantis @dmitshur

@dmitshur
Copy link
Contributor

@dmitshur dmitshur commented May 18, 2021

This test has also passed for me locally on an M1 machine with macOS 11.3.1, so it seems this being a builder issue is likely. I'll try to look at why it's failing in post-submit but not via gomote.

@dmitshur
Copy link
Contributor

@dmitshur dmitshur commented May 19, 2021

The test also passes on the builder as invoked during release tests (i.e., with something like release -target=darwin-arm64 -version=go1.17beta123 -watch -rev=ff7d5f97b3375a87e2de90f42e96983e5f0f95a4). This supports a theory that the problem could be related to coordinator using SplitMakeRun mode (where it compiles, then snapshots, then runs tests) during TryBots and post-submit builds.

@bcmills
Copy link
Member

@bcmills bcmills commented May 19, 2021

This supports a theory that the problem could be related to coordinator using SplitMakeRun mode

That suggests a possible connection to #33598.

@dmitshur
Copy link
Contributor

@dmitshur dmitshur commented May 19, 2021

I've investigated this more today. The test is passing on a physical darwin/arm64 machine, via sequential (non-sharded) test execution, when executed on its own, but failing specifically when executed by cmd/coordinator in sharded test mode on this builder. It's really nice that it fails so reproducibly and gives us a chance to investigate it, as it's likely the root cause for this problem may shared across more builder issues, though the feedback loop is quite slow.

Getting to the bottom of it will take longer, so I'll send a CL to skip the test for now, so that its failure doesn't mask other darwin/arm64 regressions in the mean time.

@gopherbot
Copy link

@gopherbot gopherbot commented May 19, 2021

Change https://golang.org/cl/321349 mentions this issue: misc/cgo/testplugin: skip TestIssue25756pie on darwin/arm64 builder

gopherbot pushed a commit that referenced this issue May 20, 2021
This test is known to be broken on the darwin/arm64 builder.
Skip it while it's being investigated so it doesn't mask other failures.

For #46239.
Updates #43228.

Change-Id: I8fe57a0636bba84c3100337146dcb96cc264e524
Reviewed-on: https://go-review.googlesource.com/c/go/+/321349
Trust: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
@toothrot toothrot added this to Planned in Go Release Team Jun 8, 2021
@dmitshur
Copy link
Contributor

@dmitshur dmitshur commented Jun 10, 2021

Based on investigation above, this is a bug in the build system rather than a test failure. The problematic builder is skipped via CL 321349, so this is no longer blocking Go 1.17 release.

I think this is a good opportunity for us to investigate further, likely after the 1.17 release, so retitling and moving to next milestone.

@dmitshur dmitshur changed the title misc/cgo/testplugin: TestIssue25756pie fails on darwin-arm64 builder x/build/cmd/coordinator: misc/cgo/testplugin.TestIssue25756pie fails on darwin-arm64 builder deterministically but only in sharded mode Jun 10, 2021
@dmitshur dmitshur modified the milestones: Go1.17, Go1.18 Jun 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants