Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/cmd/release: green-screen on Windows ARM64 builder with SkipTests: false #47017

Closed
toothrot opened this issue Jul 1, 2021 · 3 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@toothrot
Copy link
Contributor

toothrot commented Jul 1, 2021

It's unclear what is causing this issue on the Windows ARM64 builders, only during the release process. The builders are otherwise passing tests on the Go dashboard.

Discovered while testing #42604.

What version of Go are you using (go version)?

Go tip: 4711bf3

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

windows/arm64

What did you do?

@@ -206,10 +206,6 @@ var builds = []*Build{
                Arch:    "arm64",
                Race:    false, // Not supported as of 2021-06-01.
                Builder: "windows-arm64-10",
-               // TODO(golang.org/issue/46406, golang.org/issue/46502): Fix
-               // or skip failing tests, ensure the builder is fast enough to
-               // complete tests, then remove SkipTests here.
-               SkipTests: true,
        },
go run ./cmd/release -rev master -target windows-arm64 -version go1.17beta900

What did you expect to see?

A successful build with passing tests

What did you see instead?

2021/06/30 12:39:40 windows-arm64: Building (all.bash to ensure tests pass).
 /status
2021/06/30 12:45:17 user-rakoczy-windows-arm64-10-1: timeout after 10s waiting for headers for /status
2021/06/30 12:45:37 user-rakoczy-windows-arm64-10-1: timeout after 10s waiting for headers for /status
2021/06/30 12:45:37 Buildlet https://farmer.golang.org:443 failed three heartbeats; final error: timeout waiting for headers
2021/06/30 12:45:37 user-rakoczy-windows-arm64-10-1: peer dead with Buildlet https://farmer.golang.org:443 failed heartbeat after 10.001424152s; marking dead; err=timeout waiting for headers, waiting for headers for /halt
2021/06/30 12:45:37 windows-arm64: Error: Buildlet https://farmer.golang.org:443 failed heartbeat after 10.001424152s; marking dead; err=timeout waiting for headers

Stop code: MEMORY MANAGEMENT
Stop code: IRQL NOT LESS OR EQUAL

@toothrot toothrot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jul 1, 2021
@toothrot toothrot added this to the Unreleased milestone Jul 1, 2021
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Jul 1, 2021
@dmitshur dmitshur added this to In Progress in Go Release Team Jul 20, 2021
@dmitshur
Copy link
Contributor

I tried to gather more information on this issue by making use of a physical windows/arm64 machine I have access to. I ran all.bat a number of times on it, and can confirm this issue doesn't reproduce on it, as all.bat passes okay (even with GOROOT_FINAL and other environment variables set as cmd/release does) without causing any green-screens. I tested with a recent Go tip commit, b7a85e0.

@dmitshur
Copy link
Contributor

dmitshur commented Aug 25, 2021

I've repeated this exercise using the windows-arm64-10 builder, and the green-screens no longer happen reproducibly. I've still observed at least 1 instance out of 5 runs, filed #47965 for that.


windows-arm64, 1.17 + CL 344674*
--------------------------------

$ go run . -target=windows-arm64 -version=go1.17beta700 -rev=12e8b254cbd6f8c2014473cc6f44fe9a2a07d90a -watch

run 1 = "ALL TESTS PASSED"
run 2 = "ALL TESTS PASSED"
run 3 = "ALL TESTS PASSED"

* This CL removes a test that is known to be flaky, which the windows/arm64
  port seemed to run into somewhat often last time I looked into this issue.

windows-arm64, 1.17
-------------------

$ go run . -target=windows-arm64 -version=go1.17beta710 -rev=ec5170397c724a8ae440b2bc529f857c86f0e6b1 -watch

run 1 = "ALL TESTS PASSED"
run 2 = runtime.TestGcLastTime failed: "gc_test.go:126: bad last GC time: got 1629865473098425100, want [1629865473098425100, 1629865473001007600]" (issue #44820)
run 3 = "windows-arm64: Error: Buildlet https://farmer.golang.org:443 failed heartbeat after 10.005360314s; marking dead; err=timeout waiting for headers"
run 4 = "ALL TESTS PASSED"
run 5 = "ALL TESTS PASSED"

windows-arm64, 1.18 (tested at commit de1c934b9709728b15cc821a055155ee13e1d0ab)
---------------------

$ go run . -target=windows-arm64 -version=go1.18beta720 -rev=de1c934b9709728b15cc821a055155ee13e1d0ab -watch

run 1 = "ALL TESTS PASSED"
run 2 = "ALL TESTS PASSED"
run 3 = "ALL TESTS PASSED"

As long as that failure doesn't happen too often, the 3 retries should allow to compensate for it, so we can remove the SkipTests now, which can help catch real failures. Sent CL 345129.

@gopherbot
Copy link

Change https://golang.org/cl/345129 mentions this issue: cmd/release: start running release tests for windows-arm64

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Aug 25, 2021
Go Release Team automation moved this from In Progress to Done Aug 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Projects
Archived in project
Development

No branches or pull requests

3 participants