x/build: add a darwin/amd64 longtest builder #35678

bcmills · 2019-11-18T20:21:08Z

Now that we have a Windows longtest builder (#26529), I'm wondering where else we might have a coverage gap.

It would be nice to have a darwin builder running non-short tests too. I doubt that it will turn up many regressions for cmd/go in particular, but the system calls are enough different — and the installed base of cmd/go users on macOS is large enough — that I suspect it would have a net positive value.

CC @golang/release

The text was updated successfully, but these errors were encountered:

bcmills · 2019-11-18T20:22:51Z

(This message brought to you by cmd/go.tooSlow, which probably ought to just Not Be a Thing.)

go/src/cmd/go/go_test.go

Lines 49 to 53 in a23f9af

    
           // In -short mode; skip test, except run it on the {darwin,linux,windows}/amd64 builders. 
        
           if testenv.Builder() != "" && runtime.GOARCH == "amd64" && (runtime.GOOS == "linux" || runtime.GOOS == "darwin" || runtime.GOOS == "windows") { 
        
           	return 
        
           } 
        
           t.Skip("skipping test in -short mode")

bcmills · 2020-01-06T22:06:54Z

CC @matloob @jayconrod

dmitshur · 2021-11-16T23:14:06Z

I've folded the identical #26530 into this issue. I've also retitled this one to be about darwin/amd64, since #49055 now exists for darwin/arm64.

bcmills · 2023-03-07T20:20:25Z

This was moved to the Planned project state back in November — do we have an ETA on when it will happen, or is it still in the backlog?

I'd still like to get rid of the builder check in the tooSlow test helper because it fails to detect builders that really are slower (like the darwin-amd64-race builder), such as in #58918 and #58919, and I don't want to add even more complexity to the test to try to hard-code more information about specific builders.

More clarity about when this is likely to happen would help me evaluate the tradeoffs of that change.

gopherbot · 2023-03-07T21:50:18Z

Change https://go.dev/cl/474137 mentions this issue: cmd/go: don't run slow tests on non-longtest builders

Also annotate calls to tooSlow with specific reasons. This will somewhat reduce test coverage on the 'darwin' builders until we have darwin 'longtest' builders (#35678,#49055), but still seems worthwhile to avoid alert fatigue from tests that really shouldn't be running in the short configurations. Fixes #58918. Fixes #58919. Change-Id: I0000f0084b262beeec3eca3e9b8a45d61fab4313 Reviewed-on: https://go-review.googlesource.com/c/go/+/474137 Reviewed-by: Ian Lance Taylor <iant@google.com> Auto-Submit: Bryan Mills <bcmills@google.com> Run-TryBot: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>

gopherbot · 2023-03-08T20:54:32Z

Change https://go.dev/cl/474580 mentions this issue: [release-branch.go1.20] cmd/go: avoid running slow tests on non-longtest builders

gopherbot · 2023-03-08T20:57:04Z

Change https://go.dev/cl/474581 mentions this issue: [release-branch.go1.19] cmd/go: avoid running slow tests on non-longtest builders

…est builders Also annotate calls to tooSlow with specific reasons. This will somewhat reduce test coverage on the 'darwin' builders until we have darwin 'longtest' builders (#35678,#49055), but still seems worthwhile to avoid alert fatigue from tests that really shouldn't be running in the short configurations. Updates #58918. Updates #58919. Fixes #58937. Change-Id: I0000f0084b262beeec3eca3e9b8a45d61fab4313 Reviewed-on: https://go-review.googlesource.com/c/go/+/474137 Reviewed-by: Ian Lance Taylor <iant@google.com> Auto-Submit: Bryan Mills <bcmills@google.com> Run-TryBot: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> (cherry picked from commit 9f532dd) Reviewed-on: https://go-review.googlesource.com/c/go/+/474581

…est builders Also annotate calls to tooSlow with specific reasons. This will somewhat reduce test coverage on the 'darwin' builders until we have darwin 'longtest' builders (#35678,#49055), but still seems worthwhile to avoid alert fatigue from tests that really shouldn't be running in the short configurations. Updates #58918. Updates #58919. Fixes #58938. Change-Id: I0000f0084b262beeec3eca3e9b8a45d61fab4313 Reviewed-on: https://go-review.googlesource.com/c/go/+/474137 Reviewed-by: Ian Lance Taylor <iant@google.com> Auto-Submit: Bryan Mills <bcmills@google.com> Run-TryBot: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> (cherry picked from commit 9f532dd) Reviewed-on: https://go-review.googlesource.com/c/go/+/474580

heschi · 2023-03-13T18:53:45Z

It's still in the backlog, no particular ETA. Adding the builder obviously isn't too hard, but we'd have to add more machines and that means re-evaluating our budget.

bcmills · 2023-03-13T19:05:37Z

FWIW, from my perspective the builder would be useful even if it doesn't have enough machines to run at every CL.

(Many of the reverse builders are in that state, and they're still useful for detecting deterministic failures, narrowing down bisection windows, and performing SlowBot runs on particularly risky changes.)

gopherbot · 2023-03-28T17:45:32Z

Change https://go.dev/cl/479837 mentions this issue: dashboard: add darwin-amd64-longtest builder

Post-submit builds the same set as other "secondary" longtest builders, like linux-386-longtest. Trybot builds are not enabled for release branches due to lack of capacity. For golang/go#35678. Change-Id: I60f15bfe8fe74072f35b20462cd9d13a111a1b31 Reviewed-on: https://go-review.googlesource.com/c/build/+/479837 Reviewed-by: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Michael Pratt <mpratt@google.com> Reviewed-by: Heschi Kreinick <heschi@google.com> Run-TryBot: Michael Pratt <mpratt@google.com>

prattmic · 2023-04-04T21:31:32Z

This is deployed. Tests are failing because they are attempting to access the GKE proxy: https://build.golang.org/log/3f22deaeecb1b6d29789d0c3dc904a7edf34bf1c.

I don't immediately see why this is happening, or even where this value is coming from. There is no GOPROXY in the logged environment.

bcmills · 2023-04-04T21:36:31Z

https://cs.opensource.google/go/x/build/+/master:cmd/coordinator/buildstatus.go;l=1047;drc=f2d94d7ddd3020510adb472cc4377d4d0a1558e8

Overridden here for the non-GCE builders:
https://cs.opensource.google/go/x/build/+/master:dashboard/builders.go;l=908-912;drc=b88b71fe103eaec6c7372cf0a34f2bd7227435f6

bcmills · 2023-04-04T21:36:47Z

(I don't know why it isn't in the logged environment, though. 🤷‍♂️)

prattmic · 2023-04-04T21:40:37Z

Thanks, I found that but misread some conditions.

(I don't know why it isn't in the logged environment, though. 🤷‍♂️)

I think this is because we are only logging the environment when running make.bash. For tests we add GOPROXY here, only after running make.bash.

gopherbot · 2023-04-04T22:01:24Z

Change https://go.dev/cl/482335 mentions this issue: dashborad: remove needsGoProxy

This field was added in CL 166218 but never used. For golang/go#35678. Change-Id: I197fd3835136e7649b20a7ca47d4179591cb49a5 Reviewed-on: https://go-review.googlesource.com/c/build/+/482335 Auto-Submit: Michael Pratt <mpratt@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Michael Pratt <mpratt@google.com>

gopherbot · 2023-04-05T18:31:51Z

Change https://go.dev/cl/482339 mentions this issue: dashboard,cmd/coordinator: unify and simplify GOPROXY setting behavior

gopherbot · 2023-04-06T18:41:38Z

Change https://go.dev/cl/482915 mentions this issue: dashboard: add known issue to darwin-amd64-longtest

The existing behavior for setting GOPROXY is rather hard to follow, and doesn't work correctly in many cases. For example, longtest on a reverse builder gets the GKE proxy. Before CL 479837 there were no longtest builders outside of GCE, so this case was never covered. Fixing this is the motivation of this CL. They way configuration works today is: 1. buildstatus.go unconditionally sets GOPROXY to the GKE proxy [1]. 2. st.conf.ModuleEnv potentially overrides GOPROXY with a more reasonable setting, with a bunch of complex conditions. Unify and simplify this process by moving it into buildstatus.go, where their is now a strict ordering of possible GOPROXY values. Notable changes: * The GKE proxy is never used outside of GCE. * There is a consistent default/fallback of proxy.golang.org. I initially tried to split this into two CLs: one unifying the implementation and the next changing the behavior, but the old behavior is so mind-boggling that the first CL doesn't really make much sense. The annoying part of this CL is that tests move from dashboard to cmd/coordinator, requiring us to export additional fields so cmd/coordinator tests can configure the builders. The test cases themselves are unchanged except for the addition of a non-GCE longtest builder case. [1] Except in runSubrepoTests, which avoids doing so for reverse builders. This was a workaround for private proxy builders in CL 275412, but wasn't extended to other callers because only subrepo tests were seeing a regression. More strangeness. For golang/go#35678. Change-Id: I6090c8c5e91ce6be9bfc07c16f36ed339c9d27ae Reviewed-on: https://go-review.googlesource.com/c/build/+/482339 Reviewed-by: Bryan Mills <bcmills@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org> Run-TryBot: Michael Pratt <mpratt@google.com>

For golang/go#35678. Change-Id: I1d408c6547e689702311d73b5bf56adf4d1cd150 Reviewed-on: https://go-review.googlesource.com/c/build/+/482915 Run-TryBot: Michael Pratt <mpratt@google.com> Reviewed-by: Carlos Amedee <carlos@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>

prattmic · 2023-04-06T19:43:56Z

The latest failures are in crypto/x509:

https://build.golang.org/log/8815b334c89791bc3c30410757d4c1020e927e96

--- FAIL: TestPlatformVerifier (2.85s)
    --- FAIL: TestPlatformVerifier/revoked_leaf (0.19s)
        root_darwin_test.go:116: unexpected verification error: got "x509: certificate has expired or is not yet valid: “revoked.badssl.com” certificate is expired", want "x509: “revoked.badssl.com” certificate is revoked"
    --- FAIL: TestPlatformVerifier/leaf_missing_SCTs (0.74s)
        root_darwin_test.go:118: unexpected verification success: want "x509: “no-sct.badssl.com” certificate is not standards compliant"
FAIL
FAIL	crypto/x509	4.436s

@rolandshoemaker any idea what is going on here? Do the builders need some certificate verification system dependency installed/enabled?

bcmills · 2023-04-06T19:45:39Z

FWIW, that's #57428.

prattmic · 2023-04-06T19:46:48Z

Ah, nice! So that test needs a skip for now it seems.

gopherbot · 2023-04-10T15:30:18Z

Change https://go.dev/cl/482165 mentions this issue: crypto/x509: skip broken darwin root tests

For #57428. For #35678. Change-Id: I806c16d3ff3815b8681916753338356c444970d2 Reviewed-on: https://go-review.googlesource.com/c/go/+/482165 Reviewed-by: Bryan Mills <bcmills@google.com> Auto-Submit: Michael Pratt <mpratt@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Michael Pratt <mpratt@google.com>

gopherbot · 2023-04-12T21:50:19Z

Change https://go.dev/cl/484216 mentions this issue: cmd/go: skip TestScript/list_goroot_symlink on darwin

The list_goroot_symlink test relies on fsys.Walk (and ultimately syscall.Lstat) conforming to POSIX pathname resolution semantics. POSIX requires that symlinks ending in a slash be fully resolved, but it appears that lstat in current darwin kernels does not fully resolve the last pathname component when it is a symlink to a symlink. For #59586. For #35678. Change-Id: I37526f012ba94fa1796b33109a41c3226c967d3e Reviewed-on: https://go-review.googlesource.com/c/go/+/484216 Run-TryBot: Bryan Mills <bcmills@google.com> Reviewed-by: Michael Pratt <mpratt@google.com> Auto-Submit: Bryan Mills <bcmills@google.com> TryBot-Bypass: Bryan Mills <bcmills@google.com>

gopherbot · 2023-04-13T03:27:14Z

Change https://go.dev/cl/484295 mentions this issue: cmd/internal/moddeps: preserve PWD more carefully in commands

On macOS, TMPDIR is typically a symlink, and the GOROOT for the buildlet is in TMPDIR as well. PWD must be preserved in order for os.Getwd (and functions based on it) to report paths that remain relative to GOROOT, and paths relative to GOROOT are necessary in order for filepath.Rel to report subdirectories as subdirectories (rather than paths with long "../../…" prefixes). Fortunately, the (*Cmd).Environ method added for #50599 makes preserving PWD somewhat easier. This fixes 'go test cmd/internal/moddeps' on the new darwin-amd64-longtest builder. For #35678. Change-Id: Ibaa458bc9a94b44ba455519bb8da445af07fe0d2 Reviewed-on: https://go-review.googlesource.com/c/go/+/484295 Auto-Submit: Bryan Mills <bcmills@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Michael Pratt <mpratt@google.com>

gopherbot · 2023-04-13T17:58:35Z

Change https://go.dev/cl/484476 mentions this issue: dashboard: remove darwin-amd64-longtest known issue

cagedmantis · 2023-05-02T20:23:46Z

Just mentioning that this post-submit builder is failing in the 1.19 and 1.20 release branches.

gopherbot · 2023-05-03T15:11:25Z

Change https://go.dev/cl/492055 mentions this issue: dashboard: make darwin longtest builder like others as of Go 1.21

The darwin longtest builder was added during the Go 1.21 dev cycle and needed some development work before it would pass all tests. Disable it on older release branches that don't have said work, and make it run on Go 1.21 and newer release branches as a TryBot like all other longtest builders. This makes it consistent with other -longtest builders and will help avoid entering broken state when backporting to future release branches. Also remove redundant version check from windows longtest tryBot policy functions since the version check is already there in buildsRepo policy. For golang/go#35678. For golang/go#37827. Change-Id: I09c479f1be2b3635c385b504b0c86b50e65e696b Reviewed-on: https://go-review.googlesource.com/c/build/+/492055 TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Michael Pratt <mpratt@google.com> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>

gopherbot · 2023-06-15T14:56:41Z

Change https://go.dev/cl/503758 mentions this issue: internal/releasetargets: set LongTestBuilder for darwin-amd64

A longtest builder for the darwin/amd64 port was added in the Go 1.21 development cycle. Start using it as for non-advisory release testing. For golang/go#40561. For golang/go#35678. For golang/go#29252. Change-Id: Ic65e84e5e10bcb786cb28c36c1d1b137e2f6202e Reviewed-on: https://go-review.googlesource.com/c/build/+/503758 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Heschi Kreinick <heschi@google.com> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>

bcmills added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. new-builder labels Nov 18, 2019

gopherbot added this to the Unreleased milestone Nov 18, 2019

gopherbot added the Builders x/build issues (builders, bots, dashboards) label Nov 18, 2019

ALTree mentioned this issue Oct 19, 2021

x/build: add longtest builder for darwin/arm64 #49055

Open

dmitshur mentioned this issue Nov 16, 2021

x/build: add MacOS builder for long tests #26530

Closed

dmitshur changed the title ~~x/build: add a darwin longtest builder~~ x/build: add a darwin/amd64 longtest builder Nov 16, 2021

bcmills mentioned this issue Nov 17, 2022

x/build: add longtest builders for all first class ports #56822

Closed

dmitshur mentioned this issue Dec 21, 2022

crypto/x509: TestPlatformVerifier/revoked_leaf fails on macOS 13.1 #57428

Open

bcmills added the OS-Darwin label Mar 8, 2023

prattmic self-assigned this Mar 28, 2023

dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Mar 28, 2023

prattmic mentioned this issue Apr 10, 2023

net/http: "Test appears to have leaked a Transport" on darwin-amd64-longtest #59526

Closed

gopherbot closed this as completed in golang/build@17fb1f8 Apr 13, 2023

bcmills mentioned this issue Apr 14, 2023

x/tools: multiple tests timing out on darwin-amd64-longtest #59634

Open

dmitshur mentioned this issue Jun 19, 2023

x/build/cmd/coordinator: include -longtest and -race builders for trybots on release branches #37827

Open

5 tasks

x/build: add a darwin/amd64 longtest builder #35678

x/build: add a darwin/amd64 longtest builder #35678

bcmills commented Nov 18, 2019 •

edited by dmitshur

bcmills commented Nov 18, 2019

bcmills commented Jan 6, 2020

dmitshur commented Nov 16, 2021

bcmills commented Mar 7, 2023

gopherbot commented Mar 7, 2023

gopherbot commented Mar 8, 2023

gopherbot commented Mar 8, 2023

heschi commented Mar 13, 2023

bcmills commented Mar 13, 2023

gopherbot commented Mar 28, 2023

prattmic commented Apr 4, 2023

bcmills commented Apr 4, 2023

bcmills commented Apr 4, 2023

prattmic commented Apr 4, 2023

gopherbot commented Apr 4, 2023

gopherbot commented Apr 5, 2023

gopherbot commented Apr 6, 2023

prattmic commented Apr 6, 2023

bcmills commented Apr 6, 2023

prattmic commented Apr 6, 2023

gopherbot commented Apr 10, 2023

gopherbot commented Apr 12, 2023

gopherbot commented Apr 13, 2023

gopherbot commented Apr 13, 2023

cagedmantis commented May 2, 2023

gopherbot commented May 3, 2023

gopherbot commented Jun 15, 2023

x/build: add a darwin/amd64 longtest builder #35678

x/build: add a darwin/amd64 longtest builder #35678

Comments

bcmills commented Nov 18, 2019 • edited by dmitshur

bcmills commented Nov 18, 2019

bcmills commented Jan 6, 2020

dmitshur commented Nov 16, 2021

bcmills commented Mar 7, 2023

gopherbot commented Mar 7, 2023

gopherbot commented Mar 8, 2023

gopherbot commented Mar 8, 2023

heschi commented Mar 13, 2023

bcmills commented Mar 13, 2023

gopherbot commented Mar 28, 2023

prattmic commented Apr 4, 2023

bcmills commented Apr 4, 2023

bcmills commented Apr 4, 2023

prattmic commented Apr 4, 2023

gopherbot commented Apr 4, 2023

gopherbot commented Apr 5, 2023

gopherbot commented Apr 6, 2023

prattmic commented Apr 6, 2023

bcmills commented Apr 6, 2023

prattmic commented Apr 6, 2023

gopherbot commented Apr 10, 2023

gopherbot commented Apr 12, 2023

gopherbot commented Apr 13, 2023

gopherbot commented Apr 13, 2023

cagedmantis commented May 2, 2023

gopherbot commented May 3, 2023

gopherbot commented Jun 15, 2023

bcmills commented Nov 18, 2019 •

edited by dmitshur