x/build: dragonfly-amd64 builders may need a longer TEST_TIMEOUT_SCALE #45216

bcmills · 2021-03-24T15:32:04Z

I've noticed frequent cmd/go test timeouts on the dragonfly-amd64 builders recently (#45215). Some other tests tend to time out on these builders too (#34034).

I notice that the dragonfly-amd64 builders are reverse builders, so I wonder if the builders are just on the slow side. Perhaps they need a GO_TEST_TIMEOUT_SCALE to allow the more intensive tests to complete?

(See also #44584.)

CC @tuxillo @golang/release

The text was updated successfully, but these errors were encountered:

tuxillo · 2021-04-11T08:42:55Z

@bcmills is that some configuration option? Let me know if you think I should change something.

dmitshur · 2021-04-11T16:55:42Z

@tuxillo Yes, it allows providing a custom multiplier for the default test timeout. See:

go/src/cmd/dist/test.go

Lines 149 to 161 in 189c694

    
           t.timeoutScale = 1 
        
           switch goarch { 
        
           case "arm": 
        
           	t.timeoutScale = 2 
        
           case "mips", "mipsle", "mips64", "mips64le": 
        
           	t.timeoutScale = 4 
        
           } 
        
           if s := os.Getenv("GO_TEST_TIMEOUT_SCALE"); s != "" { 
        
           	t.timeoutScale, err = strconv.Atoi(s) 
        
           	if err != nil { 
        
           		fatalf("failed to parse $GO_TEST_TIMEOUT_SCALE = %q as integer: %v", s, err) 
        
           	} 
        
           }

It's useful for cases where a builder has limited CPU/IO/network resources that result in default test timeouts being too aggressive, since it gives tests more time to finish. See golang/build@3689a13 and golang/build@3a0fcf7 for past examples where that configuration was adjusted.

tuxillo · 2021-04-14T07:51:58Z

Ok, I'll submit a patch to build with a timescale of 2, unless you want to set another value?

dmitshur · 2021-04-28T23:05:24Z

@tuxillo Sure, let's try going with 2x and see how it goes. We can always follow up if needed.

Please feel free to send a CL and make me a reviewer. You can look at CL 300870 as a recent example.

tuxillo · 2021-04-29T23:01:41Z

golang/build#35

This sets the GO_TEST_TIMEOUT_SCALE=2 for dragonfly builders in order to try to solve intermittent test failures. For golang/go#45215. For golang/go#34034. Fixes golang/go#45216.

gopherbot · 2021-04-30T13:35:31Z

Change https://golang.org/cl/315329 mentions this issue: dashboard: increase GO_TEST_TIMEOUT_SCALE for dragonfly builders

bcmills · 2021-11-02T15:07:14Z

Unfortunately the timeouts are still quite frequent:

greplogs --dashboard -md -l -e '(?ms)\Adragonfly-amd64.*panic: test timed out' --since=2021-06-01

gopherbot · 2021-11-02T15:24:03Z

Change https://golang.org/cl/360735 mentions this issue: internal/testenv: treat dragonfly-amd64 as a slow machine

Updates golang/go#45216 Change-Id: I9e714582c4d8c3c7ce7571359d7f9bbe2c6c9497 Reviewed-on: https://go-review.googlesource.com/c/tools/+/360735 Trust: Bryan C. Mills <bcmills@google.com> Run-TryBot: Bryan C. Mills <bcmills@google.com> Reviewed-by: Robert Findley <rfindley@google.com> gopls-CI: kokoro <noreply+kokoro@google.com> TryBot-Result: Go Bot <gobot@golang.org>

tuxillo · 2021-11-02T23:40:07Z

We've moved the builder to a host with a faster disk (on October 18th), it should be a bit better, isn't it?

bcmills · 2021-11-03T20:20:34Z

From the test logs above, it looks like things improved somewhat on Oct. 18 but then got worse again around Nov. 1.

Curiously, all of the failures since Nov. 1 include at least one deadline error in Test_Issue38211 (#44098). I wonder if that's somehow related... 🤔

tuxillo · 2021-11-03T22:39:17Z

I have noticed that the time in the VM we use gets screwed somehow (see a similar case in https://bugs.dragonflybsd.org/issues/3299). We're not sure yet why this happens. I've enabled an ntp client now and it's making a lot of adjustments, which looks suspicious.

/var/log/messages:Nov  3 18:44:28 gobuilder dntpd[524]: issuing offset adjustment: 0.002025s
/var/log/messages:Nov  3 19:05:27 gobuilder dntpd[524]: issuing offset adjustment: -0.002612s
/var/log/messages:Nov  3 19:25:19 gobuilder dntpd[524]: issuing offset adjustment: 0.002456s
/var/log/messages:Nov  3 19:46:18 gobuilder dntpd[524]: issuing offset adjustment: -0.001033s
/var/log/messages:Nov  3 20:07:19 gobuilder dntpd[524]: issuing offset adjustment: -0.002583s
/var/log/messages:Nov  3 20:28:17 gobuilder dntpd[524]: issuing offset adjustment: 0.002426s
/var/log/messages:Nov  3 22:31:44 gobuilder dntpd[524]: issuing offset adjustment: -0.002397s

In any case, is it performing better right now? It definitely should.

bcmills added OS-Dragonfly Builders NeedsInvestigation labels Mar 24, 2021

gopherbot added this to the Unreleased milestone Mar 24, 2021

cagedmantis mentioned this issue Apr 12, 2021

x/build: dragonfly-amd64 builder missing #45312

Closed

dmitshur assigned tuxillo and dmitshur Apr 28, 2021

dmitshur added this to In Progress in Go Release Team Apr 28, 2021

dmitshur mentioned this issue Apr 28, 2021

cmd/go: frequent test timeouts on dragonfly-amd64 builder #45215

Open

tuxillo mentioned this issue Apr 30, 2021

dashboard: increase GO_TEST_TIMEOUT_SCALE for dragonfly builders golang/build#35

Closed

gopherbot closed this as completed in golang/build@249a248 Apr 30, 2021

Go Release Team automation moved this from In Progress to Done Apr 30, 2021

rsc unassigned tuxillo and dmitshur Jun 23, 2022

x/build: dragonfly-amd64 builders may need a longer TEST_TIMEOUT_SCALE #45216

x/build: dragonfly-amd64 builders may need a longer TEST_TIMEOUT_SCALE #45216

bcmills commented Mar 24, 2021 •

edited by dmitshur

tuxillo commented Apr 11, 2021

dmitshur commented Apr 11, 2021

tuxillo commented Apr 14, 2021

dmitshur commented Apr 28, 2021

tuxillo commented Apr 29, 2021

gopherbot commented Apr 30, 2021

bcmills commented Nov 2, 2021

gopherbot commented Nov 2, 2021

tuxillo commented Nov 2, 2021

bcmills commented Nov 3, 2021 •

edited

tuxillo commented Nov 3, 2021

x/build: dragonfly-amd64 builders may need a longer TEST_TIMEOUT_SCALE #45216

x/build: dragonfly-amd64 builders may need a longer TEST_TIMEOUT_SCALE #45216

Comments

bcmills commented Mar 24, 2021 • edited by dmitshur

tuxillo commented Apr 11, 2021

dmitshur commented Apr 11, 2021

tuxillo commented Apr 14, 2021

dmitshur commented Apr 28, 2021

tuxillo commented Apr 29, 2021

gopherbot commented Apr 30, 2021

bcmills commented Nov 2, 2021

gopherbot commented Nov 2, 2021

tuxillo commented Nov 2, 2021

bcmills commented Nov 3, 2021 • edited

tuxillo commented Nov 3, 2021

bcmills commented Mar 24, 2021 •

edited by dmitshur

bcmills commented Nov 3, 2021 •

edited