Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: dragonfly-amd64 builders may need a longer TEST_TIMEOUT_SCALE #45216

Closed
bcmills opened this issue Mar 24, 2021 · 11 comments
Closed

x/build: dragonfly-amd64 builders may need a longer TEST_TIMEOUT_SCALE #45216

bcmills opened this issue Mar 24, 2021 · 11 comments
Labels
Builders NeedsInvestigation OS-Dragonfly
Milestone

Comments

@bcmills
Copy link
Member

@bcmills bcmills commented Mar 24, 2021

I've noticed frequent cmd/go test timeouts on the dragonfly-amd64 builders recently (#45215). Some other tests tend to time out on these builders too (#34034).

I notice that the dragonfly-amd64 builders are reverse builders, so I wonder if the builders are just on the slow side. Perhaps they need a GO_TEST_TIMEOUT_SCALE to allow the more intensive tests to complete?

(See also #44584.)

CC @tuxillo @golang/release

@bcmills bcmills added OS-Dragonfly Builders NeedsInvestigation labels Mar 24, 2021
@gopherbot gopherbot added this to the Unreleased milestone Mar 24, 2021
@tuxillo
Copy link
Contributor

@tuxillo tuxillo commented Apr 11, 2021

@bcmills is that some configuration option? Let me know if you think I should change something.

@dmitshur
Copy link
Contributor

@dmitshur dmitshur commented Apr 11, 2021

@tuxillo Yes, it allows providing a custom multiplier for the default test timeout. See:

go/src/cmd/dist/test.go

Lines 149 to 161 in 189c694

t.timeoutScale = 1
switch goarch {
case "arm":
t.timeoutScale = 2
case "mips", "mipsle", "mips64", "mips64le":
t.timeoutScale = 4
}
if s := os.Getenv("GO_TEST_TIMEOUT_SCALE"); s != "" {
t.timeoutScale, err = strconv.Atoi(s)
if err != nil {
fatalf("failed to parse $GO_TEST_TIMEOUT_SCALE = %q as integer: %v", s, err)
}
}

It's useful for cases where a builder has limited CPU/IO/network resources that result in default test timeouts being too aggressive, since it gives tests more time to finish. See golang/build@3689a13 and golang/build@3a0fcf7 for past examples where that configuration was adjusted.

@tuxillo
Copy link
Contributor

@tuxillo tuxillo commented Apr 14, 2021

Ok, I'll submit a patch to build with a timescale of 2, unless you want to set another value?

@dmitshur
Copy link
Contributor

@dmitshur dmitshur commented Apr 28, 2021

@tuxillo Sure, let's try going with 2x and see how it goes. We can always follow up if needed.

Please feel free to send a CL and make me a reviewer. You can look at CL 300870 as a recent example.

@tuxillo
Copy link
Contributor

@tuxillo tuxillo commented Apr 29, 2021

golang/build#35

tuxillo added a commit to tuxillo/build that referenced this issue Apr 30, 2021
This sets the GO_TEST_TIMEOUT_SCALE=2 for dragonfly builders in order
to try to solve intermittent test failures.

For golang/go#45215.
For golang/go#34034.
Fixes golang/go#45216.
@gopherbot
Copy link

@gopherbot gopherbot commented Apr 30, 2021

Change https://golang.org/cl/315329 mentions this issue: dashboard: increase GO_TEST_TIMEOUT_SCALE for dragonfly builders

Go Release Team automation moved this from In Progress to Done Apr 30, 2021
@bcmills
Copy link
Member Author

@bcmills bcmills commented Nov 2, 2021

Unfortunately the timeouts are still quite frequent:

greplogs --dashboard -md -l -e '(?ms)\Adragonfly-amd64.*panic: test timed out' --since=2021-06-01

2021-11-02T06:25:39-ee08195-088bb4b/dragonfly-amd64
2021-11-02T06:19:33-ee08195-4633b2d/dragonfly-amd64
2021-11-02T05:43:05-ee08195-a97c527/dragonfly-amd64
2021-11-02T03:55:19-ee08195-6f1e9a9/dragonfly-amd64
2021-11-01T21:27:46-ee08195-f9cb33c/dragonfly-amd64
2021-11-01T15:20:08-b7d65fb-4a84298/dragonfly-amd64
2021-11-01T13:54:03-42daa65-4a84298/dragonfly-amd64
2021-10-17T04:27:13-98f6e03-cf51fb5/dragonfly-amd64
2021-10-15T19:54:44-98f6e03-22951fb/dragonfly-amd64
2021-10-15T18:07:49-98f6e03-8331f25/dragonfly-amd64
2021-10-15T16:00:01-98f6e03-fad4a16/dragonfly-amd64
2021-10-15T14:09:01-98f6e03-c580180/dragonfly-amd64
2021-10-15T07:13:43-288d0b3-1cbec68/dragonfly-amd64
2021-10-15T02:45:22-288d0b3-3da0ff8/dragonfly-amd64
2021-10-15T02:45:22-288d0b3-2ac3bdf/dragonfly-amd64
2021-10-15T02:02:50-ce04ca3-3da0ff8/dragonfly-amd64
2021-10-15T00:16:20-ce04ca3-9e8ed86/dragonfly-amd64
2021-10-14T22:21:32-3a269dc-c580180/dragonfly-amd64
2021-10-14T22:21:32-3a269dc-2ac3bdf/dragonfly-amd64
2021-10-14T22:15:24-e69ba9d-b59467e/dragonfly-amd64
2021-10-14T19:24:35-e69ba9d-4cbc0a5/dragonfly-amd64
2021-10-14T18:44:21-e69ba9d-85a068f/dragonfly-amd64
2021-10-14T17:38:39-e69ba9d-011fd00/dragonfly-amd64
2021-10-14T15:17:03-e69ba9d-fbdf830/dragonfly-amd64
2021-10-14T14:04:14-e69ba9d-2ac3bdf/dragonfly-amd64
2021-10-14T12:38:31-88e80df-2feb2cc/dragonfly-amd64
2021-10-14T07:19:29-acac939-2feb2cc/dragonfly-amd64
2021-10-14T01:51:22-e13a265-276fb27/dragonfly-amd64
2021-10-14T01:51:22-acac939-276fb27/dragonfly-amd64
2021-10-13T20:31:25-378b9e1-b90d258/dragonfly-amd64
2021-10-13T18:15:48-e13a265-40f82f8/dragonfly-amd64
2021-10-13T03:12:54-24389d4-c580180/dragonfly-amd64
2021-10-13T03:12:54-24389d4-2ac3bdf/dragonfly-amd64
2021-10-12T22:26:24-074820e-732f6fa/dragonfly-amd64
2021-10-12T22:26:24-074820e-2ac3bdf/dragonfly-amd64
2021-10-12T22:12:04-94178a2-2ac3bdf/dragonfly-amd64
2021-10-12T21:30:45-0b930fb-3283d1a/dragonfly-amd64
2021-10-12T18:52:28-c83725b-26b6833/dragonfly-amd64
2021-10-12T18:00:16-c83725b-6e0adde/dragonfly-amd64
2021-10-12T06:55:50-c83725b-d887d3b/dragonfly-amd64
2021-10-12T04:35:19-c83725b-6372e7e/dragonfly-amd64
2021-10-11T20:46:14-18fa840-7023535/dragonfly-amd64
2021-08-22T22:30:38-d4cc65f-29d7e54/dragonfly-amd64
2021-08-09T17:59:45-4869184-acbe242/dragonfly-amd64
2021-08-06T19:38:52-d529aec-fa6aa87/dragonfly-amd64
2021-08-04T21:03:22-f367f01-acbe242/dragonfly-amd64
2021-08-04T21:03:22-f367f01-6e73886/dragonfly-amd64
2021-08-04T15:26:45-309db04-6e73886/dragonfly-amd64
2021-08-04T13:50:16-309db04-7921829/dragonfly-amd64
2021-08-04T13:50:16-309db04-16ab7e4/dragonfly-amd64
2021-08-03T20:45:05-2f64839-8a7ee4c/dragonfly-amd64
2021-08-03T19:59:22-32c652e-7921829/dragonfly-amd64
2021-08-03T19:59:22-32c652e-16ab7e4/dragonfly-amd64
2021-08-03T17:18:10-7a2ec09-7921829/dragonfly-amd64
2021-08-03T16:06:16-594b3a2-8a7ee4c/dragonfly-amd64
2021-08-03T16:06:16-594b3a2-7921829/dragonfly-amd64
2021-08-02T22:28:48-3395cb0-818c5f6/dragonfly-amd64
2021-08-02T22:26:52-3395cb0-3d5afa9/dragonfly-amd64
2021-08-02T22:23:25-55fae94-b81d75f/dragonfly-amd64
2021-08-02T19:06:04-a668498-8a7ee4c/dragonfly-amd64
2021-07-29T13:34:27-ab1fe72-ed8cbbc/dragonfly-amd64
2021-07-27T22:01:54-07bc1bf-7cd10c1/dragonfly-amd64
2021-07-26T23:46:13-07bc1bf-840e583/dragonfly-amd64
2021-07-26T20:36:31-07bc1bf-ed8cbbc/dragonfly-amd64
2021-07-26T20:36:31-07bc1bf-ecaa681/dragonfly-amd64
2021-07-23T18:10:55-46d1522-0914646/dragonfly-amd64
2021-07-23T18:08:39-46d1522-c6d89db/dragonfly-amd64
2021-07-22T17:23:26-c740bfd-c6d89db/dragonfly-amd64
2021-07-22T17:23:26-c740bfd-bc51e93/dragonfly-amd64
2021-07-22T17:13:07-251092d-c6d89db/dragonfly-amd64
2021-07-22T17:01:15-412ee17-fdb45ac/dragonfly-amd64
2021-07-21T23:55:41-412ee17-3e48c03/dragonfly-amd64
2021-07-21T20:52:30-7f68387-3e48c03/dragonfly-amd64
2021-07-21T16:31:48-7aa8294-bc51e93/dragonfly-amd64
2021-07-19T17:27:58-6e9046b-6298cfe/dragonfly-amd64
2021-07-18T21:03:43-6e9046b-a66190e/dragonfly-amd64
2021-07-18T21:00:03-6e9046b-650fc21/dragonfly-amd64
2021-07-15T17:35:20-6e9046b-c1cc9f9/dragonfly-amd64
2021-07-15T13:35:41-6e9046b-21a04e3/dragonfly-amd64
2021-07-13T20:49:08-5061c41-c6d89db/dragonfly-amd64
2021-07-13T20:49:08-5061c41-a985897/dragonfly-amd64
2021-07-13T20:15:39-ef97713-bc51e93/dragonfly-amd64
2021-07-13T16:40:23-8e85a28-bc51e93/dragonfly-amd64
2021-07-13T13:36:40-d36a54b-c6d89db/dragonfly-amd64
2021-07-13T12:52:46-3844600-a985897/dragonfly-amd64
2021-07-12T21:21:15-ccff732-c6d89db/dragonfly-amd64
2021-07-12T17:19:11-a7dfe3d-c77980b/dragonfly-amd64
2021-07-12T16:18:15-a7dfe3d-fb052db/dragonfly-amd64
2021-07-12T16:18:15-a7dfe3d-3d1d066/dragonfly-amd64
2021-07-12T16:16:36-980829d-ab4085c/dragonfly-amd64
2021-07-12T16:16:36-980829d-3d1d066/dragonfly-amd64
2021-07-10T00:41:40-cb1acef-3d1d066/dragonfly-amd64
2021-07-10T00:32:22-5b540d3-ab4085c/dragonfly-amd64
2021-07-09T19:50:55-e33c0f2-ab4085c/dragonfly-amd64
2021-07-09T17:21:04-8e32e9f-fb052db/dragonfly-amd64
2021-07-09T17:21:04-8e32e9f-3d1d066/dragonfly-amd64
2021-07-08T23:16:08-6994825-3d1d066/dragonfly-amd64
2021-07-08T21:00:45-cae92d5-fb052db/dragonfly-amd64
2021-07-08T21:00:45-cae92d5-296ddf2/dragonfly-amd64
2021-07-08T19:58:23-41a65bd-fb052db/dragonfly-amd64
2021-07-08T19:58:23-41a65bd-3d1d066/dragonfly-amd64
2021-07-08T19:56:07-71eae3a-3d1d066/dragonfly-amd64
2021-07-08T19:56:07-71eae3a-296ddf2/dragonfly-amd64
2021-07-08T19:53:41-c979f92-3d1d066/dragonfly-amd64
2021-07-08T16:45:40-55cd480-0579cf1/dragonfly-amd64
2021-07-08T16:04:41-77c1b7a-cb4cd9e/dragonfly-amd64
2021-07-08T02:47:23-640c1de-b51bf4f/dragonfly-amd64
2021-07-07T22:29:01-fd00574-5c59e11/dragonfly-amd64
2021-07-07T22:27:48-fd00574-11f5df2/dragonfly-amd64
2021-07-07T20:33:11-fd00574-186a3bb/dragonfly-amd64
2021-07-07T20:25:29-fd00574-f264879/dragonfly-amd64
2021-07-07T20:06:16-fd00574-cb4cd9e/dragonfly-amd64
2021-07-07T20:06:16-fd00574-c96833e/dragonfly-amd64
2021-07-02T16:25:10-7edcfe5-6125d0c/dragonfly-amd64
2021-07-02T16:14:50-7edcfe5-b51bf4f/dragonfly-amd64
2021-07-01T19:09:57-20dafe5-03761ed/dragonfly-amd64
2021-07-01T08:39:23-20dafe5-cb4cd9e/dragonfly-amd64
2021-07-01T01:38:42-f0847e0-9d65578/dragonfly-amd64
2021-06-30T22:02:09-f0847e0-4711bf3/dragonfly-amd64
2021-06-30T16:45:04-f0847e0-cb4cd9e/dragonfly-amd64
2021-06-30T01:29:49-72e4d1b-c45e800/dragonfly-amd64
2021-06-29T19:12:30-72e4d1b-cb4cd9e/dragonfly-amd64
2021-06-29T19:12:30-72e4d1b-3463852/dragonfly-amd64
2021-06-29T17:38:08-100b229-d4ee025/dragonfly-amd64
2021-06-28T23:31:13-12f8456-4bb0847/dragonfly-amd64
2021-06-28T21:17:28-12f8456-5385e23/dragonfly-amd64
2021-06-25T21:07:44-00129ff-ed01cea/dragonfly-amd64
2021-06-25T19:21:47-00129ff-0e7012e/dragonfly-amd64
2021-06-25T19:21:17-fe2294a-d01bc57/dragonfly-amd64
2021-06-25T15:37:24-636017e-fd129a6/dragonfly-amd64
2021-06-25T15:05:05-48cad5e-37f9a8f/dragonfly-amd64
2021-06-25T01:08:06-da404b5-37f9a8f/dragonfly-amd64
2021-06-24T20:04:22-da404b5-c309c89/dragonfly-amd64
2021-06-24T18:58:27-da404b5-cce6214/dragonfly-amd64
2021-06-24T18:12:53-da404b5-600a2a4/dragonfly-amd64
2021-06-24T04:20:31-da404b5-a9bb382/dragonfly-amd64
2021-06-24T04:20:08-4833ac5-fd129a6/dragonfly-amd64
2021-06-24T04:20:08-4833ac5-0e7012e/dragonfly-amd64
2021-06-23T15:16:55-d824a74-0ebd5a8/dragonfly-amd64
2021-06-22T19:12:15-6d3e439-fd129a6/dragonfly-amd64
2021-06-22T19:12:15-6d3e439-0ebd5a8/dragonfly-amd64
2021-06-22T19:12:15-6d3e439-0e7012e/dragonfly-amd64
2021-06-22T16:07:17-4c651fc-fd129a6/dragonfly-amd64
2021-06-22T16:07:17-4c651fc-0e7012e/dragonfly-amd64
2021-06-22T16:01:58-d25f906-5bd09e5/dragonfly-amd64
2021-06-22T11:31:57-d25f906-63daa77/dragonfly-amd64
2021-06-22T02:44:43-d25f906-197a5ee/dragonfly-amd64
2021-06-22T02:37:43-d25f906-9afd158/dragonfly-amd64
2021-06-21T21:21:09-d25f906-1bd5a20/dragonfly-amd64
2021-06-21T17:37:39-d25f906-20bdfba/dragonfly-amd64
2021-06-21T15:39:45-d25f906-ced0fdb/dragonfly-amd64
2021-06-19T00:46:21-d25f906-b73cc4b/dragonfly-amd64
2021-06-18T22:05:09-d25f906-9401172/dragonfly-amd64
2021-06-17T02:42:36-463a76b-881b6ea/dragonfly-amd64
2021-06-16T06:05:23-463a76b-785a8f6/dragonfly-amd64
2021-06-16T04:45:46-463a76b-a752bc0/dragonfly-amd64
2021-06-16T01:01:29-116feae-b4a2af2/dragonfly-amd64
2021-06-15T20:59:42-4b484fb-d77f4c0/dragonfly-amd64
2021-06-14T21:46:05-4b484fb-0fd20ed/dragonfly-amd64
2021-06-14T19:26:24-4b484fb-d4f34f8/dragonfly-amd64
2021-06-14T07:12:21-4b484fb-3249b64/dragonfly-amd64
2021-06-11T16:09:15-e2dc6dd/dragonfly-amd64
2021-06-11T16:06:07-9a55cb1-7677616/dragonfly-amd64
2021-06-11T16:06:07-9a55cb1-2f11284/dragonfly-amd64
2021-06-11T15:30:39-490eac8-2f11284/dragonfly-amd64
2021-06-10T20:01:45-9aa007e-b4a2af2/dragonfly-amd64
2021-06-10T17:33:43-f6327c5-7677616/dragonfly-amd64
2021-06-09T21:40:20-13cf844-ab7f829/dragonfly-amd64
2021-06-09T21:40:20-13cf844-182157c/dragonfly-amd64
2021-06-09T14:15:55-890984b-aa5540c/dragonfly-amd64
2021-06-08T22:32:06-4e58f8f-63dcab2/dragonfly-amd64
2021-06-08T16:06:52-4e58f8f-7677616/dragonfly-amd64
2021-06-07T23:44:22-716319f-39c39ae/dragonfly-amd64
2021-06-07T21:12:46-716319f-909dd5e/dragonfly-amd64
2021-06-07T15:22:48-126df1d-ab7f829/dragonfly-amd64
2021-06-05T19:52:26-1225b6f-e1fa260/dragonfly-amd64
2021-06-04T20:34:05-1225b6f-831f937/dragonfly-amd64
2021-06-04T17:45:44-7295a4e-ab7f829/dragonfly-amd64
2021-06-04T17:21:49-bf13205-3a9d906/dragonfly-amd64
2021-06-04T01:02:31-bf13205-6d98301/dragonfly-amd64
2021-06-03T19:38:56-8f2cf6c-6d98301/dragonfly-amd64
2021-06-03T18:03:08-8f2cf6c-ab7f829/dragonfly-amd64
2021-06-03T17:14:44-4abb1e2-7677616/dragonfly-amd64
2021-06-03T16:41:39-4abb1e2-6b411e9/dragonfly-amd64
2021-06-03T16:41:39-4abb1e2-0d6115c/dragonfly-amd64
2021-06-03T03:03:28-1c2154a-dd7ba3b/dragonfly-amd64
2021-06-02T23:08:29-384c392-6b411e9/dragonfly-amd64
2021-06-02T19:45:53-384c392-fbf844b/dragonfly-amd64
2021-06-02T15:04:26-7271753-e3c9537/dragonfly-amd64
2021-06-02T00:20:16-377464f-cae6870/dragonfly-amd64
2021-06-01T22:58:04-377464f-0410005/dragonfly-amd64

@gopherbot
Copy link

@gopherbot gopherbot commented Nov 2, 2021

Change https://golang.org/cl/360735 mentions this issue: internal/testenv: treat dragonfly-amd64 as a slow machine

gopherbot pushed a commit to golang/tools that referenced this issue Nov 2, 2021
Updates golang/go#45216

Change-Id: I9e714582c4d8c3c7ce7571359d7f9bbe2c6c9497
Reviewed-on: https://go-review.googlesource.com/c/tools/+/360735
Trust: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
@tuxillo
Copy link
Contributor

@tuxillo tuxillo commented Nov 2, 2021

We've moved the builder to a host with a faster disk (on October 18th), it should be a bit better, isn't it?

@bcmills
Copy link
Member Author

@bcmills bcmills commented Nov 3, 2021

From the test logs above, it looks like things improved somewhat on Oct. 18 but then got worse again around Nov. 1.

Curiously, all of the failures since Nov. 1 include at least one deadline error in Test_Issue38211 (#44098). I wonder if that's somehow related... 🤔

@tuxillo
Copy link
Contributor

@tuxillo tuxillo commented Nov 3, 2021

I have noticed that the time in the VM we use gets screwed somehow (see a similar case in https://bugs.dragonflybsd.org/issues/3299). We're not sure yet why this happens. I've enabled an ntp client now and it's making a lot of adjustments, which looks suspicious.

/var/log/messages:Nov  3 18:44:28 gobuilder dntpd[524]: issuing offset adjustment: 0.002025s
/var/log/messages:Nov  3 19:05:27 gobuilder dntpd[524]: issuing offset adjustment: -0.002612s
/var/log/messages:Nov  3 19:25:19 gobuilder dntpd[524]: issuing offset adjustment: 0.002456s
/var/log/messages:Nov  3 19:46:18 gobuilder dntpd[524]: issuing offset adjustment: -0.001033s
/var/log/messages:Nov  3 20:07:19 gobuilder dntpd[524]: issuing offset adjustment: -0.002583s
/var/log/messages:Nov  3 20:28:17 gobuilder dntpd[524]: issuing offset adjustment: 0.002426s
/var/log/messages:Nov  3 22:31:44 gobuilder dntpd[524]: issuing offset adjustment: -0.002397s

In any case, is it performing better right now? It definitely should.

@rsc rsc unassigned tuxillo and dmitshur Jun 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders NeedsInvestigation OS-Dragonfly
Projects
Development

Successfully merging a pull request may close this issue.

4 participants