Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net: *TCPConn deadlines missed by more than 25% on NetBSD and OpenBSD #50189

Open
bcmills opened this issue Dec 15, 2021 · 2 comments
Open

net: *TCPConn deadlines missed by more than 25% on NetBSD and OpenBSD #50189

bcmills opened this issue Dec 15, 2021 · 2 comments

Comments

@bcmills
Copy link
Member

@bcmills bcmills commented Dec 15, 2021

In CL 366176 (for #36108), I increased the timeout slop in net.TestWriteTimeoutFluctuation and net.TestReadTimeoutFluctuation to 33% even for tests with very generous (multiple-second) timeouts. At that scale, that margin of slop should be trivial for even a heavily-loaded builder to hit.

Unfortunately, the NetBSD and OpenBSD builders still do not reliably hit it, even on an n1 instance that does not appear to be affected by #49209.

Given the other issues we've had with NetBSD and OpenBSD, I suspect a kernel bug. I plan to further raise the slop for those two platforms, narrow it everywhere else, and call it at that without investigating further. However, I suggest that folks who care about these kernels (@bsiegert, @coypoop, @4a6f656c?) may want to look into whether the underlying system calls may be adding unnecessary slop in their timeout handling.

greplogs --dashboard -md -l -e 'FAIL: TestWriteTimeoutFluctuation' --since=2021-11-23

2021-12-13T16:09:22-d198a36/openbsd-amd64-70-n2d
2021-11-29T19:45:58-f598e29/netbsd-amd64-9_0-n1
2021-11-29T16:08:23-a59ab29/openbsd-amd64-68
2021-11-25T00:07:28-f7e34e7/openbsd-amd64-70-n1
2021-11-25T00:07:11-c58243a/openbsd-amd64-70-n1

@gopherbot
Copy link

@gopherbot gopherbot commented Dec 15, 2021

Change https://golang.org/cl/372215 mentions this issue: net: increase timing slop in TimeoutFluctuation tests on NetBSD and OpenBSD

@bcmills
Copy link
Member Author

@bcmills bcmills commented Dec 15, 2021

Expanding the regexp a bit to catch related tests:

greplogs --dashboard -md -l -e 'FAIL: Test(?:Read|ReadFrom|Write)TimeoutFluctuation' --since=2021-11-19

2021-12-13T16:09:22-d198a36/openbsd-amd64-70-n2d
2021-12-03T14:28:11-a174638/netbsd-386-9_0-n2d
2021-12-02T05:25:23-00dbcb3/netbsd-386-9_0
2021-12-01T22:20:39-c3a7fb2/openbsd-amd64-70
2021-11-30T20:04:58-7ccbcc9/openbsd-amd64-70-n1
2021-11-29T22:02:51-ebd0b77/openbsd-amd64-70
2021-11-29T22:02:45-1970e3e/openbsd-amd64-70
2021-11-29T19:45:58-f598e29/netbsd-amd64-9_0-n1
2021-11-29T16:08:23-a59ab29/openbsd-amd64-68
2021-11-27T23:27:52-9f2a075/openbsd-amd64-70-n1
2021-11-27T19:49:32-a142d65/openbsd-amd64-70-n1
2021-11-25T00:07:28-f7e34e7/openbsd-amd64-70-n1
2021-11-25T00:07:11-c58243a/openbsd-amd64-70-n1
2021-11-24T21:09:36-7e5331a/openbsd-amd64-70-n1
2021-11-24T20:53:54-a3b8f62/openbsd-amd64-70-n1
2021-11-22T18:55:55-2d7ae3f/openbsd-amd64-70
2021-11-22T18:50:19-9e94cc3/netbsd-amd64-9_0-n1
2021-11-22T18:50:19-9e94cc3/openbsd-amd64-70
2021-11-22T16:53:57-cd0bf38/netbsd-386-9_0-n1
2021-11-22T16:53:57-cd0bf38/netbsd-amd64-9_0
2021-11-22T16:15:36-6275b54/netbsd-386-9_0
2021-11-22T16:15:36-6275b54/netbsd-amd64-9_0-n1
2021-11-22T15:54:11-e73c6c8/netbsd-amd64-9_0
2021-11-22T12:30:26-b2aa138/netbsd-amd64-9_0-n1
2021-11-22T12:30:26-b2aa138/openbsd-amd64-70
2021-11-22T12:29:44-ffb6c79/netbsd-386-9_0
2021-11-22T12:29:44-ffb6c79/netbsd-amd64-9_0-n1
2021-11-22T12:29:44-ffb6c79/openbsd-amd64-70
2021-11-22T04:27:29-e30ebaa/netbsd-amd64-9_0
2021-11-22T03:24:07-a287c4a/openbsd-amd64-68
2021-11-22T03:24:01-0c3b4a3/netbsd-amd64-9_0
2021-11-22T03:24:01-0c3b4a3/openbsd-amd64-68
2021-11-22T03:24:01-0c3b4a3/openbsd-amd64-70
2021-11-20T08:47:36-91abe4b/netbsd-386-9_0-n1
2021-11-20T08:47:36-91abe4b/netbsd-amd64-9_0
2021-11-20T08:47:36-91abe4b/openbsd-amd64-70
2021-11-20T00:37:28-d2f4c93/netbsd-386-9_0
2021-11-20T00:32:49-57aba32/netbsd-amd64-9_0
2021-11-19T21:59:14-5e774b0/netbsd-386-9_0-n1
2021-11-19T21:59:14-5e774b0/openbsd-amd64-70

gopherbot pushed a commit that referenced this issue Dec 15, 2021
…penBSD

Decrease the slop everywhere else, since NetBSD and OpenBSD seem to be
the only ones that miss by that much.

For #50189
Updates #36108

Change-Id: I22ac39cc7c254e40358fcd933b5a6016629602c3
Reviewed-on: https://go-review.googlesource.com/c/go/+/372215
Trust: Bryan Mills <bcmills@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants