Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: flaky network or bad 'git' version on linux-386-longtest builder #49953

Open
bcmills opened this issue Dec 3, 2021 · 2 comments
Open

x/build: flaky network or bad 'git' version on linux-386-longtest builder #49953

bcmills opened this issue Dec 3, 2021 · 2 comments

Comments

@bcmills
Copy link
Member

@bcmills bcmills commented Dec 3, 2021

The cmd/go tests involving git requests to external services (mainly golang.org, github.com, gopkg.in) are expected to have a slightly nonzero failure rate due to external service outages. However, the failure rate on the linux-386-longtest builder in particular seems to have increased significantly starting in September, without a corresponding increase in the failure rate on the linux-amd64-longtest builder, and many of the failures are to the Go project's own Git server (go.googlesource.com).

It isn't clear to me whether these failures are due to network drops, or perhaps bugs in the builder's git implementation causing the upstream git server to drop the connection.

@golang/release, were there any builder image or configuration changes around September that might explain the increase in failure rate? Could we talk with the go.googlesource.com admins about whether they have also seen a higher error rate on the server side in the same timeframe?

greplogs --dashboard -md -l -e 'fatal: The remote end hung up unexpectedly' --since=2021-01-01

2021-12-02T22:05:56-e533b57/linux-386-longtest
2021-12-02T22:04:50-bcb98f1/linux-386-longtest
2021-12-02T22:03:54-25f06cb/linux-386-longtest
2021-12-02T07:04:05-d34051b/linux-386-longtest
2021-12-01T17:01:41-8ebb8c9/linux-386-longtest
2021-11-30T15:28:46-682435d/linux-386-longtest
2021-11-29T16:08:23-a59ab29/linux-amd64-longtest
2021-11-24T21:11:45-b77f5f9/linux-amd64-longtest
2021-11-24T21:11:34-8cdfe40/linux-386-longtest
2021-11-24T20:59:14-67dd9ee/linux-386-longtest
2021-11-24T20:57:46-c25bf0d/linux-386-longtest
2021-11-24T20:54:12-696515e/linux-386-longtest
2021-11-24T20:54:07-6ea17aa/linux-386-longtest
2021-11-16T19:41:37-f659183/linux-386-longtest
2021-11-16T15:38:59-7f4a946/linux-386-longtest
2021-11-15T18:02:28-1dc9af5/linux-386-longtest
2021-11-15T17:10:25-f986191/linux-amd64-longtest
2021-11-11T15:34:02-8c73f80/linux-386-longtest
2021-11-11T11:47:08-e9ef931/linux-386-longtest
2021-11-05T21:27:19-58ec925/linux-386-longtest
2021-11-05T16:51:14-c58417b/linux-386-longtest
2021-11-04T23:56:29-0e5f287/linux-386-longtest
2021-11-03T18:37:22-cfd016d/linux-386-longtest
2021-11-02T15:51:21-c3cb1ec/linux-386-longtest
2021-11-01T22:55:50-02e5913/linux-amd64-longtest
2021-10-29T19:30:03-4592933/linux-amd64-longtest
2021-10-28T16:08:36-6f0185b/linux-386-longtest
2021-10-26T21:17:38-091db63/linux-amd64-longtest
2021-10-21T18:19:51-6508fda/linux-386-longtest
2021-09-16T23:56:28-265b59a/linux-386-longtest
2021-09-15T16:32:27-72bb818/linux-386-longtest
2021-09-14T23:03:28-3a72175/linux-386-longtest
2021-09-14T23:03:28-3a72175/linux-amd64-longtest
2021-09-14T00:29:16-71adc65/linux-386-longtest
2021-09-13T22:57:34-9a58aa2/linux-386-longtest
2021-09-02T17:30:32-7609b50/linux-386-longtest
2021-08-12T17:54:49-5805efc/linux-amd64-longtest
2021-06-28T23:31:05-1519271/linux-amd64-longtest
2021-05-26T22:43:54-1d5298d/linux-amd64-longtest
2021-04-13T21:13:11-e69f022/linux-386-longtest
2021-03-30T03:31:44-bb2fc21/linux-386-longtest
2021-03-15T21:49:19-de2b27d/linux-amd64-longtest
2021-03-08T18:56:25-b478720/linux-386-longtest
2021-02-09T05:28:50-e0ac989/linux-amd64-longtest
2021-01-27T21:11:22-00f2ff5/linux-386-longtest
2021-01-27T21:11:22-00f2ff5/linux-amd64-longtest
2021-01-27T21:04:09-35334ca/linux-386-longtest
2021-01-27T21:04:09-35334ca/linux-amd64-longtest
2021-01-27T20:44:24-a5a5e2c/linux-386-longtest
2021-01-27T20:44:24-a5a5e2c/linux-amd64-longtest

@bcmills

This comment was marked as off-topic.

@bcmills

This comment was marked as off-topic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants