x/build,cmd/go: frequent connection timeouts to github.com since 2022-03-04 #52545

bcmills · 2022-04-25T14:35:06Z

# cd $WORK/tmp/d2/src/github.com/myitcv; git clone -- https://github.com/myitcv/vgo_example_compat $WORK/tmp/d2/src/github.com/myitcv/vgo_example_compat
            Cloning into '$WORK/tmp/d2/src/github.com/myitcv/vgo_example_compat'...
            fatal: unable to access 'https://github.com/myitcv/vgo_example_compat/': Failed to connect to github.com port 443: Connection timed out

greplogs --dashboard -md -l -e 'github.com.* Connection timed out' --since=2021-01-01

@golang/release: were there changes to the linux-.*-longtest image or network configuration around 2022-03-04 that might explain the increase in timeouts at that point?

The text was updated successfully, but these errors were encountered:

heschi · 2022-04-26T20:54:13Z

Not that I'm aware of.

bcmills · 2022-05-03T15:13:37Z

As with the other network-related failures, this failure mode is much more common on the linux-386-longtest builder.

bcmills · 2022-05-04T14:18:11Z

greplogs -l -e 'github.com.* Connection timed out' --since=2022-05-03
2022-05-04T00:51:08-8b389eb/linux-amd64-longtest
2022-05-03T21:36:22-64b6e44/linux-386-longtest
2022-05-03T14:51:51-920b9ab/linux-386-longtest

bcmills · 2022-05-05T15:16:24Z

greplogs -l -e 'github.com.* Connection timed out' --since=2022-05-04
2022-05-05T01:33:01-ed462a6/linux-amd64-longtest
2022-05-04T15:38:23-c6688b5/linux-amd64-longtest
2022-05-04T00:51:08-8b389eb/linux-amd64-longtest

bcmills · 2022-05-06T13:59:22Z

greplogs -l -e 'github.com.* Connection timed out' --since=2022-05-05T01:34:00
2022-05-06T04:22:04-3ea3bc0/linux-386-longtest
2022-05-05T21:52:34-50b1add/linux-386-longtest
2022-05-05T18:48:01-53647a1/linux-amd64-longtest
2022-05-05T14:44:31-485a572/linux-386-longtest

bcmills · 2022-05-06T15:28:29Z

This is causing a huge amount of noise in the longtest builders, and affects the most comprehensive builders for two first-class ports (linux/386 and linux/amd64).

At this failure rate, I suspect that if this issue affected end users we'd be hearing about it by now. So my hypothesis is that this is some kind of GCE networking issue that affects the builder hosts.

Marking as release-blocker for Go 1.19. We need to figure out what's happening with the network and either fix it (if it's a GCE or builder problem) or figure out a workaround (if it's a GitHub problem).

(Note that most of these failures come from the git binary itself; #46693 may explain why we're not seeing this on the Windows builders.)

bcmills · 2022-05-06T15:42:21Z

When these timeouts occur, the tests are consistently at 120s or longer of running time.
That suggests that raising the connection timeout would not be helpful.

bcmills · 2022-05-06T17:00:20Z

Here's a theory, inspired by https://askubuntu.com/questions/218728/connection-timeout-when-accessing-github.

Maybe this timeout is some kind of rate-limiting countermeasure on the GitHub side, spurred by bursts of traffic from the Go builders running more SlowBots and post-commit CLs.

Maybe the 5-month gap from October 2021 to March 2022 is an artifact of the Go development cycle: the Go release freeze started 2021-11-01 and ended 2022-02-28. The last failure before the gap was a couple of weeks before the freeze began, and the first failure after the gap was about a week after it ended. During the freeze, the CL rate is naturally much lower, which might keep the builders below whatever threshold GitHub is using for rate-limiting.

adonovan · 2022-05-06T20:48:08Z

Hi @theojulienne, do you know who might be able to cast an eye on this from the GitHub side?

bcmills · 2022-05-11T13:28:19Z

I filed a GitHub support ticket (#1616480), but their conclusion was that it's probably a network issue on the GCE side.

Unfortunately, the GCE network architecture is pretty opaque — traceroute doesn't even work properly (https://cloud.google.com/compute/docs/faq#networking). 😵

I think we need to escalate this on the GCE side to at least get enough routing information to identify where along the route the connection is being dropped.

gopherbot · 2022-05-11T14:46:10Z

Change https://go.dev/cl/405714 mentions this issue: cmd/go: add timestamps to script test output

Go tests don't include timestamps by default, but we would like to have them in order to correlate builder failures with server and network logs. Since many of the Go tests with external network and service dependencies are script tests for the 'go' command, logging timestamps here adds a lot of logging value with one simple and very low-risk change. For #50541. For #52490. For #52545. For #52851. Change-Id: If3fa86deb4a216ec6a1abc4e6f4ee9b05030a729 Reviewed-on: https://go-review.googlesource.com/c/go/+/405714 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Auto-Submit: Bryan Mills <bcmills@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Bryan Mills <bcmills@google.com>

bcmills · 2022-05-19T16:55:13Z

bcmills · 2022-05-19T17:07:52Z

The failures starting 2022-05-11 have timestamps, so we (at least theoretically) now have something we can correlate with network and server logs.

gopherbot added the Builders label Apr 25, 2022

gopherbot added this to the Unreleased milestone Apr 25, 2022

bcmills added the NeedsInvestigation label Apr 25, 2022

bcmills mentioned this issue May 3, 2022

build dashboard triage log #52653

Open

bcmills added the release-blocker label May 6, 2022

bcmills removed this from the Unreleased milestone May 6, 2022

bcmills added this to the Go1.19 milestone May 6, 2022

bcmills mentioned this issue May 9, 2022

cmd/go: TestScript/mod_invalid_version failures on linux-386-longtest due to missing ancestor tags #52796

Open

heschi added this to Planned in Go Release Team May 10, 2022

dmitshur added the GoCommand label May 13, 2022

golang / go Public

x/build,cmd/go: frequent connection timeouts to github.com since 2022-03-04 #52545

x/build,cmd/go: frequent connection timeouts to github.com since 2022-03-04 #52545

bcmills commented Apr 25, 2022 •

edited

heschi commented Apr 26, 2022

bcmills commented May 3, 2022

bcmills commented May 4, 2022

bcmills commented May 5, 2022

bcmills commented May 6, 2022

bcmills commented May 6, 2022 •

edited

bcmills commented May 6, 2022

bcmills commented May 6, 2022

adonovan commented May 6, 2022

bcmills commented May 11, 2022

gopherbot commented May 11, 2022

bcmills commented May 19, 2022

bcmills commented May 19, 2022

golang / go Public

x/build,cmd/go: frequent connection timeouts to github.com since 2022-03-04 #52545

x/build,cmd/go: frequent connection timeouts to github.com since 2022-03-04 #52545

Comments

bcmills commented Apr 25, 2022 • edited

heschi commented Apr 26, 2022

bcmills commented May 3, 2022

bcmills commented May 4, 2022

bcmills commented May 5, 2022

bcmills commented May 6, 2022

bcmills commented May 6, 2022 • edited

bcmills commented May 6, 2022

bcmills commented May 6, 2022

adonovan commented May 6, 2022

bcmills commented May 11, 2022

gopherbot commented May 11, 2022

bcmills commented May 19, 2022

bcmills commented May 19, 2022

bcmills commented Apr 25, 2022 •

edited

bcmills commented May 6, 2022 •

edited