x/build: build failures due to `410 Gone` responses from `symbolic-datum-552` GOPROXY

[bcmills]

:: Running /workdir/go/bin/go with args ["/workdir/go/bin/go" "run" "golang.org/x/benchmarks/cmd/bench"] and env ["HOSTNAME=buildlet-linux-amd64-perf-rn8418630" "DEBIAN_FRONTEND=noninteractive" "HOME=/root" "USER=root" "GO_STAGE0_NET_DELAY=0s" "GO_STAGE0_DL_DELAY=100ms" "WORKDIR=/workdir" "GO_BUILDER_NAME=linux-amd64-perf" "GOROOT_BOOTSTRAP=/go1.4" "BENCH_BASELINE_GOROOT=/workdir/gobaseline" "BENCH_BRANCH=master" "GOROOT=/workdir/go" "GOPATH=/workdir/gopath" "GOPROXY=http://gk3-services-nap-jetxd907-8f582974-sbzf.c.symbolic-datum-552.internal:30157" "TMPDIR=/workdir/tmp" "GOCACHE=/workdir/gocache" "PATH=/workdir/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" "PWD=/workdir/gopath/src/golang.org/x/benchmarks"] in dir /workdir/gopath/src/golang.org/x/benchmarks
…
[sweet] Setting up benchmark: go-build
[sweet] error: build go-build for experiment: exit status 1. stderr:
go: github.com/golang-migrate/migrate/v4@v4.6.2 requires
	cloud.google.com/go@v0.37.4 requires
	google.golang.org/genproto@v0.0.0-20190404172233-64821d5d2107: reading http://gk3-services-nap-jetxd907-8f582974-sbzf.c.symbolic-datum-552.internal:30157/google.golang.org/genproto/@v/v0.0.0-20190404172233-64821d5d2107.mod: 410 Gone
	server response: not found: temporarily unavailable
go: downloading github.com/go-redis/redis/v8 v8.0.0
go: downloading github.com/google/safehtml v0.0.2
go: downloading cloud.google.com/go v0.66.0
go: downloading cloud.google.com/go/storage v1.10.0
go: downloading golang.org/x/net v0.0.0-20210226172049-e18ecbb05110
go: downloading cloud.google.com/go/logging v1.0.0
go: downloading contrib.go.opencensus.io/integrations/ocsql v0.1.4
go: downloading github.com/jackc/pgx/v4 v4.12.0
go: downloading google.golang.org/genproto v0.0.0-20200923140941-5646d36feee1
go: downloading contrib.go.opencensus.io/exporter/prometheus v0.1.0
go: downloading contrib.go.opencensus.io/exporter/stackdriver v0.13.4
go: downloading go.opencensus.io v0.22.4
go: downloading golang.org/x/mod v0.3.1-0.20200828183125-ce943fd02449
go: downloading github.com/go-redis/redis_rate/v9 v9.0.2
go: downloading google.golang.org/api v0.32.0
go: downloading github.com/microcosm-cc/bluemonday v1.0.5
go: downloading github.com/russross/blackfriday/v2 v2.0.1
go: downloading github.com/yuin/goldmark v1.4.0
go: downloading github.com/yuin/goldmark-emoji v1.0.1
go: downloading github.com/golang/protobuf v1.4.2
go: downloading google.golang.org/grpc v1.32.0
go: downloading github.com/Masterminds/squirrel v1.4.0
go: downloading github.com/golang-migrate/migrate/v4 v4.6.2
go: downloading github.com/google/licensecheck v0.3.1
go: downloading github.com/lib/pq v1.10.2
go: downloading golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208
go: downloading github.com/jackc/pgconn v1.9.0
go: downloading github.com/go-git/go-billy/v5 v5.0.0
go: downloading github.com/go-git/go-git/v5 v5.1.0
go: downloading github.com/evanw/esbuild v0.8.57
go: downloading google.golang.org/protobuf v1.25.0
go: downloading github.com/google/go-cmp v0.5.2
go: downloading golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97
go: downloading github.com/aws/aws-sdk-go v1.34.29
go: github.com/golang-migrate/migrate/v4@v4.6.2 requires
	cloud.google.com/go@v0.37.4 requires
	google.golang.org/genproto@v0.0.0-20190404172233-64821d5d2107: reading http://gk3-services-nap-jetxd907-8f582974-sbzf.c.symbolic-datum-552.internal:30157/google.golang.org/genproto/@v/v0.0.0-20190404172233-64821d5d2107.mod: 410 Gone
	server response: not found: temporarily unavailable

greplogs -l -e '\[sweet\] error: build .*(?:\n.*)*410 Gone' --since=2022-01-01
2022-06-22T16:18:44-0fc6e7c-92c9b81/linux-amd64-perf

It's not obvious to me how the symbolic-datum-552 proxy gets populated or whether perhaps the x/benchmarks/sweet module dependencies are somehow non-hermetic or unstable.

(attn @prattmic @mknyszek; CC @golang/release)

[prattmic]

I got very similar failures in x/tools trybots (linux and windows) on https://go.dev/cl/413578 on 2022-06-23 around 15:22 UTC.

Different packages, and no x/benchmarks involved, but same 410 Gone:

https://storage.googleapis.com/go-build-log/2e773a38/linux-amd64_f7fad36c.log

linux-amd64 at 2e773a3894fba7af744090d7d42968f4993018e2 building build at d36736901e40cbd378c11bb2535ea9e16f39a7a3

:: Running /workdir/go/bin/go with args ["/workdir/go/bin/go" "test" "-short" "golang.org/x/build/..."] and env ["HOSTNAME=buildlet-linux-bullseye-rnf676e4b" "DEBIAN_FRONTEND=noninteractive" "HOME=/root" "USER=root" "GO_STAGE0_NET_DELAY=0s" "GO_STAGE0_DL_DELAY=100ms" "WORKDIR=/workdir" "GO_BUILDER_NAME=linux-amd64" "GOROOT_BOOTSTRAP=/go1.4" "GO_DISABLE_OUTBOUND_NETWORK=1" "GOROOT=/workdir/go" "GOPATH=/workdir/gopath" "GOPROXY=http://gk3-services-nap-jetxd907-8f582974-sbzf.c.symbolic-datum-552.internal:30157" "GO111MODULE=on" "TMPDIR=/workdir/tmp" "GOCACHE=/workdir/gocache" "PATH=/workdir/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" "PWD=/workdir/gopath/src/golang.org/x/build"] in dir /workdir/gopath/src/golang.org/x/build

go: downloading cloud.google.com/go/compute v1.3.0
go: downloading github.com/google/go-github v17.0.0+incompatible
go: downloading github.com/google/go-cmp v0.5.8
go: downloading github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7
go: downloading golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
go: downloading golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8
go: downloading go4.org v0.0.0-20180809161055-417644f6feb5
go: downloading google.golang.org/grpc v1.44.0
go: downloading github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51
go: downloading golang.org/x/term v0.0.0-20220526004731-065cf7ba2467
go: downloading cloud.google.com/go/storage v1.21.0
go: downloading golang.org/x/crypto v0.0.0-20220307211146-efcb8507fb70
go: downloading github.com/bradfitz/go-smtpd v0.0.0-20170404230938-deb6d6237625
go: downloading github.com/jellevandenhooff/dkim v0.0.0-20150330215556-f50fe3d243e1
go: downloading google.golang.org/api v0.70.0
go: downloading cloud.google.com/go/datastore v1.1.0
go: downloading github.com/NYTimes/gziphandler v1.1.1
go: downloading github.com/jackc/pgx/v4 v4.13.0
go: downloading github.com/shurcooL/githubv4 v0.0.0-20220520033151-0b4e3294ff00
go: downloading github.com/golang/protobuf v1.5.2
go: downloading contrib.go.opencensus.io/exporter/prometheus v0.3.0
go: downloading contrib.go.opencensus.io/exporter/stackdriver v0.13.5
go: downloading go.opencensus.io v0.23.0
go: downloading cloud.google.com/go v0.100.2
go: downloading google.golang.org/genproto v0.0.0-20220222213610-43724f9ea8cf
go: downloading golang.org/x/text v0.3.7
go: downloading cloud.google.com/go/errorreporting v0.2.0
go: downloading github.com/gliderlabs/ssh v0.3.3
go: downloading golang.org/x/time v0.0.0-20210723032227-1f47c861a9ac
go: downloading golang.org/x/perf v0.0.0-20220408152633-b570d1a8e7a7
go: downloading cloud.google.com/go/secretmanager v1.3.0
go: downloading github.com/influxdata/influxdb-client-go/v2 v2.8.0
go: downloading cloud.google.com/go/bigquery v1.8.0
go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.0.1-0.20190118093823-f849b5445de4
go: downloading github.com/googleapis/google-cloud-go-testing v0.0.0-20200911160855-bcd43fbb19e8
go: downloading github.com/creack/pty v1.1.15
go: downloading golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd
go: downloading github.com/aws/aws-sdk-go v1.30.15
go: downloading google.golang.org/protobuf v1.27.1
go: downloading github.com/google/uuid v1.2.0
go: downloading github.com/googleapis/gax-go/v2 v2.1.1
go: downloading github.com/jackc/pgconn v1.11.0
go: downloading github.com/go-sql-driver/mysql v1.5.0
go: downloading google.golang.org/appengine v1.6.7
go: downloading github.com/mattn/go-sqlite3 v1.14.6
go: downloading github.com/aclements/go-gg v0.0.0-20170118225347-6dbb4e4fefb0
go: downloading gopkg.in/inf.v0 v0.9.1
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading github.com/coreos/go-systemd v0.0.0-20190719114852-fd7a80b32e1f
go: downloading github.com/dghubble/oauth1 v0.7.0
go: downloading github.com/esimov/stackblur-go v1.1.0
go: downloading github.com/sendgrid/sendgrid-go v3.11.1+incompatible
go: downloading github.com/yuin/goldmark v1.4.12
go: downloading golang.org/x/image v0.0.0-20210628002857-a66eb6448b8d
go: downloading github.com/golang-migrate/migrate/v4 v4.15.0-beta.3
go: downloading github.com/julienschmidt/httprouter v1.3.0
go: downloading golang.org/x/sys v0.0.0-20220209214540-3681064d5158
go: downloading github.com/shurcooL/graphql v0.0.0-20220520033453-bdb1221e171e
go: downloading github.com/jackc/pgproto3/v2 v2.2.0
go: downloading github.com/jackc/puddle v1.1.3
go: downloading github.com/google/go-querystring v1.0.0
go: downloading github.com/prometheus/client_golang v1.9.0
go: downloading github.com/prometheus/statsd_exporter v0.20.0
go: downloading cloud.google.com/go/monitoring v1.4.0
go: downloading cloud.google.com/go/trace v1.2.0
go: downloading github.com/census-instrumentation/opencensus-proto v0.2.1
go: downloading cloud.google.com/go/iam v0.1.1
go: downloading golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
go: downloading github.com/anmitsu/go-shlex v0.0.0-20200514113438-38f4b401e2be
go: downloading github.com/deepmap/oapi-codegen v1.8.2
go: downloading github.com/pkg/errors v0.9.1
go: downloading gopkg.in/yaml.v2 v2.4.0
go: downloading github.com/jackc/pgio v1.0.0
go: downloading github.com/jackc/pgtype v1.8.1
go: downloading github.com/jackc/chunkreader/v2 v2.0.1
go: downloading github.com/jackc/pgpassfile v1.0.0
go: downloading github.com/jackc/pgservicefile v0.0.0-20200714003250-2b9c44734f2b
go: downloading github.com/aclements/go-moremath v0.0.0-20210112150236-f10218a38794
go: downloading gonum.org/v1/plot v0.10.0
go: downloading github.com/sendgrid/rest v2.6.9+incompatible
go: downloading github.com/hashicorp/golang-lru v0.5.4
go: downloading github.com/prometheus/common v0.15.0
go: downloading cloud.google.com/go/container v1.2.0
go: downloading github.com/beorn7/perks v1.0.1
go: downloading github.com/cespare/xxhash/v2 v2.1.1
go: downloading github.com/prometheus/client_model v0.2.0
go: downloading github.com/prometheus/procfs v0.2.0
go: downloading github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e
go: downloading github.com/hashicorp/go-multierror v1.1.0
go: downloading github.com/jackc/pgerrcode v0.0.0-20201024163028-a0d42d470451
go: downloading go.uber.org/atomic v1.6.0
go: downloading github.com/influxdata/line-protocol v0.0.0-20200327222509-2487e7298839
go: downloading github.com/gonum/matrix v0.0.0-20181209220409-c518dec07be9
go: downloading github.com/fogleman/gg v1.3.0
go: downloading github.com/jmespath/go-jmespath v0.4.0
go: downloading github.com/sirupsen/logrus v1.8.1
go: downloading gopkg.in/alecthomas/kingpin.v2 v2.2.6
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1
go: downloading github.com/go-fonts/liberation v0.2.0
go: downloading github.com/go-latex/latex v0.0.0-20210823091927-c0d11ff05a81
go: downloading github.com/ajstarks/svgo v0.0.0-20210923152817-c3b6e2f0c527
go: downloading github.com/go-pdf/fpdf v0.5.0
go: downloading github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751
go: downloading github.com/hashicorp/errwrap v1.0.0
go: downloading github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d
go: downloading github.com/gonum/blas v0.0.0-20181208220705-f22b278b28ac
go: downloading github.com/gonum/floats v0.0.0-20181209220543-c233463c7e82
go: downloading github.com/gonum/internal v0.0.0-20181124074243-f884aa714029
go: downloading github.com/gonum/lapack v0.0.0-20181123203213-e4cdc5a0bff9
go: downloading github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0
google.golang.org/grpc@v1.44.0: reading http://gk3-services-nap-jetxd907-8f582974-sbzf.c.symbolic-datum-552.internal:30157/google.golang.org/grpc/@v/v1.44.0.mod: 410 Gone
	server response: not found: temporarily unavailable

[prattmic]

perhaps the x/benchmarks/sweet module dependencies are somehow non-hermetic or unstable.

I don't think so. This part of go-build is probably https://cs.opensource.google/go/x/benchmarks/+/master:sweet/harnesses/go-build.go;l=93. i.e., we are just doing a plain go build of a fixed commit of either kubelet, istioctl, or the pkgsite frontend (we fail to log which one; I'll fix this). I think this should be stable unless there is something odd going on.

[gopherbot]

Change https://go.dev/cl/414396 mentions this issue: sweet/harnesses: log which go-build package fails to build

[bcmills]

Different packages, and no x/benchmarks involved, but same 410 Gone:

Indeed, and here's another one building x/build: https://build.golang.org/log/c532106b8e7169a892bbdc77978e904684480c53

Updating the search regexp, we have:
greplogs -l -e 'reading .*: 410 Gone\n\s+server response: not found: temporarily unavailable'
2022-06-24T23:50:20-60cf787-41e1d90/linux-amd64
2022-06-22T16:18:44-0fc6e7c-92c9b81/linux-amd64-perf

It's not clear to me whether this is a problem in the symbolic-datum-552 proxy or a backend that it connects to (perhaps proxy.golang.org?).

[prattmic]

https://go.dev/ref/mod#private-module-privacy

The default value of GOPROXY is: https://proxy.golang.org,direct With this setting, when the go command downloads a module or module metadata, it will first send a request to proxy.golang.org... If the proxy responds with a 404 (Not Found) or 410 (Gone) status, the go command will attempt to connect directly to the version control system providing the module.

The coordinator sets GOPROXY without a direct fallback: https://cs.opensource.google/go/x/build/+/master:cmd/coordinator/buildstatus.go;l=1244

This explains why 410 causes failure. But it isn't clear to me (1) if it is intentional that we don't have a direct fallback, or (2) why this hasn't been occurring for a long time.

[dmitshur]

Re 1, I think we don't have a direct fallback because, if GO_DISABLE_OUTBOUND_NETWORK is working (it might not be, see #51444), then any fallback direct connections would be refused by the firewall anyway.

[prattmic]

Thanks, that makes sense. I figured that might be the case when I realized that we are just providing a reverse proxy to proxy.golang.org.

cc @heschi @suzmue if they are aware of changes to the proxy that may have triggered this.

One possible theory is that over time the versions of the packages we are fetching have become sufficiently unpopular globally that they are now more likely to fall out of cache in the proxy.

So far, it seems the versions we have received 410 on are:

• github.com/docker/docker@v17.12.0-ce-rc1.0.20210128214336-420b1d36250f+incompatible
• google.golang.org/genproto@v0.0.0-20190404172233-64821d5d2107

[bcmills]

So it looks like this path is hitting proxy.golang.org? Then the question is, why is the proxy returning 410s for these URLs instead of blocking until they are fetched?

(Maybe a bad interaction with some bug or outage on the module fetch path? But at that point I would expect to see something describing the fetch error, not just temporarily unavailable...)

[heschi]

This is a semi-known bug in the module proxy. @suzmue should know more.