Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Go checksum server responds with 410 Gone despite tag existing #41611

Closed
mholt opened this issue Sep 24, 2020 · 4 comments
Closed

Go checksum server responds with 410 Gone despite tag existing #41611

mholt opened this issue Sep 24, 2020 · 4 comments

Comments

@mholt
Copy link

@mholt mholt commented Sep 24, 2020

What version of Go are you using (go version)?

$ go version
go version go1.15.2 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/matt/.cache/go-build"
GOENV="/home/matt/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/matt/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/matt/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/matt/Dev/caddyserver/caddy/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build115081725=/tmp/go-build -gno-record-gcc-switches"

What did you do?

$ go get github.com/caddyserver/caddy/v2@v2.2.0

What did you expect to see?

Successful pull:

go: finding github.com/caddyserver/caddy/v2 v2.2.0

What did you see instead?

verifying github.com/caddyserver/caddy/v2@v2.2.0/go.mod: github.com/caddyserver/caddy/v2@v2.2.0/go.mod: reading https://sum.golang.org/lookup/github.com/caddyserver/caddy/v2@v2.2.0: 410 Gone

According to seankhliao on Gophers Slack, right now proxy responds gone, fallbacks to direct, then it errors out because the sum server responds gone.

And according to @bcmills:

It seems to work fine using GOPRIVATE=github.com/caddyserver, so this certainly looks like something specific to the proxy, and my prior is that “something weird having to do with the proxy” is almost always the cache TTL.

But this is the first pull of that module and tag. (We've been using this release flow for about 6 months with no problems.)

Bryan also suggested issue #38065 might be related, but I am skeptical, since again, I have no clue as to why this would be cached already. Nor does it make sense to cache a non-entry. (Bug in the cache/proxy?)

This is blocking the v2.2.0 release of Caddy: https://github.com/caddyserver/caddy/runs/1161405190

Edit: After about 20 minutes it is working now. But this is still a bit unsettling because it blocks releases. What causes this, and how can this be avoided in the future?

@heschik
Copy link
Contributor

@heschik heschik commented Sep 24, 2020

This is the exact same situation you reported in #36183. I checked the logs again, and the first request we've seen for v2.2.0 came in today, not too long before you filed this issue. I think that the most likely explanation by far is that some part of the release process requested it early. I suppose it is possible that someone took the trouble to delay the release half an hour.

Do you have a specific question that was not answered in #36183? I already pointed to why it's necessary to have negative caching.

@mholt
Copy link
Author

@mholt mholt commented Sep 24, 2020

The thing is, this keeps happening, and although I'm pretty sure I didn't make a mistake last time, I'm 100% sure there wasn't a mistake (by me) this time, since our release process is automated and has been working fine, except occasionally the module cache has for some reason cached a non-entry. Maybe somebody set up a client to poll for the new tag or something. Whatever it is, it's getting in the way.

I believe that the ability for any random person on the Internet to break or delay releases -- especially security releases (not this one, fortunately) -- is something that needs to be fixed.

@heschik
Copy link
Contributor

@heschik heschik commented Sep 24, 2020

As I said, nobody has been polling for the new release: the first request came in shortly before you filed this issue. It's either a huge coincidence or a consequence of the release process.

proxy/sum.golang.org perform no authentication, so everyone is a "random person" to them, and that is not likely to change. Without that, there isn't going to be a good way to expire the negative cache entry. The TTL is less than an hour, which should be a tolerable, though admittedly unfortunate, amount of time to wait.

I can leave this open if you like but I don't think it's likely we'll change anything without evidence that it's causing widespread problems. I have seen almost none in the ~year since you filed the last issue.

@hyangah @katiehockman

@mholt
Copy link
Author

@mholt mholt commented Sep 24, 2020

It's either a huge coincidence or a consequence of the release process.

The release process is triggered by a push of the tag, so it's not a consequence of it. It's the other way around.

it's likely we'll change anything without evidence

Well. That is unfortunate (and disappointing, for something relied upon so heavily).

@mholt mholt closed this Sep 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.