cmd/go: GOPROXY default can make 'get -u' lag upstream repository

[mvdan]

What version of Go are you using (go version)?

$ go version
go version devel +623d653db7 Sat Jun 29 13:17:15 2019 +0000 linux/amd64

Does this issue reproduce with the latest release?

No, because Go 1.12 doesn't have a GOPROXY set up by default.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN="/home/mvdan/go/bin"
GOCACHE="/home/mvdan/go/cache"
GOENV="/home/mvdan/.config/go/env"
GOEXE=""
GOFLAGS="-ldflags=-w"
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/mvdan/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org"
GOROOT="/home/mvdan/tip"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/mvdan/tip/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build558320792=/tmp/go-build -gno-record-gcc-switches"
GOROOT/bin/go version: go version devel +623d653db7 Sat Jun 29 13:17:15 2019 +0000 linux/amd64
GOROOT/bin/go tool compile -V: compile version devel +623d653db7 Sat Jun 29 13:17:15 2019 +0000
uname -sr: Linux 5.1.15-arch1-1-ARCH
/usr/lib/libc.so.6: GNU C Library (GNU libc) stable release version 2.29.
gdb --version: GNU gdb (GDB) 8.3

What did you do?

Module A and B live in separate repositories, and they are both v0. A depends on B.

I push a commit to B, and then run go get -u in A to update the dependency to that new master commit. Alternatively, tried go get B@master.

What did you expect to see?

A's go.mod updated to show B's latest pseudo-version with the master commit just pushed.

What did you see instead?

B's version staying at an older version, pushed hours or days before. Presumably because proxy.golang.org caches the version for @latest and @master.

This went unnoticed for a while and someone was scratching their head until they realised go get -u hadn't done what they were used to.

GOPROXY=direct go get -u had to be used to work around this issue in the end. The other option was to manually copy the commit hash to force the proxy to fetch that git ref, which is a bit cumbersome.

In the end, I worry that the new GOPROXY default in 1.13 is going to confuse people who are used to pushing to multiple repos that depend on each other. I understand why the proxy server can't fetch from git repeatedly every single time, but I wonder if there's something we can do to not silently break this.

/cc @bcmills @heschik @jayconrod

[myitcv]

@mvdan @rogpeppe and I were discussing this offline including various alternatives to the GOPROXY=direct "solution".

I think there is a change in workflow here that might affect a not-insignificant number of users. That's just a feeling, because I don't have specific numbers.

In 1.12, go get -u works in the scenario described.

In 1.13, are we saying GOPROXY=direct go get -u is only guaranteed way to deal with the outlined scenario?

[mewmew]

I think there is a change in workflow here that might affect a not-insignificant number of users. That's just a feeling, because I don't have specific numbers.

Not that this adds much to the discussion, but I'd be one of those users. Thanks for raising this issue.

[bcmills]

One option would be for the proxy not to serve branch names at all. It could serve a 404 or 410 for the .info file for the branch, and then the default fallback to direct would resolve that to the correct version.

We would lose some of the efficiency benefits of the proxy that way, but those benefits are most important for transitive dependencies — and transitive dependencies should always be pre-resolved to proper versions or pseudo-versions anyway.

CC @hyangah @katiehockman

[myitcv]

One option would be for the proxy not to serve branch names at all.

This would presumably include @latest so that also falls back to direct?

[hyangah]

@bcmills That is an interesting idea. Will the fallback to 'direct' mode be permanent? It seems like go commands are starting to depend on the fallback more and more.
What do you think about tags which don't look like semver or incompatible?
If we decide to take this path, maybe the go command even stops sending requests to proxy.golang.org for such requests to avoid extra network round trips.

@myitcv Is @latest version meant the one from the go command's perspective (the default when version is not specified), or the proxy endpoint /@latest for pseudo versions?
The go command currently picks up the latest version from the proxy's /@v/list endpoints and if there is no released version, the go command hits /@latest version to find any usable latest pseudo version.

If you meant the former, most go users will start fetching modules, packages, programs without version specified and the go command will treat them (and some of the dependencies) as @latest. As a result, the users will still face the problems proxy wanted to solve. (e.g. left-pad issues, vcs client requirements, ...).

--
Someone suggested a GONOPROXY trick for the repos I am actively working and testing on.

That will cause the module I am working on to be fetched directly (without any roundtrip to the proxy) but does verification against sum.golang.org.

We are also planning to improve data freshness by utilizing the new 1.13 feature (#32239) to detect the latest pseudo version.

[tbpg]

If we decide to take this path, maybe the go command even stops sending requests to proxy.golang.org for such requests to avoid extra network round trips.

I really like this idea. I'm slightly concerned about different versions of go returning different versions for @latest. The true latest is probably better than the proxy's latest, though.

[myitcv]

@hyangah

... for pseudo versions?

I'm not sure this issue is specific to pseudo versions is it? Because it occurs whenever a VCS releases a new version that the proxy has not yet seen, pseudo-version or otherwise.

The issue (at least the aspect I'm worried about) is that in in 1.13, we are saying users either have to:

• GOPROXY=direct go get -u
• or, as you suggested, set GONOPROXY

to avoid getting a stale latest version.

Both solutions work, but both represent a change to what is, I suspect, a common workflow.

I just want to be sure a) my understanding is correct and b) we are comfortable with pushing this change in workflow.

[hyangah]

@myitcv For non-pseudo versions, proxy.golang.org includes it in the /@v/list results as soon as it observes the new version. The module owners can teach proxy.golang.org about their new released version by simply running go mod download or go get with the newly released version. I hope that becomes part of the best practice for module release. The future go release tool may be helpful.

Once the semver-tagged module lands in the proxy.golang.org, it should be available to other users almost immediately (a couple of minutes in worst case). Otherwise, please file a bug.

[bcmills]

@hyangah

Will the fallback to 'direct' mode be permanent? It seems like go commands are starting to depend on the fallback more and more.

I'm not sure. I certainly expect it to be around for a while, at least.

What do you think about tags which don't look like semver or incompatible?

It seems fairly harmless to let those drop to direct too, since the go command will resolve them to canonical versions or pseudo-versions in the steady state.

If we decide to take this path, maybe the go command even stops sending requests to proxy.golang.org for such requests to avoid extra network round trips.

That would be fine for proxy.golang.org, but not for proxies in general: for private proxies we still want to give the proxy the opportunity to reject the module outright (for security, privacy, or other policy reasons).

[bcmills]

This would presumably include @latest so that also falls back to direct?

Probably not, for the reasons @hyangah mentioned. Plus, @latest is not as rare as, say, a branch-name or non-canonical tag: if someone runs go get -u they'll end up fetching the latest version of potentially a very large number of modules, so it's more important to get the efficiency boost from the proxy.