cmd/go: improve error message when a module is successfully fetched but not in the sum db

[tbpg]

What version of Go are you using (go version)?

$ go version
go version devel +0f897f916a Tue May 28 02:52:39 2019 +0000 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/tbp/.cache/go-build"
GOENV="/home/tbp/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/tbp/go"
GOPROXY="https://proxy.golang.org"
GOROOT="/home/tbp/code/gotip"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/tbp/code/gotip/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/tmp/bar/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build702923268=/tmp/go-build -gno-record-gcc-switches"
GOROOT/bin/go version: go version devel +0f897f916a Tue May 28 02:52:39 2019 +0000 linux/amd64
GOROOT/bin/go tool compile -V: compile version devel +0f897f916a Tue May 28 02:52:39 2019 +0000
uname -sr: Linux 4.19.28-2rodete1-amd64
Distributor ID:	Debian
Description:	Debian GNU/Linux rodete
Release:	rodete
Codename:	rodete
/lib/x86_64-linux-gnu/libc.so.6: GNU C Library (Debian GLIBC 2.24-12) stable release version 2.24, by Roland McGrath et al.

What did you do?

$ GOPROXY=https://proxy.golang.org GONOPROXY=github.com/tbpg go get github.com/tbpg/modules-testing

verifying github.com/tbpg/modules-testing@v0.0.0-20190528133524-a4881321cb96/go.mod: github.com/tbpg/modules-testing@v0.0.0-20190528133524-a4881321cb96/go.mod: reading https://sum.golang.org/lookup/github.com/tbpg/modules-testing@v0.0.0-20190528133524-a4881321cb96: 410 Gone

What did you expect to see?

I'm not sure we should make this change (might not want to make it obvious, for some sense of the word, how to disable the sum db).

The solution, in this case, is to add GONOSUMDB=github.com/tbpg. Could we mention GONOSUMDB in the error message?

What did you see instead?

410 Gone

cc @FiloSottile @bcmills

Related to #32184, which would have prevented this error.

[bcmills]

CC @jayconrod @rsc

[bcmills]

I'm on the fence about this one, but suggested that @tbpg file an issue regardless.

We don't want to train users to add GONOSUMDB entries without considering the implications whenever the proxy returns a 404 or 410, but we do want users to be aware of GONOSUMDB when it is actually appropriate.

A missing sumdb entry seems much more likely to result from a private repo than from a genuine MITM attack, so GONOSUMDB is probably going to be the first hit from a Google or StackOverflow search regardless.

[FiloSottile]

At the very least there needs to be a speed bump, like having to click on a link to the wiki. We don't want "here's some error about something checksum, here's the line to get on with your day". For example:

If you are trying to fetch a private module, see https://golang.org/wiki/PrivateModules. Otherwise, please report this issue at https://golang.org/issue/new.

[tbpg]

Makes sense to me! Thanks.

Next steps are to figure out the right location for that documentation, write it, then update this error message to link to it. No ETA yet.

[rsc]

We should definitely detect the 404/410 and print a nicer error.

[djgilcrease]

This is going to make it very hard to use with any private repo (internal git.company.local, or just private repo on github/gitlab). I do not, and should not need to maintain an env var that is a comma separated lists of private repos just to be able to build my golang project, and setting GOSUMDB to off does not seem advisable either. I need something in go.mod that will skip the sumdb stuff so when I run go mod download which I cannot pass insecure to will work without me needing to maintain a comma seperated list of private repos and set the env var

[thepudds]

@djadala

should not need to maintain an env var

Just to make sure you are aware of options, you can choose to do something akin to:

go env -w 'GOPRIVATE=*.corp.example.com'

...which persists that configuration information so that you don't need to set an actual environment variable. From the doc:

The -w flag requires one or more arguments of the form NAME=VALUE and changes the default settings of the named environment variables to the given values.

Some more details here:

https://tip.golang.org/cmd/go/#hdr-Module_configuration_for_non_public_modules

[djgilcrease]

go env -w 'GOPRIVATE=*.corp.example.com'

sure, for my local box, but in build systems I need the env var, and I need to document this in the readme or some such and maintain it since this is not maintained in the source of truth on the modules go.mod

Maybe add something to to the go.mod to maintain this

go 1.13
require (
    gitlab.company.local/namespace/group/project/lib v1.0.0+private
    gitlab.company.local/namespace/group/project2/lib v2.0.0+private.incompatible
)

[icemarkom]

For what is worth, I just spent a better part of the evening and the morning trying to find a solution to the problem of "401 gone" for the module in my private GitHub repo.

I believe this could be better documented, and also believe that an environment variable is a very blunt tool to use for this purpose. It would be better to have a control from go.mod. Perhaps it would be cleaner if there was a separate section from the require. Something along these lines:

go 1.13

private (
  private.repo/namespace/module1
  private.repo/namespace/module2
)

require (
  ...
)

[FiloSottile]

Using the go.mod is being discussed in #33985. This issue is about the error message.

[bcmills]

Probably the error message should direct users to some (stable) documentation link.

Figuring out the stable links for module documentation is #33637.