cmd/go: 'go get all' no longer works when a deleted transitive dependency is no longer needed after upgrading

[bcmills]

go get -d all used to work in some situations in which it no longer does.

It appears that as of Go 1.16, it first resolves all to a package pattern, then attempts to install the latest version of each such package. If some of those packages have been deleted, it fails.

In Go 1.15 and earlier, it did something closer to what go get -u -t ./... does today: it upgraded (some subset of) direct imports and then upgraded transitive dependencies from there.

As a workaround, go get -d -u -t ./... achieves essentially the same set of upgrades, but go get -d all ought to be a reasonable alternative.

$ go1.17rc1 get -d all
go get all: module google.golang.org/protobuf@upgrade found (v1.27.1), but does not contain package google.golang.org/protobuf/internal/fieldsort
go get all: module google.golang.org/protobuf@upgrade found (v1.27.1), but does not contain package google.golang.org/protobuf/internal/mapsort

$ go1.16.6 get -d all
go get all: module google.golang.org/protobuf@upgrade found (v1.27.1), but does not contain package google.golang.org/protobuf/internal/fieldsort
go get all: module google.golang.org/protobuf@upgrade found (v1.27.1), but does not contain package google.golang.org/protobuf/internal/mapsort

$ go1.15.14 get -d all
go: github.com/google/go-cmp upgrade => v0.5.6
go: google.golang.org/protobuf upgrade => v1.27.1
go: golang.org/x/xerrors upgrade => v0.0.0-20200804184101-5ec99f83aff1

$ go1.17rc1 get -d -u -t ./...
go get: upgraded google.golang.org/protobuf v1.25.0 => v1.27.1

-- go.mod --
module example.com/m

go 1.16

require google.golang.org/protobuf v1.25.0
-- main.go --
package main

import _ "google.golang.org/protobuf/proto"

[bcmills]

CC @jayconrod @matloob

[bcmills]

Since nobody reported this against Go 1.16, and given that go get -d -u -t ./... works as a workaround, I suspect that it is not severe enough to warrant a backport. (That said, if the fix patches cleanly onto Go 1.16 I think it would be fine to backport too. Go 1.15 is not affected either way.)

[ianlancetaylor]

@bcmills @matloob This issue is marked for 1.19. It's just been rolling forward in milestones. Should it move to Backlog?

[gopherbot]

Change https://go.dev/cl/449035 mentions this issue: all: update all dependencies to @latest

[bcmills]

Note that the -d flag no longer exists (it is the default), but this bug does still exist: go get all will first identify the existing packages matching the all pattern, then try to resolve their transitive dependencies — even if intermediate steps in resolving those dependencies end up changing which packages are in all.

Probably cmd/go/internal/modget ought to treat the all pattern specially so that it can re-resolve the pattern at each iteration.