cmd/go: 'go list -m all' doesn't diagnose conflicts in non-imported packages

[yihuang]

Currently vgo2nix is using go list -mod mod -json -m all to find out all the dependencies of current project, but we find this command will output conflicts modules.

A simple example:

cat > go.mod << EOF
module testmod1

go 1.15

require (
	github.com/ugorji/go v1.1.4 // indirect
	github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8 // indirect
)
EOF
cat > main.go << EOF
package main

import "fmt"

func main() {
	fmt.Println("hello world")
}
EOF

These too dependencies are in conflict, but not imported by code, so go build and go mod vendor runs just fine.
But go list -mod mod -m all will list them both.

Is it considered an issue of go list, or is there a command to serve our purpose?

[andybons]

@jayconrod @bcmills @matloob

[bcmills]

go mod vendor copies in non-test dependencies matching the all package pattern. go list -m all reports the all module pattern.

The package namespace and module namespace are related, but not quite the same. The all module pattern includes (as you have seen) all of the dependencies imposed by the go.mod file, but the all package pattern (the one used by go build all and go mod vendor!) examines only those packages transitively imported by the main module. So there is no conflict in the dependencies of this module, because the only package that conflicts is not in the all package pattern.

The general solution is to give the go command the source code that you intend to build, so that go mod vendor can trace the import statements accurately. If that general solution does not suit your needs, we'll need to know more about the concrete problem you're trying to solve.

[yihuang]

The general solution is to give the go command the source code that you intend to build, so that go mod vendor can trace the import statements accurately. If that general solution does not suit your needs, we'll need to know more about the concrete problem you're trying to solve.

what vgo2nix do is try to get the locked dependency version informations from current source directory, and convert that information to nix expression and build in GOPATH mode, to get a more re-producible build (from nix perspective).
so i think maybe one possibility is to work on the all package list, and try to figure out the module list and versions?
another approach is just rely on go mod vendor, and treat the whole vendor directory as dependency, which is taken by buildGoModule, I wonder if the vgo2nix approach gonna work out?

[bcmills]

The all package list seems like a good starting point. Once we have lazy loading (#36460), you may be able to do something simpler using the original go.mod file and replace directives.