Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: 'go mod download' pulls in test dependencies of dependencies #41431

Open
prashantv opened this issue Sep 16, 2020 · 4 comments
Open

cmd/go: 'go mod download' pulls in test dependencies of dependencies #41431

prashantv opened this issue Sep 16, 2020 · 4 comments
Labels
Milestone

Comments

@prashantv
Copy link
Contributor

@prashantv prashantv commented Sep 16, 2020

What version of Go are you using (go version)?

$ go version
go version go1.15.2 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/prashant/.cache/go-build"
GOENV="/home/prashant/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/prashant/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/prashant/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/prashant/.gimme/versions/go1.15.2.linux.amd64"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/prashant/.gimme/versions/go1.15.2.linux.amd64/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/prashant/go/src/github.com/prashantv/go-mod-download-repro/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build327180583=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Run go mod download for a module (called go-mod-download-repro in the example) that takes on a dependency on another module (go.uber.org/atomic in the example), which only has external dependencies in tests.

go.mod:

module github.com/prashantv/go-mod-download-repro

go 1.14

require go.uber.org/atomic v1.7.0

main.go:

package main

import (
        "fmt"

        "go.uber.org/atomic"
)

func main() {
        var i atomic.Int32
        i.Inc()
        fmt.Println(i.Load())
}

go.uber.org/atomic has test dependencies on testify.

Download to a local directory to see what is downloaded: $ GOMODCACHE=$(pwd)/cache go mod download

What did you expect to see?

Expected to see all dependencies of go-mod-download-repro to be downloaded, but not the test dependencies of go.uber.org/atomic.

What did you see instead?

Only dependenceis required to build/test go-mod-download-repro.

If this behaviour is intentional, would it be possible to add an option to limit the download to just packages needed to build, or to packages needed to test the current module?

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Sep 16, 2020

cc @bcmills @matloob

This will be resolved by lazy module loading (#36460), but only when the main module and all of its dependencies have go 1.16 or later in their go.mod files.

It's expected behavior, but it does overfetch, and maybe we can improve it. Perhaps, in addition to #36460, we could narrow the scope of go mod download to just modules that provide packages in all (the new all in 1.16, i.e., packages needed to build packages in the main module, and tests of those packages).

@jayconrod jayconrod added this to the Backlog milestone Sep 16, 2020
@bcmills
Copy link
Member

@bcmills bcmills commented Sep 16, 2020

go mod download intentionally does not load packages: it examines only the module graph (both as implemented today, and with lazy module loading).

If you want to download only those modules that provide packages in all, you can run go list all with either -json or a suitable -f format string, which will list the corresponding modules. (go list all will download the relevant modules as a side-effect.)

@bcmills
Copy link
Member

@bcmills bcmills commented Sep 16, 2020

(And note that if your go.mod file specifies go 1.16 or higher, go list all already omits the dependencies of tests outside your module when using a go command built from head, such as golang.org/dl/gotip.)

@bcmills
Copy link
Member

@bcmills bcmills commented Sep 17, 2020

Finally, today you should be able to use go list -test -deps ./... to download only the modules transitively imported by the packages in your module.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.