Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: 'go mod why' should have an answer for every module in 'go list -m all' #27900

Open
bcmills opened this issue Sep 27, 2018 · 5 comments
Open

Comments

@bcmills
Copy link
Member

@bcmills bcmills commented Sep 27, 2018

Currently, if you run go mod why -m on the output of every module in go list -m all, some modules may come back with the answer (main module does not need module […]), even after a go mod tidy.

The reason is that the module graph is conservative: go list -m all answers the question “which module versions are implied by the requirements of the main module?”, not “which modules does the main module need to download to complete go list all?”.¹

In contrast, go mod why explicitly ties its answer to go list all.

We should have some flag to go mod why to answer the related go list -m all question, “what path of requirements imposes the version requirement on module x?”


¹The command to answer the latter question, as it turns out, is:

go list -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}' all | sort -u
@myitcv
Copy link
Member

@myitcv myitcv commented Nov 12, 2018

In contrast, go mod why explicitly ties its answer to go list all.

I don't think this is true, because go list all is subject to build constraints. The statement would be true if we were to say the union of go list all for all possible build constraints, or the hypothetical go list -nobuild all which would enable go list to list package irrespective of build constraints.

Consider github.com/myitcvscratch/depanalysis/cmd/analyzer. The main package whose import path coincides with the module path effectively sits behind a js,wasm build constraint. As does one of its dependencies.

This is therefore unsurprising:

$ go list
can't load package: package github.com/myitcvscratch/depanalysis/cmd/analyzer: build constraints exclude all Go files in /tmp/depanalysis/cmd/analyzer

If we provide the constraints we can list the main package:

$ GOOS=js GOARCH=wasm go list
github.com/myitcvscratch/depanalysis/cmd/analyzer

Similarly, unless we satisfy the constraints, go list all won't work either:

$ go list -f "{{if not .Standard}}{{.ImportPath}}{{end}}" all | sort -u
go: warning: "all" matched no packages

Again, fixed:

$ GOOS=js GOARCH=wasm go list -f "{{if not .Standard}}{{.ImportPath}}{{end}}" all | sort -u
github.com/myitcvscratch/depanalysis/cmd/analyzer
github.com/myitcvscratch/depanalysis/jsdep
github.com/myitcvscratch/depanalysis/normdep

go list -m is not subject to build constraints; it's operating at the module level, and therefore outside the build:

$ go list -m all
github.com/myitcvscratch/depanalysis/cmd/analyzer
github.com/myitcvscratch/depanalysis v0.0.1
github.com/myitcvscratch/depanalysis/distant v0.0.1

Hence it is not equivalent to a go list -f where we output the module path:

$ go list -f "{{if not .Standard}}{{.Module.Path}}{{end}}" all | sort -u
go: warning: "all" matched no packages

Because when listing packages we need to consider build constraints:

$ GOOS=js GOARCH=wasm go list -f "{{if not .Standard}}{{.Module.Path}}{{end}}" all | sort -u
github.com/myitcvscratch/depanalysis
github.com/myitcvscratch/depanalysis/cmd/analyzer

go list -m gives the nodes of the requirement graph output by go mod graph:

$ go mod graph
github.com/myitcvscratch/depanalysis/cmd/analyzer github.com/myitcvscratch/depanalysis@v0.0.1
github.com/myitcvscratch/depanalysis@v0.0.1 github.com/myitcvscratch/depanalysis/distant@v0.0.1

The docs for go mod why say:

By default, why queries the graph of packages matched by "go list all",
which includes tests for reachable packages.

But that does not appear to be the case. go mod why instead seems to correspond to the hypothetical go list -nobuild all mentioned above:

$ go mod why github.com/myitcvscratch/depanalysis/jsdep
# github.com/myitcvscratch/depanalysis/jsdep
github.com/myitcvscratch/depanalysis/cmd/analyzer
github.com/myitcvscratch/depanalysis/jsdep

$ go mod why github.com/myitcvscratch/depanalysis/normdep
# github.com/myitcvscratch/depanalysis/normdep
github.com/myitcvscratch/depanalysis/cmd/analyzer
github.com/myitcvscratch/depanalysis/normdep

$ go mod why github.com/myitcvscratch/depanalysis/unused
# github.com/myitcvscratch/depanalysis/unused
(main module does not need package github.com/myitcvscratch/depanalysis/unused)

$ go mod why image/jpeg
# image/jpeg
github.com/myitcvscratch/depanalysis/cmd/analyzer
fmt
reflect
reflect.test
encoding/json
encoding/json.test
image
image.test
image/jpeg

$ go mod why -m github.com/myitcvscratch/depanalysis
# github.com/myitcvscratch/depanalysis
github.com/myitcvscratch/depanalysis/cmd/analyzer
github.com/myitcvscratch/depanalysis/jsdep

$ go mod why -m github.com/myitcvscratch/depanalysis/distant
# github.com/myitcvscratch/depanalysis/distant
(main module does not need module github.com/myitcvscratch/depanalysis/distant)

As you say, we need the ability to answer the question "why does github.com/myitcvscratch/depanalysis/distant appear in our module requirement graph?"

I'm less clear we need something like a -nobuild flag for go list. Instead, on the (rather dangerous) assumption my analysis correct, I think the docs for go mod why need a slight tweak.

@mwf
Copy link

@mwf mwf commented Feb 13, 2019

Please, keep in mind the case with test dependencies - would be cool if go mod why could tell if the dependency is used transitively in tests - #30206

@aofei
Copy link
Contributor

@aofei aofei commented Jan 7, 2020

And I think the note (main module does not need module ...) is flawed. If a module isn't needed by our main module, then it shouldn't cause our main module to be broken (see #36423 for example).

@gopherbot
Copy link

@gopherbot gopherbot commented Feb 19, 2020

Change https://golang.org/cl/220080 mentions this issue: design/36460: add design for lazy module loading

gopherbot pushed a commit to golang/proposal that referenced this issue Mar 5, 2020
Updates golang/go#36460
Updates golang/go#27900
Updates golang/go#26955
Updates golang/go#30831
Updates golang/go#32058
Updates golang/go#32380
Updates golang/go#32419
Updates golang/go#33370
Updates golang/go#33669
Updates golang/go#36369

Change-Id: I1d4644e3e8b4e688c2fc5a569312495e5072b7d7
Reviewed-on: https://go-review.googlesource.com/c/proposal/+/220080
Reviewed-by: Russ Cox <rsc@golang.org>
@stapelberg
Copy link
Contributor

@stapelberg stapelberg commented Apr 29, 2020

“which modules does the main module need to download to complete go list all?”

¹The command to answer the latter question, as it turns out, is:

go list -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}' all | sort -u

I don’t think that’s correct. When I run this command in ~/distri, and ensure that the reported modules are present in the reported versions in my build environment, I’m encountering an error when running go install ./cmd/...:

go: golang.org/x/exp@v0.0.0-20200331195152-e8c3332aa8e5 requires
	dmitri.shuralyov.com/gpu/mtl@v0.0.0-20190408044501-666a987793e9: module lookup disabled by GOPROXY=off

Package dmitri.shuralyov.com/gpu/mtl is only used from files behind the darwin build tag (I’m working with linux), so that’s likely why go list all doesn’t show it, but it seems like go install still requires it to be present. #29410 (comment) is related.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants