cmd/go: provide straightforward way to see non-test dependencies

[rogpeppe]

It's important to be able to see the full set of dependencies used by production code exclusive of tests, particularly with different major versions as distinct modules, as inadvertently using an import with a different major version can introduce bugs (also, some external dependencies can be toxic to have in production code).

The go mod why command provides a -vendor flag to exclude tests; perhaps go list could provide a similar flag to exclude testing dependencies.

In addition to the above, it is also useful to be able to be able to see modules used by local production code and tests, but not transitively by tests in external dependencies, as external dependencies are harder to change, and inadvertently forked modules are still an issue for tests too. I'm not sure of a good way to specify that, though.

[thepudds]

An example where a benchmark-only dependency appearing in go.mod caused consternation:
https://github.com/scylladb/go-set/pull/3/files#r208130318

In that example, one of the main points of this new project is to resuscitate a dead project (in a cleaner form), and there was surprise at seeing the dead project listed as a dependency of the new project. (There was surprise and consternation on the part of the authors, but one could also imagine that being a deterrent to a user evaluating whether or not to download and/or use this new project if it appears the new project relies on a dead project).

[flibustenet]

We should see if it's better to have this information directly in the go.mod file (like // test or separate sections of require). Or if it's better to use a cmd or a web site like godoc.org ?

[myitcv]

@flibustenet - just in case you haven't seen #26913 (comment) yet.

I think it's worth focussing on what people are trying to achieve here... which I agree has some overlap with their workflow and hence the tools they are using, but in the case of using godoc.org, I believe the underlying question that someone is trying to answer is "what dependencies does module M have?" and making some judgement on whether or not to use M using that information. Getting more detail on that will, I think, help to steer us towards the building blocks we then need in place.

As I noted on Slack:

Because all of these sorts of queries are possible using go list.... but the very fact we're having this discussion means that that solution is probably too esoteric. And if the queries are common enough it probably makes sense to provide something more accessible for day-to-day use

That said, we don't need to load everything into the go tool; if it provides the building blocks for some other solution that answers these same questions but via a different UI/UX then that's just as good.

[rogpeppe]

As an example of the issue here, I just noticed a dependency on github.com/jtolds/gls which I wasn't expecting. The go mod why output looked like this:

# github.com/jtolds/gls
github.com/juju/charmstore-client/cmd/charm
github.com/juju/juju/juju/osenv
github.com/juju/juju/juju/osenv.test
github.com/juju/juju/testing
github.com/juju/juju/api/block
github.com/juju/juju/api/block.test
github.com/juju/juju/state
github.com/juju/juju/state.test
github.com/juju/juju/component/all
github.com/juju/juju/cmd/juju/commands
github.com/juju/juju/provider/all
github.com/juju/juju/provider/ec2
gopkg.in/ini.v1
gopkg.in/ini.v1.test
github.com/smartystreets/goconvey/convey
github.com/jtolds/gls

So the shortest path to this dependency takes us through four levels of external test dependencies. In practice I almost never care about test dependencies of external dependencies.

[wsc1]

Hi, I'm redirecting the comment here

Russ's comment about scopes is compelling to have unified go.mod, but not compelling to hide scopes (build tags x test).

I think having go mod why not refer to scopes can be confusing or worse: go mod why is arguably not providing the functionality advertised (i.e. explaining why) if it doesn't also give scope information, at least for the main module.

[dmitris]

I would like to be able to look at go.mod and find out which packages are needed to build and run tests in the current module. In particularly, this is needed if you have a policy that all open-source / "external" (outside of your company control) packages must be mirrored on internal servers.

One idea is for go to add comments like github.com/gopherjs/gopherjs v0.0.0-20181004151105-1babbf986f6f // indirect-test for indirect dependencies that are dependencies from tests in the dependencies' modules. Then one could look at go.mod and see that packages/modules marked as // indirect-test are needed only if you want to run go test all or go test ... but not for the "regular" workflow of go install -v .... && go test -v ./... (to build and test your own code).

[flibustenet]

Tests dependencies should be visible, even for direct test dependencies. It's a valuable information. I'd like to choose a lib with as few dependency as possible to run but the most tests (and test dependencies if needed) as possible...

[bcmills]

In particularly, this is needed if you have a policy that all open-source / "external" (outside of your company control) packages must be mirrored on internal servers.

That's probably a task better suited to GOPROXY than to go.mod. (In particular, I would expect the company to provide a GOPROXY server containing all of the approved/vetted versions of external modules.)

[wsc1]

I would like to be able to look at go.mod and find out which packages are needed to build and run tests in the current module.

I think the go tool would be better. But, in theory the problem of deciding whether or not a file has satisfiable build tags is NP-complete :)