New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: provide straightforward way to see non-test dependencies #26955

Open
rogpeppe opened this Issue Aug 13, 2018 · 12 comments

Comments

Projects
None yet
7 participants
@rogpeppe
Contributor

rogpeppe commented Aug 13, 2018

It's important to be able to see the full set of dependencies used by production code exclusive of tests, particularly with different major versions as distinct modules, as inadvertently using an import with a different major version can introduce bugs (also, some external dependencies can be toxic to have in production code).

The go mod why command provides a -vendor flag to exclude tests; perhaps go list could provide a similar flag to exclude testing dependencies.

In addition to the above, it is also useful to be able to be able to see modules used by local production code and tests, but not transitively by tests in external dependencies, as external dependencies are harder to change, and inadvertently forked modules are still an issue for tests too. I'm not sure of a good way to specify that, though.

@rogpeppe rogpeppe added the modules label Aug 13, 2018

@thepudds

This comment has been minimized.

thepudds commented Aug 13, 2018

An example where a benchmark-only dependency appearing in go.mod caused consternation:
https://github.com/scylladb/go-set/pull/3/files#r208130318

In that example, one of the main points of this new project is to resuscitate a dead project (in a cleaner form), and there was surprise at seeing the dead project listed as a dependency of the new project. (There was surprise and consternation on the part of the authors, but one could also imagine that being a deterrent to a user evaluating whether or not to download and/or use this new project if it appears the new project relies on a dead project).

@flibustenet

This comment has been minimized.

flibustenet commented Aug 13, 2018

We should see if it's better to have this information directly in the go.mod file (like // test or separate sections of require). Or if it's better to use a cmd or a web site like godoc.org ?

@myitcv

This comment has been minimized.

Member

myitcv commented Aug 13, 2018

@flibustenet - just in case you haven't seen #26913 (comment) yet.

I think it's worth focussing on what people are trying to achieve here... which I agree has some overlap with their workflow and hence the tools they are using, but in the case of using godoc.org, I believe the underlying question that someone is trying to answer is "what dependencies does module M have?" and making some judgement on whether or not to use M using that information. Getting more detail on that will, I think, help to steer us towards the building blocks we then need in place.

As I noted on Slack:

Because all of these sorts of queries are possible using go list.... but the very fact we're having this discussion means that that solution is probably too esoteric. And if the queries are common enough it probably makes sense to provide something more accessible for day-to-day use

That said, we don't need to load everything into the go tool; if it provides the building blocks for some other solution that answers these same questions but via a different UI/UX then that's just as good.

@rogpeppe

This comment has been minimized.

Contributor

rogpeppe commented Aug 22, 2018

As an example of the issue here, I just noticed a dependency on github.com/jtolds/gls which I wasn't expecting. The go mod why output looked like this:

# github.com/jtolds/gls
github.com/juju/charmstore-client/cmd/charm
github.com/juju/juju/juju/osenv
github.com/juju/juju/juju/osenv.test
github.com/juju/juju/testing
github.com/juju/juju/api/block
github.com/juju/juju/api/block.test
github.com/juju/juju/state
github.com/juju/juju/state.test
github.com/juju/juju/component/all
github.com/juju/juju/cmd/juju/commands
github.com/juju/juju/provider/all
github.com/juju/juju/provider/ec2
gopkg.in/ini.v1
gopkg.in/ini.v1.test
github.com/smartystreets/goconvey/convey
github.com/jtolds/gls

So the shortest path to this dependency takes us through four levels of external test dependencies. In practice I almost never care about test dependencies of external dependencies.

@bcmills bcmills added this to the Go1.12 milestone Sep 18, 2018

@wsc1

This comment has been minimized.

wsc1 commented Sep 29, 2018

Hi, I'm redirecting the comment here

Russ's comment about scopes is compelling to have unified go.mod, but not compelling to hide scopes (build tags x test).

I think having go mod why not refer to scopes can be confusing or worse: go mod why is arguably not providing the functionality advertised (i.e. explaining why) if it doesn't also give scope information, at least for the main module.

@dmitris

This comment has been minimized.

Contributor

dmitris commented Oct 8, 2018

I would like to be able to look at go.mod and find out which packages are needed to build and run tests in the current module. In particularly, this is needed if you have a policy that all open-source / "external" (outside of your company control) packages must be mirrored on internal servers.

One idea is for go to add comments like github.com/gopherjs/gopherjs v0.0.0-20181004151105-1babbf986f6f // indirect-test for indirect dependencies that are dependencies from tests in the dependencies' modules. Then one could look at go.mod and see that packages/modules marked as // indirect-test are needed only if you want to run go test all or go test ... but not for the "regular" workflow of go install -v .... && go test -v ./... (to build and test your own code).

@flibustenet

This comment has been minimized.

flibustenet commented Oct 8, 2018

Tests dependencies should be visible, even for direct test dependencies. It's a valuable information. I'd like to choose a lib with as few dependency as possible to run but the most tests (and test dependencies if needed) as possible...

@bcmills

This comment has been minimized.

Member

bcmills commented Oct 8, 2018

In particularly, this is needed if you have a policy that all open-source / "external" (outside of your company control) packages must be mirrored on internal servers.

That's probably a task better suited to GOPROXY than to go.mod. (In particular, I would expect the company to provide a GOPROXY server containing all of the approved/vetted versions of external modules.)

@wsc1

This comment has been minimized.

wsc1 commented Oct 18, 2018

I would like to be able to look at go.mod and find out which packages are needed to build and run tests in the current module.

I think the go tool would be better. But, in theory the problem of deciding whether or not a file has satisfiable build tags is NP-complete :)

@bcmills

This comment has been minimized.

Member

bcmills commented Nov 9, 2018

perhaps go list could provide a similar flag to exclude testing dependencies.

I'm curious: does go list -deps -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}' ./... | sort -u produce the list that you want? (It's possible that the same flag can address both this use-case and #27900.)

@myitcv

This comment has been minimized.

Member

myitcv commented Nov 12, 2018

I'm curious: does go list -deps -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}' ./... | sort -u produce the list that you want? (It's possible that the same flag can address both this use-case and #27900.)

Per my answer in #27900 (comment), I think it depends if you expect/want that answer to be irrespective of build constraints.

@rogpeppe

This comment has been minimized.

Contributor

rogpeppe commented Nov 12, 2018

I'm curious: does go list -deps -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}' ./... | sort -u produce the list that you want? (It's possible that the same flag can address both this use-case and #27900.)

Yes, this is pretty much exactly what I was after (I wasn't aware of the -deps flag). For the request in my final paragraph, go list -test -deps -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}'| sort -u does the job. Not that either of these are particularly easy to type, but I can write a script. It would be nice if go mod why could work with this call graph instead of the regular one though.

PS I'd be happier if the go command never looked at external test dependencies (they can be enormous and it's often impossible or uneconomic to run all tests in all third party modules), but that's another story.

@bcmills bcmills modified the milestones: Go1.12, Go1.13 Nov 15, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment