New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: separate direct and indirect dependencies in go.mod #27887

Open
jamesgoodhouse opened this Issue Sep 26, 2018 · 9 comments

Comments

Projects
None yet
8 participants
@jamesgoodhouse

jamesgoodhouse commented Sep 26, 2018

To summarize, the ordering of the go.mod file, with direct and indirect dependencies intermixed, makes it hard to view direct dependencies that I actually care about.

A proposed fix would be...

module github.com/org/repo

require (
	github.com/some/dependency v1.2.3
	github.com/another/dependency v4.5.6

	github.com/some-indirect/dependency v1.2.3 // indirect
	github.com/another-indirect/dependency v4.5.6 // indirect
)

What version of Go are you using (go version)?

go1.11 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/<<REDACTED>>/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/<<REDACTED>>/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.11/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.11/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/<<REDACTED>>/go/src/<<REDACTED>>/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/lq/y1k6czrx4gld0j4wrjvy1w6r0000gn/T/go-build624209139=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Run go mod tidy in a newly initialized project.

What did you expect to see?

An orderly go.mod file, with dependencies I care about and use directly separated out from indirect dependencies.

What did you see instead?

Direct and indirect dependencies intermixed in the go.mod file, leading toward difficulty in see modules I use and care about.

@agnivade agnivade changed the title from go.mod ordering to cmd/go: separate direct and indirect dependencies in go.mod Sep 27, 2018

@agnivade agnivade added this to the Unplanned milestone Sep 27, 2018

@agnivade

This comment has been minimized.

Member

agnivade commented Sep 27, 2018

@thepudds

This comment has been minimized.

thepudds commented Oct 21, 2018

@gopherbot, please add label modules

@gopherbot gopherbot added the modules label Oct 21, 2018

@thepudds

This comment has been minimized.

thepudds commented Oct 21, 2018

@jamesgoodhouse you might also want to see the discussion in #26955 and #26913, especially this comment #26913 (comment)

@myitcv

This comment has been minimized.

Member

myitcv commented Oct 21, 2018

Note, the list of indirect dependencies in your go.mod will approach zero over time as the set of transitive dependencies adopt Go modules.

That said, a more useful way to look at your dependencies is golang.org/x/tools/cmd/digraph. e.g.

$ GO111MODULE=off go get golang.org/x/tools/cmd/digraph
$ cd /tmp
$ git clone https://github.com/gobuffalo/buffalo
Cloning into 'buffalo'...
$ cd buffalo/
$ go mod download
go: finding github.com/gobuffalo/mw-contenttype v0.0.0-20180802152300-74f5a47f4d56
go: finding github.com/gobuffalo/mw-paramlogger v0.0.0-20181005191442-d6ee392ec72e
...
$ go mod graph | digraph succs github.com/gobuffalo/buffalo
github.com/codegangsta/negroni@v1.0.0
github.com/dgrijalva/jwt-go@v3.2.0+incompatible
github.com/dustin/go-humanize@v1.0.0
github.com/fatih/color@v1.7.0
github.com/gobuffalo/buffalo-plugins@v1.6.1
github.com/gobuffalo/buffalo-pop@v1.1.2
github.com/gobuffalo/envy@v1.6.5
github.com/gobuffalo/events@v1.1.1
github.com/gobuffalo/genny@v0.0.0-20181019144442-df0a36fdd146
github.com/gobuffalo/github_flavored_markdown@v1.0.5
github.com/gobuffalo/httptest@v1.0.2
github.com/gobuffalo/makr@v1.1.5
github.com/gobuffalo/mw-basicauth@v1.0.3
github.com/gobuffalo/mw-contenttype@v0.0.0-20180802152300-74f5a47f4d56
github.com/gobuffalo/mw-csrf@v0.0.0-20180802151833-446ff26e108b
github.com/gobuffalo/mw-forcessl@v0.0.0-20180802152810-73921ae7a130
github.com/gobuffalo/mw-i18n@v0.0.0-20180802152014-e3060b7e13d6
github.com/gobuffalo/mw-paramlogger@v0.0.0-20181005191442-d6ee392ec72e
github.com/gobuffalo/mw-tokenauth@v0.0.0-20181001105134-8545f626c189
github.com/gobuffalo/packr@v1.13.7
github.com/gobuffalo/plush@v3.7.20+incompatible
github.com/gobuffalo/pop@v4.8.4+incompatible
github.com/gobuffalo/tags@v2.0.11+incompatible
github.com/gobuffalo/x@v0.0.0-20181007152206-913e47c59ca7
github.com/gorilla/context@v1.1.1
github.com/gorilla/mux@v1.6.2
github.com/gorilla/sessions@v1.1.3
github.com/kr/pty@v1.1.3
github.com/markbates/deplist@v1.0.5
github.com/markbates/grift@v1.0.4
github.com/markbates/inflect@v1.0.1
github.com/markbates/oncer@v0.0.0-20181014194634-05fccaae8fc4
github.com/markbates/refresh@v1.4.10
github.com/markbates/sigtx@v1.0.0
github.com/markbates/willie@v1.0.9
github.com/monoculum/formam@v0.0.0-20180901015400-4e68be1d79ba
github.com/nicksnyder/go-i18n@v1.10.0
github.com/pkg/errors@v0.8.0
github.com/sirupsen/logrus@v1.1.1
github.com/spf13/cobra@v0.0.3
github.com/spf13/pflag@v1.0.3
github.com/spf13/viper@v1.2.1
github.com/stretchr/testify@v1.2.2
golang.org/x/crypto@v0.0.0-20181015023909-0c41d7ab0a0e
golang.org/x/sync@v0.0.0-20180314180146-1d60e4601c6f
golang.org/x/tools@v0.0.0-20181019005945-6adeb8aab2de
gopkg.in/alexcesaro/quotedprintable.v3@v3.0.0-20150716171945-2caba252f4dc
gopkg.in/mail.v2@v2.0.0-20180731213649-a0242b2233b4

digraph -help gives more details:

digraph: queries over directed graphs in text form.

Graph format:

  Each line contains zero or more words.  Words are separated by
  unquoted whitespace; words may contain Go-style double-quoted portions,
  allowing spaces and other characters to be expressed.

  Each field declares a node, and if there are more than one,
  an edge from the first to each subsequent one.
  The graph is provided on the standard input.

  For instance, the following (acyclic) graph specifies a partial order
  among the subtasks of getting dressed:

        % cat clothes.txt
        socks shoes
        "boxer shorts" pants
        pants belt shoes
        shirt tie sweater
        sweater jacket
        hat

  The line "shirt tie sweater" indicates the two edges shirt -> tie and
  shirt -> sweater, not shirt -> tie -> sweater.

Supported queries:

  nodes
        the set of all nodes
  degree
        the in-degree and out-degree of each node.
  preds <label> ...
        the set of immediate predecessors of the specified nodes
  succs <label> ...
        the set of immediate successors of the specified nodes
  forward <label> ...
        the set of nodes transitively reachable from the specified nodes
  reverse <label> ...
        the set of nodes that transitively reach the specified nodes
  somepath <label> <label>
        the list of nodes on some arbitrary path from the first node to the second
  allpaths <label> <label>
        the set of nodes on all paths from the first node to the second
  sccs
        all strongly connected components (one per line)
  scc <label>
        the set of nodes nodes strongly connected to the specified one

Example usage:

   Show the transitive closure of imports of the digraph tool itself:
   % go list -f '{{.ImportPath}}{{.Imports}}' ... | tr '[]' '  ' |
         digraph forward golang.org/x/tools/cmd/digraph

   Show which clothes (see above) must be donned before a jacket:
   %  digraph reverse jacket <clothes.txt
@thepudds

This comment has been minimized.

thepudds commented Oct 24, 2018

@jamesgoodhouse I understand this is not exactly what you asked for, but note that go list is pretty flexible, and can show you your direct dependencies today, I believe. For example:

$ go list -f '{{if not .Indirect}}{{.}}{{end}}' -m all                        
 github.com/you/hello
 rsc.io/quote v1.5.2

That -f filter defaults to showing the current module (the first line of output), and then shows your direct dependencies (along with their current versions). If you wanted something different more focused, there is also a .Main boolean (if you wanted to exclude the main/current module), and .Path if you only wanted to the module name without the version. You can see more details in the module-specific portion of this section of the doc:

https://tip.golang.org/cmd/go/#hdr-List_packages_or_modules

Here is the go.mod for that particular above example:

$ cat go.mod
module github.com/you/hello

require rsc.io/quote v1.5.2

In contrast, for that same example above, a go list -m all also shows indirect dependencies:

$ go list -m all
github.com/you/hello
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c
rsc.io/quote v1.5.2
rsc.io/sampler v1.3.0

Again, I understand it might be more convenient to be able to more easily see direct vs. indirect in go.mod itself (for example if you are browsing someone else's repo), but wanted to at least mention some things that are supported today in the core go tooling already.

@DemonWav

This comment has been minimized.

Contributor

DemonWav commented Nov 1, 2018

The build definition file in any system should be easily readable and understandable. I understand that typically go.mod is read and handled by the go tool, and that the need for // indirect dependencies will decrease as adoption of modules increases. However, this is still something that we will have to deal with for a while, and requiring an external tool just to read your dependencies easily isn't a good solution.

Cluttering the require block with // indirect dependencies surrounding the direct dependencies (as they are sorted alphabetically) makes it much more difficult to actually find the direct dependencies, which is the more relevant information for people when looking at the file.

Simply adding a blank line and separating the direct and indirect dependencies would go a long way in cleaning up the presentation of the go.mod file.

@myitcv

This comment has been minimized.

Member

myitcv commented Nov 2, 2018

Adding some notes per a discussion with @DemonWav on Slack.

In my experience with modules thus far, a) I never made changes to a go.mod by hand and b) any questions I have about my dependencies I ask via tools.

I agree that it's easier to glance at your go.mod to check your immediate dependencies (ignoring // indirect for a second), but I don't know what question it helps you answer. Because that list includes your test dependencies... and doesn't include your transitive non-test dependencies.

@bcmills

This comment has been minimized.

Member

bcmills commented Nov 2, 2018

@DemonWav, the go.mod file oriented toward machines, not people. go list is for people, and even then you need to be specific about which question you actually want to answer. (The common questions tend to get their own flags, so feedback on which operations are useful-but-difficult is valuable.)

To answer the question “what are the modules that provide the direct imports of my non-test packages?”, you could do something like:

go list -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}' \
	$(go list -f '{{range .Imports}}{{if ne . "C"}}{{.}} {{end}}{{end}}' ./...) \
	| sort -u

See also #27900.

@flibustenet

This comment has been minimized.

flibustenet commented Nov 2, 2018

It's valuable to can read go.mod on github before cloning the lib. To have an idea of the dependencies of a new lib for example.

https://research.swtch.com/vgo-module
| My goals for the file format were that it be (1) clear and simple, (2) easy for people to read

However about the annoying // indirect it's a motivation to send PR to make the old libs module aware!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment