Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: behavior of go list ... is not documented in module mode #37227

Open
perillo opened this issue Feb 14, 2020 · 17 comments
Open

cmd/go: behavior of go list ... is not documented in module mode #37227

perillo opened this issue Feb 14, 2020 · 17 comments

Comments

@perillo
Copy link

@perillo perillo commented Feb 14, 2020

go help packages documents the special path all both for gopath and module mode:

"all" expands to all packages found in all the GOPATH trees. For example, 'go list all' lists all the packages on the local system. When using modules, "all" expands to all packages in the main module and their dependencies, including dependencies needed by tests of any of those.

However the ... pattern is documented only for gopath module:

An import path is a pattern if it includes one or more "..." wildcards, each of which can match any string, including the empty string and strings containing slashes. Such a pattern expands to all package directories found in the GOPATH trees with names matching the patterns.

How is the ... pattern supposed to work in module mode?

As a test, I ran go list ... | wc -l inside the github.com/golang/mod (golang.org/x/mod) directory with different versions of the go tool. The directory is outside GOPATH. The results are:

  • go1.11.13: 501
  • go1.12.16: 526
  • go1.13.8: 632
  • go1.14rc1: 634

As a side note, how can I tell go list to list only packages in the main module?

@perillo

This comment has been minimized.

Copy link
Author

@perillo perillo commented Feb 14, 2020

Unrelated to this issue, I also noted that with GO111MODULE=off and GOPATH set to a not existing directory:

  • go1.13.8 list ...: 300
  • go1.13.8 list std cmd: 376

This seems to contrast with the documentation saying Such a pattern expands to all package directories found in the GOPATH, since it is matching package directories in GOROOT.

The result is the same with GO111MODULE=on and inside an empty module (where go list all matches no packages).

@dmitshur dmitshur added this to the Backlog milestone Feb 14, 2020
@dmitshur

This comment has been minimized.

Copy link
Member

@dmitshur dmitshur commented Feb 14, 2020

@perillo

This comment has been minimized.

Copy link
Author

@perillo perillo commented Feb 14, 2020

I think that go list ... should never match packages outside the main and active modules.

The difference between go list ... and go list std cmd is that the latter also includes the vendor directories.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Feb 14, 2020

It is true that we have not documented the ... package pattern in module mode. The intent is for it to have the same meaning as in GOPATH mode: every package that can be found in any of the available package roots. (In GOPATH mode, the available package roots are GOROOT/src, and GOPATH/src for each entry in GOPATH. In module mode, the available package roots are GOROOT/src and each active module.)

The increase from Go 1.11 to 1.12 is due to the change from vendor/golang_org/x to internal/x for packages vendored into the standard library (see CL 147443).

The increase from Go 1.12 to 1.13 is due to the inclusion of the cmd subtree plus the addition of crypto/ed25519. (The previous lack of cmd packages was a bug present since Go 1.11: note that the cmd subtree was always included in GOPATH mode.)

The minor changes from Go 1.13 to 1.14 are due to changes in internal packages plus the addition of hash/maphash.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Feb 14, 2020

The difference between GOPATH mode and module mode in Go 1.13 is due to the requirement that each package have a unique path in module mode: for example, the package reported as cmd/vendor/golang.org/x/sys/unix would be just one of many possible packages known as golang.org/x/sys/unix in GOPATH mode.

Arguably the lack of such a listing entirely in GOPATH mode is a bug, but it is a fairly minor one and at this point almost certainly not worth fixing.

@perillo

This comment has been minimized.

Copy link
Author

@perillo perillo commented Feb 14, 2020

There is still something missing, in module mode.

For go1.13, the number of packages in GOROOT is 376 (and ... only matches 300 packages).

Inside the golang.org/x/mod module:

  • go1.13 list all: 170
  • go1.13 list std cmd all: 392
  • go1.13 list std cmd ...: 708

Am I missing something?

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Feb 14, 2020

Per https://tip.golang.org/cmd/go/#hdr-Package_lists_and_patterns:

When using modules, "all" expands to all packages in the main module and their dependencies, including dependencies needed by tests of any of those.

Since no package in the main module can import a package main from cmd, that is a strict subset of ....

@perillo

This comment has been minimized.

Copy link
Author

@perillo perillo commented Feb 14, 2020

Is it only me that thinks this is really complex?

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Feb 14, 2020

The ... pattern is just not all that useful. It's an edge-case of a general wildcard. So, yeah: edge cases are complex, but generally avoidable. (The all pattern is usually what you want instead, and that one is well-defined.)

@perillo

This comment has been minimized.

Copy link
Author

@perillo perillo commented Feb 14, 2020

It could be make useful if ... matches only the packages in the main module, since there are no other means to get that list.

@myitcv

This comment has been minimized.

Copy link
Member

@myitcv myitcv commented Feb 14, 2020

@perillo can you not do

go list $(go list -m)/...
@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Feb 14, 2020

I think we do need a pattern that matches only the packages in the main module, but ... is not that pattern.

(./... is such a pattern, but it only works in the module root.)

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Feb 14, 2020

@myitcv: go list $(go list -m)/... will catch packages in nested modules too.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Feb 14, 2020

@jayconrod, @matloob, and I were discussing such a pattern this morning, in fact.

(We're thinking maybe mod, since it's a three-letter shortening like std and cmd and less likely than main to be used as the name of someone's local toy module.)

But that really ought to be filed as a separate issue.

@perillo

This comment has been minimized.

Copy link
Author

@perillo perillo commented Feb 14, 2020

@bcmills I tested go list $(go list -m)/... from inside the golang.org/x/tools directory and it does not list the golang.org/x/tools/gopls sub module.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Feb 14, 2020

It would if x/tools depended on gopls, which it (intentionally) does not.

I think cloud.google.com/go has that sort of structure, though.

@myitcv

This comment has been minimized.

Copy link
Member

@myitcv myitcv commented Feb 14, 2020

Ok, one last try 😄

go list -f "{{\$p := .}}{{with \$p.Module}}{{if eq .Path \"$(go list -m)\"}}{{\$p.ImportPath}}{{end}}{{end}}" $(go list -m)/...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.