Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: add a flag to ignore build constraints when listing packages #42504

Open
mvdan opened this issue Nov 11, 2020 · 12 comments
Open

cmd/go: add a flag to ignore build constraints when listing packages #42504

mvdan opened this issue Nov 11, 2020 · 12 comments

Comments

@mvdan
Copy link
Member

@mvdan mvdan commented Nov 11, 2020

I realise that the point of go list is to list packages, and I realise that one must generally obey build constraints when loading packages. For example, it makes no sense to try to type-check all the Go files in a package while ignoring build constraints, because there will likely be duplicate definition errors if declarations are split by GOOS or GOARCH.

Having said that, it can be very useful to merely list or traverse a set of packages in a way that build constraints are completely ignored. The most common use case is: what packages are potentially imported by the current package, directly or indirectly, across all build configurations?

This question is valid, for example, if one wants to copy a package and all of its transitive package dependencies. go list -deps <package> does not work in general, because if I run the command on Linux, I would not be including imports which are only used on Windows.

The closest thing we have right now is go mod vendor, which copies all transitively imported packages by all the packages in the current module into the vendor folder, across all build constraints. So cmd/go already has the machinery to traverse a package dependency graph while ignoring build constraints, it seems.

--

Here ends the problem statement, and begins my initial suggestion for a solution: a go list flag to ignore build constraints. Here are some example use cases:

  • List all the packages potentially depended on by a given package: go list -deps -newflag <package>

  • List all the packages under the current directory tree, not just for the current platform: go list -newflag ./...

I'm not sure what this new flag could be called. Perhaps -nobuild or -anybuild.

The flag would also restrict what go list can do. For example, using -newflag along with -export or -compiled would be an error, because it does not make sense to load/build packages when we're ignoring build constraints. Similarly, -newflag -json would always omit some fields like IgnoredGoFiles, since they make sense only when following build constraints.

This idea has been discussed briefly before, but as far as I know no problem statement or solution has been raised as an issue yet. I'm not using a proposal title and label; as far as I know that's only necessary if we need the proposal review committee to intervene.

cc @bcmills @jayconrod @matloob @rsc for cmd/go
cc @ianthehat @dominikh @myitcv @rogpeppe @kardianos for golang-tools and some previous issues

@mvdan
Copy link
Member Author

@mvdan mvdan commented Nov 11, 2020

The main counter-argument I see to this flag solution is that it could make go list harder to understand. Right now, it always obeys build constraints when looking at packages, and it only has a second mode: -m for instead working with modules.

However, I think this third mode would be fairly niche, and we could explain it rather simply: list packages while ignoring build constraints, hence not loading packages as if they were to be built. This helps us limit go list -newflag to only support basic operations.

@rogpeppe
Copy link
Contributor

@rogpeppe rogpeppe commented Nov 11, 2020

I support the idea here. I've often wanted to enumerate dependencies across all build configurations and it's awkward to do currently (I've built custom solutions to do it in the past).

As for the name of the flag, how about -all-builds ?

@mvdan
Copy link
Member Author

@mvdan mvdan commented Nov 11, 2020

The only reason I tried to avoid "all builds" as a flag name is because, to me, that implies "load and possibly build all build constraint combinations". As if go list -export -all-builds would then build an export file for each GOOS/GOARCH platform, for example.

@kardianos
Copy link
Contributor

@kardianos kardianos commented Nov 11, 2020

For govendor, years ago, I had to re-build go list functionality because of this very issue, but that was specific for vendoring that as you note go mod handles correctly. Initially go code completion tools also (unfortunately) followed build constraints, but that seems to have been fixed now in recent gopls. So I agree there is strong data points support this need in general. I cannot comment on the addition to go list itself. But I could easily believe this would be useful to have.

@mvdan
Copy link
Member Author

@mvdan mvdan commented Nov 11, 2020

I hadn't thought about gopls. Perhaps they have some code that could be eventually replaced with this new flag. cc @stamblerre

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Nov 11, 2020

This would be useful. We already have some of the code for this in cmd/go for module commands.

In go mod tidy, the go command loads packages in all and their tests, ignoring build constraints with one exception: the tag ignore is still considered false. That's useful for examples and small one-file programs that are in the same directory as another package. If there's a flag to ignore build constraints, I think it should have the same semantics.

As for the CLI, maybe overload -tags? Something like -tags=all or -tags=*?

This should be restricted to go list, too, and I agree about restricting -export and -compiled.

@mvdan
Copy link
Member Author

@mvdan mvdan commented Nov 11, 2020

the tag ignore is still considered false. [...] If there's a flag to ignore build constraints, I think it should have the same semantics.

That would be fine for my use case. The packages I want to list will eventually be loaded in some build configuration, so the ignore tag will never really play a part in that. I guess that, if someone has a very niche use case where they want all files including those with +build ignore, they could use go list -f {{.Dir}} -newflag and then grab/copy the entire directory.

As for the CLI, maybe overload -tags?

Overloading -tags could be a nice idea. I only worry that it's a generic build flag, so what would go build -tags=all do? Do we define that go list's -tags flag is actually different than all the other occurrences of the same flag?

@mvdan
Copy link
Member Author

@mvdan mvdan commented Nov 11, 2020

they could use go list -f {{.Dir}} -newflag and then grab/copy the entire directory.

To clarify in case anyone wonders - this kind of workaround about copying entire directories doesn't help solve the issue in this thread. For example, think packages which contain zero files matching the current build constraints, which wouldn't show up in the go list output at all.

@bcmills
Copy link
Member

@bcmills bcmills commented Nov 11, 2020

We will need a bit of care with import cycles, though. When ignoring build tags, it is possible to end up with import cycles that are otherwise rejected.
(See https://github.com/golang/go/blob/master/src/cmd/go/testdata/script/mod_tagged_import_cycle.txt.)

@ianthehat
Copy link

@ianthehat ianthehat commented Nov 11, 2020

It transforms the ouptut of go list from a DAG to a true graph, and there are real world cases of this. It would never be feasible to have a version of the go/packages API that sat on top of this for instance (which if it was just a special tag would end up happening)
I do think that it would be a valuable feature, but it really does need to be called out as a special case that you have to hold carefully, because its the kind of thing that is easy to use in a way that works on your testing and dies in the real world in hard to comprehend ways.

@bcmills
Copy link
Member

@bcmills bcmills commented Nov 11, 2020

@ianthehat, FWIW I suspect that the existing -e flag also transforms the output from a DAG to a graph, or at least would if it worked as often as we want it to. (Cycles cause an error, but -e suppresses errors.) So tools built on top of go list -e probably already need to know how to handle cycles, and I'm not sure how many tools build on go list today don't want -e.

@stamblerre
Copy link
Contributor

@stamblerre stamblerre commented Nov 12, 2020

To answer #42504 (comment), gopls actually doesn't have any handling for build tags (hence the longstanding #29202). Users must specify their build tags in their configuration before using build tagged files. If it were possible to use this in go/packages, gopls might be able to use it when we do get around to #29202 -- but there are a lot of other complications with type-checking that we would need to figure out first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants
You can’t perform that action at this time.