cmd/go: allow forcing tags on/off during go mod vendor, tidy #25873
In https://golang.org/cl/117258 the "vendor ALL" was introduced to include all files when walking the source tree, except for ignore build tags. This is great.
In https://golang.org/cl/118317 the command introduces flags for "mod -fix" and "mod -sync" (but waits for another CL to implement them). These flags use the knowledge of complete source tree (when trimming or adding requirements) to operate.
Lastly, it is important to note that the "go.mod" exclude directive only operates on the current module. This is good and to the benefit of this proposal.
This proposal adds two things:
This would be used when the entire source tree (ALL) contains irrelevant platforms to the direct user of the module (common examples would be "exclude appengine" and "exclude solaris". Because exclude directives are ignored by other modules, they do not define limits of the module's use. In addition, these exclude tags are not used when running "build" or "install".
There are two optional features of this proposal as well, but not intrinsic to the base proposal:
The implementation in govendor supports these cases here:
Two ways to specify tags would be to call any "exclude" that has no "/" a tag. Another way would be to prefix the tag with a "tag:".
Can you give some more detail about why that's useful?
If I understand correctly, it would avoid vendoring in (some subset of) unused source code, at the cost of making a particular module unusable with the excluded tags (presumably because it's already incompatible with them to begin with?). What fraction of the vendored code is that, in the common cases you mentioned?
The module would still be usable with the excluded tags.
There are three common cases for excluding tags when calculating the dependency tree for dependency analysis (not building):
As you can see, this is most useful for trimming what is copied into the vendor folder. It has a secondary benefit for not adding un-tested "require" directives for un-tested platforms with "-sync".
However, it is not a blacklist in anyway. "[v]go build" will continue to operate as-is. These exclude tags are only used in the "[v]go mod" sub-command.
Very common. Many google network libraries for instance support appengine. Many apps that use these libraries are CLI applications or other non-appengine services that will never run on appengine. Note, this doesn't preclude an appengine application from using another modules that has "exclude appengine", it just requires the consuming module specify a few more requires for the appengine specific modules.
That being said, I have not done a quantitative analysis on the impact here.
Sure, but the point of
What happens to users of the package with
It's not just a cache: it's a consistent version. If you
(I think that's what you mean by “I have no opinion on these dependency versions,” but that's meaningfully different from caching.)
referenced this issue
Jun 21, 2018
Any updates on this?
I have what appears to be the same issue: I have a file guarded by build tags that import a package that will only exist on the user's machine i.e. the same situation as the
In general I am going to push back very hard against adding new settings to go.mod. One by one they might be reasonable but the end result would be to turn go.mod into a complex config file, when one of my favorite things about Go is that code follows conventions instead of having configs that differ from project to project. Also, providing fine-grained control over default settings often masks the defaults simply being wrong. Maybe @DisposaBoy sees the conventions as hacks and special cases, but I see them as making Go projects consistent in a way they wouldn't be if the tool were endlessly customizable.
@kardianos pointed out these reasons this fine-grained control over tags might be needed:
As of CL 122256 earlier today, test data is excluded from (1) unconditionally, so that's no longer a concern. This is an example of the default being wrong and getting a chance to correct it precisely because it's not possible to override.
I have been meaning to do something about (2) as well. If you're running with Go 1.10 it seems fine to assume that the go1.8 and earlier build tags are satisfied, possibly even go1.9.
I would be interested in numbers about the amount of code (3) and (4) are causing to be copied unnecessarily. What convinced me about (1) was that the test code added a dramatic size increase to the vendor directory when trying vgo on k8s.io/kubernetes.
That leaves (5). I would like to learn more about the case @DisposaBoy has in mind to see whether it's really like appengine or not and to figure out whether there's a general answer. The special case for "appengine/*" packages is unfortunate but not directly related to the appengine build tag. It's because those import paths predate GOPATH, so the App Engine compile servers essentially added those packages to a modified standard library. Is that what you are doing too @DisposaBoy? Assuming not, how is the go command supposed to build the files on the user's machine? Or if the go command isn't ever going to build the files on the user's machine, what does consume them? Do the paths begin with domain names or plain words?
One possible way to resolve (5) generally would be to say that anything that doesn't start with a dotted name and isn't covered by existing modules in go.mod will just be ignored during mod -sync. But I don't know whether that would address @DisposaBoy's situation or not.
@rsc If you want, I can try and make a repo demonstrating what I'm doing tomorrow, but it's essentially a compile-time plugin system.
In simple terms: If the tag
In the real code this is all automated, but the effect is that a user can
The problem is that when I try to vendor the dependencies,
For reasons I prefer to use the name
@rsc I just came across the same idea (this feature request).
I'm using the latest vgo from the vgo repository (not the one integrated in the golang repository).
I have just added
grep -r "mysql"
grep -r "sqlite"
grep -r "pq"
As you can see, only test files are actually importing packages
I've been thinking more about @DisposaBoy's situation described in #25873 (comment), and while I'm not completely opposed to some more heuristics here, I don't think they would solve his situation. The general problem with exceptions is that if 'go mod vendor' and 'go mod tidy' (formerly 'go mod -sync') cannot understand where the imports come from, then neither can 'go build'. Putting in an exception would not change the fact that the code is unbuildable. So putting in an exception does not really solve anything.
There is not going to be a way to "create the package my_plugin". In the old GOPATH world you created GOPATH/src/my_plugin. That door is closed in module world: you have to do something in go.mod now.
What I would suggest is to make a separate module in its own repo, with a path you actually control (maybe github.com/you/user-plugin) that actually exists and compiles and is the no-op plugin. Then plugins could be enabled always, they would just get the no-op plugin with no effect. At this point everything builds, there are no magic dangling imports, all the commands work.
But there is still some magic in the world.
Users can clone that repo, make changes, and then run, from within the clone,
The go get will add your/app to the repo's go.mod and build it in that context, and in that context, the modified clone is the de facto authority for github.com/you/user-plugin. The "go get", because it is executing inside the modified clone, will build your app using the modified clone of the plugin instead of the original no-op plugin. This gives the same effect as your compile-time plugin. It even lets users manage multiple different plugins (in different clones) easily, without having to make a new GOPATH for each one.
Rereading the reasons from #25873 (comment), it looks like the main motivations left would be (3) and (4). These only really apply to vendoring. Now that go mod vendor is its own command, we could envision adding a flag to say 'assume these tags are on or off', like maybe -tags aws,!azure to force aws on and azure off. This would be a command-line flag, not extra configuration in go.mod, for the reasons I mentioned in the earlier comment.
Leaving for decision in Go 1.12.