cmd/go: exclude vendor directories from wildcard matches? #11659
Comments
ianlancetaylor
changed the title
|
What package imports |
|
@adg it's pulled in from |
|
That would require special-casing |
adg
changed the title
|
When I run |
|
@technosophos +1 on that problem. @adg IMO that makes the vendor experiment nearly useless (for me anyway). I'll either need to make godep much more intelligent wrt package handling and/or try to discern what to build through additional package introspection. It's also just a bad user experience IMO. |
|
@technosophos yes, that is the correct behavior as of now. @freeformz there is an easy work around. Instead of using or use |
|
On the testing front, I think it is valuable to know that your dependencies' tests pass. |
|
Yeah, I don't find any of this particularly disconcerting. I just wanted to verify that it was the intended behavior. |
|
@adg Heh, thanks. I should have thought of that. :-( Are there any plans to special case vendor/ at all, say in the 1.6 timeframe? |
|
Possibly. That's what this issue is now tracking. We're going to gather people's experience with the 1.5 vendoring experiment before making such plans. |
|
@freeformz After looking at this issue carefully, I would say the current behavior would be my desired behavior. While I understand Godep copies over the entire repository to vendor, I don't think that is what I would encourage. I think copying over each package as needed by the vendor tool is where the analysis work needs to be. At least that is what vendor will do. |
|
@kardianos The issue really has nothing to do with Godep, even though Godep's current behavior triggers it. |
|
A couple reasons I would prefer When I choose to take on a dependency, I've already verified it serves my need. That includes verifying automated tests are run by that dependency's overall project, running them myself, or similar. With things in Second, It's possible a dependency's tests will never build, work, or be relevant in my particular environment. This could be due to the way the tests are written, differences in developing vs deploying platform, etc. I develop projects on my OS X laptop that use libcontainer that would probably be good examples for this. |
|
@dpiddy Tests that are specific to an environment shouldn't run on a different environment anyway. If they try to run file a bug against the package. I would also argue when copying packages over the vendor tool user should decide if they want to bring over the test files or not The case where my reasoning falls down is where people choose to use a vcs features like sub-modules. (Not directly trying to argue here, just state other sides.) |
|
@kardianos good point, tools could do that. Re the original issue of Vendoring tools could also skip |
|
@dpiddy Of note, I've had at least one person explicitly want to vendor "main" packages. What the kardianos/vendor tool does it to categorize packages by a "status": Missing (packages referenced but not found) These are two overlapping issues I hope to address soon (location and status), but that's the general idea. Then my tool can work on package status rather then just paths. For instance, if vendor added an install command, you could run "vendor install -status local". I'm not convinced that adding a "status" concept to the go tool would ever be a desirable thing. "go" already supports "std". The syntax "go install ./...!local" could be interesting I think. |
|
This has come up repeatedly, but it seems like a bad idea. |
|
I missed Andrew's comment earlier about using this issue to track feedback. That's fine, but if so it should not be marked as milestone Go 1.5. Reopening and reclassifying. |
|
@rsc Thanks for re-opening. I hope other's provide additional feedback ... My general thoughts on your last comments are: I "technically" agree with you, but it makes for a poor user experience. |
|
Having |
|
@jacobsa Sure, but don't confuse the issue of "go test ./..." with a tool, godep. There (will be) many tools. There will be some that analyze dependencies well. |
|
This isn't just about testing. Here are my thoughts to date.... When The issues I have with this are:
Some, but not all of the above stuff can be handled by vendoring tools. IMO if we're going to cause this much disruption then Go itself should provide a vendoring tool such as govendor/godeps/something new/whatever as part of the standard tooling (i.e. |
|
I second @freeformz's points FWIW.. |
If I add dependency X that is only a sub-package of a larger package Y, the whole thing ends up in the vendored folder, even if I only need the subpackage and its dependencies. If I test the whole project with You may ask why I add the all of Y if I only need X. The reason is that I use git submodules to add the repo, and |
|
I share all of @freeformz's concerns. I think it is telling that at least godep has always used an ignored @bradfitz that is what I figured. I fear if it takes a major release to change the experiment's behavior enough momentum (whether because it's good or not) will be built with the current behavior that it will be unlikely to ever change. |
|
My main concern, before I can start to think about others, is that of the import paths. As I understand Are other people okay with the idea that I thought the point of vendoring was to put a copy of the original package (e.g., |
|
I've been dealing with the repercussions of the vendoring experiment for a few weeks now. I like having fine-grained control over my dependencies, and it’s awesome to be able to commit the vendor source along with my project source without having to change all the vendored import paths. I think I voice the same concern as most of the non-googlers here, which is that I didn’t expect to have golint, go vet, and go test blow up in my face when I moved everything to the vendor directory. It’s frustrating to have to contort my workflow to accommodate external code that didn’t previously require it. However, I wouldn’t have known about many of these problems with the third-party libraries had I not moved them into the vendor directory. It’s been a pain to get everything to play nice, and I understand that not everyone has the luxury of being able to fix other-people’s-code, but the new methodology incentivizes quality code… which is one reason I believe Go is as successful as it is today. If I compare the two solutions from a long-term perspective, I see two very different outcomes. Allowing people to ignore bad code results in a complex solution whereby some code is whitelisted, some is blacklisted, the lists require regular maintenance, and you have no guarantee of a minimum level of quality. The other solution is to develop good code, fork bad code, and take the penalty of due diligence up front. I’m not saying either solution is right or wrong, but since this issue is still open for feedback, I’m of the opinion that the short-term pain of broken libraries will eventually subside as the authors feel the pain and/or new authors emerge. I believe that the long-term result will be a better foundation from which to build future solutions. |
|
I was asked to provide my opinion here, so here it is. I agree with @rsc's comment in #11659 (comment). Further, it seems renaming the directory to _vendor would make it harder to operate on the vendored packages. For example, it's useful (at least to me) to be able to run Personally, I think the current behavior is ok. It hasn't really been a problem for me (though I understand why some find it inconvenient). For example, when I run interactive commands to test or vet packages, I type |
|
Thanks for all of your input. I think everything has been covered here that could be covered. I'm satisfied with the outcome, although I still think it ignores usability a bit, so I'm going to close this issue (since I opened it). |
|
FYI, see https://godoc.org/rsc.io/gt for a go test wrapper
that could help with this issue.
|
|
That doesn't help with the issue where |
|
it's always possible for go list to output invalid import paths.
try run `go list` outside of GOPATH or GOROOT.
|
|
But I said I have multiple tools that use the output of All tools that prefer to preserve previous behavior of not including vendored package copies (e.g., like they did not include the vendored packages in |
|
@shurcooL In hopes of being helpful, you might try to migrate your tools to use something else. For instance you could try http://godoc.org/github.com/kardianos/govendor/context create a new context then call ctx.Status(), then filter all the returned items by their "status" which includes if they are in a vendor directory or not. If that doesn't quite fit your need, but is close, feel free to open an issue. |
See bug here: golang/go#11659
|
I have been doing research and setting up a continuous integration & testing pipeline for Go during most of the day, on different tools, and this is an issue that really bothers me. I love Go for it's minimalist and clean approach. In line with that, I use the clean architecture approach (or something close to it) for most of my projects. This means I will usually have four subpackages, with the main package in the root folder. Sometimes, there are some small additional tools in the cmd folder as well. I was under the impression that this is a configuration that most projects use. I now have to use a dirty workaround to get CI to build and test properly, all due to the fact that ./.. includes folders in vendor. This is counter-intuitive and not well thought-out at all. Which is contrary to what I've come to expect and appreciate about Go. The default case should not require any workaround. There should be a coherent approach where all go tools exclude vendor when using ./... if the GO15VENDOREXPERIMENT environment variable is set to 1. That would be consistent. As someone mentioned before, you can still do "cd vendor && go test ./..." if you want to include the subpackages. But you will have to be explicit about it, which is fine since it constitutes a non-default workflow imo. If the go tool deals with vendoring using the environment flag (and by default in Go 1.6), then it should also deal with the resulting consequences and solve them in a coherent way. Let's keep it simple, folks. For now, I'm forced to switch back to godep to keep a clean non-hacky script in my repository. |
|
I think there are two cases:
I think it's a mistake to skip the test in (1). And I think (2) indicates a The counter-argument to this is that you can't omit individual packages |
|
@rsc I disagree even with your first point. If you vendor your dependencies, you will usually pin a specific version, so it is nonsense to run the tests each and every time you push a commit and your CI kicks in. You should test your dependencies once, manually, each time you update them. That's completely besides the point of automatic testing of your own commits and pull requests. |
|
Agree with @awishformore, similar to what I said in #11659 (comment). How many projects that use net/http run those tests every time? Would that be reasonable? I don't think so, because presumably you've verified that the Go release you're using is good and you know net/http isn't changing out from under you. I feel things in |
|
As others have noted, there is an easy workaround if you just want to test Depending on your configuration, it may be appropriate or worthwhile to use I also still think that for the specific case of testing, if we could solve |
|
@rsc Please don't qualify it as an "easy" workaround. It's a dirty workaround for something that should work out-of-the-box. Why? Because someone decided that the Go tool should deal with vendoring, and since it does, the default way should correspond to the most intuitive and common approach. The workaround should be required if you do something out of the ordinary, like testing dependencies on each build or, like you mention, customizing dependencies. In that case, a workaround (i.e. cd vendor && go test ./...) seems perfectly fine. I don't understand why you argue to make the default more annoying in favour of some edge cases. One of the most wonderful things about Go is that defaults usually work in intuitive ways and hardly ever annoy me. |
It's more convenient and simpler (for both the user and for the go tool's implementation) to test them every time you run your own tests. I'd love to know, concretely, why some folks are so resistant to this idea. Is it because running more tests is slower? (If so, see @rsc's last comment above.)
From my point of view, it does work out of the box. That's because I like to run the tests for my dependencies. Of course, anyone is free to prefer otherwise, but that preference doesn't necessarily imply that the go tool's design is wrong. |
In case it's helpful, here's an easy workaround if you want to test dependencies: |
|
@kr build time is certainly a factor with current Go versions. I also gave my reasons in #11659 (comment) and other comments. Perhaps my mental model of what |
|
Of course I'm forgetting that an easy way to exclude vendored tests is to do just that: exclude them when vendoring things. At least godep and govendor already support this. Same for unwanted Curious to see how things like |
|
The fact that everyone is proposing workarounds just points to the fact that this should be fixed. Renaming it to _vendor just works. If you want to test or install stuff from your vendor directory, you can by just cd'ing into that directory.... but if you don't want to, then you can Keep go simple. Make the vendor directory _vendor so that tools skip it. I shouldn't have to use some annoyingly arcane command to just build and test my code. |
|
"vendor" is not being renamed. That ship has long sailed. I think everything that can be said on this issue has been said in the many comments above. I think it's time we move this to other forums (for general discussion) or new bugs (for specific new bugs that aren't just a dup of this one). Ultimately this bug is a policy/behavior question and not actually a bug report. |
and others
go version go1.5beta1 darwin/amd64
using this repo/branch as a test of vendor/: https://github.com/heroku/log-shuttle/tree/vendor2
GO15VENDOREXPERIMENT=1 go install -v ./...
What did you expect to see?
What did you see instead?
I expected go to skip un-needed stuff (non imported packages) in the vendor/ directory. Perhaps this is a problem with my expectations, but it was a surprise.
The text was updated successfully, but these errors were encountered: