Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
nested vendor folders management. #303
I'm quite sure somehow this was already discussed but I'd like to put down my view.
Glide is flattening the dependency tree in the base
If a dependant package also vendors its dependencies, glide doesn't remove them and these ones take the precedence and are used for compilation instead of the one in the base project
I'm temped to call this "soft flattening" .
It's clear that many projects (many monorepo so they also provides libraries) wants to vendor their dependencies (where their
Given this I can see two different possibilities to manage nested vendor folders that will have different impacts and behaviors:
What are your thoughts on this?
I can't see a default solution, perhaps the user should be able to force one or the other behavior per package dependency? (but this will mean that the user must explicitly define a repo and also subpackages, also if not an explicit dep but an inherited one, in glade.yml and define something like
Additionally, will removing just the nested vendor folder break some licenses like explained in http://engineeredweb.com/blog/2016/go-why-not-strip-unused-pkgs/ ?
This is a good and difficult question.
I don't think there's a license issue with removing vendored packages from a dependency. But, that's the least of the problems.
Here are a couple examples.
This makes it hard for a tool to automate things.
As a rule, libraries should not
I'm not sure how to do this better than we do today for the
@mattfarina Thanks for your answer.
I just noticed other tools do (or want to do) "hard" flattening. For example I found some issues on godep like tools/godep#428 and tools/godep#224. In this last one @thockin talks about something similar to this issue (soft vs hard flattening) and, in the end, he prefers "hard" flattening (I apologize if I misunderstood something).
Probably there isn't an automatic solution given pros and cons of soft vs hard flattening and actually it's very difficult if not impossible to detect when a nested vendor is needed or not (since I can put anything inside a vendor folder without any vendoring tool information to detect the package version, sometime also without a source repo etc...).
Probably the unique solution is to let the user choose what's better.
What I'd like to propose is to add some options to Glide to give the user the ability to choose the preferred behavior but I'm not sure how to specify it (options? config file?. What I had in mind was:
If this actually doesn't fit directly inside Glide, I'm temped to initially add this to https://github.com/sgotti/glide-vc. Something like
What do you think about this approach?
Returning to your questions:
Good point. Will be possible to improve the
See above (explicit keeping of some nested vendor directories).
I can understand it, but on the other side it's difficult to define why a library should not vendor a specific version of a package if it needs to.
@sgotti at the moment I would prefer something like glide-vc handle stripping nested vendor directories. I may be persuaded to change my mind but I don't really want to add a bunch of complexity into Glide for that right now.
Dirty would be hard to detect for a vendored repo. If the VCS data isn't available the VCS can't easily do it for us. This is complicated. What's dirty? Missing files? Changed files? With a VCS it's pretty easy. Without it we just ignore it which is, in part, why updating vendored is opt-in.
I think stripping vendor directories should be in the hands of the developer. They know what they're building and can deal with the consequences. This is really why libraries should not vendor. This is, in part, why storing outside packages in your VCS isn't common (or even allowed) in some other toolchains.
Yet, I understand the issues k8s and others are having that leads to this.
I was running into this today, too. I want glide to see that a dep has a
As an example: Kubernetes vendors heapster. heapster vendors a client lib
Stripping out deps' vendor/ and Godeps/ and so on will save a lot of
On Fri, Mar 11, 2016 at 12:06 PM, Matt Farina email@example.com
@thockin This is where things get hard.
This is why the discussions in places like the go nuts mailing list have been to not vendor outside dependencies in your project if you're building a library for others to use.
Bringing this back to Glide, we've been wary to add something that alters a codebase to Glide which is what you're asking will do. The project glide-vc is working to be able to do this with numerous options. With this installed you'll be able to do things like run:
$ glide vc
We're also going to add a scripts section to Glide. Consider this in the
scripts: setup: go get -u github.com/sgotti/glide-vc install: - glide install --update-vendored --strip-vcs - glide vc
Then you can run:
$ glide run setup $ glide run install
We need the scripts for varying reasons such as installing applications used by generators. None of the scripts would be run automatically (for security reasons). It would be great to just use make but that's not always available or cross platform.
Thoughts? This is something we may want to get on a hangout to discuss in more detail.
Yeah, that all makes sense EXCEPT when you're vendoring something that vendors.
That's great in theory. A large number of libs that we use are part of a repo
Given that, what happens once everyone uses glide (world domination, yay)?
At this point you've gone full-recursive. There's not ACTUALLY a cyclical
Something has to be the turtle at the bottom. Godep respects go packages (plus
I think any vendoring tool HAS TO break the false cycle here, and the way to do
Well, our lawyers were pretty clear that this is not an alteration in the
Regarding glide-vc: if that's the best we can do, we'll do it, but I'll go on
Scripts is a cute hack, but it means that everyone's repo will be different
@mattfarina Thanks for listening and working in this. I think that the work done for glide 0.10 (strip-vcs and strip-vendor) is a good base solution for this issue.
As also explained in #349 handling some incompatible packages with same name could be done installing them as separate nested vendors dirs (some like in npm v3) but as explained this is a tricky problem (causing possible incompatibilities).
And now glide-vc can be used just to remove all unneeded packages and files (better than doing vendor management inside it) if license permits this and to preserve space when committing the
If you want this issue should be closed and perhaps future updates will go in new issues.
fwiw, in the new SAT solver-based engine, the intention is to flatten as an (aggressive) default - because it's ultimately the only path to a sane system - but to allow exceptions in cases where it's provably safe automatically...and then probably in other, perhaps user-chosen situations as well.
also, this pretty much describes my view:
I'm new to glide and trying to understand how nested vendor directories are handled presently.
From what I can understand, "soft-flattening" is now happening by default, but "hard-flattening" is optional. Do this sound accurate?
For example, all dependencies end up at the top level vendor, but all nested vendor directories can be removed on request by specifying