New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
build: use vendor
directory for dependencies
#4182
Comments
Does it? Masterminds/glide#172 |
There's this line in the readme: "In addition to fetching packages, Glide builds the packages with go install", which I took to mean that you could list a |
As it turns out, it's possible to vendor standard library bits. From https://docs.google.com/document/d/1Bz5-UB7g2uPBdOx-rw5t9MxJwkfpx90cqG9AFL0JAYo/edit:
This means we can vendor |
Wow, I'm surprised this is actually possible. Positively. Though it's a bit On Wed, Feb 10, 2016 at 9:23 PM Tamir Duberstein notifications@github.com
|
vendor
directory for dependenciesvendor
directory for dependencies
@petermattis I didn't try glide because @tamird reported having problems with it. I've used govendor on a few projects before and really liked it. I think it's better than godep because of how it arranges code and how easy it is to manipulate the vendor manifest and dependencies. |
glide was just horribly slow. |
As @mjibson pointed out on #4620, something to consider as we evaluate these approaches, in addition to how we treat a potentially shared GOPATH, is that we're currently depending on the maintainers of our dependencies to continue to distribute them (and at the revisions we depend on). Some vendoring approaches obviously address the distribution of our dependencies directly -- particularly the "check everything directly into /vendor" -- while we could probably adapt others too as well e.g. fetch from our own fork of every dep when checking out submodules or whatever. |
Another option, especially as we move towards a more formal release process, is to keep the main repo pulling from the upstream version control, but whenever we make a release we commit all of that release's |
Although I've dropped off the face of the earth from roachdb, I'd like to update this issue. One option is to use Updating packages can be done with Lastly, govendor is flexible. By default the migration process from glock makes all deps "tree" deps, in that it doesn't prune unused packages. However you can remove them and add them again without that to just select used packages: |
(I'm willing to propose a PR if interested in a |
Hi Daniel, have you seen #4620? I'm personally all for vendoring (but haven't looked at it much); maybe some of the comments (that led to eventually closing that PR) can be instructive. Essentially, it was not desired to check in a snapshot of all dependencies in our main repo. I'm also not terribly interested in introducing submodules, and we've talked about eliminating the dependency on upstream vendors' repositories. Maybe a single subtree/submodule'd In any case, I'm glad you're interesting in improving the state of affairs here, and I hope some of the others involved in this thread will chime in as well. |
We (@mjibson, @dt, @paperstreet) discussed having a
@bdarnell @tamird do either of you object to this solution (since you objected to the previous one)? |
Relevant thread with @tamird in it: golang/go#7764 |
@tschottdorf Yes I have. That was before govendor grew "sync" and "fetch" sub-commands. You can now use govendor without needing to checking dep sources. Actually, You could run You would probably want to finesse the actual deps in the vendor folder, but that would be very workable. |
or
Excellent, glad to hear things have improved. |
I don't think a I think that for reproducible builds, we should be producing an archive of all the source used for each of our releases, including the entire vendor tree (perhaps in another git repo). This keeps the main repo optimized for development and ensures that we have at least coarse-grained reproducibility. We don't get complete reproducibility of historical builds for use with tools like |
Looking at it again, we don't have to use a submodule at all. We can keep the vendor deps at This may need to be discussed offline, but we may just fundamentally disagree here. One of the features you want is to be able to easily hack on packages. I'm not sure of a reasonable way to do that without those packages being in the $GOPATH and not in our vendor directory. This means that they also have to be at a specific revision, which is one of the specific things I'm trying to remove. I think all of us want to remove the glock Is there another solution here? I've now tried a few times and am out of ideas. |
But then we'd need to do something to tie each version of the main repo to a version of the vendor repo, which is basically rolling submodules by hand. Yeah, there's a philosophical difference: i would like per-project GOPATH to become the norm just like e.g. virtualenv in python; IMHO all this effort to make different projects coexist in a single GOPATH is misdirected. But I concede the point; we shouldn't be stepping on other things in the GOPATH (this includes both
That's a good point, and may render all my objections moot. I'm not very familiar with the operation of the tools when a vendor directory is used, but it looks like what I want to do won't work. @kardianos, what's the workflow for using a local modification to a dependency? Do I hack on it in If we use |
Here's one more vendoring approach, centered around submodules: https://github.com/kovetskiy/manul |
While figuring out how to distribute The most obvious of these is go itself -- the go team has unfortunately decided that the expansion of Any vendoring solution will need to figure out how linters and tools that traverse over the packages we write can be excluded from inspecting Checking out/building an entirely separate |
(revisiting this now that master/develop split is over) To recap:
A big open question / challenge though is linters and other tools that want to traverse all the code we author, but need to avoid traversing
This mess can be sidestepped by avoiding having the
|
Just in case someone else is going to try this, I was trying to be clever: mkdir src
cd src
ln -s ../storage ./storage
cd ..
go list ./src/... # finds nothing, unfortunately |
How about setting up our linters so that they operate on each package individually? We almost never create new top-level packages. We could even automate this with something a la |
Some linters can't do their work without operating on all the packages at On Mon, Oct 3, 2016 at 12:12 PM, Tobias Schottdorf <notifications@github.com
|
Yeah, behaviors vary across linters, some (like unused) want to see all packages at once, others can't handle more than one argument, many use go package specs, but others use directories -- my previous proposed PRs have attempted to individually do whatever it took to make each happy, using a |
Next silly idea: we check in a file |
yeah, @tamird and I discussed that (maintaining it either with a CI linter, or in the builder before starting docker), and if we do go the "make the linters happy" with approach, we'll probably do something like that, but since not all our linters can even take more than one arg, it'll still add some additional overhead to adding and maintaining linters vs just having one dir we can say is the one to lint. |
As for the root cause: "docker I/O is slow". On Mon, Oct 3, 2016 at 1:32 PM, David Taylor notifications@github.com
|
One more silly one: Can we write a fast replacement for |
You'd think that, but docker never ceases to amaze:
|
As far as directory locations and their tool friendliness goes, looking at the options above, I guess the important questions are:
If the answers to those questions are "very important" and "too gross", then by elimination we're down to figuring out how to make all the linters happy, and performant (e.g. the denormalized |
I personally don't care about multiple checkouts, but being go-gettable would be good and having the dependencies closely pinned is too.
It's kind of gross, but that seems to be the only sane choice given the insane choice of including With Cockroach, we have the privilege that almost all code is internal, so in theory we could move everything into Between the peer-repo, the linter tax and the subdirectory, I think the subdirectory is by far the best. |
I'll whip up a vendoring PR that nests all our packages under something to see how it looks in practice... name bikeshed for this subdir: |
Moving everything into a subdir sounds reasonable to me. The name -1 on using |
kubernetes also uses the name |
nice, I like |
Once we move to go 1.6 (and the vendor "experiment" becomes official) we should consider using the vendor system for our dependencies.
glock
(in its current version) can be problematic for people who only use a singleGOPATH
since it checks out our dependency revisions into what is supposed to be a shared space (so it may conflict with other projects that use a similar pinning strategy or justgo get -u
).There is an unreleased branch of
glock
that supports thevendor
directory (robfig/glock#28), butglide
appears to be emerging as the de-facto standard and we should consider switching. It looks like it has an equivalent toglock
'scmd
feature (which is missing from other dependency tools).The text was updated successfully, but these errors were encountered: