build: use `vendor` directory for dependencies

[bdarnell]

Once we move to go 1.6 (and the vendor "experiment" becomes official) we should consider using the vendor system for our dependencies. glock (in its current version) can be problematic for people who only use a single GOPATH since it checks out our dependency revisions into what is supposed to be a shared space (so it may conflict with other projects that use a similar pinning strategy or just go get -u).

There is an unreleased branch of glock that supports the vendor directory (robfig/glock#28), but glide appears to be emerging as the de-facto standard and we should consider switching. It looks like it has an equivalent to glock's cmd feature (which is missing from other dependency tools).

[tamird]

It looks like it has an equivalent to glock's cmd feature

Does it? Masterminds/glide#172

[bdarnell]

There's this line in the readme: "In addition to fetching packages, Glide builds the packages with go install", which I took to mean that you could list a main package in a subpackages listing and it would be installed. (into $GOPATH/bin instead of the vendor subtree, so you could still get version conflicts if multiple projects share a GOPATH, but it's something). I haven't tested this, though, and based on that issue it sounds like I'm wrong.

[tamird]

As it turns out, it's possible to vendor standard library bits. From https://docs.google.com/document/d/1Bz5-UB7g2uPBdOx-rw5t9MxJwkfpx90cqG9AFL0JAYo/edit:

Update, January 2016: These rules do not apply to the “C” pseudo-package, which is processed earlier than normal import processing. They do, however, apply to standard library packages. If someone wants to vendor (and therefore hide the standard library version of) “math” or even “unsafe”, they can.

This means we can vendor net/http and locally patch golang/go#14221 so that we can experiment with multiplexing RPC/PG on a single port during the Go 1.6 cycle.

[tbg]

Wow, I'm surprised this is actually possible. Positively. Though it's a bit
scary. This also opens up funny new venues like improved deadlock
detection, ...

On Wed, Feb 10, 2016 at 9:23 PM Tamir Duberstein notifications@github.com
wrote:

As it turns out, it's possible to vendor standard library bits. From
https://docs.google.com/document/d/1Bz5-UB7g2uPBdOx-rw5t9MxJwkfpx90cqG9AFL0JAYo/edit
:

Update, January 2016: These rules do not apply to the “C” pseudo-package,
which is processed earlier than normal import processing. They do, however,
apply to standard library packages. If someone wants to vendor (and
therefore hide the standard library version of) “math” or even “unsafe”,
they can.

This means we can vendor net/http and locally patch golang/go#14221
golang/go#14221 so that we can experiment
with multiplexing RPC/PG on a single port during the Go 1.6 cycle.

—
Reply to this email directly or view it on GitHub
#4182 (comment)
.

[petermattis]

@mjibson Wondering what led to trying to use govender in #4620 vs glide or some other alternative.

[maddyblue]

@petermattis I didn't try glide because @tamird reported having problems with it. I've used govendor on a few projects before and really liked it. I think it's better than godep because of how it arranges code and how easy it is to manipulate the vendor manifest and dependencies.

[tamird]

glide was just horribly slow.

[dt]

As @mjibson pointed out on #4620, something to consider as we evaluate these approaches, in addition to how we treat a potentially shared GOPATH, is that we're currently depending on the maintainers of our dependencies to continue to distribute them (and at the revisions we depend on).

Some vendoring approaches obviously address the distribution of our dependencies directly -- particularly the "check everything directly into /vendor" -- while we could probably adapt others too as well e.g. fetch from our own fork of every dep when checking out submodules or whatever.

[bdarnell]

Another option, especially as we move towards a more formal release process, is to keep the main repo pulling from the upstream version control, but whenever we make a release we commit all of that release's $GOPATH/src (to a separate repo). This would ensure we never lose source that is necessary to reproduce a build without adding clutter to the main repo or making it more difficult to submit PRs upstream to dependencies.

[kardianos]

Although I've dropped off the face of the earth from roachdb, I'd like to update this issue. One option is to use govendor to help manage deps. For instance you can run govendor migrate in the cockroach folder and it will switch the revision annotations over to the vendor meta-data file. You can then run govendor sync to pull down the stated revisions to the vendor folder. Subsequent calls to `govendor sync are cheap because the hash of the files are checked so no network operations are needed, unless the files are found to be out of sync.

Updating packages can be done with govendor fetch pkg@revision/version. Ignoring the source files in the vendor folder can be done by adding the ignore rule vendor/*/.

Lastly, govendor is flexible. By default the migration process from glock makes all deps "tree" deps, in that it doesn't prune unused packages. However you can remove them and add them again without that to just select used packages: govendor remove gopkg.in/inf.v0 && govendor fetch gopkg.in/inf.v0/...@3887ee99ecf07df5b447e9b00d9c0b2adaa9f3e4 only vendor used packages. You can also choose to ignore test files or other build tags. Commands can also be vendored just fine.