title |
---|
FAQ |
The FAQ predated the introduction of the rest of the documentation. If something in here conflicts with other guides or reference documents, it's probably here that it's wrong - please file a PR!
- Does
dep
replacego get
? - Why is it
dep ensure
instead ofdep install
? - What is a direct or transitive dependency?
- What is the difference between Gopkg.toml (the "manifest") and Gopkg.lock (the "lock")?
- How do I constrain a transitive dependency's version?
- How do I change the version of a dependency?
- Can I put the manifest and lock in the vendor directory?
- How do I get
dep
to authenticate to agit
repo? - How do I get
dep
to consume privategit
repos using a GitHub Token?
- How does
dep
decide what version of a dependency to use? - What is the default
dep ensure -update
behavior for dependencies that are imported but not included as a[[Constraint]]
inGopkg.toml
? - What external tools are supported?
- Why is
dep
ignoring a version constraint in the manifest? - Why did
dep
use a different revision for package X instead of the revision in the lock file? - Why is
dep
slow? - How does
dep
handle symbolic links? - Does
dep
support relative imports? - How do I make
dep
resolve dependencies from myGOPATH
? - Will
dep
let me use git submodules to store dependencies invendor
? - How does
dep
work without changing my packages imports?
- Should I commit my vendor directory?
- How do I roll releases that
dep
will be able to use? - What semver version should I use?
- Is it OK to make backwards-incompatible changes now?
- My dependers don't use
dep
yet. What should I do? - How do I configure a dependency that doesn't tag its release
- How do I use
dep
with Docker? - How do I use
dep
in CI?
No. dep
and go get
serve mostly different purposes.
Here are some suggestions for when you could use dep
or go get
:
I would say that dep doesn't replace go get, but they both can do similar things. Here's how I use them:
go get
: I want to download the source code for a go project so that I can work on it myself, or to install a tool. This clones the repo under GOPATH for all to use.
dep ensure
: I have imported a new dependency in my code and want to download the dependency so I can start using it. My workflow is "add the import to the code, and then run dep ensure so that the manifest/lock/vendor are updated". This clones the repo under my project's vendor directory, and remembers the revision used so that everyone who works on my project is guaranteed to be using the same version of dependencies.
The long term vision is a sane, overall-consistent go tool. My general take is that
go get
is for people consuming Go code, and dep-family commands are for people developing it.
Yeah, we went round and round on names. A lot.
The idea of "ensure" is roughly, "ensure that all my local states - code tree, manifest, lock, and vendor - are in sync with each other." When arguments are passed, it becomes "ensure this argument is satisfied, along with synchronization between all my local states."
We opted for this approach because we came to the conclusion that allowing the tool to perform partial work/exit in intermediate states ended up creating a tool that had more commands, had far more possible valid exit and input states, and was generally full of footguns. In this approach, the user has most of the same ultimate control, but exercises it differently (by modifying the code/manifest and re-running dep ensure).
- Direct dependencies are dependencies that are imported directly by your project: they appear in at least one import statement from your project.
- Transitive dependencies are the dependencies of your dependencies. Necessary to compile but are not directly used by your code.
The manifest describes user intent, and the lock describes computed outputs. There's flexibility in manifests that isn't present in locks..., as the "branch": "master" constraint will match whatever revision master HAPPENS to be at right now, whereas the lock is nailed down to a specific revision.
This flexibility is important because it allows us to provide easy commands (e.g.
dep ensure -update
) that can manage an update process for you, within the constraints you specify, AND because it allows your project, when imported by someone else, to collaboratively specify the constraints for your own dependencies.
First, if you're wondering about this because you're trying to keep the version
of the transitive dependency from changing, then you're working against dep
's
design. The lock file, Gopkg.lock
, will keep the selected version of the
transitive dependency stable, unless you explicitly request an upgrade or it's
impossible to find a solution without changing that version.
If that isn't your use case and you still need to constrain a transitive dependency, you have a couple of options:
- Make the transitive dependency a direct one, either with a dummy import or an entry in the
required
list inGopkg.toml
. - Use an override.
Overrides are a sledgehammer, and should only be used as a last resort. While
constraints and overrides are declared in the same way in Gopkg.toml
, they
behave differently:
- Constraints:
- Can be declared by any project's manifest, yours or a dependency
- Apply only to direct dependencies of the project declaring the constraint
- Must not conflict with the
constraint
entries declared in any other project's manifest
- Overrides:
- Are only utilized from the current/your project's manifest
- Apply globally, to direct and transitive dependencies
- Supersede constraints declared in all manifests, yours or a dependency's
Overrides are also discussed with some visuals in the gps docs.
If you want to:
- Change the allowed
version
/branch
/revision
- Switch to using a fork
for one or more dependencies, do the following:
-
Manually edit your
Gopkg.toml
. -
Run
$ dep ensure
No.
Placing these files inside
vendor/
would concretely bind us tovendor/
in the long term. We prefer to treat thevendor/
as an implementation detail.
dep
currently uses the git
command under the hood, so configuring the credentials
for each repository you wish to authenticate to will allow dep
to use an
authenticated repository.
First, configure git
to use the credentials option for the specific repository.
For example, if you use GitLab, and you wish to access https://gitlab.example.com/example/package.git
,
then you would want to use the following configuration:
$ git config --global credential.https://gitlab.example.com.example yourusername
In the example the hostname gitlab.example.com.example
string seems incorrect, but
it's actually the hostname plus the name of the repo you are accessing which is username
.
The trailing 'yourusername' is the username you would use for the actual authentication.
You also need to configure git
with the authentication provider you wish to use. You can get
a list of providers, with the command:
$ git help -a | grep credential-
credential-cache remote-fd
credential-cache--daemon remote-ftp
credential-osxkeychain remote-ftps
credential-store remote-http
You would then choose an appropriate provider. For example, to use the osxkeychain, you would use the following:
git config --global credential.helper osxkeychain
If you need to do this for a CI system, then you may want to use the "store" provider. Please see the documentation on how to configure that: https://git-scm.com/docs/git-credential-store
After configuring git
, you may need to use git
manually once to have it store the
credentials. Once you've checked out the repo manually, it will then use the stored
credentials. This at least appears to be the behavior for the osxkeychain provider.
Another alternative to make dep
work with private repos is to use a Personal GitHub
Token
and configure it inside the .netrc
file
as the following example:
machine github.com
login [YOUR_GITHUB_USERNAME]
password [YOUR_GITHUB_TOKEN]
Once you have set that up, dep will automatically use that Token to authenticate to the repositories.
You can rewrite the repo url and use the git+ssh shema with follow example:
git config --global url."git@github.yourEnterprise.com:".insteadOf "https://github.yourEnterprise.com/"
The full algorithm is complex, but the most important thing to understand is
that dep
tries versions in a certain
order,
checking to see a version is acceptable according to specified constraints.
- All semver versions come first, and sort mostly according to the semver 2.0
spec, with one exception:
- Semver versions with a prerelease are sorted after all non-prerelease semver. Within this subset they are sorted first by their numerical component, then lexicographically by their prerelease version.
- The default branch(es) are next; the semantics of what "default branch" means
are specific to the underlying source type, but this is generally what you'd
get from a
go get
. - All other branches come next, sorted lexicographically.
- All non-semver versions (tags) are next, sorted lexicographically.
- Revisions, if any, are last, sorted lexicographically. Revisions do not typically appear in version lists, so the only invariant we maintain is determinism - deeper semantics, like chronology or topology, do not matter.
So, given a slice of the following versions:
- Branch:
master
devel
- Semver tags:
v1.0.0
v1.1.0
v1.1.0-alpha1
- Non-semver tags:
footag
- Revision:
f6e74e8d
Sorting for upgrade will result in the following slice:
[v1.1.0 v1.0.0 v1.1.0-alpha1 master devel footag f6e74e8d]
There are a number of factors that can eliminate a version from consideration,
the simplest of which is that it doesn't match a constraint. But if you're
trying to figure out why dep
is doing what it does, understanding that its
basic action is to attempt versions in this order should help you to reason
about what's going on.
What is the default dep ensure -update
behavior for dependencies that are imported but not included as a [[Constraint]]
in Gopkg.toml
?
dep
updates the dependency to the latest semver tag. If there are no semver tags, dep
uses the tip of master.
During dep init
configuration from other dependency managers is detected
and imported, unless -skip-tools
is specified.
The following tools are supported: glide
, godep
, vndr
, govend
, gb
, gvt
, govendor
and glock
.
See #186 for how to add support for another tool.
Only your project's directly imported dependencies are affected by a constraint
entry
in the manifest. Transitive dependencies are unaffected. See How do I constrain a transitive dependency's version?
Sometimes the revision specified in the lock file is no longer valid. There are a few ways this can occur:
- When you generated the lock file, you had an unpushed commit in your local copy of package X's repository in your
GOPATH
. (This case will be going away soon) - After generating the lock file, new commits were force pushed to package X's repository, causing the commit revision in your lock file to no longer exist.
To troubleshoot, you can revert dep's changes to your lock, and then run dep ensure -v -n
.
This retries the command in dry-run mode with verbose logs enabled. Check the output
for a warning like the one below, indicating that a commit in the lock is no longer valid.
Unable to update checked out version: fatal: reference is not a tree: 4dfc6a8a7e15229398c0a018b6d7a078cccae9c8
The lock file represents a set of precise, typically immutable versions for the entire transitive closure of dependencies for a project. But "the project" can be, and is, decomposed into just a bunch of arguments to an algorithm. When those inputs change, the lock may need to change as well.
Under most circumstances, if those arguments don't change, then the lock remains fine and correct. You've hit one of the few cases where that guarantee doesn't apply. The fact that you ran dep ensure and it DID a solve is a product of some arguments changing; that solving failed because this particular commit had become stale is a separate problem.
There are two things that really slow dep
down. One is unavoidable; for the other, we have a plan.
The unavoidable part is the initial clone. dep
relies on a cache of local
repositories (stored under $GOPATH/pkg/dep
), which is populated on demand.
Unfortunately, the first dep
run, especially for a large project, may take a
while, as all dependencies are cloned into the cache.
Fortunately, this is just an initial clone - pay it once, and you're done.
The problem repeats itself a bit when you're running dep
for the first time
in a while and there's new changesets to fetch, but even then, these costs are
only paid once per changeset.
The other part is the work of retrieving information about dependencies. There are three parts to this:
- Getting an up-to-date list of versions from the upstream source
- Reading the
Gopkg.toml
for a particular version out of the local cache - Parsing the tree of packages for import statements at a particular version
The first requires one or more network calls; the second two usually mean
something like a git checkout
, and the third is a filesystem walk, plus
loading and parsing .go
files. All of these are expensive operations.
Fortunately, we can cache the second and third. And that cache can be permanent when keyed on an immutable identifier for the version - like a git commit SHA1 hash. The first is a bit trickier, but there are reasonable staleness tradeoffs we can consider to avoid the network entirely. There's an issue to implement persistent caching that's the gateway to all of these improvements.
There's another major performance issue that's much harder - the process of picking versions itself is an NP-complete problem in dep
's current design. This is a much trickier problem 😜
because we're not crazy people who delight in inviting chaos into our lives, we need to work within one
GOPATH
at a time. -@sdboyer in #247
Out of convenience, one might create a symlink to a directory within their GOPATH/src
, e.g. ln -s ~/go/src/github.com/user/awesome-project ~/Code/awesome-project
.
When dep
is invoked with a project root that is a symlink, it will be resolved according to the following rules:
- If the symlink is outside
GOPATH
and links to a directory within aGOPATH
, or vice versa, thendep
will choose whichever path is withinGOPATH
. - If the symlink is within a
GOPATH
and the resolved path is within a differentGOPATH
, then an error is thrown. - If both the symlink and the resolved path are in the same
GOPATH
, then an error is thrown. - If neither the symlink nor the resolved path are in a
GOPATH
, then an error is thrown.
This is the only symbolic link support that dep
really intends to provide. In keeping with the general practices of the go
tool, dep
tends to either ignore symlinks (when walking) or copy the symlink itself, depending on the filesystem operation being performed.
No.
dep simply doesn't allow relative imports. this is one of the few places where we restrict a case that the toolchain itself allows. we disallow them only because:
- the toolchain already frowns heavily on them
- it's worse for our case, as we start venturing into dot dot hell territory when trying to prove that the import does not escape the tree of the project
For a refresher on Go's recommended workspace organization, see the "How To Write Go Code" article in the Go docs. Organizing your code this way gives you a unique import path for every package.
dep init
provides an option to scan the GOPATH
for dependencies by doing
dep init -gopath
, which falls back to network mode when the packages are not
found in GOPATH
. dep ensure
doesn't work with projects in GOPATH
.
No, with just one tiny exception: dep
preserves /vendor/.git
, if it exists. This was added at cockroachdb's request, who rely on it to keep vendor
from bloating their primary repository.
The reasons why git submodules will not be a part of dep are best expressed as a pro/con list:
Pros
- git submodules provide a well-structured way of nesting repositories within repositories.
Cons
- The nesting that git submodules perform is no more powerful or expressive than what dep already does, but dep does it both more generally (for bzr and hg) and more domain-specifically (e.g. elimination of nested vendor directories).
- Incorporating git submodules in any way would new fork new paths in the logic to handle the submodule cases, meaning nontrivial complexity increases.
- dep does not currently know or care if the project it operates on is under version control. Relying on submodules would entail that dep start paying attention to that. That it would only be conditionally does not make it better - again, more forking paths in the logic, more complexity.
- Incorporating submodules in a way that is at all visible to the user (and why else would you do it?) makes dep's workflows both more complicated and less predictable: sometimes submodule-related actions are expected; sometimes submodule-derived workflows are sufficient.
- Nesting one repository within another implies that changes could, potentially, be made directly in that subrepository. This is directly contrary to dep's foundational principle that
vendor
is dead code, and directly modifying anything in there is an error.
dep
doesn't require imports (or the $GOPATH
) to be updated because go has native support for a vendor directory since version 1.5. You do not need to update import paths to be relative. For instance, import github.com/user/awesome-project
will be found in the project's /vendor/github.com/user/awesome-project
before looking to $GOPATH/src/github.com/user/awesome-project
.
It's up to you:
Pros
- It's the only way to get truly reproducible builds, as it guards against upstream renames, deletes and commit history overwrites.
- You don't need an extra
dep ensure
step to syncvendor/
withGopkg.lock
after most operations, such asgo get
, cloning, getting latest, merging, etc.
Cons
- Your repo will be bigger, potentially a lot bigger,
though
prune
can help minimize this problem. - PR diffs will include changes for files under
vendor/
whenGopkg.lock
is modified, however files invendor/
are hidden by default on GitHub.
In short: make sure you've committed your Gopkg.toml
and Gopkg.lock
, then
just create a tag in your version control system and push it to the canonical
location. dep
is designed to work automatically with this sort of metadata
from git
, bzr
, and hg
.
It's strongly preferred that you use semver-compliant tag names. We hope to develop documentation soon that describes this more precisely, but in the meantime, the npm docs match our patterns pretty well.
This can be a nuanced question, and the community is going to have to work out some accepted standards for how semver should be applied to Go projects. At the highest level, though, these are the rules:
- Below
v1.0.0
, anything goes. Use these releases to figure out what you want your API to be. - Above
v1.0.0
, the general Go best practices continue to apply - don't make backwards-incompatible changes - exported identifiers can be added to, but not changed or removed. - If you must make a backwards-incompatible change, then bump the major version.
It's important to note that having a v1.0.0
does not preclude you from having
alpha/beta/etc releases. The semver spec allows for prerelease
versions, and dep
is careful to not allow
such versions unless Gopkg.toml
contains a range constraint that explicitly
includes prereleases: if there exists a version v1.0.1-alpha4
, then the
constraint >=1.0.0
will not match it, but >=1.0.1-alpha1
will.
Some work has been done towards a tool to that will analyze and compare your code with the last release, and suggest the next version you should use.
Yes. But.
dep
will make it possible for the Go ecosystem to handle
backwards-incompatible changes more gracefully. However, dep
is not some
magical panacea. Version and dependency management is hard, and dependency hell
is real. The longstanding community wisdom about avoiding breaking changes
remains important. Any v1.0.0
release should be accompanied by a plan for how
to avoid future breaking API changes.
One good strategy may be to add to your API instead of changing it, deprecating old versions as you progress. Then, when the time is right, you can roll a new major version and clean out a bunch of deprecated symbols all at once.
Note that providing an incremental migration path across breaking changes (i.e., shims) is tricky, and something we don't have a good answer for yet.
For the most part, you needn't do anything differently.
The only possible issue is if your project is ever consumed as a library. If
so, then you may want to be wary about committing your vendor/
directory, as
it can cause
problems.
If your dependers are using dep
, this is not a concern, as dep
takes care of
stripping out nested vendor
directories.
Add a constraint to Gopkg.toml
that specifies branch: "master"
(or whichever branch you need) in the [[constraint]]
for that dependency. dep ensure
will determine the current revision of your dependency's master branch, and place it in Gopkg.lock
for you. See also: What is the difference between Gopkg.toml and Gopkg.lock?
dep ensure -vendor-only
creates the vendor folder from a valid Gopkg.toml
and Gopkg.lock
without checking for Go code.
This is especially useful for builds inside docker utilizing cache layers.
Sample Dockerfile:
FROM golang:1.9 AS builder
RUN curl -fsSL -o /usr/local/bin/dep https://github.com/golang/dep/releases/download/vX.X.X/dep-linux-amd64 && chmod +x /usr/local/bin/dep
RUN mkdir -p /go/src/github.com/***
WORKDIR /go/src/github.com/***
COPY Gopkg.toml Gopkg.lock ./
# copies the Gopkg.toml and Gopkg.lock to WORKDIR
RUN dep ensure -vendor-only
# install the dependencies without checking for go code
...
Since dep
is expected to change until v1.0.0
is released, it is recommended to rely on a released version.
You can find the latest binary from the releases page.
Sample configuration for Travis CI:
# ...
env:
- DEP_VERSION="X.X.X"
before_install:
# Download the binary to bin folder in $GOPATH
- curl -L -s https://github.com/golang/dep/releases/download/v${DEP_VERSION}/dep-linux-amd64 -o $GOPATH/bin/dep
# Make the binary executable
- chmod +x $GOPATH/bin/dep
install:
- dep ensure
Caching can also be enabled but there are a couple of caveats you should be aware of:
Until recently, we have had intermittent cache corruption that would have been super annoying if it was breaking Travis build too.
Also according to https://docs.travis-ci.com/user/caching/#Things-not-to-cache, they don't recommend it for larger caches.
https://docs.travis-ci.com/user/caching/#How-does-the-caching-work%3F
Note that this makes our cache not network-local, it's still bound to network bandwidth and DNS resolutions for S3. That impacts what you can and should store in the cache. If you store archives larger than a few hundred megabytes in the cache, it's unlikely that you'll see a big speed improvement.
If you are sure you want to enable caching on Travis, it can be done by adding $GOPATH/pkg/dep
, the default location for dep
cache, to the cached directories:
# ...
cache:
directories:
- $GOPATH/pkg/dep