Skip to content

proposal: Vendor specification and experimental repository fetch code #13517

Closed
@kardianos

Description

@kardianos

Proposal: Vendor specification and experimental repository fetch code

Author(s): Daniel Theophanes

Last updated: 2015-12-06

Abstract

Establish a specification file format that lists dependency revisions and
a package in the golang.org/x/exp repository that discovers, reads, and downloads
packages at a given revision. Tools may continue to use other formats to generate
this file.

Background

Many developers wish to specify revisions of vendor dependencies without copying
them into the repository. For a case study I will bring up two:

A) https://github.com/cockroachdb/cockroach

B) https://github.com/gluster/glusterd2

(A) uses github.com/robfig/glock which specifies revisions for each remote repository
in file in the project root called "GLOCKFILE". A partial list of the file is:

cmd golang.org/x/tools/cmd/stress
cmd golang.org/x/tools/cmd/stringer
github.com/agtorre/gocolorize f42b554bf7f006936130c9bb4f971afd2d87f671
github.com/biogo/store 3b4c041f52c224ee4a44f5c8b150d003a40643a0
github.com/cockroachdb/c-rocksdb bf15ead80bdc205a19b3d33415b23c156a3cf371
github.com/cockroachdb/c-snappy 5c6d0932e0adaffce4bfca7bdf2ac37f79952ccf
github.com/cockroachdb/yacc 443154b1852a8702b07d675da6cd97cd9177a316
github.com/coreos/etcd a423a55b142c2b9a82811604204cddbccd0a9cf9

(B) uses github.com/Masterminds/glide which specifies revisions for each remote
repository in a file in the project root called "glide.yaml". This file contains:

parent: null
package: github.com/gluster/glusterd2
import:
- package: github.com/gorilla/context
  version: 1c83b3eabd45b6d76072b66b746c20815fb2872d
- package: gopkg.in/tylerb/graceful.v1
  version: 48afeb21e2fcbcff0f30bd5ad6b97747b0fae38e
- package: github.com/pborman/uuid
  version: cccd189d45f7ac3368a0d127efb7f4d08ae0b655
- package: github.com/gorilla/mux
  version: ad4d7a5882b961e07e2626045eb995c022ac6664
- package: golang.org/x/net
  version: b4e17d61b15679caf2335da776c614169a1b4643
- package: github.com/docker/libkv
  version: 93099f38de7421e6979983652730a81e2bafd578
- package: github.com/codegangsta/negroni
  version: c7477ad8e330bef55bf1ebe300cf8aa67c492d1b
- package: golang.org/x/sys
  subpackages:
  - /unix
- package: github.com/meatballhat/negroni-logrus
  version: dd89490b0057cca7fe3fa3885f82935dfd430c2e
- package: github.com/Sirupsen/logrus
  version: v0.8.7
- package: github.com/hashicorp/consul
  version: v0.5.2

I would like to point out a few features these tools provide:

  • Specify commands to fetch.
  • Specify repositories at a given revision.
  • Specify repositories at a given version.
  • Specify a sub-tree of packages in a given repository.

Right now each vendor tool specifies these same properties in different formats.
A common tool cannot be built that reads a single file and downloads the needed
dependencies. This isn't a huge burden on a dedicated developer, but for a
user passing by who just wants to build the source quickly, it is an impediment.

Proposal

I propose specifying a single file format that will describe packages sourced
outside the project repository. I also propose adding a packge to the
golang.org/x/exp repository that discovers, reads, and optionally downloads
third party packages.

Furthermore I propose using the specification found at
https://github.com/kardianos/vendor-spec with one addition as the basis for this
specification. The addition is:

Package []struct {
    ...

    // Tree indicates that the specified folder, along with all sub-folders
    // are required.
    Tree bool `json:"tree"`

    ...
}

Both the specification and the proposed package will be considered experimental
and subject to change or retraction until at least go1.7. This process will be
done with an eye to possibly adding this feature to go get.

Rationale

The vendor file format needs to be able to be read and written with standard
go packages. This adds to the possibly that go get could fetch packages
automatically.

Vendor tools exist today that download packages from a specification. They are just
incompatible with each other despite using the same information to download the
dependencies. If we can agree on a single format for tools to write to, even if
it isn't the primary format for that tool, all tools and possibly go get can
download dependencies.

Existing vendor tools and their formats don't always handle corner cases or
different approaches. For example current tool file formats can't handle the
case of vendoring a patched version of a standard library package (this
would have been useful for crypto/tls forks for detecting the heartbleed
attack and for accessing MS Azure).

I am proposing a file format that "govendor" uses. I'm not trying to put my own
tool as central. Infact, "govendor" was built to validate the "vendor-spec"
proposal. The "vendor-spec" has received significant external contributions
and as such "govendor" has changed to match the spec (and will continue to do so).

Compatibility

This will be standardization of existing practices. There is no go1 compatibility
issues. Existing tools can treat the specification as a write only file.

Implementation

A file format to describe vendor packages should be accepted when this
proposal is accepted. Should this proposal be accepted a new package
should be added to the "golang.org/x/exp" repository to support reading
the vendor file and downloading packages. The author of this proposal
offers to create or assist in creating this package. This would be created
within 2 months of the proposal being accepted.

Risks

It would be ideal if other vendor tool package authors could agree to at least
write to a standard file format informally and collaboratively. Indeed the largest
risk is if vendor tools fail to write the common file format. However I think
unless there is a tangible benefit (such as go get support) there will continue
to not be a reason to collaborate on a standard.

Open issues

The proposed standard file format uses JSON, which might be better then XML, but
harder to write by then something like TOML. Tools that want the vendor file
to be hand created will be forced to generate this file from a different file.

The file format specifies packages, not repositories. Repositories can be specified
by using the root path to the repository and specifying "tree": true, but it isn't
the default for the format. Some people may take issue with that as they are used
to or desire tools that only work at the repository level. This could be a point
of division. From experience I absolutely love vendoring at the package level
(this is what github.com/kardianos/govendor does by default).

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions