[RFC] Adding support for creating gb projects that don't contain main packages

[davecheney]

This is a blanket issue to cover questions on the go-pm mailing list and other issues.

Supporting gb projects that don't have a main package

Currently a gb project is defined as a collection of packages and main packages (commands), along with supporting source code in vendor/src/.

At the GDG presentation in Berlin in April I said that gb was not suitable for libraries (projects consisting of non main packages, to use an imprecise definition), after reflecting on this for a few days I think this restriction may have been premature and probably motivated by trying to simplify the story for the GDG audience.

Specifically there is nothing preventing a library author using gb, and in fact many of the reasons why a library author would want to take advantage of the reproducibility guarantees that a gb project attempts to offer.

Distinguishing between projects and library projects

In principal a gb project, and a gb library project (need to find a better description) are the same, they both contain some code in $PROJECT/src/, they both may contain some supporting dependencies in vendor/src/. The only remaining point of differentiation is gb build executed in the former may produce some output in $PROJECT/bin/, while the latter most likely will not.

In short, this prohibition was short sighted and unnecessary.

However, with this prohibition relaxed, a new question arises: how can one gb project consume another gb project as a dependency ? It turns out that this isn't a very difficult proposition, but it is regrettably manual at the moment. Consider this scenario, mongodb/ is the project root of some library, twitter is the project root of a gb project that consumes mongodb's packages as a dependency. The names are completely arbitrary.

mongodb/
    src/
        client/
            client.go
    vendor/
        src/
            bson/
                bson.go
            gocheck/
                tests.go

twitter/
    src/
        cmd/
            twitter/
                main.go
    vendor/
        src/

mongodb and twitter are developed separately, they live in different respositories. The mongodb client has a dependency on bson, and includes its source in its' vendor/src/ directory. mongodb also has a dependency on gocheck for tests.

The process for vendoring mongodb into twitter/vendor/src/ would be to copy the contents of mongodb/src/* into twitter/vendor/src/ and also selectively copying mongodb/vendor/src/* into twitter/vendor/src/. The project owner of twitter may choose to omit the gocheck dependency of mongodb, of they may choose to include it for completeness.

Open issues

The process described above is not difficult, but it does require the package owner to be cognoscente of their actions when importing new dependencies into their project. Automating this initial import with tools is difficult at the moment due to a lack of version information in Go code in general (see #20 for a distinction between version and revision). Automating the orderly upgrade of a set of vendored dependencies is more difficult again.

Tackling these issues are outside the scope of this specific bug, and arguably outside the code mandate of gb -- it just provides the mechanism to work with a "project" of Go code, it is mute on how that source is assembled -- but feels like ample fuel for a plugin that tackles these complicated issues.

[andrewchambers]

For this reason, libraries in cargo don't specify which version they go with, in gb's case that would be analogous to not allowing (or at least discouraging) vendoring when the project is meant as a library.

[davecheney]

@andrewchambers i don't think there is anything needed here. Go code already indicates the names of the packages it builds upon, there is no formal way for a Go package to say it needs a specific revision (nee version) of a dependency, so there is no need for us to discourage the use of something that doesn't exist.

[andrewchambers]

Its just annoying if the library has one vendors commit in its manifest and you have another in yours. Gb can just use precedence or something, I don't mind. I am pretty happy with Gb being program only.

[bruth]

From experience in other languages, secondary dependencies of explicit dependencies in a project are included as is unless the same dependency is already present. Many of the existing dependency/manifest files allow for specifying a range of versions (revisions) that are valid for the library to behave correctly. This of course conflicts with the current notion of vendoring since a specific snapshot of the dependency is versioned with the source code (as well as the code at a particular version being available). There may need to be a trade-off defined for libraries that allow a range of versions to pull those dependencies in dynamically. The basic logic would be:

• Fetch project dependency
• Check if project has dependencies that overlap with a dependency's dependencies
• Check if the version requirements conflict between the two
• If not, fetch the most recent common version
• Otherwise return an error and push it on the developer to update their dependency version locally or contact the library maintainer to see if the version requirement is necessary.

The bottom line is that libraries are generally not as strict with dependency versions as projects, so that needs to be considered when vendoring a library in the gb structure. A project's manifest file could include a dependency's dependencies with the version [ranges] specified.

[davecheney]

Thank you for your comment. I want to point out that I believe that Go
packages do not have versions, at least not in the way that we commonly
accept maven poms, java jars, npm modules, python eggs, etc to have
versions.

Because of this, it is not possible to specify version ranges as no such
thing exists in the Go ecosystem at this time.

This is of course not to say that such things should exist, as they are
clearly useful, but they don't exist today, in June 2015, so that's the
hand that gb has been dealt.

On Wed, Jun 10, 2015 at 12:21 PM, Byron Ruth notifications@github.com
wrote:

From experience in other languages, secondary dependencies of explicit
dependencies in a project are included as is unless the same dependency is
already present. Many of the existing dependency/manifest files allow for
specifying a range of versions (revisions) that are valid for the
library to behave correctly. This of course conflicts with the current
notion of vendoring since a specific snapshot of the dependency is
versioned with the source code (as well as the code at a particular version
being available). There may need to be a trade-off defined for libraries
that allow a range of versions to pull those dependencies in dynamically.
The basic logic would be:

• Fetch project dependency
• Check if project has dependencies that overlap with a dependency's
  dependencies
• Check if the version requirements conflict between the two
• If not, fetch the most recent common version
• Otherwise return an error and push it on the developer to update
  their dependency version locally or contact the library maintainer to see
  if the version requirement is necessary.

The bottom line is that libraries are generally not as strict with
dependency versions as projects, so that needs to be considered when
vendoring a library in the gb structure. A project's manifest file
could include a dependency's dependencies with the version [ranges]
specified.

—
Reply to this email directly or view it on GitHub
#49 (comment).

[bruth]

So then it may be that, in June 2015, gb should either pave the way for standardizing the notion of a version or simply does not support libraries. I support the former since gb already has momentum with "deviating from the norm" (go get).

On Jun 9, 2015, at 10:24 PM, Dave Cheney notifications@github.com wrote:

Thank you for your comment. I want to point out that I believe that Go
packages do not have versions, at least not in the way that we commonly
accept maven poms, java jars, npm modules, python eggs, etc to have
versions.

Because of this, it is not possible to specify version ranges as no such
thing exists in the Go ecosystem at this time.

This is of course not to say that such things should exist, as they are
clearly useful, but they don't exist today, in June 2015, so that's the
hand that gb has been dealt.

On Wed, Jun 10, 2015 at 12:21 PM, Byron Ruth notifications@github.com
wrote:

From experience in other languages, secondary dependencies of explicit
dependencies in a project are included as is unless the same dependency is
already present. Many of the existing dependency/manifest files allow for
specifying a range of versions (revisions) that are valid for the
library to behave correctly. This of course conflicts with the current
notion of vendoring since a specific snapshot of the dependency is
versioned with the source code (as well as the code at a particular version
being available). There may need to be a trade-off defined for libraries
that allow a range of versions to pull those dependencies in dynamically.
The basic logic would be:

• Fetch project dependency
• Check if project has dependencies that overlap with a dependency's
  dependencies
• Check if the version requirements conflict between the two
• If not, fetch the most recent common version
• Otherwise return an error and push it on the developer to update
  their dependency version locally or contact the library maintainer to see
  if the version requirement is necessary.

The bottom line is that libraries are generally not as strict with
dependency versions as projects, so that needs to be considered when
vendoring a library in the gb structure. A project's manifest file
could include a dependency's dependencies with the version [ranges]
specified.

—
Reply to this email directly or view it on GitHub
#49 (comment).

—
Reply to this email directly or view it on GitHub.

[davecheney]

gb projects can be libraries, which I interpret as projects which don't have command (package mains) packages, that exists today.

What is missing, not just from gb, but the whole Go ecosystem is a widely accepted notion of releasing a project (or package, or set of packages). That is, taking a specific copy of some Go source code, possibly identified by a revision hash or marker in a VCS system, and giving that a human (and possibly machine) readable version identifier.

This is not something gb seeks to do, gb is only concerned with building source it finds on disk in the prescribed project format.

With that said, this is absolutely something that needs to be done, but that is a different discussion, for a different issue tracker.

[bruth]

I understand that and agree gb should focus on what it has be designed to do (and as you made very clear when describing the gb-vendor tool). By paving the way I simply mean that the gb community appears to be in a good position to define:

taking a specific copy of some Go source code, possibly identified by a revision hash or marker in a VCS system, and giving that a human (and possibly machine) readable version identifier.

Based on the constraints you've declared for the tool, I am not sure there is anything that could or should be done for using gb in a library context.

[davecheney]

Based on the constraints you've declared for the tool, I am not sure there is anything that could or should be done for using gb in a library context.

Can you please clarify what you mean by library for me. Gophers can use gb today to develop gb projects which are libraries, ie, they are projects that contain no main package and there is an expectation that their source will be vendored into another project.

As for versioning Go packages, this is not something that gb is focused on, but it is not to say that I am not working on it.

[bruth]

Sorry:

I am not sure there is anything [more] that could or should be done for using gb in a library context.

I simply mean that what has been described in this issue sounds to all be centered around dealing with versions. As you stated this is not something that the gb tool should directly deal with, but rather a plugin.

[davecheney]

ok. Thanks for confirming.

To be clear, the lack of a version identifier for Go packages is a very big
problem, but I'm addressing it in other forums.

On Wed, Jun 10, 2015 at 8:27 PM, Byron Ruth notifications@github.com
wrote:

Sorry:

I am not sure there is anything [more] that could or should be done for
using gb in a library context.

I simply mean that what has been described in this issue sounds to all be
centered around dealing with versions. As you stated this is not something
that the gb tool should directly deal with, but rather a plugin.

—
Reply to this email directly or view it on GitHub
#49 (comment).

[mmlb]

@davecheney what forums would those be? Can I haz link(s)?

[davecheney]

https://groups.google.com/forum/?hl=en-GB#!forum/go-package-management
http://getgb.io/rationale/

And it mainly consists of me pointing out that go packages don't have
versions.

On Fri, Jun 12, 2015 at 12:42 AM, Manuel Mendez notifications@github.com
wrote:

@davecheney https://github.com/davecheney what forums would those be?
Can I haz link(s)?

—
Reply to this email directly or view it on GitHub
#49 (comment).

[mmlb]

ok, I've seen those just thought there may have been something more substantial or something I may have missed.

Thanks

[davecheney]

nope. i can only boil one ocean at a time.

On Fri, Jun 12, 2015 at 1:28 AM, Manuel Mendez notifications@github.com
wrote:

ok, I've seen those just thought there may have been something more
substantial or something I may have missed.

Thanks

—
Reply to this email directly or view it on GitHub
#49 (comment).

[mmlb]

Welp, seems like its working. Keep making soup sir.

[bruth]

Arguably relevant to the discussion (posted on HN): https://groups.google.com/forum/#!msg/golang-dev/74zjMON9glU/4lWCRDCRZg0J

[aviflax]

Sorry if I’m missing something, but how do I include a gb library dependency via gb fetch? It puts the whole project into my vendor path, which means my import paths will need to include stuff like project/src/project which is quite ugly.

[davecheney]

@aviflax this isn't supported by gb vendor yet. It's on the roadmap. For the moment you'll have to vendor manually by copying the relevant parts from one gb project to another.

[aviflax]

Got it, thanks!

[nvcnvn]

Hi @davecheney I just try gb for some day and far from understand what gb build do.
Can gb recursive check if an folder in /vendor is a gb project or not?
Can gb build and copy package file of the twitter/vendor/src/mongodb/src/ only and move the pacakges file to the correct folder (twitter/pkg/linux-amd64/mongodb/client.a)?
Can gb build with only the package file?

[davecheney]

@nvcnvn gb does not use the vendor mechanism that is used by go get.

[yoshiwaan]

I know this is old and maybe not the right place for it, but I have a question on what is the recommended best way to deal with libraries that vendor themselves.
I just spent a pretty unhappy few hours trying to get both github.com/docker/docker and github.com/docker/libcompose to play nicely in a gb project I'm working on.
In the end I had to use the version of docker in github.com/docker/libcompose/vendor/src/github.com/docker/docker as anything which I sourced via gb vendor fetch or go get had the wrong API.

My question is, how do you recommend to handle this sort of painful situation right now with gb?

Is it possible that if gb vendor fetch has to recursively fetch dependencies that it looks in the vendor directory of the library you asked to fetch first before it goes and gets it via URL?