Including minimal subset of github.com/hlandau/degoutils/* for better distributability

[petercolberg]

I would like to package acmetool for Debian, as a robust alternative to the official Let’s Encrypt client.

In Debian each Go source package is packaged as a separate Debian package, which works well for almost all dependencies of acmetool. The degoutils packages however require altogether ~50 external modules for compilation and testing, while only a tiny subset is actually used by acmetool. Would you consider including the mimimal required subset of files from degoutils in the acme repository?

This would pave the way for shipping acmetool with Debian and Debian derivatives such as Ubuntu.

[hlandau]

This is a problem only created by Debian's very silly way of packaging Go dependencies. Which, come to think of it, I need to write an article about at some point...

There doesn't seem to be much documentation on their policies, so what is their policy on vendorization? Plenty of Go packages use vendorization. If, purely hypothetically, acmetool were to switch to vendored packages, would this fix the issue, or would Debian require a policy of aggressive de-vendorization of package dependencies?

[petercolberg]

I am not sure whether a policy exists. The Debian Go Packaging guidelines focus on technical aspects.

I have not packaged Go packages for Debian before. My goal is to get acmetool into Debian, and otherwise work with the existing Debian Go packaging tools as is. That said, I see two major points for creating separate Debian packages for each Go source package and thus avoiding code duplication.

First, this way all source code is tested using the upstream unit tests shipped with each package.
Second, the most laborious task of packaging software for Debian is creating the machine-readable debian/copyright file, which lists authorship and licenses for each and every source file. This is important to many Debian users (e.g., Debian derivatives), since it ensures that they may modify and/or redistribute the packages. Packaging each Go package separately ensures that this task has to be done once only.

Fortunately all of the above is not relevant for this request. Instead I would like to avoid packaging degoutils separately, since most of its functionality would be unused, and, as you state in the README, the package is not intended for public use anyway. The question is how to best include the required functionality in acme.

As I understand, the Debian dh_golang helper supports only one Go package prefix per Debian source package (e.g., github.com/hlandau/acme). Thus it seems easiest if the required files were integrated into acme, e.g., as github.com/hlandau/acme/utils.

acme imports the following packages from degoutils:

github.com/hlandau/degoutils/buildinfo
github.com/hlandau/degoutils/clock
github.com/hlandau/degoutils/net
github.com/hlandau/degoutils/os
github.com/hlandau/degoutils/text
github.com/hlandau/degoutils/xlogconfig

clock contains a single file that could be copied to acme/utils. From text only the function parseBoolUser() with dflt = true is used. A subset only of net and os is needed. Maybe xlogconfig could be integrated into the https://github.com/hlandau/xlog package?

[petercolberg]

Just to make sure there is no confusion: My request is about degoutils only. All other imported modules are perfectly fine as external modules, and I have already packaged them as dependencies of acmetool.

[hlandau]

What I suggest is you use the Go1.5+ vendor directory system. This will avoid the need to depend on Debian packages.

This script may help you construct a vendor directory, which should be placed at $GOPATH/src/github.com/hlandau/acme/vendor and which will need to contain all dependencies, not just degoutils, to work properly.

[petercolberg]

Sigh, I thought I was close to packaging acmetool, but degoutils appears to be an impossible hurdle.

Debian frowns upon vendorization for many reasons, two given above.

Why did you decide to lump together so many unrelated files in one module?

Go at Google: Language Design in the Service of Software Engineering
https://talks.golang.org/2012/splash.article#TOC_7.

Through the design of the standard library, great effort was spent on controlling dependencies. It can be better to copy a little code than to pull in a big library for one function.

[hlandau]

Go has no concept of 'modules', only packages. A repository may have many packages, but where the demarcation of a repository ends matters only to go get, and certainly not to go build. In particular, importing a package under degoutils does not pull the entirety of the repository into the binary. go build makes no semantic distinction between the difference between $GOPATH/src/github.com/foo/bar and $GOPATH/src/github.com/foo/baz and $GOPATH/src/github.com/foo/bar/alpha and $GOPATH/src/github.com/foo/bar/beta. If Debian has decided to package by repository, that's ultimately an arbitrary decision taken by Debian, not the design of the Go toolchain.

Suppose, for example that I depend on some package foo, and so does some other program. My program requires foo v1. The other requires foo v2. How is Debian supposed to package both of these? They have decided to (quite pointlessly) package the source, but of course the GOPATH can either contain v1 or v2, but not both. Since all Go binaries are statically linked, this process serves no purpose. By comparison, if Debian had simply decided to build a new GOPATH for each binary (perhaps based on a list of commit hashes, like the tool I linked above generates), this problem wouldn't exist and they'd still have a list of dependencies for whatever purposes they like, tracking security updates, etc.

From my perspective, what Debian is doing is vendorization. It is Debian's methodologies here which resemble the issues with vendorization. Debian's entire way of packaging Go software is nonsensical.

The entire thing is a farce. If Go dependency foo in Debian is locked at v1 because binary bar depends on v1, why does that mean another binary baz which prefers (or requires) v2 has to suffer for it? Nothing in Go requires this.

Debian may "frown" on vendorization. The question is whether they'll reject it. If not, I suggest you take that route. I have no intention of checking a vendor directory into the acme repository, so any vendor directory you assemble is as fresh as you like. There is no meaningful difference between providing packages for import by putting them in $GOPATH/src/github.com/hlandau/acme/vendor and putting them in $GOPATH/src.

[petercolberg]

On Sun, Mar 06, 2016 at 10:52:09AM -0800, Hugo Landau wrote:

Go has no concept of 'modules', only packages. A repository may have many packages, but where the demarcation of a repository ends matters only to go get, and certainly not to go build. In particular, importing a package under degoutils does not pull the entirety of the repository into the binary. go build makes no semantic distinction between the difference between $GOPATH/src/github.com/foo/bar and $GOPATH/src/github.com/foo/baz and $GOPATH/src/github.com/foo/bar/alpha and $GOPATH/src/github.com/foo/bar/beta.

Yes, I know that go build of cmd/acmetool only pulls in the required files.

If Debian has decided to package by repository, that's ultimately an arbitrary decision taken by Debian, not the design of the Go toolchain.

Yes, that is a decision of the Debian Go team. It is not arbitrary though. Debian wishes to provide secure, tested software with clear licensing terms to its users. One way to do that for the Go universe is to map each repository to a package, and thus perform the packaging work (e.g., assembling debian/copyright) only once per Go library (which comprises one or multiple Go packages sharing the same package prefix).

Suppose, for example that I depend on some package foo, and so does some other program. My program requires foo v1. The other requires foo v2. How is Debian supposed to package both of these?

I am new to Go programming, but this appears to be a solved problem. https://gopkg.in/ provides a redirection service for authors that wish to make breaking API changes, i.e. increase the major version of their library. Debian packages major versions separately; e.g., gopkg.in/hlandau/svcutils.v1 is packaged as golang-gopkg-hlandau-svcutils.v1.

They have decided to (quite pointlessly) package the source, but of course the GOPATH can either contain v1 or v2, but not both. Since all Go binaries are statically linked, this process serves no purpose. By comparison, if Debian had simply decided to build a new GOPATH for each binary (perhaps based on a list of commit hashes, like the tool I linked above generates), this problem wouldn't exist and they'd still have a list of dependencies for whatever purposes they like, tracking security updates, etc.

The static linking is indeed an issue. However, packaging each Go library separately still makes it much easier to provide security updates. Each Go package contains a Build-Using: field that indicates the exact package versions of its build dependencies. So when one Go library is updated to fix a security issue, all that is needed is to trigger an automatic rebuild of all Go binary packages that contain that library in Build-Using:.

From my perspective, what Debian is doing is vendorization. It is Debian's methodologies here which resemble the issues with vendorization.

Debian is certainly not doing vendorization, rather the opposite, avoiding source code copies at all cost.

Debian may "frown" on vendorization. The question is whether they'll reject it. If not, I suggest you take that route. I have no intention of checking a vendor directory into the acme repository, so any vendor directory you assemble is as fresh as you like. There is no meaningful difference between providing packages for import by putting them in $GOPATH/src/github.com/hlandau/acme/vendor and putting them in $GOPATH/src.

I will try that route then. I am sure Debian will be fine with including a copy of degoutils, since it is a private library with an unstable API. Though I still think that the imported files are so trivial that you could just include them to avoid the whole dependency.

[petercolberg]

The vendoring of degoutils works, but has the disadvantage that the upstream version information is lost.

I settled on separately packaging the subset needed by acmetool. git archive allows selecting specific files while retaining the git commit id and date, which produces a reproducible partial upstream tarball.