Regarding use of gopkg.in

[alimoeeny]

Hi there,
1- keep up the great work.
2- I was wondering if we can remove the use of gopkg.in. In /log (and maybe other places), a few packages are imported through gopkg.in like gopkg.in/logfmt.v0 and gopkg.in/stack.v1 .
Although there is nothing wrong with this, it is yet another point of possible failure and possibility for attacks.
I am basically suggesting we import them directly with their github urls like using https://github.com/go-logfmt/logfmt instead of gopkg.in/logfmt.v0

If you are happy with this approach I'd be delighted to send a PR.

[peterbourgon]

I am moderately to strongly in favor of this, but I know that @ChrisHines believes in the benefits of gopkg.in. Chris, would you be very much against using GitHub import paths directly?

[ChrisHines]

I do see value in gopkg.in at least until the Go community has a better story for API versioning. I also see it as an implicit declaration of compatibility with the Go kit API Stability Policy.

As for those two specific packages:

gopkg.in/stack.v1 is very stable and importing from the master branch via github.com/go-stack/stack will work fine. Having said that, I am tracking the discussion in Go issue 11432 about a possible new API for call stack information in package runtime. That will more than likely just change the implementation rather than the API.

gopkg.in/logfmt.v0 is incomplete, meaning it hasn't met all of its goals yet. The portion used by Go kit is likely to be stable. I have just started working on this package again and hope to get it to a v1.0.0 soon (for some definition of soon). Importing it via github.com/go-logfmt/logfmt will also work fine and in fact looking at the repo now I see that the test code currently uses that path in the black-box tests which not the best practice when using gopkg.in, mea culpa.

[ChrisHines]

@peterbourgon asked:

Chris, would you be very much against using GitHub import paths directly?

I think it depends on how seriously we take the Go kit API stability policy. I think that in practice we won't have any problems using the github.com import paths, but it would be playing a bit loose with the rules we've established.

I'm happy to discuss and debate the topic to reach a consensus.

[alimoeeny]

I honestly don't know what best fits the communities' needs and expectations, I of course personally prefer github imports.
But I thought this sentence in the README is relevant:

and doesn't recommend use of third-party import proxies.

[ChrisHines]

I see. I didn't notice when that language was added to the README six months ago. Thanks for pointing that out.

Here is a timely thread that discusses the difficulty of the issues here: https://forum.golangbridge.org/t/how-to-fix-the-go-package-management-problem/2026/7

😢

[ChrisHines]

I continue to consider the forces involved here in search of the best least bad way to manage these libraries. There is an approach that may not be totally horrible.

Bumping a package to a new major version would entail making a copy of the repository within the same github user/organization and including the major version number in the name. Let's say that github.com/go-logfmt/logfmt needed a new API that was not backwards compatible, we could copy it to github.com/go-logfmt/logfmt-v2. Of note, the package name declared inside the repo would still be logfmt so that client code would still read nicely and would only need to change import paths and code affected by the breaking changes.

Using this strategy means that each major version of the package could be compliant with the Go kit API stability policy without using gopkg.in while still giving package authors an avenue for evolving APIs based on experience. Newer versions of a repo could retain commit history back through older versions (although probably not minor revisions made after the fork). The main downside is the fragmenting of issue tracking across versions.

[peterbourgon]

The implicit contract between Go kit and its dependencies is and has been to track the dependencies' latest revision. This is largely driven by necessity¹ but I think is also virtuous: it puts pressure on us to keep our work up-to-date within the broader ecosystem. Tools like gopkg.in, which promote API stability as we've defined it, may counterindicate this goal by allowing our package to continue to build with an outdated version of a dependency. Concretely: I would expect that a foopkg-v2 is intended to be a strictly superior version of the original foopkg-v1 (modulo unintentional bugs) and, consequently, we should do our best to adopt it.

Of course, if the v1 package will continue to be maintained, then the impetus to switch is not as strong. But my sense is this situation is very rare. Thoughts?

__
¹ i.e. in the current state of Go package management, libraries cannot pragmatically vendor their dependencies, and so in order to be good citizens must track the version of the dep most likely to be installed by their own importers.

[ChrisHines]

I agree that Go kit should try to stay up to date with the latest, best revisions of its dependencies. The use of gopkg.in is not intended to promote use of old APIs, but to satisfy the API stability policy while allowing for the evolution of the API within a single repository. The API stability policy boils down to this assertion: for a given import path each API revision will be backwards compatible with all prior revisions.

So what I'm trying to work out is the best approach to managing a library that tries to satisfy that assertion while keeping the library repo sane from a maintainer's perspective. Without something like gopkg.in then breaking changes must go in a new repository, which has it's own downsides.

[ORBAT]

One option could also be to vendor dependencies instead of using gopkg.in, but that would only work with Go >= 1.5

[peterbourgon]

Vendoring is a solution for binaries, i.e. Go kit consumers. It's generally inappropriate for libraries, especially those in open ecosystems. See the dependency management section of the README.

[peterbourgon]

Without something like gopkg.in then breaking changes must go in a new repository, which has it's own downsides.

From the perspective of Go kit, I don't mind very much if our dependencies introduce breaking changes. I'm fine with having that detected on a CI build, and just fixing Go kit as quickly as possible. Does this perspective change yours, @ChrisHines? Or perhaps I'm overlooking something important? :)

[ChrisHines]

From the perspective of Go kit, I don't mind very much if our dependencies introduce breaking changes.

That statement seems to contradict the API stability policy linked in the main README.

My perspective remains as I stated in #197 (comment).

Perhaps we need to revisit the API stability policy in the light of vendoring becoming the accepted practice.

[peterbourgon]

Thanks for that. Yes, I agree that the API stability policy is in conflict. My intuition is to update it, or remove it altogether.

[peterbourgon]

I've removed the API stability policy; see #226. All else equal, I would prefer to use direct (github.com) over indirect (gopkg.in) import paths, but I defer to the package author. go-logfmt/logfmt encourages use of gopkg.in so I'm happy to oblige. I think that's a reasonably well-defined rule for this subject, so I'll close the issue. Thanks for the discussion! Please feel free to re-open if there's something I've overlooked.

[ChrisHines]

I have just updated github.com/go-logfmt/logfmt and it no longer recommends gopkg.in.