Better Package Versioning

[slimsag]

From @slimsag on September 16, 2014 12:7

One of the things that really bothers me about Go is it's lack of official package versioning. Today, Azul3D uses a versioning scheme like gopkg.in does -- except on the azul3d.org domain.

Pros of versioned import paths (in my opinion):

• Reproducible builds are easier -- but not guaranteed ("The new text renderer has the same API but renders text at half the resolution and now it looks all ugly!" or "I ran into some hidden corner case.").
• It looks like a correct approach to newcomers.

Cons of versioned import paths (in my opinion):

• A lot of cases require bumping the version number (i.e. "I just changed one tiny thing -- and now we must release a new version of the package with just one tiny change") and it looks silly.
• If new features are added to a package, say foo.v1, if the API is the same it keeps the same version number (v1) -- so, yes, you have to update your v1 package multiple times to get new features which looks silly.
• Changing import paths is not easy -- but tools do exist to do it for you.

Other solutions are:

• Go-Package-Store -- Web GUI that shows available updates to all your packages.
• godep -- tool to save and restore dependencies from a JSON file for truly reproducible builds.

I want to better understand how the rest of the Go community feels about this, so I've created a survey we can all fill out to get a better understanding of the most liked solutions tot his.

Take the survey

View the results

Copied from original issue: azul3d-legacy/issues#25

[slimsag]

From @robertmeta on September 16, 2014 16:26

A lot of your concerns / questions in a more general way are spoken about on http://semver.org

[slimsag]

From @robertmeta on September 16, 2014 16:27

"Reproducible builds are easier" -- another way of thinking about it is -- "bug fixes actually get distributed!". When people pin to a specific commit, they end up falling farther and farther behind and get scared to update. When you you get a .v3 ... you are saying "I want anything that works with the v3 API I am already using, bring the bug fixes and corrections!". This isn't specific to go, a lot of build systems have the idea of generalized deps to major versions (3, >2<4, 3, etc.)

"A lot of cases require bumping the version number" -- seems like you are saying -- "I like to make breaking changes inside major version numbers" to which I say PLEASE STOP. That is just cruel, how the hell am I supposed to know when you are breaking things? The entire point of having a version is a promise of compatibility -- if you don't want to do this, just leave everything v0 and have it be the wild wild west. Additionally, if you change the text rendering (even without breaking the API) and you consider that important, create a new version, you are the god-king of your little versioning world.

"feature... the API is the same it keeps the same version number" -- that is a choice, if you think new features merit a new version number, do it. I promise you, you won't run out of version numbers for a long time, I for one would be delighted to be using .v514 if it was better than v513

[slimsag]

From @dskinner on September 16, 2014 16:43

Mostly what @robertmeta said except I'd keep src local. What something like gopkg.in does is try to shoehorn in version resolving into go get, which i think is nice enough, but is not totally necessary. Versioning can be independent of go get use. So if I have version 1.2 copied local to my super-important-project, I can look at the repo and see the latest 1.x tag is 1.8 and feel confident that the transition should be mostly painless while receiving important fixes and updates.

[slimsag]

From @cespare on September 16, 2014 16:56

My 2 cents: all of the options in common use in the Go community today more or less suck.

For projects that are important (e.g., all the code at $dayjob), I will pin specific versions of library code using my tool glp, because I don't trust any third-party author to not introduce breaking changes.

So I don't particularly mind if libraries don't have any form of versioning. If they use some kind of semver scheme, such as via gopkg.in, it is a convenience for me (because then if I update versions I can anticipate whether or not there will be breaking changes, although it's still my job to test it out and make sure -- trust but verify).

In any case, as Azul3d is a set of libraries, I don't think that gopep is applicable. Like glp, it's best suited for applications making use of third-party libraries, not the libraries themselves.

I also don't see how Go-Package-Store is relevant as far as package versioning is concerned.

To summarize my opinion:

• What you're doing right now is okay. Please don't make any drastic changes.
• I would slightly prefer semver numbering to the single-integer system you're currently using, as it would give me more information about changes.
• People who pull in the latest version of any third-party library, even with the promise of versioning, are responsible for checking what updates they're getting, and if their code gets broken by go get -u they deserved it.

[slimsag]

From @niemeyer on September 16, 2014 17:30

Agree with what others have said, and specially @robertmeta.

Just one minor note: please do not support vN.M style number in the URL. It is okay to support it in the project itself, though, via tags and whatever else. The background is here.

[slimsag]

@robertmeta

"A lot of cases require bumping the version number" -- seems like you are saying -- "I like to make breaking changes inside major version numbers" to which I say PLEASE STOP. ...

I think there is a miss-communication going on here and I apologize for it. I've read over my exact wording and I'm not sure how you are getting that idea from it -- so allow me to elaborate: I agree with everything you said. Here is where I was confused originally and what that statement was intended to represent:

For the azul3d.org/audio.v1 package, we need to (see azul3d-legacy/audio#3) to change the definition of one single method, for example from Seek(sample uint64) error to Seek(offset int64, whence int) (int64, error).

What my statement intended to say about the above situation -- is that we would have to bump from v1 to v2 in order to make that change, because it is an API incompatible change. To me it feels strange that there would be a new package version for such a small change (when most likely no other changes need to be made to the package over the next few months).

[slimsag]

@cespare

I also don't see how Go-Package-Store is relevant as far as package versioning is concerned.

I placed this here under the context of keeping up-to-date with changes made remotely -- but you are absolutely correct.

What you're doing right now is okay. Please don't make any drastic changes.

My intent wasn't to make any drastic changes at all -- I am very sorry that I gave off this impression to everyone.

[slimsag]

@niemeyer

Just one minor note: please do not support vN.M style number in the URL. It is okay to support it in the project itself, though, via tags and whatever else. The background is here.

Sadly, we already do this with azul3d.org/native/glfw.v3.1 (the intent was to align the package version with the literal GLFW version).

I hadn't thought about any of that background though regarding conflicting minor versions bringing two versions of the same package into a program.

Still, it's interesting that you say this and then @cespare says:

I would slightly prefer semver numbering to the single-integer system you're currently using, as it would give me more information about changes.

@cespare: Do you mean via git tags (i.e. internally) or within the import path (e.g. azul3d.org/pkg.v1.1)?

[slimsag]

From @niemeyer on September 16, 2014 20:22

@slimsag I don't see the misunderstanding from @robertmeta. He addressed exactly your point:

For the azul3d.org/audio.v1 package, we need to (see azul3d-legacy/audio#3) to change the definition of one single method, for example from Seek(sample uint64) error to Seek(offset int64, whence int) (int64, error).

In other words, you're suggesting that you would "like to make breaking changes inside major version numbers".

I would not do that. Instead, if you want to preserve work on v1 for a little longer, introduce a new method with the signature you want, and preserve the old one so code continues working. Then, when you bump to v2, polish that and any other similar changes you've done. I do that sort of thing all the time.

[slimsag]

From @niemeyer on September 16, 2014 20:23

Sadly, we already do this with azul3d.org/native/glfw.v3.1 (the intent was to align the package version with the literal GLFW version).

That's a bad idea, for all the reasons described here:

niemeyer/gopkg#14

[slimsag]

From @dskinner on September 16, 2014 20:23

I dont know what @cespare meant, but I know I was referring to git tags and
then if pkg.v1 is gotten via go get under such a scheme, it pulls in
the latest minor revision, just as gopkg.in works.

On Tue, Sep 16, 2014 at 3:20 PM, Stephen Gutekanst <notifications@github.com

wrote:

@niemeyer https://github.com/niemeyer

Just one minor note: please do not support vN.M style number in the URL.
It is okay to support it in the project itself, though, via tags and
whatever else. The background is here.

Sadly, we already do this with azul3d.org/native/glfw.v3.1 (the intent
was to align the package version with the literal GLFW version).

I hadn't thought about any of that background though regarding conflicting
minor versions bringing two versions of the same package into a program.

Still, it's interesting that you say this and then @cespare
https://github.com/cespare says:

I would slightly prefer semver numbering to the single-integer system
you're currently using, as it would give me more information about changes.

@cespare https://github.com/cespare: Do you mean via git tags (i.e.
internally) or within the import path (e.g. azul3d.org/pkg.v1.1)?

—
Reply to this email directly or view it on GitHub
azul3d-legacy/issues#25 (comment).

[slimsag]

From @cespare on September 16, 2014 20:33

@slimsag I was talking about git tags; I don't think you can reasonably make the full semver package part of the import URL for the reasons @niemeyer gave in niemeyer/gopkg#14.

[slimsag]

In other words, you're suggesting that you would "like to make breaking changes inside major version numbers".

Ahh, I see it now. It was a misunderstanding on my part.

That's a bad idea, for all the reasons described here:

You're absolutely right. I think the proper resolution is to have versions not be in sync with GLFW ones. I.e. GLFW v3.1 becomes just azul3d.org/native/glfw.v3, maybe GLFW v3.2 will become azul3d.org/native/glfw.v4 if such changes in the Go API warrant it. (the good news is -- not a lot of people use the old import path directly right now, so we have the opportunity to cut this practice sooner rather than later.)

@niemeyer I want to kind of personally apologize to you if it seemed like any of this was trying to derail or bash gopkg.in in any way. Looking back on it now it's very clear to me how it could be easily perceived as such. Your work on gopkg.in has actually been very inspiring to me.

I did a really bad job communicating most of this cleanly, and it became apparent to me after only a few hours of posting that the survey probably wasn't warranted. I didn't do my research well enough.

Nonetheless, I think the results of the survey speak highly in favor of the gopkg.in versioning scheme (in combination with the use of godep).

Going forward I will be looking at making azul3d.org use the same exact versioning scheme as gopkg.in rather than just similar to it like it is right now.

[slimsag]

From @niemeyer on September 16, 2014 23:0

@slimsag Thanks for being nice, but there's really no need to apologize. The status quo of gopkg.in was only achieved with debate, and the push is indeed an attempt to solve the problem at a reasonable scale so we can benefit of each other's packages with some sanity. That kind of conversation is a good way to make sure we're still on the right track for that.

@dskinner If you mean using a tag/branch such as v1.3 under the URL gopkg.in/mypkg.v1 (no minor), that's supported already. The system will automatically select the most recent version. You can see that mapping on the right hand side for packages that use it (for example: https://gopkg.in/fsnotify/fsnotify.v1). But if you mean actually having v1.3 in the URL, then we can't for the reasons described in issue niemeyer/gopkg#24

[slimsag]

From @nathany on September 20, 2014 18:4

Going forward I will be looking at making azul3d.org use the same exact versioning scheme as gopkg.in rather than just similar to it like it is right now.

@slimsag You might even consider running a self-hosted version of gopkg.in.
niemeyer/gopkg#16

[slimsag]

@nathany At first sight it seems to be a good idea, but I see some issues taking that approach:

• We have our own homepage (i.e. we need http://azul3d.org to show up -- not gopkg.in's front page).
• We have a bunch of other pages that would need to be served from azul3d.github.io.
• For better or worse, we do have our own online docs (see e.g. http://azul3d.org/gfx.v1) and gopkg would have to send those (I've been meaning to ask @nf if azul3d.org could be removed from the godoc.org / gddo blacklist, see Use GLFW instead of Chippy #26).
• Lastly, future changes to any dynamic part of the website would require changes to gopkg which I think is not ideal.

I think a more interesting solution could be turning gopkg into a usable package itself, exposing an API to e.g. test if a request comes from go get or git and have gopkg handle the response appropriately.

[slimsag]

From @shurcooL on September 20, 2014 18:56

Why was azul3d.org blacklisted from godoc.org?

[slimsag]

@shurcooL (I originally posted here but moved it to #26)

[slimsag]

From @cespare on September 20, 2014 19:55

@slimsag Please ask him again -- I'm sure he'll be happy to un-blacklist.

[slimsag]

From @nathany on September 24, 2014 13:29

@slimsag I would also want to use my own homepage (http://fsnotify.org/) if self-hosting gopkg.in.

I like the idea of making gopkg a pkg.

[slimsag]

I am marking this as closed now, but feel free to leave feedback here.

The news article has more information: Better Package Docs, Versioning.

Relevant commits (for those interested):

• slimsag/semver@f6195a6
• azul3d-legacy/appengine@81d9442
• azul3d/webgen@2cea2cd8f4e51c3e0c63f600b9b45923a723d9a6
• azul3d/webgen@fe71c558bb1ef29607fe1f02e9540f9f11251e25
• azul3d/webgen@f401af676e3aa89ee1244be7314ad86de67872da

[nathany]

Wow. This conversation is from quite a while ago.

Personally I'm beginning to move away from gopkg.in as I feel it complicates development of libraries and is less necessary now that the vendor/ folder is a standard part of the Go toolchain.

Right now I'm still using semver and tagging. I remain curious as to what other tools could help identify API changes. If it were possible to reliably depend on a v0 library thanks to vendoring and helpful tools, there may come a time when version numbers are no longer that important.

@shurcooL's Go-Package-Store for the vendor/ folder with some Sourcegraph smarts for API changes could a very useful tool.

[niemeyer]

@nathany Moving away from gopkg.in because you're using vendoring is mixing two independent problems with two independent solutions. When you vendor someone's work, you are at the consuming side. When you offer people a stable API via gopkg.in or whatever else, you're at the producing side, trying to help people out when the use your package bare, or when they vendor it.

By all means, I'm not trying to prevent you from moving away from gopkg.in. But I care a lot about having proper stable APIs to work with, rather than an ecosystem of daily breakages everywhere.

[dskinner]

I don't use gopkg.in, I'm familiar with the history and arguments, and overall I think it can be a good thing for a particular circumstance. Mgo is such a circumstance. And it's true, it does make things hard for the library maintainer, you'll need to give more consideration to your changes and how they might play into working systems you do not own.

I think the main issue here with azul3d is that there is no suitable version and there won't be until a major piece of software is developed using all parts. At such a point, you might actually tag a version 1 on all it's parts and say "see, this is what can be done with v1", and then you can provide support for that.

At that point, gopkg.in provides one way to handle that and plays nice with things like go get. With the addition of the vendor directory, perhaps what @nathany is thinking, one could use their vcs tool to clone a package into that directory by a given tag/branch instead.

But ultimately, I think there's harm to be had by trying to give versions to things prematurely.

[nathany]

Hi Gustavo.

While having major version numbers in import paths is applicable for bare packages and vendoring, I feel it provides more value when using packages without vendoring.

I do care about stable APIs too. But when APIs inevitably do change, I'm interested in what other tools could help. Tools beyond incrementing a major version number. Tools that other language communities haven't dreamt up yet.

[nathany]

you'll need to give more consideration to your changes and how they might play into working systems you do not own

I think that's always important for all library authors if the library has a reasonable amount of usage.

The complexity I'm speaking of is having version numbers in project relative import paths that need to be adjusted for each major version. Not that breaking changes should happen often, but it's still extra work and added complexity for the library author. Work that provides little value if all end users are using the vendor/ folder anyway.

(Not that I have any insight into how many people have adopted vendor/ vs. using bare packages)

[dmitshur]

Right now I'm still using semver and tagging. I remain curious as to what other tools could help identify API changes.

I want to ask @nathany, but do you consider the Go compiler such a tool? If not, why not? IMO, it's quite a good tool for that. When you update a dependency, you can use it to check if everything still compiles. If there are breaking API changes (that cause compilation errors), the compiler will tell you when and where things need to be updated.

That's how I stay on latest master of most libraries I use (together with Go Package Store to have more visibility on what updates are available and their commits).

@shurcooL's Go-Package-Store for the vendor/ folder with some Sourcegraph smarts for API changes could a very useful tool.

I can imagine an enhancement for G-P-S that also displays which commits contain API changes. It would have to rely on an external API to provide that information, but it is possible. Each Go release uses this to find out all the API changes to the standard library.

G-P-S also supports /vendor/ folder (assuming you have a Godeps.json or vendor.json file), are you already aware of that? Just need to provide -godeps or -govendor flag.

[nathany]

Thanks @shurcooL. I haven't tried out G-P-S recently with a /vendor folder project. Will do.

The compiler certainly makes API changes apparent. You're right. Combined with running tests and the ability to rollback to the previous /vendor, it's a pretty okay solution.

What I like about G-P-S is that you can see all the changes for all the dependencies on one page. What I have in mind is a sort of augmented CHANGELOG that gives people a look at new/changed/removed APIs. Something that could even be dynamic based on the current vendor snapshot and the tag/tip being pulled in.

It's just an idea though. Hard to say how well it work in practice, but it seems like it could be especially useful for v0 libraries.