Improper Module Versioning

[FryDay]

Bug Report

Importing concourse packages into code is especially painful due to improper module versioning.

Steps to Reproduce

go get github.com/concourse/concourse in an existing project using go modules

Expected Results

Latest version of concourse is pulled.

Actual Results

Old version (4.2?) is pulled.

Additional Context

The go documentation states:

If the module is version v2 or higher, the major version of the module must be included as a /vN at the end of the module paths used in go.mod files (e.g., module github.com/my/mod/v2, require github.com/my/mod/v2 v2.0.0) and in the package import path (e.g., import "github.com/my/mod/v2/mypkg").

[vito]

Hmm, what part of the repo are you trying to import?

We're not really intending to provide the whole repo as an importable package/module - Concourse is primarily an application, and we just use Go modules for its dependency management. The tags on the repo correspond to product versions, not an API version. I don't really want to wrangle all our import statements every time we ship a major version, since the version doesn't even really indicate any sort of Go interface compatibility.

We do however provide a go-concourse client, so I can see how the current structure would be problematic for importing it. Maybe we should extract it out into a separate module/repo? I wonder if you can have modules within modules? 🤔

[FryDay]

That is exactly what we were trying to do. You can, but you can't have a root module AND submodules. You have to either have a single module per repo or each module has to be separated.

[FryDay]

And if you were to break off go-concourse into it's own module or even it's own repo, you will run into the same issue. The release is >v5 but since the module is not properly versioned, it won't pull correctly.

[jmelfi]

@vito you can see details about go modules here.

The fact that there is a major version tag that is trying to be used here is why go is grabbing the older version. This is due to how golang works with versioning and modules.

As of right now the go-concourse client is not really usable as you can't get the right code for the repo in with how that is written currently. Golang having the version "vX" in the path is explicitly listed as the pattern that the golang tools will expect.

[vito]

Makes sense. I get how the module versioning stuff works, I just don't want to apply it to the entire project because the versions would be dishonest from a semver client standpoint (they're product versions, not client/API versions) and it'd result in a lot of pointless internal code churn every time we do a major version bump.

I think we should extract go-concourse and give it a proper versioning scheme that matches our API version, not product version. We don't yet have a publicly supported or properly versioned API, but we could at least start by pulling go-concourse out and stamping it with v0.1.0 or something until then. I just wanted to make sure you were only trying to use go-concourse.

@pivotal-jwinters Does that sound like a good starting point for API versioning? We'd also need to figure out what to do with the bits of atc it uses, since it can't import github.com/concourse/concourse. 🤔

[vito]

I'm also willing to cede the point of extracting the repo if it ends up more pragmatic to just do proper module versioning - hoping others on the team have opinions here. Maybe the noise isn't too bad if our major bumps are infrequent enough. API versioning could end up being unnecessary and/or overkill depending on how people will be using it.

edit: are there any other examples of large Go project which are primarily applications that have run into this?

[stale]

Beep boop! This issue has been idle for long enough that it's time to check in and see if it's still important.

If it is, what is blocking it? Would anyone be interested in submitting a PR or continuing the discussion to help move things forward?

If no activity is observed within the next week, this issue will be exterminated closed, in accordance with our stale issue process.

[TiagoJMartins]

What's the current state of this? I'm developing an application that allows me to control Concourse programatically, however, I'd like to avoid having to basically duplicate the whole API on my project and simply use go-concourse.
How would I go about doing that with the project's current state? Is it even possible?

[vito]

@TiagoJMartins We would have to extract go-concourse out and give it proper semantic versioning. We haven't done this yet because we're still "moving fast and breaking things" and are pretty reluctant to commit to an API that we actually want people to use externally. We can spend some time on this, but since we don't yet intend to support our API in a backwards-compatible way I suppose we'd make it v0.0.1 and just not worry about breaking things for now.

I'll re-open to keep tabs on this (and just let the stale bot keep doing its thing). In the meantime you could try doing go get github.com/concourse/concourse@master - that will forego the Go module versioning semantics and let you just depend directly on our code, which is just about as much stability as you'd get from us extracting it out anyway.

[TiagoJMartins]

@vito That's enough for my use case for now as I don't really mind if you guys break things down the line. This being the case I agree there's no need to extract it and version it as 0.0.1 just for the sake of it being versioned and would argue that's a job better done when the project actually requires it. Thanks a lot for your help! Happy holidays!

[mmwtsn]

We would have to extract go-concourse out and give it proper semantic versioning. We haven't done this yet because we're still "moving fast and breaking things" and are pretty reluctant to commit to an API that we actually want people to use externally. We can spend some time on this, but since we don't yet intend to support our API in a backwards-compatible way I suppose we'd make it v0.0.1 and just not worry about breaking things for now.

There's an interesting question in here about whether the Go source itself should be a separate version than whatever the current semantic version that Concourse has more broadly. I think the two are arguably the same. If I wanted to use the fly package programmatically rather from a Go CLI, rather than call out to an executable, the features available to me are versioned in the documentation based on the project version. If there is a separate Go SDK for interacting with the project that is versioned differently as a library, that makes sense, but the API implementation version is the same.

If the intention was to keep the project private, those modules should all move under an internal/ directory to ensure they cannot be imported. It may be worth doing that and releasing a properly-versioned 8.0.0 with a valid module path if the team does not want to commit to maintaining a public API. I think having those modules public is great though, as it allows for users to programmatically administer their Concourse pipelines in Go, which is in the spirit of the project!