change go.mod to make current version (v2) a different module path

[GeorgeZhai]

Proposal

Change the first line of go.mod to:
module github.com/prometheus/prometheus/v2

Use case. Why is this important?
it's the recommended way of releasing V2 and beyond
We have multiple projects using both v1 and v2 Prometheus in our monorepo. sharing the same import path for the incompatible versions is causing the problem

[beorn7]

This is a known issue with Go Modules. The semantic versioning of Prometheus versions the behavior of Prometheus as a server, not its code as a library. By changing the module path to v2, we would suggest that Prometheus obeys the contract of Go Modules as a library, but it doesn't, i.e. there are many breaking changes to expect even in a minor release.

We had a lot of discussions around this, without a clear result yet. @roidelapluie is working on starting a design doc to summarize the results and come up with a conclusion.

[beorn7]

See also #8513 #8417 #7991 #7663 #6048 #5356 😞

[GeorgeZhai]

I see, thanks for the explanation.
Considering the popularity of prometheus being used both as server and library, the benefit is obvious to follow the Go Modules convention 😄
Really hope we can have this soon

[bschaeffer]

If you have a repo using go modules, what is the suggested way of building a write adapter if you can't add the module as a dependency. Unless we vendor the library, we cannot do as suggested in example documentation. Maybe the protos could be moved to prometheus/common in that case?

[rakyll]

Is there any update on this issue? Prometheus is used heavily used as a library (even though it is not intended to be so) and cannot be imported from the latest release at the point. This presents a huge risk such as not being able to patch a security issue if a vulnerability is found in the packages.

[roidelapluie]

Prometheus was not intended to be used as a library. Now that has changed, and it is intended to be used as such, even if we do not accept all general-purpose contributions.

The go official wordings are clear that adding /v2 in the module name would tie is to using semver for the public API's. Having a dedicated go.mod with internal replace directives and releasing cmd/ independently requires an go.sum change for every time the code changes.

On the other hand, users of Prometheus as a library require more frequent updates than users of Prometheus. And if there is a security issue in prometheus libraries that do not affect Prometheus users, tagging a new release would make everyone lose a lot of time by upgrading Prometheus for a noop change.

I have still not really found a really great solution, it looks like the /v2 in module path would be the lowest cost for us and would fix 90% of the issues.

[roidelapluie]

After looking deeper, I do think that we should eventually make the move to v2, to keep things tagged. It would help casual downstream users, by clearly specifying which version of Prometheus they build against.

Overall, we would need to clearly document that we do not follow semver on our interfaces, and that breaking changes to the code happen between minor releases of Prometheus.

[beorn7]

I'm not sure if moving to module github.com/prometheus/prometheus/v2 in go.mod would reduce the sum of pain. It would definitely shift it to somewhere else, but that alone isn't worth the overhead of changing.

Status quo

To go get prometheus/prometheus as a dependency, you have to pick a specific commit hash, as described earlier:

~/testgoprom $ go get github.com/prometheus/prometheus@657ba532e42f1db8d7c77bf802378643da0d3118
go: downloading github.com/prometheus/prometheus v1.8.2-0.20200724121523-657ba532e42f
go: github.com/prometheus/prometheus 657ba532e42f1db8d7c77bf802378643da0d3118 => v1.8.2-0.20200724121523-657ba532e42f

Pro:

• No change needed.
• It is probably the clearest (or least unclear) way of saying "prometheus/prometheus uses Go Modules, but it is not meant to be a Go Module itself". (If you use prometheus/prometheus naively as a Go Module, the dissonance between a tagged v2.x and not using the v2 import path component breaks things in a more or less noticeable way.)

Con:

• The v1.8.2 in go.mod is confusing.
• You don't get the dependency management as intended by Go Modules but have to hand-pick the used prometheus/prometheus version.

Suggested in this issue

Change go.mod's first line to module github.com/prometheus/prometheus/v2.

__ Pro:__

• Things would work "normally", at least at first, as long as you are not updating anything.

Con:

• You still don't get the dependency management as intended by Go Modules, because our tagged v2.x versions don't follow semver on the API level. What's arguable worse than before is that we now suggest to be a fully fledged Go Module, but we aren't. The possible breakages are way more subtle: An innocent go get might pull in breaking changes, and all hell will break loose if different of your dependencies require different versions of prometheus/prometheus.

Another "easy" idea

How about changing our future release tags to a format that Go Modules doesn't recognize? E.g. use release-2.33.1 rather than v2.33.1? We could then change go.mod's first line to module github.com/prometheus/prometheus/v2 and add independent vX.Y style tags that take into account semver on an API level. (In practice, it probably means to bump the major version very often, with almost each minor release of Prometheus.)

Pro:

• We would be a "real" Go Module. Everything would work as intended, for better or worse.

Con:

• Since prometheus/prometheus is a really large repo, we would have breaking changes on the API level quite often, and would require a major version bump with almost each release. This requires importers of prometheus/prometheus to update their import paths very often. (But this is a problem with Go Modules in general. Every Go Module with a broad set of features suffers from it, i.e. major version bumps on the whole module with all the subsequent import path hassle even if only small parts of it are affected.)

More involved approaches

To stay faithful to Go Modules, we needed to cut prometheus/prometheus into many modules, e.g. one for TSDB, one for PromQL, etc. We discussed this before. It would be a lot of additional work for the maintainers. Plus, a multi-module repository is considered to be pretty painful to handle. An attempt to get out of that is the modularise project, but it appears that despite all efforts, it hasn't got traction.

Conclusion

I'm really not sure what to do. All options seem to be bad in different ways. In that situation, I would just stay with the status quo because it is least work and at least doesn't create new sources of confusion.

Personally, I could imagine that the separate tagging (what I titled "Another 'easy" idea" above) might be slightly less bad than the status quo and wouldn't require much additional work, but I might be missing something.

[roidelapluie]

I think that the "breakages" we make are often low-risk for consumers (it won't compile). Some parts of Prometheus are intended to be a library nowadays, so I would really like to do the change for v2, which makes it at the end easier for downstream users to know which version of Prometheus they are building against.

[beorn7]

I think that the "breakages" we make are often low-risk for consumers (it won't compile).

Yeah, that's the best case, but it's still pretty bad for a user expecting Go Modules semantics.

And it gets really bad if prometheus/prometheus is an indirect dependency. You use module foo (which depends on Prometheus v2.34) and module bar (which depends on Prometheus v2.35). You might end up in a situation where you cannot compile your code unless you change the code of your dependencies. (The same problem exist today, but at least we make it as obvious as we can that we aren't follow Go Module semantics, see above. That's why I think adding the v2 isn't a clear win. Perhaps it's even a net loss.)

Some parts of Prometheus are intended to be a library nowadays

Yes, that's why I tend to do the "right thing", if we do anything at all. I.e. bite the bullet and maintain a multi-module repo. Or try out my idea with separate tagging for releases of Prometheus as a program vs. tags for the Go Module.

[roidelapluie]

I think separate tagging would have pretty bad side effects on downstream users that are following tags, it seems like it would cost them unnecessary efforts to adapt their patterns, probably an order of magnitude more annoying than moving to v2/.

[beorn7]

It appears we are both just restating what we said before. We only disagree in the impact of the various annoyances.

We could go with changing to module github.com/prometheus/prometheus/v2, and if it wreaks havoc as I expect, we should be ready to invoke plan B and add my idea.

Or we declare it all not worth the risk and stay with the status quo.

We should also take into account that the v2 (etc.) import path renaming is fairly disputed in the Go community. (Some projects never tag anything beyond v0.x to avoid the whole thing.)

[bboreham]

I would love to have something constructive to say. I've tried to solve this kind of problem a few times over the past 30 years and I do not know of any nice solutions where "force all users to upgrade now" is not an option.

it gets really bad if prometheus/prometheus is an indirect dependency

+10

[tomwilkie]

Alternative suggestion:

As per @beorn7, change our release tags to a format that Go Modules doesn't recognise. But keep the code itself "un versioned" eg pre-v1. This sends the signal that our released binaries follow semver - but that the code doesn't, and that updating the code between any version can cause breakage (see https://go.dev/ref/mod):

A version is considered unstable if its major version is 0 or it has a pre-release suffix. Unstable versions are not subject to compatibility requirements. For example, v0.2.0 may not be compatible with v0.1.0, and v1.5.0-beta may not be compatible with v1.5.0.

This would require us to rename / change old tags, which may be a big no-no - but worth considering?

---

it gets really bad if prometheus/prometheus is an indirect dependency

I agree; not a complete solution but one way to improve things is to make it possible for automated tools to update dependancies, and evangelise that. This way the we can find out if we break things more quickly, but also we can encourage everyone to be one the latest release. Right now eg dependabot won't update unversioned code, but renovatebot will.

An extension to the above, to make this easier (and to be considered separately) would be to prefix release branches with v0. (eg v0.2.33). This would allow automated tools to update code which depends on prometheus. We don't include the patch version in the release branch anyway, so we're not loosing fidelity. A downside would be people that look at the git branches may get confuses when the look at the tags.

WDYT?

[bboreham]

Can you clarify: your suggestion includes deleting all the v1 tags in this repo?

[tomwilkie]

All the historic tags would be changed to release-X.Y.Z instead of X.Y.Z or vX.Y.Z. Is that even possible?

[roidelapluie]

I would not change historic tags.

What would be the effect of retracting the following releases from go.mod:

retract v1.0.0-rc.0
retract v1.0.0
retract v1.0.1
retract v1.0.2
retract v1.1.0
retract v1.1.1
retract v1.1.2
retract v1.1.3
retract v1.2.0
retract v1.2.1
retract v1.2.2
retract v1.2.3
retract v1.3.0-beta.0
retract v1.3.0
retract v1.3.1
retract v1.4.0
retract v1.4.1
retract v1.5.0
retract v1.5.1
retract v1.5.2
retract v1.5.3
retract v1.6.0
retract v1.6.1
retract v1.6.2
retract v1.6.3
retract v1.7.0
retract v1.7.1
retract v1.7.2
retract v1.8.0
retract v1.8.1
retract v1.8.2
retract v2.0.0-alpha.0+incompatible
retract v2.0.0-alpha.1+incompatible
retract v2.0.0-alpha.2+incompatible
retract v2.0.0-alpha.3+incompatible
retract v2.0.0-beta.0+incompatible
retract v2.0.0-beta.1+incompatible
retract v2.0.0-beta.2+incompatible
retract v2.0.0-beta.3+incompatible
retract v2.0.0-beta.4+incompatible
retract v2.0.0-beta.5+incompatible
retract v2.0.0-rc.0+incompatible
retract v2.0.0-rc.1+incompatible
retract v2.0.0-rc.2+incompatible
retract v2.0.0-rc.3+incompatible
retract v2.0.0+incompatible
retract v2.1.0+incompatible
retract v2.2.0-rc.0+incompatible
retract v2.2.0-rc.1+incompatible
retract v2.2.0+incompatible
retract v2.2.1+incompatible
retract v2.3.0+incompatible
retract v2.3.1+incompatible
retract v2.3.2+incompatible
retract v2.4.0-rc.0+incompatible
retract v2.4.0+incompatible
retract v2.4.1+incompatible
retract v2.4.2+incompatible
retract v2.4.3+incompatible
retract v2.5.0-rc.0+incompatible
retract v2.5.0-rc.1+incompatible
retract v2.5.0-rc.2+incompatible
retract v2.5.0+incompatible

[tomwilkie]

Oh I didn't know about retract! We could even continue to tag the release with semver and add a retract for everyone... scratch that I still think there is value in doing the v0.2.33 tagging so eg dependabot works and other projects don't just use the latest master.

Should we do a fork with these changes and see how it works?

(Sorry keep clicking the wrong button. Blame covid brain)

[roidelapluie]

My preference is to just move prometheus to v2. As said before, it would enable dependabot to work, and clarify which version of prometheus you are consuming. I think the cons- are not worth blocking us from doing so.

[beorn7]

If retract allows us to present a v0.x module to the Go Modules framework, that might be the way to go. I still think pretending to fulfill the Go Module contract by presenting a formally valid v2.x module but then still release API-breaking changes within the same v2.x major release is much worse than not playing along at all in an obvious fashion.

[invidian]

I usually find adding /v2 to Go module a good way to bring some old project up to recent Go module requirements. Of course from this point, in case of any breaking changes, it should be updated to v3 and so on. v2 code can be then managed in a release branch. This way you can keep supporting "stable" users, you can add deprecation messages and you can roll new versions. IIRC I think this also works when you want to import both v2 and v3 at the same time. I don't see any major disadvantage of not having a v2 directory in the source tree etc (not many project does that in general).

[beorn7]

@invidian That's essentially what I proposed as "another 'easy' idea" above. Note that it requires to use a new tag format for our normal product releases (because we don't want to signal a breaking change to the users where there is on breaking change for users).

Also note (as also said above) that the prometheus/prometheus as a set of libraries is pretty broad. We will have changes that are breaking somewhere with almost each minor product release. Which means we have to bump the Go Module major release more or less every six weeks. And this requires developers using prometheus/prometheus as a library to edit their import path very frequently to get updates, even if the part of the codebase they are using doesn't even have a breaking change.

Go Modules have the problem of conflicting interests: On the one hand, you want small modules with a well defined and interrelated feature set so that a major version bump makes sense throughout the module. On the other hand, you want large modules to not face the complexity of managing a million small dependencies.

[invidian]

We will have changes that are breaking somewhere with almost each minor product release.

That sounds like a code-smell to me. Usually it should be possible to rollout backward compatible changes by expanding the API + deprecating older calls. If that's really the case, perhaps indeed some kind of new module (generic prometheus library) should be rolled out with this repo adopting it eventually, which will have either more stable API or will be able to get version bumps more frequently? Maybe some stronger contracts should be defined around this before manifesting that using Go modules compatibility? The library could also start versioning from v0.x to account for frequent breaking changes.

[roidelapluie]

I think we could move the repository to a multi-module repository, with each part being v0.x unless / which only contains cmd/.
The trick to achieve this in an efficient way would be to use go 1.18 workspaces. I guess I will look into this further down as this option has not been considered yet.

[beorn7]

That sounds like a code-smell to me. Usually it should be possible to rollout backward compatible changes by expanding the API + deprecating older calls.

That's true if you see the primary goal of prometheus/prometheus to provide a library for others to use. However, the actual primary goal is to provide a world-leading metrics-based monitoring system and maximize the innovation and dev speed for it. That's why there was so much quite controversial discussion how much effort the Prometheus devs should invest into making prometheus/prometheus also a good library for other devs to use. The priorities are obviously shifting here, with projects that are using prometheus/prometheus as a library gaining more and mori importance in the wider Prometheus ecosystem.

Essentially, if we had the people do invest the effort, we could totally do a lot of things to improve the situation. It's mostly the question how much effort we can invest for it without harming the primary goals of the project.

[roidelapluie]

prometheus 2.35.0-rc1 is avaible in go.mod as 0.35.0-rc1.

I tried many things to make retract work, but it seems it just does not work for our usecase. Opened an issue at golang/go#52359

[thepudds]

Hi 👋 , I just commented in golang/go/issues#52359, but three quick additional comments.

I've only skimmed this issue here, so sorry if this is off base.

1. You probably want to include a comment in the retract directives, which helps consumers understand what is happening and why. From the go mod ref:

Each retract directive should have a comment explaining the rationale for the retraction, though this is not mandatory. The go command may display rationale comments in warnings about retracted versions and in go list output

2. Just to make sure you are aware, you don't need to list every version separately, and you can do a closed intervals such as this (made up) example:

retract (
    v1.0.0
    [v1.2.0, v1.9.9]
)

3. You should definitely test the upgrade path for existing consumers who are using one of the retracted tags, including to make sure you understand what will happen along with making sure the project README or similar is accurate. Short description of upgrade (also from the ref):

To prevent users from upgrading to v1.0.0, the author can add two retract directives to go.mod, then tag v1.0.1 with the retractions.

retract (
v1.0.0 // Published accidentally.
v1.0.1 // Contains retractions only.
)

When a user runs go get example.com/m@latest, the go command reads retractions from v1.0.1, which is now the highest version. Both v1.0.0 and v1.0.1 are retracted, so the go command will upgrade (or downgrade!) to the next highest version, perhaps v0.9.5.

[roidelapluie]

Thank you! That fixed the issue!!!! Awesome!

[roidelapluie]

To everyone on this issue:

We will from now on mirror Prometheus releases to go.mod.
We will mirror 2.x releases onto 0.x releases on go.mod.

prometheus release	go.mod release
v2.35.0	v0.35.0
v2.35.1	v0.35.1
v2.36.0-rc.1	v0.36.0-rc.1
v2.43.0	v0.43.0

We have also retracted other releases from pkg.go.dev. The latest documentation is therefore available:

https://pkg.go.dev/github.com/prometheus/prometheus

Thank you all for your input and ideas.