cmd/go: add package version support to Go toolchain #24301

rsc · Mar 7, 2018

proposal: add package version support to Go toolchain

It is long past time to add versions to the working vocabulary of both Go developers and our tools.
The linked proposal describes a way to do that. See especially the Rationale section for a discussion of alternatives.

This GitHub issue is for discussion about the substance of the proposal.

Other references:

https://research.swtch.com/vgo, the detailed designs
A Tour of Versioned Go, a walkthrough of what it's like to use
go get golang.org/x/vgo, the prototype implementation
https://golang.org/wiki/vgo, links to related posts, videos, etc
vgo milestone on issue tracker

Frequently Asked Questions

This issue comment answers the most frequently asked questions, whether from the discussion below or from other discussions. Other questions from the discussion are in the next issue comment.

Why is the proposal not “use Dep”?

At the start of the journey that led to this proposal, almost two years ago, we all believed the answer would be to follow the package versioning approach exemplified by Ruby's Bundler and then Rust's Cargo: tagged semantic versions, a hand-edited dependency constraint file known as a manifest, a separate machine-generated transitive dependency description known as a lock file, a version solver to compute a lock file satisfying the manifest, and repositories as the unit of versioning. Dep follows this rough plan almost exactly and was originally intended to serve as the model for go command integration. However, the more I understood the details of the Bundler/Cargo/Dep approach and what they would mean for Go, especially built into the go command, and the more I discussed those details with others on the Go team, a few of the details seemed less and less a good fit for Go. The proposal adjusts those details in the hope of shipping a system that is easier for developers to understand and to use. See the proposal's rationale section for more about the specific details we wanted to change, and also the blog post announcing the proposal.

Why must major version numbers appear in import paths?

To follow the import compatibility rule, which dramatically simplifies the rest of the system. See also the blog post announcing the proposal, which talks more about the motivation and justification for the import compatibility rule.

Why are major versions v0, v1 omitted from import paths?

v1 is omitted from import paths for two reasons. First, many developers will create packages that never make a breaking change once they reach v1, which is something we've encouraged from the start. We don't believe all those developers should be forced to have an explicit v1 when they may have no intention of ever releasing v2. The v1 becomes just noise. If those developers do eventually create a v2, the extra precision kicks in then, to distinguish from the default, v1. There are good arguments about visible stability for putting the v1 everywhere, and if we were designing a system from scratch, maybe that would make it a close call. But the weight of existing code tips the balance strongly in favor of omitting v1.

v0 is omitted from import paths because - according to semver - there are no compatibility guarantees at all for those versions. Requiring an explicit v0 element would do little to ensure compatibility; you'd have to say v0.1.2 to be completely precise, updating all import paths on every update of the library. That seems like overkill. Instead we hope that developers will simply look at the list of modules they depend on and be appropriately wary of any v0.x.y versions they find.

This has the effect of not distinguishing v0 from v1 in import paths, but usually v0 is a sequence of breaking changes leading to v1, so it makes sense to treat v1 as the final step in that breaking sequence, not something that needs distinguishing from v0. As @Merovius put it (#24301 (comment)):

By using v0.x, you are accepting that v0.(x+1) might force you to fix your code. Why is it a problem if v0.(x+1) is called v1.0 instead?

Finally, omitting the major versions v0 and v1 is mandatory - not optional - so that there is a single canonical import path for each package.

Why must I create a new branch for v2 instead of continuing to work on master?

You don't have to create a new branch. The vgo modules post unfortunately gives that impression in its discussion of the "major branch" repository layout. But vgo doesn't care about branches. It only looks up tags and resolves which specific commits they point at. If you develop v1 on master, you decide you are completely done with v1, and you want to start making v2 commits on master, that's fine: start tagging master with v2.x.y tags. But note that some of your users will keep using v1, and you may occasionally want to issue a minor v1 bug fix. You might at least want to fork a new v1 branch for that work at the point where you start using master for v2.

Won't minimal version selection keep developers from getting important updates?

This is a common fear, but I really think if anything the opposite will happen. Quoting the "Upgrade Speed" section of https://research.swtch.com/vgo-mvs:

Given that minimal version selection takes the minimum allowed version of each dependency, it's easy to think that this would lead to use of very old copies of packages, which in turn might lead to unnecessary bugs or security problems. In practice, however, I think the opposite will happen, because the minimum allowed version is the maximum of all the constraints, so the one lever of control made available to all modules in a build is the ability to force the use of a newer version of a dependency than would otherwise be used. I expect that users of minimal version selection will end up with programs that are almost as up-to-date as their friends using more aggressive systems like Cargo.

For example, suppose you are writing a program that depends on a handful of other modules, all of which depend on some very common module, like gopkg.in/yaml.v2. Your program's build will use the newest YAML version among the ones requested by your module and that handful of dependencies. Even just one conscientious dependency can force your build to update many other dependencies. This is the opposite of the Kubernetes Go client problem I mentioned earlier.

If anything, minimal version selection would instead suffer the opposite problem, that this “max of the minimums” answer serves as a ratchet that forces dependencies forward too quickly. But I think in practice dependencies will move forward at just the right speed, which ends up being just the right amount slower than Cargo and friends.

By "right amount slower" I was referring to the key property that upgrades happen only when you ask for them, not when you haven't. That means that code only changes (in potentially unexpected and breaking ways) when you are expecting that to happen and ready to test it, debug it, and so on.

See also the response #24301 (comment) by @Merovius.

If $GOPATH is deprecated, where does downloaded code live?

Code you check out and work on and modify can be stored anywhere in your file system, just like with essentially every other developer tool.

Vgo does need some space to hold downloaded source code and install binaries, and for that it does still use $GOPATH, which as of Go 1.9 defaults to $HOME/go. So developers will never need to set $GOPATH unless they want these files to be in a different directory. To change just the binary install location, they can set $GOBIN (as always).

Why are you introducing the `// import` comment?

We're not. That was a pre-existing convention. The point of that example in the tour was to show how go.mod can deduce the right module paths from import comments, if they exist. Once all projects use go.mod files, import comments will be completely redundant and probably deprecated.

rsc · Mar 7, 2018

Frequently Asked Questions

This issue comment answers the most frequently asked questions, whether from the discussion below or from other discussions. Other questions from the discussion are in the next issue comment.

Why is the proposal not “use Dep”?

At the start of the journey that led to this proposal, almost two years ago, we all believed the answer would be to follow the package versioning approach exemplified by Ruby's Bundler and then Rust's Cargo: tagged semantic versions, a hand-edited dependency constraint file known as a manifest, a separate machine-generated transitive dependency description known as a lock file, a version solver to compute a lock file satisfying the manifest, and repositories as the unit of versioning. Dep follows this rough plan almost exactly and was originally intended to serve as the model for go command integration. However, the more I understood the details of the Bundler/Cargo/Dep approach and what they would mean for Go, especially built into the go command, and the more I discussed those details with others on the Go team, a few of the details seemed less and less a good fit for Go. The proposal adjusts those details in the hope of shipping a system that is easier for developers to understand and to use. See the proposal's rationale section for more about the specific details we wanted to change, and also the blog post announcing the proposal.

Why must major version numbers appear in import paths?

To follow the import compatibility rule, which dramatically simplifies the rest of the system. See also the blog post announcing the proposal, which talks more about the motivation and justification for the import compatibility rule.

Why are major versions v0, v1 omitted from import paths?

v1 is omitted from import paths for two reasons. First, many developers will create packages that never make a breaking change once they reach v1, which is something we've encouraged from the start. We don't believe all those developers should be forced to have an explicit v1 when they may have no intention of ever releasing v2. The v1 becomes just noise. If those developers do eventually create a v2, the extra precision kicks in then, to distinguish from the default, v1. There are good arguments about visible stability for putting the v1 everywhere, and if we were designing a system from scratch, maybe that would make it a close call. But the weight of existing code tips the balance strongly in favor of omitting v1.

v0 is omitted from import paths because - according to semver - there are no compatibility guarantees at all for those versions. Requiring an explicit v0 element would do little to ensure compatibility; you'd have to say v0.1.2 to be completely precise, updating all import paths on every update of the library. That seems like overkill. Instead we hope that developers will simply look at the list of modules they depend on and be appropriately wary of any v0.x.y versions they find.

This has the effect of not distinguishing v0 from v1 in import paths, but usually v0 is a sequence of breaking changes leading to v1, so it makes sense to treat v1 as the final step in that breaking sequence, not something that needs distinguishing from v0. As @Merovius put it (#24301 (comment)):

By using v0.x, you are accepting that v0.(x+1) might force you to fix your code. Why is it a problem if v0.(x+1) is called v1.0 instead?

Finally, omitting the major versions v0 and v1 is mandatory - not optional - so that there is a single canonical import path for each package.

Why must I create a new branch for v2 instead of continuing to work on master?

You don't have to create a new branch. The vgo modules post unfortunately gives that impression in its discussion of the "major branch" repository layout. But vgo doesn't care about branches. It only looks up tags and resolves which specific commits they point at. If you develop v1 on master, you decide you are completely done with v1, and you want to start making v2 commits on master, that's fine: start tagging master with v2.x.y tags. But note that some of your users will keep using v1, and you may occasionally want to issue a minor v1 bug fix. You might at least want to fork a new v1 branch for that work at the point where you start using master for v2.

Won't minimal version selection keep developers from getting important updates?

This is a common fear, but I really think if anything the opposite will happen. Quoting the "Upgrade Speed" section of https://research.swtch.com/vgo-mvs:

Given that minimal version selection takes the minimum allowed version of each dependency, it's easy to think that this would lead to use of very old copies of packages, which in turn might lead to unnecessary bugs or security problems. In practice, however, I think the opposite will happen, because the minimum allowed version is the maximum of all the constraints, so the one lever of control made available to all modules in a build is the ability to force the use of a newer version of a dependency than would otherwise be used. I expect that users of minimal version selection will end up with programs that are almost as up-to-date as their friends using more aggressive systems like Cargo.

For example, suppose you are writing a program that depends on a handful of other modules, all of which depend on some very common module, like gopkg.in/yaml.v2. Your program's build will use the newest YAML version among the ones requested by your module and that handful of dependencies. Even just one conscientious dependency can force your build to update many other dependencies. This is the opposite of the Kubernetes Go client problem I mentioned earlier.

If anything, minimal version selection would instead suffer the opposite problem, that this “max of the minimums” answer serves as a ratchet that forces dependencies forward too quickly. But I think in practice dependencies will move forward at just the right speed, which ends up being just the right amount slower than Cargo and friends.

By "right amount slower" I was referring to the key property that upgrades happen only when you ask for them, not when you haven't. That means that code only changes (in potentially unexpected and breaking ways) when you are expecting that to happen and ready to test it, debug it, and so on.

See also the response #24301 (comment) by @Merovius.

If $GOPATH is deprecated, where does downloaded code live?

Code you check out and work on and modify can be stored anywhere in your file system, just like with essentially every other developer tool.

Vgo does need some space to hold downloaded source code and install binaries, and for that it does still use $GOPATH, which as of Go 1.9 defaults to $HOME/go. So developers will never need to set $GOPATH unless they want these files to be in a different directory. To change just the binary install location, they can set $GOBIN (as always).

Why are you introducing the `// import` comment?

We're not. That was a pre-existing convention. The point of that example in the tour was to show how go.mod can deduce the right module paths from import comments, if they exist. Once all projects use go.mod files, import comments will be completely redundant and probably deprecated.

Discussion Summary (last updated 2017-04-25)

This issue comment holds a summary of the discussion below.

How can we handle migration?

[#24301 (comment) by @ChrisHines.]

Response #24301 (comment) by @rsc. The original proposal assumes the migration is handled by authors moving to subdirectories when compatibility is important to them, but of course that motivation is wrong. Compatibility is most important to users, who have little influence on authors moving. And it doesn't help older versions. The linked comment, now also #25069, proposes a minimal change to old "go build" to be able to consume and build module-aware code.

How can we deal with singleton registrations?

[#24301 (comment) by @jimmyfrasche.]

Response #24301 (comment) by @rsc. Singleton registration collisions (such as http.Handle of the same path) between completely different modules are unaffected by the proposal. For collisions between different major versions of a single module, authors can write the different major versions to expect to coordinate, usually by making v1 call into v2, and then use a requirement cycle to make sure v2 is not used with older v1 that don't know about the coordination.

How should we install a versioned command?

[#24301 (comment) by @leonklingele.]

Response #24301 (comment) by @rsc. In short, use go get. We still use $GOPATH/bin for the install location. Remember that $GOPATH now defaults to $HOME/go, so commands will end up in $HOME/go/bin, and $GOBIN can override that.

Why are v0, v1 omitted in the import paths? Why must the others appear? Why must v0, v1 never appear?

[#24301 (comment) by @justinian.]
[#24301 (comment) by @jayschwa.]
[#24301 (comment) by @mrkanister.]
[#24301 (comment) by @mrkanister.]
[#24301 (comment) by @kaikuehne.]
[#24301 (comment) by @kaikuehne.]
[#24301 (comment) by @Merovius.]
[#24301 (comment) by @kaikuehne.]

Added to FAQ above.

Why are zip files mentioned in the proposal?

[#24301 (comment) by @nightlyone.]

The ecosystem will benefit from defining a concrete interchange format. That will enable proxies and other tooling. At the same time, we're abandoning direct use of version control (see rationale at top of this post). Both of this motivate describing the specific format. Most developers will not need to think about zip files at all; no developers will need to look inside them, unless they're building something like godoc.org.

Doesn't putting major versions in import paths violate DRY?

[#24301 (comment) by @jayschwa.]

No, because an import's semantics should be understandable without reference to the go.mod file. The go.mod file is only specifying finer detail. See the second half of the semantic import versions section of the proposal, starting at the block quote.

Also, if you DRY too much you end up with fragile systems. Redundancy can be a good thing. So "violat[ing] DRY" - that is to say, limited repeating yourself - is not always bad. For example we put the package clause in every .go file in the directory, not just one. That caught honest mistakes early on and later turned into an easy way to distinguish external test packages (package x vs package x_test). There's a balance to be struck.

Which timezone is used for the timestamp in pseudo-versions?

[#24301 (comment) by @tpng.]

UTC. Note also that you never have to type a pseudo-version yourself. You can type a git commit hash (or hash prefix) and vgo will compute and substitute the appropriate pseudo-version.

Will vgo address non-Go dependencies, like C or protocol buffers? Generated code?

[#24301 (comment) by @AlexRouSg.]
[#24301 (comment) by @stevvooe.]
[#24301 (comment) by @nim-nim.]

Non-Go development continues to be a non-goal of the go command, so there won't be support for managing C libraries and such, nor will there be explicit support for protocol buffers.

That said, we certainly do understand that using protocol buffers with Go is too difficult, and we'd like to see that addressed separately.

As for generated code more generally, a real cross-language build system is the answer, specifically because we don't want every user to need to have the right generators installed. Better for the author to run the generators and check in the result.

Won't minimal version selection keep developers from getting important updates?

[#24301 (comment) by @TocarIP.]
[#24301 (comment) by @nim-nim.]
[#24301 (comment) by @Merovius.]

Added to FAQ.

Can I use master to develop v1 and then reuse it to develop v2?

[#24301 (comment) by @mrkanister.]
[#24301 (comment) by @aarondl.]

Yes. Added to FAQ.

What is the timeline for this?

[#24301 (comment) by @flibustenet.]

Response in #24301 (comment) by @rsc. In short, the goal is to land a "technology preview" in Go 1.11; work may continue a few weeks into the freeze but not further. Probably don't send PRs adding go.mod to every library you can find until the proposal is marked accepted and the development copy of cmd/go has been updated.

How can I make a backwards-incompatible security change?

[#24301 (comment) by @buro9.]

Response in #24301 (comment) by @rsc. In short, the Go 1 compatibility guidelines do allow breaking changes for security reasons to avoid bumping the major version, but it's always best to do so in a way that keeps existing code working as much as possible. For example, don't remove a function. Instead, make the function panic or log.Fatal only if called improperly.

If one repo holds different modules in subdirectories (say, v2, v3, v4), can vgo mix and match from different commits?

[#24301 (comment) by @jimmyfrasche.]
[#24301 (comment) by @AlexRouSg.]

Yes. It treats each version tag as corresponding only to one subtree of the overall repository, and it can use a different tag (and therefore different commit) for each decision.

What if projects misuse semver? Should we allow minor versions in import paths?

[#24301 (comment) by @pbx0.]
[#24301 (comment) by @powerman.]
[#24301 (comment) by @pbx0.]
[#24301 (comment) by @powerman.]

As @powerman notes, we definitely need to provide an API consistency checker so that projects at least can be told when they are about to release an obviously breaking change.

Can you determine if you have more than one package in a build?

[#24301 (comment) by @pbx0.]

The easiest thing to do would be to use goversion -m on the resulting binary. We should make a go option to show the same thing without building the binary.

Concerns about vgo reliance on proxy vs vendor, especially open source vs enterprise.

[#24301 (comment) by @joeshaw.]
[#24301 (comment) by @kardianos.]
[#24301 (comment) by @Merovius.]
[#24301 (comment) by @joeshaw.]
[#24301 (comment) by @jamiethermo.]
[#24301 (comment) by @Merovius.]

Response: [#24301 (comment) by @rsc.] Proxy and vendor will both be supported. Proxy is very important to enterprise, and vendor is very important to open source. We also want to build a reliable mirror network, but only once vgo becomes go.

Concerns about protobuild depending on GOPATH semantics.

[#24301 (comment) by @stevvooe.]

Response [#24301 (comment) by @rsc] asked for more details in a new issue, but that issue does not seem to have been filed.

Suggestion to add special `vgo-v1-lock` tag.

[#24301 (comment) by @kybin.]

It seems appealing at first but leads to special cases that are probably not worth taking on. Full response in #24301 (comment).

How does one patch a deep dependency without vendoring?

[#24301 (comment) by @chirino.]

Response [#24301 (comment) by @kardianos.] By using a replace directive.

What will we do about module names changing?

[#24301 (comment) by @jimmyfrasche.]

Response [#24301 (comment) by @rsc.]
These are real, pre-existing problems that the vgo proposal does not attempt to address directly, but clearly we should address them eventually. The answer to code disappearing is to have caching proxies (mirrors) along with a reason to trust them; that's future work. (Or use vendoring in top-level project if you prefer.) The answer to code moving is to add an explicit concept of module or package redirects, much like type aliases are type redirects; that's also future work.

What about connecting to local Git servers?

[#24301 (comment) by @korya.]

Direct git access has been added back to the plan. See #24915.

What about binary-only packages?

[#24301 (comment) by @sdwarwick.]

Binary-only packages have only ever been supported in the limited circumstance of some kind of out-of-band installation into GOPATH/pkg. Go get has never supported fetching and installing a binary-only package, and it will continue not to support that. A binary-only package only works with one particular compiler and one particular copy of the dependencies, which severely limits how well it can be supported at all. The right answer is almost always to use source code instead.

Should we use path@version syntax in the go.mod file?

[#24301 (comment) by @sdwarwick.]

This is #24119. It seemed like a good idea at first but, after careful consideration, no.

rsc · Mar 7, 2018

Discussion Summary (last updated 2017-04-25)

This issue comment holds a summary of the discussion below.

How can we handle migration?

[#24301 (comment) by @ChrisHines.]

Response #24301 (comment) by @rsc. The original proposal assumes the migration is handled by authors moving to subdirectories when compatibility is important to them, but of course that motivation is wrong. Compatibility is most important to users, who have little influence on authors moving. And it doesn't help older versions. The linked comment, now also #25069, proposes a minimal change to old "go build" to be able to consume and build module-aware code.

How can we deal with singleton registrations?

[#24301 (comment) by @jimmyfrasche.]

Response #24301 (comment) by @rsc. Singleton registration collisions (such as http.Handle of the same path) between completely different modules are unaffected by the proposal. For collisions between different major versions of a single module, authors can write the different major versions to expect to coordinate, usually by making v1 call into v2, and then use a requirement cycle to make sure v2 is not used with older v1 that don't know about the coordination.

How should we install a versioned command?

[#24301 (comment) by @leonklingele.]

Response #24301 (comment) by @rsc. In short, use go get. We still use $GOPATH/bin for the install location. Remember that $GOPATH now defaults to $HOME/go, so commands will end up in $HOME/go/bin, and $GOBIN can override that.

Why are v0, v1 omitted in the import paths? Why must the others appear? Why must v0, v1 never appear?

[#24301 (comment) by @justinian.]
[#24301 (comment) by @jayschwa.]
[#24301 (comment) by @mrkanister.]
[#24301 (comment) by @mrkanister.]
[#24301 (comment) by @kaikuehne.]
[#24301 (comment) by @kaikuehne.]
[#24301 (comment) by @Merovius.]
[#24301 (comment) by @kaikuehne.]

Added to FAQ above.

Why are zip files mentioned in the proposal?

[#24301 (comment) by @nightlyone.]

The ecosystem will benefit from defining a concrete interchange format. That will enable proxies and other tooling. At the same time, we're abandoning direct use of version control (see rationale at top of this post). Both of this motivate describing the specific format. Most developers will not need to think about zip files at all; no developers will need to look inside them, unless they're building something like godoc.org.

Doesn't putting major versions in import paths violate DRY?

[#24301 (comment) by @jayschwa.]

No, because an import's semantics should be understandable without reference to the go.mod file. The go.mod file is only specifying finer detail. See the second half of the semantic import versions section of the proposal, starting at the block quote.

Also, if you DRY too much you end up with fragile systems. Redundancy can be a good thing. So "violat[ing] DRY" - that is to say, limited repeating yourself - is not always bad. For example we put the package clause in every .go file in the directory, not just one. That caught honest mistakes early on and later turned into an easy way to distinguish external test packages (package x vs package x_test). There's a balance to be struck.

Which timezone is used for the timestamp in pseudo-versions?

[#24301 (comment) by @tpng.]

UTC. Note also that you never have to type a pseudo-version yourself. You can type a git commit hash (or hash prefix) and vgo will compute and substitute the appropriate pseudo-version.

Will vgo address non-Go dependencies, like C or protocol buffers? Generated code?

[#24301 (comment) by @AlexRouSg.]
[#24301 (comment) by @stevvooe.]
[#24301 (comment) by @nim-nim.]

Non-Go development continues to be a non-goal of the go command, so there won't be support for managing C libraries and such, nor will there be explicit support for protocol buffers.

That said, we certainly do understand that using protocol buffers with Go is too difficult, and we'd like to see that addressed separately.

As for generated code more generally, a real cross-language build system is the answer, specifically because we don't want every user to need to have the right generators installed. Better for the author to run the generators and check in the result.

Won't minimal version selection keep developers from getting important updates?

[#24301 (comment) by @TocarIP.]
[#24301 (comment) by @nim-nim.]
[#24301 (comment) by @Merovius.]

Added to FAQ.

Can I use master to develop v1 and then reuse it to develop v2?

[#24301 (comment) by @mrkanister.]
[#24301 (comment) by @aarondl.]

Yes. Added to FAQ.

What is the timeline for this?

[#24301 (comment) by @flibustenet.]

Response in #24301 (comment) by @rsc. In short, the goal is to land a "technology preview" in Go 1.11; work may continue a few weeks into the freeze but not further. Probably don't send PRs adding go.mod to every library you can find until the proposal is marked accepted and the development copy of cmd/go has been updated.

How can I make a backwards-incompatible security change?

[#24301 (comment) by @buro9.]

Response in #24301 (comment) by @rsc. In short, the Go 1 compatibility guidelines do allow breaking changes for security reasons to avoid bumping the major version, but it's always best to do so in a way that keeps existing code working as much as possible. For example, don't remove a function. Instead, make the function panic or log.Fatal only if called improperly.

If one repo holds different modules in subdirectories (say, v2, v3, v4), can vgo mix and match from different commits?

[#24301 (comment) by @jimmyfrasche.]
[#24301 (comment) by @AlexRouSg.]

Yes. It treats each version tag as corresponding only to one subtree of the overall repository, and it can use a different tag (and therefore different commit) for each decision.

What if projects misuse semver? Should we allow minor versions in import paths?

[#24301 (comment) by @pbx0.]
[#24301 (comment) by @powerman.]
[#24301 (comment) by @pbx0.]
[#24301 (comment) by @powerman.]

As @powerman notes, we definitely need to provide an API consistency checker so that projects at least can be told when they are about to release an obviously breaking change.

Can you determine if you have more than one package in a build?

[#24301 (comment) by @pbx0.]

The easiest thing to do would be to use goversion -m on the resulting binary. We should make a go option to show the same thing without building the binary.

Concerns about vgo reliance on proxy vs vendor, especially open source vs enterprise.

[#24301 (comment) by @joeshaw.]
[#24301 (comment) by @kardianos.]
[#24301 (comment) by @Merovius.]
[#24301 (comment) by @joeshaw.]
[#24301 (comment) by @jamiethermo.]
[#24301 (comment) by @Merovius.]

Response: [#24301 (comment) by @rsc.] Proxy and vendor will both be supported. Proxy is very important to enterprise, and vendor is very important to open source. We also want to build a reliable mirror network, but only once vgo becomes go.

Concerns about protobuild depending on GOPATH semantics.

[#24301 (comment) by @stevvooe.]

Response [#24301 (comment) by @rsc] asked for more details in a new issue, but that issue does not seem to have been filed.

Suggestion to add special `vgo-v1-lock` tag.

[#24301 (comment) by @kybin.]

It seems appealing at first but leads to special cases that are probably not worth taking on. Full response in #24301 (comment).

How does one patch a deep dependency without vendoring?

[#24301 (comment) by @chirino.]

Response [#24301 (comment) by @kardianos.] By using a replace directive.

What will we do about module names changing?

[#24301 (comment) by @jimmyfrasche.]

Response [#24301 (comment) by @rsc.]
These are real, pre-existing problems that the vgo proposal does not attempt to address directly, but clearly we should address them eventually. The answer to code disappearing is to have caching proxies (mirrors) along with a reason to trust them; that's future work. (Or use vendoring in top-level project if you prefer.) The answer to code moving is to add an explicit concept of module or package redirects, much like type aliases are type redirects; that's also future work.

What about connecting to local Git servers?

[#24301 (comment) by @korya.]

Direct git access has been added back to the plan. See #24915.

What about binary-only packages?

[#24301 (comment) by @sdwarwick.]

Binary-only packages have only ever been supported in the limited circumstance of some kind of out-of-band installation into GOPATH/pkg. Go get has never supported fetching and installing a binary-only package, and it will continue not to support that. A binary-only package only works with one particular compiler and one particular copy of the dependencies, which severely limits how well it can be supported at all. The right answer is almost always to use source code instead.

Should we use path@version syntax in the go.mod file?

[#24301 (comment) by @sdwarwick.]

This is #24119. It seemed like a good idea at first but, after careful consideration, no.

.

rsc · Mar 7, 2018

.

rsc · Mar 7, 2018

.

Change https://golang.org/cl/101678 mentions this issue: design: add 24301-versioned-go

gopherbot · Mar 20, 2018

Change https://golang.org/cl/101678 mentions this issue: design: add 24301-versioned-go

This proposal is impressive and I like most everything about it. However, I posted the following concern on the mailing list, but never received any replies. In the meantime I've seen this issue raised by others in the Gophers slack channel for vgo and haven't seen a satisfactory answer there either.

From: https://groups.google.com/d/msg/golang-dev/Plc42fslQEk/rlfeNlazAgAJ

I am most worried about the migration path between a pre-vgo world and a vgo world going badly. I think we risk inflicting major pain on the Go community if there isn't a smooth migration path. Clearly the migration cannot be atomic across the whole community, but if I've understood all that you've written about vgo so far, there may be some situations where existing widely used packages will not be usable by both pre-vgo tools and post-vgo tools.

Specifically, I believe that existing packages that already have tagged releases with major versions >= 2 will not work with vgo until they have a go.mod file and also are imported with a /vN augmented import path. However, once those changes are made to the repository it will break pre-vgo uses of the package.

This seems to create a different kind of diamond import problem in which the two sibling packages in the middle of the diamond import a common v2+ package. I'm concerned that the sibling packages must adopt vgo import paths atomically to prevent the package at the top of the diamond from being in an unbuildable state whether it's using vgo or pre-vgo tools.

I haven't seen anything yet that explains the migration path in this scenario.

The proposal states:

Module-aware builds can import non-module-aware packages (those outside a tree with a go.mod file) provided they are tagged with a v0 or v1 semantic version. They can also refer to any specific commit using a “pseudo-version” of the form v0.0.0-yyyymmddhhmmss-commit. The pseudo-version form allows referring to untagged commits as well as commits that are tagged with semantic versions at v2 or above but that do not follow the semantic import versioning convention.

But I don't see a way for non-module-aware packages to import module-aware packages with transitive dependencies >= v2. That seems to cause ecosystem fragmentation in a way not yet addressed. Once you have a module-aware dependency that has a package >= v2 somewhere in its transitive dependencies that seems to force all its dependents to also adopt vgo to keep the build working.

Update: see also #24454

ChrisHines · Mar 20, 2018

This proposal is impressive and I like most everything about it. However, I posted the following concern on the mailing list, but never received any replies. In the meantime I've seen this issue raised by others in the Gophers slack channel for vgo and haven't seen a satisfactory answer there either.

From: https://groups.google.com/d/msg/golang-dev/Plc42fslQEk/rlfeNlazAgAJ

I am most worried about the migration path between a pre-vgo world and a vgo world going badly. I think we risk inflicting major pain on the Go community if there isn't a smooth migration path. Clearly the migration cannot be atomic across the whole community, but if I've understood all that you've written about vgo so far, there may be some situations where existing widely used packages will not be usable by both pre-vgo tools and post-vgo tools.

Specifically, I believe that existing packages that already have tagged releases with major versions >= 2 will not work with vgo until they have a go.mod file and also are imported with a /vN augmented import path. However, once those changes are made to the repository it will break pre-vgo uses of the package.

This seems to create a different kind of diamond import problem in which the two sibling packages in the middle of the diamond import a common v2+ package. I'm concerned that the sibling packages must adopt vgo import paths atomically to prevent the package at the top of the diamond from being in an unbuildable state whether it's using vgo or pre-vgo tools.

I haven't seen anything yet that explains the migration path in this scenario.

The proposal states:

Module-aware builds can import non-module-aware packages (those outside a tree with a go.mod file) provided they are tagged with a v0 or v1 semantic version. They can also refer to any specific commit using a “pseudo-version” of the form v0.0.0-yyyymmddhhmmss-commit. The pseudo-version form allows referring to untagged commits as well as commits that are tagged with semantic versions at v2 or above but that do not follow the semantic import versioning convention.

But I don't see a way for non-module-aware packages to import module-aware packages with transitive dependencies >= v2. That seems to cause ecosystem fragmentation in a way not yet addressed. Once you have a module-aware dependency that has a package >= v2 somewhere in its transitive dependencies that seems to force all its dependents to also adopt vgo to keep the build working.

Update: see also #24454

The Go project has encouraged this convention from the start of the project, but this proposal gives it more teeth: upgrades by package users will succeed or fail only to the extent that package authors follow the import compatibility rule.

It is unclear to me what this means and how it changes from the current situation. It would seem to me, that this describes the current situation as well: If I break this rule, upgrades and go-get will fail. AIUI nothing really changes and I'd suggest removing at least the mention of "more teeth". Unless, of course, this paragraph is meant to imply that there are additional mechanisms in place to penalize/prevent breakages?

Merovius · Mar 20, 2018

The Go project has encouraged this convention from the start of the project, but this proposal gives it more teeth: upgrades by package users will succeed or fail only to the extent that package authors follow the import compatibility rule.

It is unclear to me what this means and how it changes from the current situation. It would seem to me, that this describes the current situation as well: If I break this rule, upgrades and go-get will fail. AIUI nothing really changes and I'd suggest removing at least the mention of "more teeth". Unless, of course, this paragraph is meant to imply that there are additional mechanisms in place to penalize/prevent breakages?

This would also affect things like database drivers and image formats that register themselves with another package during init, since multiple major versions of the same package can end up doing this. It's unclear to me what all the repercussions of that would be.

jimmyfrasche · Mar 20, 2018

This would also affect things like database drivers and image formats that register themselves with another package during init, since multiple major versions of the same package can end up doing this. It's unclear to me what all the repercussions of that would be.

If the major version is v0 or v1, then the version number element must be omitted; otherwise it must be included.

Why is this? In the linked post, I only see the rationale that this is what developers currently do to create alternate paths when they make breaking changes - but this is a workaround for the fact that they don't initially plan for the tooling not handling versions for them. If we're switching to a new practice, why not allow and encourage (or even mandate) that new vgo-enabled packages include v0 or v1? It seems like paths lacking versions are just opportunities for confusion. (Is this a vgo-style package? Where is the module boundary? etc.)

justinian · Mar 21, 2018

If the major version is v0 or v1, then the version number element must be omitted; otherwise it must be included.

Why is this? In the linked post, I only see the rationale that this is what developers currently do to create alternate paths when they make breaking changes - but this is a workaround for the fact that they don't initially plan for the tooling not handling versions for them. If we're switching to a new practice, why not allow and encourage (or even mandate) that new vgo-enabled packages include v0 or v1? It seems like paths lacking versions are just opportunities for confusion. (Is this a vgo-style package? Where is the module boundary? etc.)

I generally like the proposal, but am hung up on requiring major versions in import paths:

It violates the DRY principle when the major version can already be known from the go.mod. Understanding what will happen if there's a mismatch between the two is also hard to intuit.
The irregularity of allowing v0 and v1 to be absent is also unintuitive.
Changing all the import paths when upgrading a dependency seems potentially tedious.

I understand that scenarios like the moauth example need to be workable, but hopefully not at the expense of keeping things simple for more common scenarios.

jayschwa · Mar 21, 2018

I generally like the proposal, but am hung up on requiring major versions in import paths:

It violates the DRY principle when the major version can already be known from the go.mod. Understanding what will happen if there's a mismatch between the two is also hard to intuit.
The irregularity of allowing v0 and v1 to be absent is also unintuitive.
Changing all the import paths when upgrading a dependency seems potentially tedious.

I understand that scenarios like the moauth example need to be workable, but hopefully not at the expense of keeping things simple for more common scenarios.

First of all: Impressive work!

One thing that is totally unclear to me and seems a bit underspecified:

Why there is a zip files in this proposal?

Layout, constraints and multiple use cases like when it is created and how it's life cycle is managed, what tools need support, how tools like linters should interact with it are also unclear, because they are not covered in the proposal.

So I would suggest to either refer to a later, still unwritten, proposal here and remove the word zip or remove the whole part from the proposal text, if you plan not discuss it at all within the scope of this proposal.

Discussing this later also enables a different audiences to contribute better here.

nightlyone · Mar 21, 2018

First of all: Impressive work!

One thing that is totally unclear to me and seems a bit underspecified:

Why there is a zip files in this proposal?

Layout, constraints and multiple use cases like when it is created and how it's life cycle is managed, what tools need support, how tools like linters should interact with it are also unclear, because they are not covered in the proposal.

So I would suggest to either refer to a later, still unwritten, proposal here and remove the word zip or remove the whole part from the proposal text, if you plan not discuss it at all within the scope of this proposal.

Discussing this later also enables a different audiences to contribute better here.

Which timezone is used for the timestamp in the pseudo-version (v0.0.0-yyyymmddhhmmss-commit)?

Edit:
It is in UTC as stated in https://research.swtch.com/vgo-module.

tpng · Mar 21, 2018

Which timezone is used for the timestamp in the pseudo-version (v0.0.0-yyyymmddhhmmss-commit)?

Edit:
It is in UTC as stated in https://research.swtch.com/vgo-module.

@rsc Will you be addressing C dependencies?

AlexRouSg · Mar 21, 2018

@rsc Will you be addressing C dependencies?

Looks like Minimal version selection makes propagation of non-breaking changes very slow. Suppose we have a popular library Foo, which is used by projects A,B and C. Someone improves Foo performance without changing API. Currently receiving updates is an opt-out process. If project A vendored Foo, but B and C didn't, author only needs to send pr with update to vendored dependency to A. So non-api breaking contributions won't have as much effect on community and are somewhat discouraged compared to current situation. This is even more problematic for security updates. If some abandoned/small/not very active project (not library) declares direct dependency on old version of e. g. x/crypto all users of that project will be vulnerable to flaw in x/crypto until project is updated, potentially forever. Currently users of such projects will receive latest fixed version, so this makes security situation worse. IIRC there were some suggestions how to fix this in maillist discussion, but, as far as I can tell this proposal doesn't mention it.

TocarIP · Mar 21, 2018

Looks like Minimal version selection makes propagation of non-breaking changes very slow. Suppose we have a popular library Foo, which is used by projects A,B and C. Someone improves Foo performance without changing API. Currently receiving updates is an opt-out process. If project A vendored Foo, but B and C didn't, author only needs to send pr with update to vendored dependency to A. So non-api breaking contributions won't have as much effect on community and are somewhat discouraged compared to current situation. This is even more problematic for security updates. If some abandoned/small/not very active project (not library) declares direct dependency on old version of e. g. x/crypto all users of that project will be vulnerable to flaw in x/crypto until project is updated, potentially forever. Currently users of such projects will receive latest fixed version, so this makes security situation worse. IIRC there were some suggestions how to fix this in maillist discussion, but, as far as I can tell this proposal doesn't mention it.

IIRC there were some suggestions how to fix [getting security patches] in maillist discussion, but, as far as I can tell this proposal doesn't mention it.

See the mention of go get -p.

jba · Mar 21, 2018

IIRC there were some suggestions how to fix [getting security patches] in maillist discussion, but, as far as I can tell this proposal doesn't mention it.

See the mention of go get -p.

See the mention of go get -p.

I've seen it, but this is still an opt-in mechanism.
I was thinking of way for library to mark all previous releases as unsafe, to force user to run go get -p or explicitly opt-in into insecure library.

TocarIP · Mar 21, 2018

See the mention of go get -p.

I've seen it, but this is still an opt-in mechanism.
I was thinking of way for library to mark all previous releases as unsafe, to force user to run go get -p or explicitly opt-in into insecure library.

If support for go get as we know it today will be deprecated and eventually removed, what's the recommended way to fetch & install (untagged) Go binaries then? Does it require git clone'ing the project first, followed by a manual go install to install the binary?
If $GOPATH is deprecated, where will these binaries be installed to?

leonklingele · Mar 21, 2018

If support for go get as we know it today will be deprecated and eventually removed, what's the recommended way to fetch & install (untagged) Go binaries then? Does it require git clone'ing the project first, followed by a manual go install to install the binary?
If $GOPATH is deprecated, where will these binaries be installed to?

@leonklingele: from my understanding, go get will not be deprecated, on the contrary.
It will be enhanced with automatic and transparent versioning capabilities. If a project depends from an untagged project, it would just take the master and "vendor" it at this exact version.
Again, my own understanding from reading just a little bit about vgo. I'm still in the process of understanding it completely.

dolanor · Mar 21, 2018

@leonklingele: from my understanding, go get will not be deprecated, on the contrary.
It will be enhanced with automatic and transparent versioning capabilities. If a project depends from an untagged project, it would just take the master and "vendor" it at this exact version.
Again, my own understanding from reading just a little bit about vgo. I'm still in the process of understanding it completely.

I wonder how this will affect the flow of working with a Git repository in general, also building on this sentence from the proposal:

If the major version is v0 or v1, then the version number element must be omitted; otherwise it must be included.

At the moment, it seems common to work on master (for me this includes short-lived feature branches) and to tag a commit with a new version every now and then. I feel this workflow is made more confusing with Go modules as soon as I release v2 of my library, because now I have a master and a v2 branch. I would expect master to be the current branch and v2 to be a maintenance branch, but it is exactly the other way around.

I know that the default branch can be changed from master to v2, but this still leaves me with the task to update that every time I release a new major version. Personally, I would rather have a master and a v1 branch, but I am not sure how exactly this would fit the proposal.

mrkanister · Mar 22, 2018

I wonder how this will affect the flow of working with a Git repository in general, also building on this sentence from the proposal:

If the major version is v0 or v1, then the version number element must be omitted; otherwise it must be included.

At the moment, it seems common to work on master (for me this includes short-lived feature branches) and to tag a commit with a new version every now and then. I feel this workflow is made more confusing with Go modules as soon as I release v2 of my library, because now I have a master and a v2 branch. I would expect master to be the current branch and v2 to be a maintenance branch, but it is exactly the other way around.

I know that the default branch can be changed from master to v2, but this still leaves me with the task to update that every time I release a new major version. Personally, I would rather have a master and a v1 branch, but I am not sure how exactly this would fit the proposal.

New major releases cause churn. If you have to change one setting in your Git repository (the default branch) whenever you make a new release, that’s a very minor cost compared to your library’s users switching to the new version.

I think this aspect of the proposal sets the right incentive: it encourages upstream authors to think about how they can do changes in a backwards-compatible way, reducing overall ecosystem churn.

stapelberg · Mar 22, 2018

New major releases cause churn. If you have to change one setting in your Git repository (the default branch) whenever you make a new release, that’s a very minor cost compared to your library’s users switching to the new version.

I think this aspect of the proposal sets the right incentive: it encourages upstream authors to think about how they can do changes in a backwards-compatible way, reducing overall ecosystem churn.

now I have a master and a v2 branch

You can instead create a v2/ subdirectory in master.

jba · Mar 22, 2018

now I have a master and a v2 branch

You can instead create a v2/ subdirectory in master.

@mrkanister

I would rather have a master and a v1 branch, but I am not sure how exactly this would fit the proposal.

According to my understanding of https://research.swtch.com/vgo-module vgo uses tags not branches to identify the versions. So you can keep development on master and branch off v1 as long as the tags point to the correct branch and commit.

AlexRouSg · Mar 22, 2018

@mrkanister

I would rather have a master and a v1 branch, but I am not sure how exactly this would fit the proposal.

According to my understanding of https://research.swtch.com/vgo-module vgo uses tags not branches to identify the versions. So you can keep development on master and branch off v1 as long as the tags point to the correct branch and commit.

New major releases cause churn. If you have to change one setting in your Git repository (the default branch) whenever you make a new release, that’s a very minor cost compared to your library’s users switching to the new version.

This is a problematic style of thinking that I think has bitten Go hard in the past. For one person on one project, switching what branch is default is simple in the moment, yes. But going against workflow conventions will mean people forget, especially when they work in several languages. And it will be one more quirky example of how Go does things totally differently that newcomers have to learn. Going against common programmer workflow conventions is not at all a minor cost.

justinian · Mar 22, 2018

New major releases cause churn. If you have to change one setting in your Git repository (the default branch) whenever you make a new release, that’s a very minor cost compared to your library’s users switching to the new version.

This is a problematic style of thinking that I think has bitten Go hard in the past. For one person on one project, switching what branch is default is simple in the moment, yes. But going against workflow conventions will mean people forget, especially when they work in several languages. And it will be one more quirky example of how Go does things totally differently that newcomers have to learn. Going against common programmer workflow conventions is not at all a minor cost.

Going against common programmer workflow conventions is not at all a minor cost.

Not following the conventional path is sometimes the necessary condition for innovation.

cznic · Mar 22, 2018

Going against common programmer workflow conventions is not at all a minor cost.

Not following the conventional path is sometimes the necessary condition for innovation.

If I understood parts of the proposal correctly, you never have to create a subdirectory or a new branch. You can potentially have only a master branch and git tag your repo from 0.0, to 1.0, to 2.0 and so on as long as you make sure to update your go.module to the correct import path for your library.

marwan-at-work · Mar 22, 2018

If I understood parts of the proposal correctly, you never have to create a subdirectory or a new branch. You can potentially have only a master branch and git tag your repo from 0.0, to 1.0, to 2.0 and so on as long as you make sure to update your go.module to the correct import path for your library.

@mrkanister I think, for dev, your clone your master (or any dev branch) and use "replace" directive (see vgo-tour) to point to it. (if i understand what you mean, no sure).

flibustenet · Mar 22, 2018

@mrkanister I think, for dev, your clone your master (or any dev branch) and use "replace" directive (see vgo-tour) to point to it. (if i understand what you mean, no sure).

@rsc I'd like to ask you to be more precise about the road map and what we should do now.
Will it follow the Go policy and feature freeze vgo at 3 month (2 now) ?
Should we now go with our pilgrim's baton asking every libs maintainer to add a go.mod file or should we wait for the proposal to be officially accepted (to be sure that name and syntax will not change) ?

flibustenet · Mar 22, 2018

@rsc I'd like to ask you to be more precise about the road map and what we should do now.
Will it follow the Go policy and feature freeze vgo at 3 month (2 now) ?
Should we now go with our pilgrim's baton asking every libs maintainer to add a go.mod file or should we wait for the proposal to be officially accepted (to be sure that name and syntax will not change) ?

@flibustenet Tools are not covered by the 1.0 policy so anything can change.

https://golang.org/doc/go1compat

Finally, the Go toolchain (compilers, linkers, build tools, and so on) is under active development and may change behavior. This means, for instance, that scripts that depend on the location and properties of the tools may be broken by a point release.

Also from the proposal

The plan, subject to proposal approval, is to release module support in Go 1.11 as an optional feature that may still change. The Go 1.11 release will give users a chance to use modules “for real” and provide critical feedback. Even though the details may change, future releases will be able to consume Go 1.11-compatible source trees. For example, Go 1.12 will understand how to consume the Go 1.11 go.mod file syntax, even if by then the file syntax or even the file name has changed. In a later release (say, Go 1.12), we will declare the module support completed. In a later release (say, Go 1.13), we will end support for go get of non-modules. Support for working in GOPATH will continue indefinitely.

AlexRouSg · Mar 22, 2018

@flibustenet Tools are not covered by the 1.0 policy so anything can change.

https://golang.org/doc/go1compat

Finally, the Go toolchain (compilers, linkers, build tools, and so on) is under active development and may change behavior. This means, for instance, that scripts that depend on the location and properties of the tools may be broken by a point release.

Also from the proposal

The plan, subject to proposal approval, is to release module support in Go 1.11 as an optional feature that may still change. The Go 1.11 release will give users a chance to use modules “for real” and provide critical feedback. Even though the details may change, future releases will be able to consume Go 1.11-compatible source trees. For example, Go 1.12 will understand how to consume the Go 1.11 go.mod file syntax, even if by then the file syntax or even the file name has changed. In a later release (say, Go 1.12), we will declare the module support completed. In a later release (say, Go 1.13), we will end support for go get of non-modules. Support for working in GOPATH will continue indefinitely.

Thanks for the feedback.

@AlexRouSg

According to my understanding of https://research.swtch.com/vgo-module vgo uses tags not branches to identify the versions. So you can keep development on master and branch off v1 as long as the tags point to the correct branch and commit.

You are correct, this will continue to work as before (just double checked to be sure), good catch!

With that out of the way, the thing that I (and apparently others) don't understand is the reasoning behind disallowing a v1 package to exist. I tried to import one using /v1 at the end of the import and also adding that to the go.mod of the package being imported, but vgo will look for a folder named v1 instead.

mrkanister · Mar 22, 2018

Thanks for the feedback.

@AlexRouSg

According to my understanding of https://research.swtch.com/vgo-module vgo uses tags not branches to identify the versions. So you can keep development on master and branch off v1 as long as the tags point to the correct branch and commit.

You are correct, this will continue to work as before (just double checked to be sure), good catch!

With that out of the way, the thing that I (and apparently others) don't understand is the reasoning behind disallowing a v1 package to exist. I tried to import one using /v1 at the end of the import and also adding that to the go.mod of the package being imported, but vgo will look for a folder named v1 instead.

@mrkanister
I think the main reason for not allowing v1 or v0 in the import path is to ensure that there is only one import path for each compatible version of a package.
Using the plain import path instead of /v1 is to ease the transition, so you don't have to update all your import paths to add /v1 at the end.

tpng · Mar 23, 2018

@mrkanister
I think the main reason for not allowing v1 or v0 in the import path is to ensure that there is only one import path for each compatible version of a package.
Using the plain import path instead of /v1 is to ease the transition, so you don't have to update all your import paths to add /v1 at the end.

Hi,

While a lot of the points in the proposal are more than welcome and will help taming the large Go codebases that emerged over time the "use minimal version" rule is quite harmful:

you want your code ecosystem to progress. That means you want people testing and using new versions and detect problems early before they accumulate.
you want new module releases, that fix security problems, to be applied as soon as possible
you want to be able to apply new module releases, that fix security problems, as soon as possible. They are not always tagged at security fixes. If you avoid new releases you also avoid those fixes
even when a new release does not contain security fixes applying its changes early means there will be less changes to vet when the next release that does contain security fixes is published (and the last thing you want when such a release is published and you need to be quick is to be bogged down in intermediary changes you didn't look at before).
applying intermediary releases is only harmful if they break compat, and they shouldn't break compat, and if they do break compat better to detect it and tell the module authors before they make it an habit for the next releases you'll eventually will absolutely need.
you do not want old bits of code to drag you down because they still specify an ancient dependency version and not one finds the time to update their manifest. Using the latest version of a major release serves this social need in other code ecosystems: force devs to test the latest version and not postpone till it's too late because “there are more important” (ie more fun) things to do.
while in theory, you can ship a limitless number of module versions so every piece of code can use the one it wants, in practice as soon as you compose two modules that use the same dep you have to choose a version so the more complex your software is, the less you'll tolerate multiple versions. So you soon hit the old problem of what to do with stragglers that slow down the whole convoy. I never met a human culture that managed this problem by telling stragglers "you're right, go as slow as you want, everyone will wait for you". It might be nice and altruistic but it's not productive.

Fighting human inertia is hard and painful, and we're fighting it because it is required to progress not because it is pleasant. Making pleasant tools that avoid the problem and incite humans to procrastinate some more is not helpful at all it will only accelerate project sedimentation and technical debt accumulation. There are already dozens of Go projects on github with most of their readme devoted to the author begging his users to upgrade because he made important fixes, defaulting to the oldest release will generalize the problem.

A good rule would be "use the latest release that matches the major release, not every intermediary commit". That would be a compromise going forward and stability. It puts the original project in command, that knows the codebase best, and can decide sanely when to switch its users to a new code state.

nim-nim · Mar 25, 2018

Hi,

While a lot of the points in the proposal are more than welcome and will help taming the large Go codebases that emerged over time the "use minimal version" rule is quite harmful:

you want your code ecosystem to progress. That means you want people testing and using new versions and detect problems early before they accumulate.
you want new module releases, that fix security problems, to be applied as soon as possible
you want to be able to apply new module releases, that fix security problems, as soon as possible. They are not always tagged at security fixes. If you avoid new releases you also avoid those fixes
even when a new release does not contain security fixes applying its changes early means there will be less changes to vet when the next release that does contain security fixes is published (and the last thing you want when such a release is published and you need to be quick is to be bogged down in intermediary changes you didn't look at before).
applying intermediary releases is only harmful if they break compat, and they shouldn't break compat, and if they do break compat better to detect it and tell the module authors before they make it an habit for the next releases you'll eventually will absolutely need.
you do not want old bits of code to drag you down because they still specify an ancient dependency version and not one finds the time to update their manifest. Using the latest version of a major release serves this social need in other code ecosystems: force devs to test the latest version and not postpone till it's too late because “there are more important” (ie more fun) things to do.
while in theory, you can ship a limitless number of module versions so every piece of code can use the one it wants, in practice as soon as you compose two modules that use the same dep you have to choose a version so the more complex your software is, the less you'll tolerate multiple versions. So you soon hit the old problem of what to do with stragglers that slow down the whole convoy. I never met a human culture that managed this problem by telling stragglers "you're right, go as slow as you want, everyone will wait for you". It might be nice and altruistic but it's not productive.

Fighting human inertia is hard and painful, and we're fighting it because it is required to progress not because it is pleasant. Making pleasant tools that avoid the problem and incite humans to procrastinate some more is not helpful at all it will only accelerate project sedimentation and technical debt accumulation. There are already dozens of Go projects on github with most of their readme devoted to the author begging his users to upgrade because he made important fixes, defaulting to the oldest release will generalize the problem.

A good rule would be "use the latest release that matches the major release, not every intermediary commit". That would be a compromise going forward and stability. It puts the original project in command, that knows the codebase best, and can decide sanely when to switch its users to a new code state.

My unanswered question copied from mailing list:

We expect that most developers will prefer to follow the usual “major branch” convention, in which different major versions live in different branches. In this case, the root directory in a v2 branch would have a go.mod indicating v2, like this:

It seems like there's subdirectories and this major branch convention that are both supported by vgo. In my anecdotal experience no repositories follow this convention in Go or other languages (can't actually think of a single one other than the ones forced to by gopkg.in which seems relatively unused these days). Master branch is whatever latest is and has v2.3.4 tags in it's history. Tags exist to separate everything (not just minor versions). If it's necessary to patch an old version, a branch is temporarily created off the last v1 tag, commits pushed, a new tag pushed, and the branch summarily deleted. There is no branch for versions, it's just current master/dev/feature branches + version tags. I know that "everything is a ref" in Git, but for other VCS the distinction may not be as fuzzy.

Having said that, I've tested the above described workflow with vgo (just having tags that say v2.0.0, v2.0.1 and no branches) and it does seem to work. So my question is: Although this works now, is it intended? As it doesn't seem as thoroughly described as the other two workflows in the blog, and I want to ensure that working without a v2/v3... branch is not accidental functionality that will disappear since as I explained above I've never seen this (or the other) described workflow in the post to be massively adopted by anyone (especially outside the Go community).

Of course my argument is coming down to preference and anecdotes, so I'd be willing to do some repo-scraping to prove this across all languages if needed. So far I've really liked the proposal posts and am generally on board with the changes, will continue to follow along and play with vgo.

Thanks for all your efforts.

aarondl · Mar 25, 2018

My unanswered question copied from mailing list:

We expect that most developers will prefer to follow the usual “major branch” convention, in which different major versions live in different branches. In this case, the root directory in a v2 branch would have a go.mod indicating v2, like this:

It seems like there's subdirectories and this major branch convention that are both supported by vgo. In my anecdotal experience no repositories follow this convention in Go or other languages (can't actually think of a single one other than the ones forced to by gopkg.in which seems relatively unused these days). Master branch is whatever latest is and has v2.3.4 tags in it's history. Tags exist to separate everything (not just minor versions). If it's necessary to patch an old version, a branch is temporarily created off the last v1 tag, commits pushed, a new tag pushed, and the branch summarily deleted. There is no branch for versions, it's just current master/dev/feature branches + version tags. I know that "everything is a ref" in Git, but for other VCS the distinction may not be as fuzzy.

Having said that, I've tested the above described workflow with vgo (just having tags that say v2.0.0, v2.0.1 and no branches) and it does seem to work. So my question is: Although this works now, is it intended? As it doesn't seem as thoroughly described as the other two workflows in the blog, and I want to ensure that working without a v2/v3... branch is not accidental functionality that will disappear since as I explained above I've never seen this (or the other) described workflow in the post to be massively adopted by anyone (especially outside the Go community).

Of course my argument is coming down to preference and anecdotes, so I'd be willing to do some repo-scraping to prove this across all languages if needed. So far I've really liked the proposal posts and am generally on board with the changes, will continue to follow along and play with vgo.

Thanks for all your efforts.

Can someone maybe clarify how the proposed alternative model to MVS would work to improve upgrade-cadence? Because it isn't clear to me. My understanding of the alternative (widely used) model is

Developer creates handcrafted manifest, listing version constraints for all used dependencies
Developer runs $solver, that creates a lockfile, listing some chosen subset of transitive dependency versions that satisfy the specified constraints
This lockfile gets committed and is used at build and install time to guarantee reproducible builds
When a new version of a dependency is released and to be used, developer potentially updates the manifest, reruns the solver and recommits the new lockfile

The proposed MVS model as I understand it is

Developer autogenerates go.mod, based on the set of import paths in the module, selecting the currently newest version of any transitive dependency
go.mod gets committed and is used to get lower bounds on versions at build and install time. MVS guarantees reproducible builds
When a new version of a dependency is released and to be used, developer runs vgo get -u, which fetches the newest versions of transitive dependencies and overwrites go.mod with the new lower bounds. That then gets submitted.

It seems I must grossly overlook something and it would be helpful if someone would point out what. Because this understanding seems to imply that due lockfiles specifying exact versions and those being used in the actual build, that MVS is better at increasing upgrade-cadence - as it doesn't allow holding back versions, in general.

Clearly I'm missing something (and will feel stupid in about 5m), what is that?

Merovius · Mar 25, 2018

Can someone maybe clarify how the proposed alternative model to MVS would work to improve upgrade-cadence? Because it isn't clear to me. My understanding of the alternative (widely used) model is

Developer creates handcrafted manifest, listing version constraints for all used dependencies
Developer runs $solver, that creates a lockfile, listing some chosen subset of transitive dependency versions that satisfy the specified constraints
This lockfile gets committed and is used at build and install time to guarantee reproducible builds
When a new version of a dependency is released and to be used, developer potentially updates the manifest, reruns the solver and recommits the new lockfile

The proposed MVS model as I understand it is

Developer autogenerates go.mod, based on the set of import paths in the module, selecting the currently newest version of any transitive dependency
go.mod gets committed and is used to get lower bounds on versions at build and install time. MVS guarantees reproducible builds
When a new version of a dependency is released and to be used, developer runs vgo get -u, which fetches the newest versions of transitive dependencies and overwrites go.mod with the new lower bounds. That then gets submitted.

It seems I must grossly overlook something and it would be helpful if someone would point out what. Because this understanding seems to imply that due lockfiles specifying exact versions and those being used in the actual build, that MVS is better at increasing upgrade-cadence - as it doesn't allow holding back versions, in general.

Clearly I'm missing something (and will feel stupid in about 5m), what is that?

@tpng

Using the plain import path instead of /v1 is to ease the transition, so you don't have to update all your import paths to add /v1 at the end.

This should actually not be necessary. Let me give an example:

A user is currently using e.g. v1.0.0 of a library, pinned by a dependency manager and the tag in the upstream repository. Now upstream decides to create a go.mod and also calls the module /v1. This should result in a new commit and a new tag (e.g. v1.0.1). Since vgo will never attempt to update dependencies on its own, this should not break anything for the user, but he/she can update consciously by also changing the import path (pr vgo can do that for him/her).

I think the main reason for not allowing v1 or v0 in the import path is to ensure that there is only one import path for each compatible version of a package.

Yes, I guess I can indeed see that point to not confuse new users of a library.

mrkanister · Mar 26, 2018

@tpng

Using the plain import path instead of /v1 is to ease the transition, so you don't have to update all your import paths to add /v1 at the end.

This should actually not be necessary. Let me give an example:

A user is currently using e.g. v1.0.0 of a library, pinned by a dependency manager and the tag in the upstream repository. Now upstream decides to create a go.mod and also calls the module /v1. This should result in a new commit and a new tag (e.g. v1.0.1). Since vgo will never attempt to update dependencies on its own, this should not break anything for the user, but he/she can update consciously by also changing the import path (pr vgo can do that for him/her).

I think the main reason for not allowing v1 or v0 in the import path is to ensure that there is only one import path for each compatible version of a package.

Yes, I guess I can indeed see that point to not confuse new users of a library.

What makes this particular class of bug special?

It's not just a bug. It's bigger than that. The library author, who is the foremost expert on their code, has told the world "yo, version 1.7 of X breaks my stuff in ways that are so bad, just don't even build with it".

Clearly, unless it's a compiler error, it's a judgement call. If 99 of your functions panic, but one doesn't... is that just a bug? Or is it a complete incompatibility? What if it's just one function that panics?

At some point, a human has to make that decision. I would much rather deal with a pre-emptive declared incompatibility that occurs at build-time, than worry about an undeclared major problem making it into production.

natefinch · May 22, 2018

What makes this particular class of bug special?

It's not just a bug. It's bigger than that. The library author, who is the foremost expert on their code, has told the world "yo, version 1.7 of X breaks my stuff in ways that are so bad, just don't even build with it".

Clearly, unless it's a compiler error, it's a judgement call. If 99 of your functions panic, but one doesn't... is that just a bug? Or is it a complete incompatibility? What if it's just one function that panics?

At some point, a human has to make that decision. I would much rather deal with a pre-emptive declared incompatibility that occurs at build-time, than worry about an undeclared major problem making it into production.

It's not just a bug. It's bigger than that. The library author, who is the foremost expert on their code, has told the world "yo, version 1.7 of X breaks my stuff in ways that are so bad, just don't even build with it".

There is another way for a library author to express this: Write a test that fails when used with an incompatible version of X. This will have the advantage of requiring no changes to the library if X releases a fixed version, as well as catching any future regressions.

neild · May 22, 2018

It's not just a bug. It's bigger than that. The library author, who is the foremost expert on their code, has told the world "yo, version 1.7 of X breaks my stuff in ways that are so bad, just don't even build with it".

There is another way for a library author to express this: Write a test that fails when used with an incompatible version of X. This will have the advantage of requiring no changes to the library if X releases a fixed version, as well as catching any future regressions.

Write a test that fails when used with an incompatible version of X.

Yeah, I thought of that. But then you're asking every top-level binary to run all the tests for all transitive dependencies, tests which might require infrastructure that you don't have. Not all test suites are 100% isolated.

There's a reason why go test ./... doesn't run tests in the vendor directory.

natefinch · May 22, 2018

Write a test that fails when used with an incompatible version of X.

Yeah, I thought of that. But then you're asking every top-level binary to run all the tests for all transitive dependencies, tests which might require infrastructure that you don't have. Not all test suites are 100% isolated.

There's a reason why go test ./... doesn't run tests in the vendor directory.

To summarize ( likely poorly) it seems like the main technical argument against vgo is that there needs to be a way for libraries to declare globally respected constraints on their own dependencies.

Why not have both sides agree to sanction a competing proposal that explores how GPS2 could fit in with the parts of vgo both sides like. Then, merge the vgo experiment in the meantime and revisit the GPS2 proposal before vgo merges mainline?

As someone who struggled with many dependency tools in the past I am excited for vgo. FWIW, as a "community" member, I feel well represented by the vgo solution far. However, I remain open to considering the arguments in favor of adding global version constraints and look forward to that argument developing further.

pbx0 · May 22, 2018

To summarize ( likely poorly) it seems like the main technical argument against vgo is that there needs to be a way for libraries to declare globally respected constraints on their own dependencies.

Why not have both sides agree to sanction a competing proposal that explores how GPS2 could fit in with the parts of vgo both sides like. Then, merge the vgo experiment in the meantime and revisit the GPS2 proposal before vgo merges mainline?

As someone who struggled with many dependency tools in the past I am excited for vgo. FWIW, as a "community" member, I feel well represented by the vgo solution far. However, I remain open to considering the arguments in favor of adding global version constraints and look forward to that argument developing further.

Just throwing this out there:

That assumes this is a compiler error or something similarly highly visible. It's a lot more likely to be a subtle behavior bug that isn't immediately apparent. You update, run your tests, and all looks good. Maybe a timeout that was 10 seconds is now 30, and that throws off the timeouts in the rest of your stack when under load.

This is an issue in software in general. NPM (the other versioning tool I personally have the most familiarity with) uses an SAT solver combined with a community that has strongly embraced semver, and this problem still exists.

As far as the transitive dependency discussion, the reality is there is no magic wand: at some point the developer must be aware of the dependencies they use and put in the time to properly test the code for which they are ultimately responsible. My employer can't be the only one where we are required to justify every library we use, including transitive libraries, for legal and other reasons.

Honestly, to my eye, a lot of the complaints towards vgo seem to be "it's not perfect, therefore it won't work". Let's not fail to implement a good solution for want of a perfect solution.

To me, it seems very aligned with the overall Go philosophy for the core team to provide a tool that is good in most situations, leaving it to the community to provide more advanced and/or specific tooling.

malexdev · May 22, 2018

Just throwing this out there:

That assumes this is a compiler error or something similarly highly visible. It's a lot more likely to be a subtle behavior bug that isn't immediately apparent. You update, run your tests, and all looks good. Maybe a timeout that was 10 seconds is now 30, and that throws off the timeouts in the rest of your stack when under load.

This is an issue in software in general. NPM (the other versioning tool I personally have the most familiarity with) uses an SAT solver combined with a community that has strongly embraced semver, and this problem still exists.

As far as the transitive dependency discussion, the reality is there is no magic wand: at some point the developer must be aware of the dependencies they use and put in the time to properly test the code for which they are ultimately responsible. My employer can't be the only one where we are required to justify every library we use, including transitive libraries, for legal and other reasons.

Honestly, to my eye, a lot of the complaints towards vgo seem to be "it's not perfect, therefore it won't work". Let's not fail to implement a good solution for want of a perfect solution.

To me, it seems very aligned with the overall Go philosophy for the core team to provide a tool that is good in most situations, leaving it to the community to provide more advanced and/or specific tooling.

@malexdev There may be a problem with your actors in the argument that:

Let's not fail to implement a good solution for want of a perfect solution.

dep is a good solution today. One that was created by the community working together. vgo, as started earlier, makes a couple assumptions:

That no one will ever break from semver, even by accident
That we will need to rework the existing codebases in the existing ecosystem

vgo is more based on the "perfect solution" in a "perfect world" while dep works today following what already works in other programming languages ecosystem while being fault tolerant.

You can see this in the history of the package management problem space I just wrote up.

mattfarina · May 22, 2018

@malexdev There may be a problem with your actors in the argument that:

Let's not fail to implement a good solution for want of a perfect solution.

dep is a good solution today. One that was created by the community working together. vgo, as started earlier, makes a couple assumptions:

That no one will ever break from semver, even by accident
That we will need to rework the existing codebases in the existing ecosystem

vgo is more based on the "perfect solution" in a "perfect world" while dep works today following what already works in other programming languages ecosystem while being fault tolerant.

You can see this in the history of the package management problem space I just wrote up.

To summarize ( likely poorly) it seems like the main technical argument against vgo is that there needs to be a way for libraries to declare globally respected constraints on their own dependencies.

I agree with this summary.

Why not have both sides agree to sanction a competing proposal that explores how GPS2 could fit in with the parts of vgo both sides like.

At this point, I remain unconvinced that GPS2 is a necessity or even a good idea. The ability you lined out above can be retrofitted to vgo on top of MVS (like this or this). Personally, I'm hoping that @sdboyer's next blog post will contain good arguments against MVS itself, but right now, I don't really see any reason for a different algorithm - especially one that would cost significant UX advantages of vgo.

Though, to be fair, I also don't see any reason against experimenting with that.

dep is a good solution today.

I'm not sure I agree. I haven't used it myself, but on slack there where several people complaining about issues that seem directly traceable to its SAT solving approach. And I do know that I'm very unhappy about deps overall design (leaving aside the question of the dependency solver itself, so general workflow and UX).

That no one will ever break from semver, even by accident

I am sorry, but I have asked repeatedly for justification of this statement and didn't get any. Why would vgo assume this in any way, shape or form more than dep? I fundamentally disagree with this statement. It is an assumption made to explain the algorithm. Just like if you'd explain dep's algorithm, you'd explain the assumptions built into semantic versions. At worst, they show the same breakages if you fail to comply with those semantics.

I think in general it makes sense to distinguish between different pieces of what we are talking about. Some of the complaints are regarding SIV, some MVS and some the general vgo-shell. For example, it is true that MVS can't handle upper version bounds, but that does not mean that vgo couldn't handle upper version bounds. SIV requires changing lots of code out there (by rewriting import statements), but again, that does not even necessarily mean vgo would require that. Though to be clear, I also don't think it's that big of a deal, as long as we can migrate. Which AFAICT we can.

Merovius · May 22, 2018

To summarize ( likely poorly) it seems like the main technical argument against vgo is that there needs to be a way for libraries to declare globally respected constraints on their own dependencies.

I agree with this summary.

Why not have both sides agree to sanction a competing proposal that explores how GPS2 could fit in with the parts of vgo both sides like.

At this point, I remain unconvinced that GPS2 is a necessity or even a good idea. The ability you lined out above can be retrofitted to vgo on top of MVS (like this or this). Personally, I'm hoping that @sdboyer's next blog post will contain good arguments against MVS itself, but right now, I don't really see any reason for a different algorithm - especially one that would cost significant UX advantages of vgo.

Though, to be fair, I also don't see any reason against experimenting with that.

dep is a good solution today.

I'm not sure I agree. I haven't used it myself, but on slack there where several people complaining about issues that seem directly traceable to its SAT solving approach. And I do know that I'm very unhappy about deps overall design (leaving aside the question of the dependency solver itself, so general workflow and UX).

That no one will ever break from semver, even by accident

I am sorry, but I have asked repeatedly for justification of this statement and didn't get any. Why would vgo assume this in any way, shape or form more than dep? I fundamentally disagree with this statement. It is an assumption made to explain the algorithm. Just like if you'd explain dep's algorithm, you'd explain the assumptions built into semantic versions. At worst, they show the same breakages if you fail to comply with those semantics.

I think in general it makes sense to distinguish between different pieces of what we are talking about. Some of the complaints are regarding SIV, some MVS and some the general vgo-shell. For example, it is true that MVS can't handle upper version bounds, but that does not mean that vgo couldn't handle upper version bounds. SIV requires changing lots of code out there (by rewriting import statements), but again, that does not even necessarily mean vgo would require that. Though to be clear, I also don't think it's that big of a deal, as long as we can migrate. Which AFAICT we can.

I just watched @rsc's GopherConSG opening keynote about versioning. He addressed the scenario where a dependency introduces a breaking change, and compared how vgo and dep would handle it, which seems to be the main concern here. (It's a great watch.)

If I understood his point correctly, dep may break the build as well if a maximum version limitation is used to avoid a bad version. I'd be very interested to see this point addressed by those here who are concerned that vgo falls short of dep in this regard.

willfaught · May 22, 2018

I just watched @rsc's GopherConSG opening keynote about versioning. He addressed the scenario where a dependency introduces a breaking change, and compared how vgo and dep would handle it, which seems to be the main concern here. (It's a great watch.)

If I understood his point correctly, dep may break the build as well if a maximum version limitation is used to avoid a bad version. I'd be very interested to see this point addressed by those here who are concerned that vgo falls short of dep in this regard.

@willfaught To us, dep breaking the build when a maximum version limitation is used to avoid a bad version is considered a success! This is what we want to happen. Russ correctly notes that this problem is not resolved automatically. A constraint on Helm>2.0.0 is not going to upgrade the user automatically to "Helm@2.1.4" but it would work successfully (downgrade grpc or trigger a build failure if that's impossible) if the user explicitly depended on "Helm==2.1.4". Personally, the first thing I usually try when encountering an issue with a library is forcing an update to the latest version. With Dep, this would inform me of the failure introduced by GRPC 1.4.0. With vgo, the only way for Helm to communicate this to me is through documentation.

To repeat the point because it continues not to be understood: neither dep nor vgo can prevent this problem from occurring or provide a foolproof solution. Rather, dep allows Helm to communicate that the problem exists, whereas vgo does not.

To summarize ( likely poorly) it seems like the main technical argument against vgo is that there needs to be a way for libraries to declare globally respected constraints on their own dependencies.

I would add some color to this summary in that these constraints are needed to deal with the unhappy path of when compatibility rules are violated. vgo establishes the import compatibility rule and then proceeds to develop a solution assuming that rule is always followed. In this idealized world, the need for upper bounds is limited or even nonexistent. In the real world, this will not happen: developers will release updates that break compatibility, either intentionally or otherwise.

I think @Merovius is on to a workable solution. MVS would proceed as specified, then after resolution is complete, vgo would check each resolved package for exclusions. If any are found, these are reported to the end user, who can choose to either alter their dependencies so as to meet these constraints or choose to override and ignore the constraints. I'm been on the flip side of this too and sometimes you know better than the maintainer; it works for your application and that's all you care about.

This restores a path for intermediary libraries to communicate an incompatibility to end users. To repeat the helm example yet again:

User@1.1: Helm @2.1.2
Helm@2.1.2: GRPC @1.3.5

==> User upgrades to Helm@2.1.3 intentionally.

User: Helm @2.1.3, GRPC@1.4.0 ("Helm updated, so I can finally use grpc 1.4!")
Helm: GRPC @1.4.0

==> Bug detected! User is seeing some problems with Helm, so check for a new version.

User: Helm@2.1.4, GRPC@1.4.0
Helm: GRPC @1.3.5, weak GRPC <1.4.0

User sees a warning that GRPC 1.4.0 is rejected by the install of Helm@2.1.4. Since Helm is broken for me and that's what I'm trying to fix, I remove my dependency on GRPC 1.4.0 (sadly loosing some useful functionality) and rerun MVS. This time, MVS resolves GRPC to 1.3.5 and Helm to 2.1.4, rechecks all the weak constraints, finds they hold, and I'm done.

I don't expect any tool to resolve these problems magically but I do expect some recourse as a middleware library. So far as I can tell, the only option in vgo is to fork and rename all my upstream dependencies (or equivalently, copy them into my project) if I want to insulate my users from compatibility issues with these dependencies. I don't think anyone wants that.

dwoodlins · May 22, 2018

@willfaught To us, dep breaking the build when a maximum version limitation is used to avoid a bad version is considered a success! This is what we want to happen. Russ correctly notes that this problem is not resolved automatically. A constraint on Helm>2.0.0 is not going to upgrade the user automatically to "Helm@2.1.4" but it would work successfully (downgrade grpc or trigger a build failure if that's impossible) if the user explicitly depended on "Helm==2.1.4". Personally, the first thing I usually try when encountering an issue with a library is forcing an update to the latest version. With Dep, this would inform me of the failure introduced by GRPC 1.4.0. With vgo, the only way for Helm to communicate this to me is through documentation.

To repeat the point because it continues not to be understood: neither dep nor vgo can prevent this problem from occurring or provide a foolproof solution. Rather, dep allows Helm to communicate that the problem exists, whereas vgo does not.

To summarize ( likely poorly) it seems like the main technical argument against vgo is that there needs to be a way for libraries to declare globally respected constraints on their own dependencies.

I would add some color to this summary in that these constraints are needed to deal with the unhappy path of when compatibility rules are violated. vgo establishes the import compatibility rule and then proceeds to develop a solution assuming that rule is always followed. In this idealized world, the need for upper bounds is limited or even nonexistent. In the real world, this will not happen: developers will release updates that break compatibility, either intentionally or otherwise.

I think @Merovius is on to a workable solution. MVS would proceed as specified, then after resolution is complete, vgo would check each resolved package for exclusions. If any are found, these are reported to the end user, who can choose to either alter their dependencies so as to meet these constraints or choose to override and ignore the constraints. I'm been on the flip side of this too and sometimes you know better than the maintainer; it works for your application and that's all you care about.

This restores a path for intermediary libraries to communicate an incompatibility to end users. To repeat the helm example yet again:

User@1.1: Helm @2.1.2
Helm@2.1.2: GRPC @1.3.5

==> User upgrades to Helm@2.1.3 intentionally.

User: Helm @2.1.3, GRPC@1.4.0 ("Helm updated, so I can finally use grpc 1.4!")
Helm: GRPC @1.4.0

==> Bug detected! User is seeing some problems with Helm, so check for a new version.

User: Helm@2.1.4, GRPC@1.4.0
Helm: GRPC @1.3.5, weak GRPC <1.4.0

User sees a warning that GRPC 1.4.0 is rejected by the install of Helm@2.1.4. Since Helm is broken for me and that's what I'm trying to fix, I remove my dependency on GRPC 1.4.0 (sadly loosing some useful functionality) and rerun MVS. This time, MVS resolves GRPC to 1.3.5 and Helm to 2.1.4, rechecks all the weak constraints, finds they hold, and I'm done.

I don't expect any tool to resolve these problems magically but I do expect some recourse as a middleware library. So far as I can tell, the only option in vgo is to fork and rename all my upstream dependencies (or equivalently, copy them into my project) if I want to insulate my users from compatibility issues with these dependencies. I don't think anyone wants that.

@willfaught To us, dep breaking the build when a maximum version limitation is used to avoid a bad version is considered a success! This is what we want to happen.

The point made in the talk is that vgo is no worse than dep in this scenario. In his example, the build isn't broken in the sense that dep can't find a solution; it's broken in the sense that dep does find a solution, and that solution includes the bad version, resulting in the same bad situation we wanted to avoid.

You really should see the video, which walks through an excellent example, but here's the gist as I understood/remember it:

Package versions (including their dependency requirements) are immutable
To add a maximum version limitation to an existing dependency for your package, you have to publish a new version of your package.
It's possible that dep will choose the previous version of your package in order to satisfy all requirements, in which case your new maximum version limitation will not be present. This allows the bad version to be used after all.

willfaught · May 22, 2018

@willfaught To us, dep breaking the build when a maximum version limitation is used to avoid a bad version is considered a success! This is what we want to happen.

The point made in the talk is that vgo is no worse than dep in this scenario. In his example, the build isn't broken in the sense that dep can't find a solution; it's broken in the sense that dep does find a solution, and that solution includes the bad version, resulting in the same bad situation we wanted to avoid.

You really should see the video, which walks through an excellent example, but here's the gist as I understood/remember it:

Package versions (including their dependency requirements) are immutable
To add a maximum version limitation to an existing dependency for your package, you have to publish a new version of your package.
It's possible that dep will choose the previous version of your package in order to satisfy all requirements, in which case your new maximum version limitation will not be present. This allows the bad version to be used after all.

I mostly agree with the proposal to add maximum version exclusions, but I do have this worry: Suppose I put in my library "use gRPC >1.4, <1.8" then in gRPC 1.9, the authors decide, "you know what, Helm was right, we made a breaking change in 1.8, we're reverting to our prior behavior in 1.9.0." Now people trying to import Helm+gRPC won't be able to use 1.9 until Helm releases a version that says "use gRPC >1.4, except 1.8.0, 1.8.1, 1.8.2, but 1.9+ is cool".

In other words, maybe exclude grpc 1.8 is sufficient because we won't know if gRPC 1.9 will be incompatible or not until it's published, at which point Helm can either add it to the exclude list or not.

carlmjohnson · May 24, 2018

I mostly agree with the proposal to add maximum version exclusions, but I do have this worry: Suppose I put in my library "use gRPC >1.4, <1.8" then in gRPC 1.9, the authors decide, "you know what, Helm was right, we made a breaking change in 1.8, we're reverting to our prior behavior in 1.9.0." Now people trying to import Helm+gRPC won't be able to use 1.9 until Helm releases a version that says "use gRPC >1.4, except 1.8.0, 1.8.1, 1.8.2, but 1.9+ is cool".

In other words, maybe exclude grpc 1.8 is sufficient because we won't know if gRPC 1.9 will be incompatible or not until it's published, at which point Helm can either add it to the exclude list or not.

I know essentially nothing about this space. But from reading the discussion here, it sounds like the biggest disagreement boils down to how to detect erroneous cases. That is, both MVS (vgo) and SAT (dep) handle normal situations more or less well, perhaps not identically, but well enough.

SAT provides an ability that MVS does not: using SAT, the author of the library package P1 can declare "P1 requires P2, and works with P2Version > 1.1 and P2Version < 1.4". MVS can only declare "P1 requires P2, and works with P2Version > 1.1", and can not express the restriction "P2Version < 1.4". In the normal case, this doesn't matter. It only matters if some operation tries to upgrade P2 to version 1.4. In that case, SAT will report an error, while MVS will not. When using MVS, if the incompatibility is not a compilation error, it may cause a failure long after the fact.

No doubt the SAT supporters see other major problems with MVS, but so far this is the one that I understand.

I think it's worth noting that if the restriction expressions are themselves versioned--if they are part of a specific release of P1--then in the normal course of events, before P2 version 1.4 is released, P1 version 2.2 will happily say "P2Version > 1.1". It is only when P2 version 1.4 is released that the P1 authors will notice the incompatibility, and release P1 version 2.3 with "P2Version > 1.1 and P2Version < 1.4". So if you are using P1 version 2.2, neither SAT nor MVS will report any problem with upgrading P2 to version 1.4, although it will fail in some possibly subtle way.

In other words, while it makes perfect sense for a release of P1 to list minimum compatible versions of P2, if the release does work correctly with the most recent version of P2, then it does not make sense for a release to list maximum compatible versions. The maximum compatible version will be either conservative, and therefore increasingly wrong as newer and better versions of P2 appear, or if P2 changes in some incompatible way in the future, will fail to specify that requirement since at the time of the release it doesn't exist.

So if we want to have a system that defines anything other than minimum version requirements, then those requirements must not be part of a specific release, but must instead be part of some sort of metadata associated with the package, metadata that can be fetched at any time without updating the package itself. And that means that the operation "update this package" must be separate from the operation "check whether my current package versions are compatible."

I would claim further--and this is definitely more tenuous than the above--that if "check whether my current package versions are compatible" fails, that it is in general unwise to trust any tool to resolve the problem. If the compatibility problem can not be solved by the simple operation "upgrade every relevant package to the current version", then it requires thought. A tool can guide in that thought, but in general it can't replace it. In particular it seems very unwise for a tool to start downgrading packages automatically.

So if we think in terms of

a way to define package metadata describing package incompatibiliities
based on that, a tool that reports whether your current packages are compatible

then perhaps some of the major differences between MVS and SAT become less important.

ianlancetaylor · May 24, 2018

I know essentially nothing about this space. But from reading the discussion here, it sounds like the biggest disagreement boils down to how to detect erroneous cases. That is, both MVS (vgo) and SAT (dep) handle normal situations more or less well, perhaps not identically, but well enough.

SAT provides an ability that MVS does not: using SAT, the author of the library package P1 can declare "P1 requires P2, and works with P2Version > 1.1 and P2Version < 1.4". MVS can only declare "P1 requires P2, and works with P2Version > 1.1", and can not express the restriction "P2Version < 1.4". In the normal case, this doesn't matter. It only matters if some operation tries to upgrade P2 to version 1.4. In that case, SAT will report an error, while MVS will not. When using MVS, if the incompatibility is not a compilation error, it may cause a failure long after the fact.

No doubt the SAT supporters see other major problems with MVS, but so far this is the one that I understand.

I think it's worth noting that if the restriction expressions are themselves versioned--if they are part of a specific release of P1--then in the normal course of events, before P2 version 1.4 is released, P1 version 2.2 will happily say "P2Version > 1.1". It is only when P2 version 1.4 is released that the P1 authors will notice the incompatibility, and release P1 version 2.3 with "P2Version > 1.1 and P2Version < 1.4". So if you are using P1 version 2.2, neither SAT nor MVS will report any problem with upgrading P2 to version 1.4, although it will fail in some possibly subtle way.

In other words, while it makes perfect sense for a release of P1 to list minimum compatible versions of P2, if the release does work correctly with the most recent version of P2, then it does not make sense for a release to list maximum compatible versions. The maximum compatible version will be either conservative, and therefore increasingly wrong as newer and better versions of P2 appear, or if P2 changes in some incompatible way in the future, will fail to specify that requirement since at the time of the release it doesn't exist.

So if we want to have a system that defines anything other than minimum version requirements, then those requirements must not be part of a specific release, but must instead be part of some sort of metadata associated with the package, metadata that can be fetched at any time without updating the package itself. And that means that the operation "update this package" must be separate from the operation "check whether my current package versions are compatible."

I would claim further--and this is definitely more tenuous than the above--that if "check whether my current package versions are compatible" fails, that it is in general unwise to trust any tool to resolve the problem. If the compatibility problem can not be solved by the simple operation "upgrade every relevant package to the current version", then it requires thought. A tool can guide in that thought, but in general it can't replace it. In particular it seems very unwise for a tool to start downgrading packages automatically.

So if we think in terms of

a way to define package metadata describing package incompatibiliities
based on that, a tool that reports whether your current packages are compatible

then perhaps some of the major differences between MVS and SAT become less important.

Thanks for saying that so well Ian. To follow up, once we have established versions and vgo, we absolutely want to have a new godoc.org (maybe a different name) that records additional information about packages, information that the go command can consult. And some of that information would be pair-wise incompatibility that the go command could report as warnings or errors in any particular build (that is, reporting the damage, not trying to hide it by working around it). But having versions at all in the core toolchain is the first step, and that, along with just minimal version requirements and semantic import versioning, is what has been accepted in this issue.

We are committed to landing this as smoothly as possible. That will require additional tooling, more educational outreach, and PRs to fix issues in existing packages. All that was blocked on accepting this proposal, since it seemed presumptuous to move forward without the overall approach being accepted. But the proposal is accepted, and work will start landing more aggressively now that the uncertainty is over.

rsc · May 25, 2018

Thanks for saying that so well Ian. To follow up, once we have established versions and vgo, we absolutely want to have a new godoc.org (maybe a different name) that records additional information about packages, information that the go command can consult. And some of that information would be pair-wise incompatibility that the go command could report as warnings or errors in any particular build (that is, reporting the damage, not trying to hide it by working around it). But having versions at all in the core toolchain is the first step, and that, along with just minimal version requirements and semantic import versioning, is what has been accepted in this issue.

We are committed to landing this as smoothly as possible. That will require additional tooling, more educational outreach, and PRs to fix issues in existing packages. All that was blocked on accepting this proposal, since it seemed presumptuous to move forward without the overall approach being accepted. But the proposal is accepted, and work will start landing more aggressively now that the uncertainty is over.

I had the same thought about external info for version compatibility... since version compatibility must be constant across , it doesn't need to be in source control (and in fact being in source control is a definite disadvantage as stated above). It would be nice if there were a proposed solution for this, since it definitely seems to be the one major problem with MVS as proposed.

natefinch · May 25, 2018

I had the same thought about external info for version compatibility... since version compatibility must be constant across , it doesn't need to be in source control (and in fact being in source control is a definite disadvantage as stated above). It would be nice if there were a proposed solution for this, since it definitely seems to be the one major problem with MVS as proposed.

It's awesome to see the discussion moving organically in this direction. It has been a one central thrust of my concerns, and it makes it so much easier to explain foundational issues when folks are already most of the way to it.

@ianlancetaylor, i think you're spot on with this observation about needing to be able to make changes to constraint information on already-released versions. As @rsc indicated, such a service is something we've discussed/i suggested in our meetings. We could do it with godoc.org, or something else, sure. But i actually don't think it entails a separate service, and it would be better without one. I made a quick reference to this in the piece i published on Friday (just up from that anchor). If nothing else, in a service, there are questions that then have to be answered about whose declaration of incompatibilities should show up in warnings, which means handling identity, and how we scope declarations to particular situations in the depgraph. Keeping the declarations inside metadata files means we don't have to worry about any of that. But more on that in a sec.

What's really important here is this point, though maybe not the way you intended it:

perhaps some of the major differences between MVS and SAT become less important.

The suggestion of a meta-tool that does this search - yes, that's a SAT search - as a solution to the problems folks are identifying is telling. It's pretty much exactly what we'll have to turn dep into, if MVS goes ahead as described. And the first thing to note there is that, if we're so concerned about these incompatibilities that we're talking about a search tool, then what we're actually saying is that MVS becomes just a step in a larger algorithm, and the grokkability benefits go right out the window.

Except it's worse than that, because no amount of meta tooling can get around the baked-in problem of information loss that arises from compacting minimum and current versions together. The big result of that is cascading rollbacks, which means that actually trying to remediate any of the incompatibilities in this list will very likely end up tossing back other parts of the dependency graph not-necessarily-related to your problem. And, developers won't be able to follow an update strategy that isn't harmful to others.. (Oh, and phantom rules, but that's just an MVS side effect in general).

This is why i've asserted that MVS is an unsuitable intermediate layer on which to build a higher-order tool like this - "not fit for purpose." It's clear that folks believe these incompatibilities will occur, so MVS Is just taking a hard problem and making it harder.

If instead, we unify the problem of an "incompatibility service" back into a metadata file, then i believe it's possible, using only a simple set of pairwise declarations, to achieve the same effect. (This is a draft of the concept, but it increasingly seems to hang together)

It would entail that parts of MVS change, but MVS could still run atop the information encoded there. That'd mostly be useful if incompatibilities truly go nuts, and you want to just avoid all of them. But the primary algorithm would start from a baseline that looks like MVS, then switch to a broader search (to be clear, MVS itself should still be considered search), without the possibility of moving into absurdly old versions.

(note, i'll be on vacation this week, so won't be responding till next weekend)

sdboyer · May 28, 2018

It's awesome to see the discussion moving organically in this direction. It has been a one central thrust of my concerns, and it makes it so much easier to explain foundational issues when folks are already most of the way to it.

@ianlancetaylor, i think you're spot on with this observation about needing to be able to make changes to constraint information on already-released versions. As @rsc indicated, such a service is something we've discussed/i suggested in our meetings. We could do it with godoc.org, or something else, sure. But i actually don't think it entails a separate service, and it would be better without one. I made a quick reference to this in the piece i published on Friday (just up from that anchor). If nothing else, in a service, there are questions that then have to be answered about whose declaration of incompatibilities should show up in warnings, which means handling identity, and how we scope declarations to particular situations in the depgraph. Keeping the declarations inside metadata files means we don't have to worry about any of that. But more on that in a sec.

What's really important here is this point, though maybe not the way you intended it:

perhaps some of the major differences between MVS and SAT become less important.

The suggestion of a meta-tool that does this search - yes, that's a SAT search - as a solution to the problems folks are identifying is telling. It's pretty much exactly what we'll have to turn dep into, if MVS goes ahead as described. And the first thing to note there is that, if we're so concerned about these incompatibilities that we're talking about a search tool, then what we're actually saying is that MVS becomes just a step in a larger algorithm, and the grokkability benefits go right out the window.

Except it's worse than that, because no amount of meta tooling can get around the baked-in problem of information loss that arises from compacting minimum and current versions together. The big result of that is cascading rollbacks, which means that actually trying to remediate any of the incompatibilities in this list will very likely end up tossing back other parts of the dependency graph not-necessarily-related to your problem. And, developers won't be able to follow an update strategy that isn't harmful to others.. (Oh, and phantom rules, but that's just an MVS side effect in general).

This is why i've asserted that MVS is an unsuitable intermediate layer on which to build a higher-order tool like this - "not fit for purpose." It's clear that folks believe these incompatibilities will occur, so MVS Is just taking a hard problem and making it harder.

If instead, we unify the problem of an "incompatibility service" back into a metadata file, then i believe it's possible, using only a simple set of pairwise declarations, to achieve the same effect. (This is a draft of the concept, but it increasingly seems to hang together)

It would entail that parts of MVS change, but MVS could still run atop the information encoded there. That'd mostly be useful if incompatibilities truly go nuts, and you want to just avoid all of them. But the primary algorithm would start from a baseline that looks like MVS, then switch to a broader search (to be clear, MVS itself should still be considered search), without the possibility of moving into absurdly old versions.

(note, i'll be on vacation this week, so won't be responding till next weekend)

@sdboyer

The suggestion of a meta-tool that does this search - yes, that's a SAT search

Can you be more specific? The sentence you quoted is right after Ian suggesting a tool to report whether the selected versions are compatible - and to the best of my knowledge, that is the main alternative suggested here (it certainly is what I intended above). That problem most definitely is not a search and it's trivial and doesn't require solving SAT (it is just evaluating a boolean formula for a given set of values, not trying to find values that satisfy it).

Merovius · May 28, 2018

@sdboyer

The suggestion of a meta-tool that does this search - yes, that's a SAT search

Can you be more specific? The sentence you quoted is right after Ian suggesting a tool to report whether the selected versions are compatible - and to the best of my knowledge, that is the main alternative suggested here (it certainly is what I intended above). That problem most definitely is not a search and it's trivial and doesn't require solving SAT (it is just evaluating a boolean formula for a given set of values, not trying to find values that satisfy it).

Right, simply reporting that there are some known-incompatible values in the formula does not require solving SAT. Taking any action on that basis, such as for a tool that assists in the process of finding a result with no such values in it. I quoted that sentence not because i think it is indicative of people having accepted search as always necessary, but because if we believe that reporting those such conditions is important, then it is because we believe it is likely we will encounter such scenarios. the problem is, once the plausibility and the importance of addressing those cases gets established, it looks like folks then make the erroneous jump that "we can just do all the search things on top of MVS, and it'll be fine." we can, but such attempts become much trickier to deal with because of the useful possible paths that MVS cuts off, by design.

…

On May 28, 2018 4:02:13 PM EDT, Axel Wagner ***@***.***> wrote: @sdboyer > The suggestion of a meta-tool that does this search - yes, that's a SAT search Can you be more specific? The sentence you quoted is right after Ian suggesting a tool to report *whether the selected versions are compatible* - and to the best of my knowledge, that is the main alternative suggested here (it certainly is what I intended above). That problem most definitely is not a search and it's trivial and doesn't require solving SAT (it is just evaluating a boolean formula for a given set of values, not trying to find values that satisfy it). -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #24301 (comment)

sdboyer · May 28, 2018

Right, simply reporting that there are some known-incompatible values in the formula does not require solving SAT. Taking any action on that basis, such as for a tool that assists in the process of finding a result with no such values in it. I quoted that sentence not because i think it is indicative of people having accepted search as always necessary, but because if we believe that reporting those such conditions is important, then it is because we believe it is likely we will encounter such scenarios. the problem is, once the plausibility and the importance of addressing those cases gets established, it looks like folks then make the erroneous jump that "we can just do all the search things on top of MVS, and it'll be fine." we can, but such attempts become much trickier to deal with because of the useful possible paths that MVS cuts off, by design.

…

On May 28, 2018 4:02:13 PM EDT, Axel Wagner ***@***.***> wrote: @sdboyer > The suggestion of a meta-tool that does this search - yes, that's a SAT search Can you be more specific? The sentence you quoted is right after Ian suggesting a tool to report *whether the selected versions are compatible* - and to the best of my knowledge, that is the main alternative suggested here (it certainly is what I intended above). That problem most definitely is not a search and it's trivial and doesn't require solving SAT (it is just evaluating a boolean formula for a given set of values, not trying to find values that satisfy it). -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #24301 (comment)

I quoted that sentence not because i think it is indicative of people having accepted search as always necessary, but because if we believe that reporting those such conditions is important, then it is because we believe it is likely we will encounter such scenarios.

To be clear: The suggestion of retrofitting upper bounds in this way is purely reactive to concerns brought up and to show that it can be done (to critically question the claim that MVS is fundamentally unfit for purpose). It seems a bit unfair to take that concession and willingness to compromise as proof that we think you were right all along.

To me, that claim (that MVS is unfit and an essentially irreversible step in the wrong direction) is what I am personally challenging and the lens I am reading your arguments through. One of those arguments was, that it's a feature if we can declare incompatibilities and have the version selection algorithm fail when they are encountered. Another fair argument is, that if they occur, it would be nice to have the algorithm be able to solve them for us (which would indeed require a SAT solver).

However, while I think those are valid and fair concerns, I don't believe they pass the bar of proving to me that MVS is fundamentally unfit. I still believe MVS as a starting point brings good and important features to the table. And that if those concerns turn out to cause significant pain in practice, there are still lots of ways we can iterate on that - from adding upper bounds (whether as part of go.mod or as a separate service) with pure failures up to and including adding a full SAT solver and lock files at some point. I.e. I agree with you that those things will happen, but a) am (maybe naively) optimistic that they won't cause as much pain as you are anticipating and b) that they are solvable problems even when we start off with MVS.

Merovius · May 28, 2018

I quoted that sentence not because i think it is indicative of people having accepted search as always necessary, but because if we believe that reporting those such conditions is important, then it is because we believe it is likely we will encounter such scenarios.

To be clear: The suggestion of retrofitting upper bounds in this way is purely reactive to concerns brought up and to show that it can be done (to critically question the claim that MVS is fundamentally unfit for purpose). It seems a bit unfair to take that concession and willingness to compromise as proof that we think you were right all along.

To me, that claim (that MVS is unfit and an essentially irreversible step in the wrong direction) is what I am personally challenging and the lens I am reading your arguments through. One of those arguments was, that it's a feature if we can declare incompatibilities and have the version selection algorithm fail when they are encountered. Another fair argument is, that if they occur, it would be nice to have the algorithm be able to solve them for us (which would indeed require a SAT solver).

However, while I think those are valid and fair concerns, I don't believe they pass the bar of proving to me that MVS is fundamentally unfit. I still believe MVS as a starting point brings good and important features to the table. And that if those concerns turn out to cause significant pain in practice, there are still lots of ways we can iterate on that - from adding upper bounds (whether as part of go.mod or as a separate service) with pure failures up to and including adding a full SAT solver and lock files at some point. I.e. I agree with you that those things will happen, but a) am (maybe naively) optimistic that they won't cause as much pain as you are anticipating and b) that they are solvable problems even when we start off with MVS.

It occurs to me that something outside of source control determining compatibility would change the determinism of an MVS system. If you have foo >= 1.5.0 as a constraint in one lib, and another lib has foo >= 1.6.0. Then put those two in a binary and they choose 1.6.0. In MVS this is all you need for a repeatable build. it'll always choose 1.6

But if you add external compatibility to the mix, then you could update that first library to say it's not compatible with 1.6, and then the algorithm would choose 1.7, even though the code hasn't changed... Which means you'd need a lock file again.

For reference, I don't think a lock file is a bad thing. It's nice to have an explicit list of exactly what you need to build. And that should make it fast. No magic logic needed.

natefinch · May 29, 2018

It occurs to me that something outside of source control determining compatibility would change the determinism of an MVS system. If you have foo >= 1.5.0 as a constraint in one lib, and another lib has foo >= 1.6.0. Then put those two in a binary and they choose 1.6.0. In MVS this is all you need for a repeatable build. it'll always choose 1.6

But if you add external compatibility to the mix, then you could update that first library to say it's not compatible with 1.6, and then the algorithm would choose 1.7, even though the code hasn't changed... Which means you'd need a lock file again.

For reference, I don't think a lock file is a bad thing. It's nice to have an explicit list of exactly what you need to build. And that should make it fast. No magic logic needed.

@natefinch If the application's go.mod file was updated to require v1.7.0 because the external compatibility tool indicated v1.6.0 was incompatible, you wouldn't need a lock file. Because specifying v1.7.0 lives in the go.mod file the author could also add a comment saying why v1.7.0 is being used and that information would be useful to readers.

leighmcculloch · May 29, 2018

@natefinch If the application's go.mod file was updated to require v1.7.0 because the external compatibility tool indicated v1.6.0 was incompatible, you wouldn't need a lock file. Because specifying v1.7.0 lives in the go.mod file the author could also add a comment saying why v1.7.0 is being used and that information would be useful to readers.

@leighmcculloch , if any files in app are updated then it is a different build and totally outside the scope of "reproducible build without lockfile" problem.

out-of-band compatibility information is proposed to reflect how knowledge develops: no incompatibilities were known at release time, but then it became apparent and extra information is published regarding already released versions. IMHO by definition this approach leads to a change how dependencies are pulled, otherwise why have this extract incompatibility information at all?

redbaron · May 29, 2018

@leighmcculloch , if any files in app are updated then it is a different build and totally outside the scope of "reproducible build without lockfile" problem.

out-of-band compatibility information is proposed to reflect how knowledge develops: no incompatibilities were known at release time, but then it became apparent and extra information is published regarding already released versions. IMHO by definition this approach leads to a change how dependencies are pulled, otherwise why have this extract incompatibility information at all?

@redbaron @natefinch

The point of the incompatibility information is for authors of libraries to communicate incompatibility information to their users. Whether that information is used at build or at release time is a different question.

In vgo's case, the current idea is to only display warnings (or potentially croak). But notably not to let it influence the choice of of versions used (as that would require solving SAT). So it actually doesn't matter, you can use it at either or both and it will fulfill its duty just fine, while retaining the property of repeatability¹.

In dep, this information is only used at release time and then recorded to a lock file, which is used at build time. So it seems that we are considering a release-time use "good enough" anyway, at least when it comes to concerns of vgo vs. dep.

I still don't think we have to actually answers those questions right now, though.

[1] Personally, I'd argue that using it at release time and only if -v is used at build time is better, because a user shouldn't have to decide whether a warning is actionable or not.

Merovius · May 29, 2018

@redbaron @natefinch

The point of the incompatibility information is for authors of libraries to communicate incompatibility information to their users. Whether that information is used at build or at release time is a different question.

In vgo's case, the current idea is to only display warnings (or potentially croak). But notably not to let it influence the choice of of versions used (as that would require solving SAT). So it actually doesn't matter, you can use it at either or both and it will fulfill its duty just fine, while retaining the property of repeatability¹.

In dep, this information is only used at release time and then recorded to a lock file, which is used at build time. So it seems that we are considering a release-time use "good enough" anyway, at least when it comes to concerns of vgo vs. dep.

I still don't think we have to actually answers those questions right now, though.

[1] Personally, I'd argue that using it at release time and only if -v is used at build time is better, because a user shouldn't have to decide whether a warning is actionable or not.

@rsc wrote:

To follow up, once we have established versions and vgo, we absolutely want to have a new godoc.org (maybe a different name) that records additional information about packages, information that the go command can consult. And some of that information would be pair-wise incompatibility that the go command could report as warnings or errors in any particular build (that is, reporting the damage, not trying to hide it by working around it).

I'm wondering if it is necessary to record pair-wise incompatibility. The way I see it currently is that any incompatibility between module A@vN and module B@vM is really because B made an incompatible change from some version vL where L < M.

If module B did not make an incompatible change, then module A just has a bug. If it did, then the issue is about B itself, not about the pairing of A and B.

So ISTM that any public repository of module metadata can record only incompatibilities of any module with previous versions of itself, which may make the problem more tractable. These incompatibility reports are quite similar to bug reports, although they're not resolvable because once a version is published, it cannot be changed.

When you upgrade your module versions, the go tool could consider the metadata and refuse to consider a version that's incompatible with any currently chosen version. I think this avoids the need to solve SAT. It could also decide that a given module has too many incompatibility reports and refuse to add it as a dependency.

A set of tuples of the form (module, oldVersion, newVersion, description) might be sufficient.

rogpeppe · May 29, 2018

@rsc wrote:

To follow up, once we have established versions and vgo, we absolutely want to have a new godoc.org (maybe a different name) that records additional information about packages, information that the go command can consult. And some of that information would be pair-wise incompatibility that the go command could report as warnings or errors in any particular build (that is, reporting the damage, not trying to hide it by working around it).

I'm wondering if it is necessary to record pair-wise incompatibility. The way I see it currently is that any incompatibility between module A@vN and module B@vM is really because B made an incompatible change from some version vL where L < M.

If module B did not make an incompatible change, then module A just has a bug. If it did, then the issue is about B itself, not about the pairing of A and B.

So ISTM that any public repository of module metadata can record only incompatibilities of any module with previous versions of itself, which may make the problem more tractable. These incompatibility reports are quite similar to bug reports, although they're not resolvable because once a version is published, it cannot be changed.

When you upgrade your module versions, the go tool could consider the metadata and refuse to consider a version that's incompatible with any currently chosen version. I think this avoids the need to solve SAT. It could also decide that a given module has too many incompatibility reports and refuse to add it as a dependency.

A set of tuples of the form (module, oldVersion, newVersion, description) might be sufficient.

the go tool could consider the metadata and refuse to consider a version that's incompatible with any currently chosen version

Of course, this doesn't work when you're adding several dependencies, which between them end up requiring using mutually incompatible versions, because the new versions aren't part of the existing module, but there might be a reasonable heuristic available. It's not crucial AFAICS, because dependencies should be added relatively rarely.

rogpeppe · May 29, 2018

the go tool could consider the metadata and refuse to consider a version that's incompatible with any currently chosen version

Of course, this doesn't work when you're adding several dependencies, which between them end up requiring using mutually incompatible versions, because the new versions aren't part of the existing module, but there might be a reasonable heuristic available. It's not crucial AFAICS, because dependencies should be added relatively rarely.

I worry that go release is becoming this discussion's "sufficiently smart compiler". What can users concretely expect from go release in Go 1.11/12? I think that makes a difference for what reasonable expectations around MVS/SIV are.

carlmjohnson · May 29, 2018

I worry that go release is becoming this discussion's "sufficiently smart compiler". What can users concretely expect from go release in Go 1.11/12? I think that makes a difference for what reasonable expectations around MVS/SIV are.

Thanks for the energy so many of you have brought to Go and this proposal in particular.

The first goal of the proposal process is to "[m]ake sure that proposals get a proper, fair, timely, recorded evaluation with a clear answer." This proposal was discussed at length and we published a summary of the discussion. After six weeks and much discussion, the proposal review committee - stepping in as arbiter because I wrote the proposal - accepted the proposal.

A single GitHub issue is a difficult place to have a wide-ranging discussion, because GitHub has no threading for different strands of the conversation and doesn't even display all the comments anymore. The only way a discussion like this works at all is by active curation of the discussion summary. Even the summary had gotten unwieldy by the time the proposal was accepted.

Now that the proposal is accepted, this issue is no longer the right place for discussion, and we're no longer updating the summary. Instead, please file new, targeted issues about problems you are having or concrete suggestions for changes, so that we can have focused discussions about each specific topic. Please prefix these new issues with “x/vgo:”. If you mention #24301 in the text of the new issue, then it will be cross-referenced here for others to find.

One last point is that accepting the proposal means accepting the idea, not the prototype implementation bugs and all. There are still details to work out and bugs to fix, and we'll continue to do that together.

Thanks again for all your help.

rsc · May 29, 2018

Thanks for the energy so many of you have brought to Go and this proposal in particular.

The first goal of the proposal process is to "[m]ake sure that proposals get a proper, fair, timely, recorded evaluation with a clear answer." This proposal was discussed at length and we published a summary of the discussion. After six weeks and much discussion, the proposal review committee - stepping in as arbiter because I wrote the proposal - accepted the proposal.

A single GitHub issue is a difficult place to have a wide-ranging discussion, because GitHub has no threading for different strands of the conversation and doesn't even display all the comments anymore. The only way a discussion like this works at all is by active curation of the discussion summary. Even the summary had gotten unwieldy by the time the proposal was accepted.

Now that the proposal is accepted, this issue is no longer the right place for discussion, and we're no longer updating the summary. Instead, please file new, targeted issues about problems you are having or concrete suggestions for changes, so that we can have focused discussions about each specific topic. Please prefix these new issues with “x/vgo:”. If you mention #24301 in the text of the new issue, then it will be cross-referenced here for others to find.

One last point is that accepting the proposal means accepting the idea, not the prototype implementation bugs and all. There are still details to work out and bugs to fix, and we'll continue to do that together.

Thanks again for all your help.

There's more work to be done (see the modules label) but the initial module implementation as proposed in this issue has been committed to the main tree, so I am closing this issue.

rsc · Jul 17, 2018

There's more work to be done (see the modules label) but the initial module implementation as proposed in this issue has been committed to the main tree, so I am closing this issue.

gopherbot added this to the Proposal milestone Mar 7, 2018

gopherbot added the Proposal label Mar 7, 2018

dsnet referenced this issue Mar 14, 2018
Closed
proto: panic: duplicate enum registered: google.protobuf.FieldDescriptorProto_Type #178

ianlancetaylor referenced this issue Mar 21, 2018
Closed
proposal: enhance import directive to allow for alternatives on failure to better support promotion of experimental packages into the standard library #22358

the42 referenced this issue Mar 22, 2018
Open
Prepare for Modules (vgo support) #237

lukechampine referenced this issue May 22, 2018
Closed
vendor: adding dependency mgmt with dep #2394
2 of 2 tasks complete

dahankzter referenced this issue May 23, 2018
Closed
Add go.mod file #46

dinos80152 referenced this issue May 24, 2018
Open
Dependency Manager Change to vgo after Go v1.11 #131

anthonynsimon referenced this issue May 24, 2018
Closed
I just run build-all #88

kytrinyx referenced this issue May 25, 2018
Open
Switch to vgo for nextercism's dependency management #535

fd0 referenced this issue May 26, 2018
Closed
release tag format, compatibility with the Go versioning proposal #985

mfridman referenced this issue May 29, 2018
Open
Reconsider tagging repo #126

igm referenced this issue May 29, 2018
Open
add go.mod for use with vgo #61

golang locked and limited conversation to collaborators May 29, 2018

rsc added the modules label Jul 17, 2018

rsc modified the milestones: Proposal, Go1.11 Jul 17, 2018

rsc closed this Jul 17, 2018

golang/go

Join GitHub today

cmd/go: add package version support to Go toolchain #24301

Comments

rsc commented Mar 7, 2018 • edited Edited 4 times rsc edited Mar 20, 2018 (most recent) rsc edited Mar 20, 2018 rsc edited Mar 20, 2018 rsc edited Mar 20, 2018

gopherbot added this to the Proposal milestone Mar 7, 2018

gopherbot added the Proposal label Mar 7, 2018

This comment has been minimized.

Frequently Asked Questions

Why is the proposal not “use Dep”?

Why must major version numbers appear in import paths?

Why are major versions v0, v1 omitted from import paths?

Why must I create a new branch for v2 instead of continuing to work on master?

Won't minimal version selection keep developers from getting important updates?

If $GOPATH is deprecated, where does downloaded code live?

Why are you introducing the // import comment?

rsc commented Mar 7, 2018 • edited Edited 7 times rsc edited Mar 30, 2018 (most recent) rsc edited Mar 30, 2018 rsc edited Mar 30, 2018 rsc edited Mar 27, 2018 rsc edited Mar 20, 2018 rsc edited Mar 20, 2018 rsc edited Mar 20, 2018

Frequently Asked Questions

Why is the proposal not “use Dep”?

Why must major version numbers appear in import paths?

Why are major versions v0, v1 omitted from import paths?

Why must I create a new branch for v2 instead of continuing to work on master?

Won't minimal version selection keep developers from getting important updates?

If $GOPATH is deprecated, where does downloaded code live?

Why are you introducing the // import comment?

This comment has been minimized.

Discussion Summary (last updated 2017-04-25)

How can we handle migration?

How can we deal with singleton registrations?

How should we install a versioned command?

Why are v0, v1 omitted in the import paths? Why must the others appear? Why must v0, v1 never appear?

Why are zip files mentioned in the proposal?

Doesn't putting major versions in import paths violate DRY?

Which timezone is used for the timestamp in pseudo-versions?

Will vgo address non-Go dependencies, like C or protocol buffers? Generated code?

Won't minimal version selection keep developers from getting important updates?

Can I use master to develop v1 and then reuse it to develop v2?

What is the timeline for this?

How can I make a backwards-incompatible security change?

If one repo holds different modules in subdirectories (say, v2, v3, v4), can vgo mix and match from different commits?

What if projects misuse semver? Should we allow minor versions in import paths?

Can you determine if you have more than one package in a build?

Concerns about vgo reliance on proxy vs vendor, especially open source vs enterprise.

Concerns about protobuild depending on GOPATH semantics.

Suggestion to add special vgo-v1-lock tag.

How does one patch a deep dependency without vendoring?

What will we do about module names changing?

What about connecting to local Git servers?

What about binary-only packages?

Should we use path@version syntax in the go.mod file?

Discussion Summary (last updated 2017-04-25)

How can we handle migration?

How can we deal with singleton registrations?

How should we install a versioned command?

Why are v0, v1 omitted in the import paths? Why must the others appear? Why must v0, v1 never appear?

Why are zip files mentioned in the proposal?

Doesn't putting major versions in import paths violate DRY?

Which timezone is used for the timestamp in pseudo-versions?

Will vgo address non-Go dependencies, like C or protocol buffers? Generated code?

Won't minimal version selection keep developers from getting important updates?

Can I use master to develop v1 and then reuse it to develop v2?

What is the timeline for this?

How can I make a backwards-incompatible security change?

If one repo holds different modules in subdirectories (say, v2, v3, v4), can vgo mix and match from different commits?

What if projects misuse semver? Should we allow minor versions in import paths?

Can you determine if you have more than one package in a build?

Concerns about vgo reliance on proxy vs vendor, especially open source vs enterprise.

Concerns about protobuild depending on GOPATH semantics.

Suggestion to add special vgo-v1-lock tag.

How does one patch a deep dependency without vendoring?

What will we do about module names changing?

What about connecting to local Git servers?

What about binary-only packages?

Should we use path@version syntax in the go.mod file?

This comment has been minimized.

rsc commented Mar 7, 2018

This comment has been minimized.

rsc commented Mar 7, 2018

dsnet referenced this issue Mar 14, 2018

This comment has been minimized.

rsc commented Mar 7, 2018 •

edited

Edited 4 times

rsc edited Mar 20, 2018 (most recent)

rsc edited Mar 20, 2018

rsc edited Mar 20, 2018

rsc edited Mar 20, 2018

Why are you introducing the `// import` comment?

rsc commented Mar 7, 2018 •

edited

Edited 7 times

rsc edited Mar 30, 2018 (most recent)

rsc edited Mar 30, 2018

rsc edited Mar 30, 2018

rsc edited Mar 27, 2018

rsc edited Mar 20, 2018

rsc edited Mar 20, 2018

rsc edited Mar 20, 2018

Why are you introducing the `// import` comment?

Suggestion to add special `vgo-v1-lock` tag.

Suggestion to add special `vgo-v1-lock` tag.

ChrisHines commented Mar 20, 2018 •

edited

Edited 3 times

ChrisHines edited Mar 21, 2018 (most recent)

ChrisHines edited Mar 20, 2018

ChrisHines edited Mar 20, 2018

justinian commented Mar 21, 2018 •

edited

Edited 2 times

justinian edited Mar 21, 2018 (most recent)

justinian edited Mar 21, 2018

tpng commented Mar 21, 2018 •

edited

Edited 1 time

tpng edited Mar 21, 2018 (most recent)

TocarIP commented Mar 21, 2018 •

edited

Edited 1 time

TocarIP edited Mar 21, 2018 (most recent)