cmd/go: allow package authors to retract older package versions as insecure, incompatible, or broken #24031

@michael-schaller I'm not sure what new functionality you are asking for.

Right now vgo will not choose a version "below" a version you specify. So if there is an insecure package version, put the minimum version selector in your package or another package and it will not choose it. Maybe For modules that build main packages, you can also specify version ranges to exclude. Maybe I'm missing something?

So if there is an insecure package version, put the minimum version selector in your package or another package and it will not choose it.

This only works if you know that a certain version is insecure. I think what he's asking for is a mechanism for package authors to broadcast to the world that a certain version is insecure; so that every time a user pulls it, they'll be warned that that version is deprecated and they'll know to update their mod file.

@ALTree That makes sense. Thanks for clarifying for me. I think that is a fine question to ask and may go hand-in-hand with the question of how to distribute the content / function of go.modverify.

@ALTree correct. :-)

One naive idea would be that 'vgo build' could check the 'go.mod' (or another machine readable file) of the latest package versions for security information. This would also be great for Continuous Integration as then a package author could notify of security issues via CI build failures that are (hopefully) monitored.

@rsc mentioned Deprecated Versions (as part of the Defining Go Modules article) which is similar to this issue. He proposed to append +deprecated to a version tag which would also be a viable solution for this issue if +insecure would be honored by vgo.

IMHO that would be a pretty bare bones solution though as I presume that people would soon want to extend that further. For an instance I could see that someone would also want +buggy for a version with a serious bug (for an example a serious memory leak) or +broken for a version that is broken under certain circumstances (for an example the Windows build is broken). Furthermore this solution lacks a way to add more context as for deprecated versions one might be interested in the deprecation announcement or timeline and for insecure versions one might be interested in CVE, severity, ... and so on.

That said I think signaling via tags if a version is deprecated, insecure, ... is not adequate. Maybe even the proposal from my previous comment isn't. Maybe the discussion should rather go into the direction of a machine readable changelog which could be managed via vgo release.

I'm not sure about the utility of this feature. If I release version 1.2.3, surely it must be fixing some bugs over 1.2.2 and I would probably mark 1.2.2. In other words, on almost every (point) release, I would mark all older versions as insecure. You might say this is only for bugs with a CVE, but I think the point still stands.

I don't see this providing much over just reporting that newer versions have been published.

@uluyol there are actually multiple points for this feature (which I sadly failed to point out so far):

1. It enables machines to automatically react:
   It's a chore to check for new versions and humans are sadly error prone, especially in burdensome cases of chore like this one. Having a machine readable form to notify of insecure, deprecated, buggy, ... versions enables machines to react so that humans can spend their time on better things. In this case vgo would be able to detect an unhealthy build and could let the build fail.

2. Cut down reaction time:
   Once machines are able to react on such issues you can cut down reaction time drastically as for an instance Continuous Integration would fail and ideally you would automatically get a bug report for the build failure with the details why the build failed. This is especially valuable for projects that aren't any longer in the active development phase but rather receive infrequent updates on demand.

3. Indicator for healthiness of new dependencies:
   If you add a new dependency to your project you get immediately a sense of the healthiness of that new dependency. If the new dependency is unhealthy vgo should fail to build and with that you know that you either need to change your project, need to file a bug report against the new dependency or you need to fork and fix the new dependency (if you still want to use it). For an instance this would catch the error that a human tries to add the new dependency with the major version v1 albeit that version is deprecated and a newer major version should be used.

4. Indicator for healthiness of indirect dependencies:
   In the end you should have a cascading effect in case a package version is marked insecure as every other package that directly and most importantly indirectly depends on it should fail to build.

I see. I agree that tags lack content, and that it would be useful to also have a reason (or changeling) listed for why a version is insecure. Then on build you could get messages like

build failed: insecure packages
oauth2: CVE-X-Y vulnerable to DoS
zstd: crash on invalid input

I can see why this would be useful. I do think that we'd want to let people override this behavior though.

A definite +1 on the letting people override this behavior part. :-)

I agree that additional in-band signalling would be useful way to let the maintainer know that action may need to be taken—but I disagree that every situation requires automatic, mechanical action.

Security fixes are important but automatically applying them can be as fraught as always ignoring them.

If the module in question is for a non-opensourced essential service using foo v1.0.5 and there's a foo v1.1.4+security that needs to be immediately investigated by a person. However, if its fixing a vulnerability introduced in foo v1.1.0 it may not necessarily be worth the effort and risk to drop everything and upgrade right now.

I would prefer if vgo would continue to work the way it does today, with another tool, perhaps vgo audit that could check online if the versions used have problems.

The idea is that this tool is run on-demand by the programmer, rather than automatically. If this tool is easy to use, vgo audit could become as natural as running go fmt.

The purpose of vgo is to add versions to the vocabulary of the toolchain, so that users and tools can talk to each other sensibly about versions. As I mentioned in the https://research.swtch.com/vgo-module article, I think it would make sense to have a v1.2.3+deprecated tag, using an annotated tag so that there's a commit message. The commit message can say anything it wants about why the release is deprecated, and we can show that to users. We could easily add a notation in the text for identifying security problems. What happens next is up to tools. Probably vgo list -m -u (tell me about pending module updates) would do well to show information about currently-used modules that have deprecation notices.

Change https://golang.org/cl/270557 mentions this issue: cmd/go: fix retract interval syntax in 'go help mod edit'

Change https://golang.org/cl/272066 mentions this issue: cmd/go: recommend 'go get' command to switch from retracted versions

warning: bikeshedding

I was a little surprised with the choice of square brackets.

retract [ v1.0.0 ]

I would have expected parenthesis.

retract v1.0.0

retract (
  v1.0.0
  v2.0.0
)

edit

There also seem to be some issues with the parser:

retract ( v1.0.0 )

# go: errors parsing go.mod:
# go.mod:5: expected '[' or version

retract (
  v1.0.0
)

# no error

retract (
  v1.0.0
  v2.0.0
)

# go: errors parsing go.mod:
# go.mod:7:3: malformed module path "": empty string

retract [
  v1.0.0
  v2.0.0
]

# go: errors parsing go.mod:
# go.mod:3: expected version after '['
# go.mod:4:3: unknown directive: v1.0.0
# go.mod:5:3: unknown directive: v2.0.0
# go.mod:6: unknown directive: ]

@icholy Square brackets denote closed intervals. For now, open and half-open intervals aren't supported because it would break the parser used in Go 1.14 and earlier. We may support those later, maybe for replace and exclude, too.

Blocks of multiple directives are still delimited by parentheses.

If you find parser bugs, please open a new issue. Those examples look like they're working as intended though.

@icholy Square brackets denote closed intervals. For now, open and half-open intervals aren't supported because it would break the parser used in Go 1.14 and earlier. We may support those later, maybe for replace and exclude, too.

That makes sense, but it's definitely not obvious from looking at it. I was expecting every retracted version to be explicitly listed.

Those examples look like they're working as intended though.

This doesn't seem like valid syntax

retract (
  v1.0.0
)

# no error

Why is it looking for a module path?

retract (
  v1.0.0
  v2.0.0
)

# go: errors parsing go.mod:
# go.mod:7:3: malformed module path "": empty string

Why is it looking for a module path?

retract (
  v1.0.0
  v2.0.0
)

# go: errors parsing go.mod:
# go.mod:7:3: malformed module path "": empty string

v2.0.0 is not a valid module version for a module whose name doesn't end in /v2. If you had a v2.0.0 tag on a non-v2 module you wanted to retract, you'd need to retract the module-ized version, e.g. v2.0.0+incompatible

Hi All, I have a problem with one of my project release, where we did a release 1.0, but it was a mistake that we later analysed and our actual stable release is 0.7 and in future also we will be doing minor changes, so we want to refrain our user to use 1.0 because if they do go get project, it will get the 1.0 version and we can't ask force someone to get the specific version using command. Better I can implement retract, So if I put retract for v 1.0 then with a new release v0.8, will it work or Do I need to have a release greater than 0.1 ? I have this confusion after reading all documentation as it is mentioned there To retract a version, a module author should add a retract directive to go.mod, then publish a new version containing that directive. The new version must be higher than other release or pre-release versions; that is, the @latest version query should resolve to the new version before retractions are considered.

@sumitkumartiwari Please open this discussion in the #modules channel on Gophers Slack or the golang-nuts mailing list. Feel free to @ me there. This design discussion is closed, but many people are still subscribed.

(To answer your question though, you should tag a v1.0.1 release that retracts v1.0.0 and v1.0.1. If it helps, try this interactive guide.)

// Release v1.0.0 should not have been tagged.
retract v1.0.0

// v1.0.1 only retracts v1.0.0.
retract v1.0.1