x/pkgsite: support retractions

[jba]

Beginning with Go 1.16, module authors will be able to retract versions of their modules. The go command will ignore retracted versions by default.

pkg.go.dev should expose that a version is retracted.

UI Ideas

• The page for a retracted version should show that it is retracted, and display the rationale if any.

• In the list of module versions, retracted versions should either be displayed at the bottom, or indicated in some other way (such as a red strikethrough). They shouldn't be hidden, because someone might be using a retracted version and want to visit its page.

• We could show which imported packages are retracted, but that depends on having precise version information about imports (that is, the build list), which we don't currently have.

Implementation

The proposal for retraction is at #24031 (comment). In brief: retraction is done by adding retract directives in go.mod files. Each directive lists a range of versions and a rationale. Only the latest go.mod file, as defined by the go command's @latest query, is authoritative.

• The modules table should have a column that contains the information in the retract directives, perhaps as JSON. The worker parses the go.mod file and populates the column during processing.

• The frontend reads the retractions from the latest version of a module and applies them to the module in the request.

This involves an extra DB query sometimes, but only when viewing a version that is not the latest, which is uncommon.

One wrinkle: the latest version can retract itself. In that case, finding the true latest version (the one that deserves the "LATEST" badge) requires another pass over the list of unretracted versions.

In an alternative design, there is a column modules.retracted which gives the state for each version, and the worker populates that for all versions when it is processing the latest one. This involves some subtleties:

• Writing rows other than the current one can result in deadlocks.
• The worker can process versions in any order, so it might see a version in a retract directive before it has processed it. Where should it store that information?

The advantages are fewer queries at serving time in some cases, and the ability to query easily for the latest unretracted version.

[gopherbot]

Change https://golang.org/cl/295430 mentions this issue: internal/proxydatasource: implement deprecation and retractions

[gopherbot]

Change https://golang.org/cl/295891 mentions this issue: internal: add experiment for retractions and deprecations

[gopherbot]

Change https://golang.org/cl/295892 mentions this issue: internal/postgres: add retraction/deprecation info to GetModuleInfo

[gopherbot]

Change https://golang.org/cl/295896 mentions this issue: internal/postgres: add retraction/deprecation info to versions

[gopherbot]

Change https://golang.org/cl/295898 mentions this issue: internal: compute deprecation when RawLatestInfo is created

[gopherbot]

Change https://golang.org/cl/295897 mentions this issue: internal: add NewRawLatestInfo method

[gopherbot]

Change https://golang.org/cl/295899 mentions this issue: internal/postgres: add retraction/deprecation info to GetNestedModules

[gopherbot]

Change https://golang.org/cl/296209 mentions this issue: internal: embed ModuleInfo into UnitMeta

[gopherbot]

Change https://golang.org/cl/296229 mentions this issue: internal/postgres: add retraction/deprecation info to GetUnitMeta

[gopherbot]

Change https://golang.org/cl/296812 mentions this issue: internal/postgres: truncate raw_latest_versions

[gopherbot]

Change https://golang.org/cl/296815 mentions this issue: content/static: add deprecated and retracted banners

[gopherbot]

Change https://golang.org/cl/297509 mentions this issue: internal/frontend: add deprecated/retracted info to versions

[gopherbot]

Change https://golang.org/cl/309610 mentions this issue: internal/postgres: remove uses of modules.deprecated_comment

[gopherbot]

Change https://golang.org/cl/309709 mentions this issue: internal: remove unit-meta-with-latest experiment

[gopherbot]

Change https://golang.org/cl/309710 mentions this issue: migrations: drop modules.deprecated_comment

[jba]

Retraction support is live.

[zx2c4]

Apparently it doesn't see the retractions in https://pkg.go.dev/golang.zx2c4.com/wireguard . The v0.0.20201119 version should retract all others and itself: https://git.zx2c4.com/wireguard-go/tree/go.mod?h=v0.0.20201119

[jba]

There's something odd going on. Did you perhaps change a tag?

According to the Go tools, the latest version of your module is indeed v0.0.20201119. But the proxy won't download that version, and here's what happens when I try to get it directly:

> GOPROXY=direct go get golang.zx2c4.com/wireguard@v0.0.20201119
go: downloading golang.zx2c4.com/wireguard v0.0.20201119
go get: golang.zx2c4.com/wireguard@v0.0.20201119: verifying module: checksum mismatch
        downloaded: h1:BbmPvlD3yvKCF1ybLUCHD+JYFn3my0MKPY24rJi41bY=
        sum.golang.org: h1:WJ4IKfT3SLG+YbM9aeZlgYB+X7hKzO66GEGBmxJPhjE=

SECURITY ERROR
This download does NOT match the one reported by the checksum server.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.

For more information, see 'go help module-auth'.

The best approach would be to tag a new, later version that has a go.mod with all the desired retractions.

[zx2c4]

You're right. I had changed a tag, after I realized that proxy.golang.org won't cache things without at least a single .go file beyond a few days. So that explains that.

Current status is that retractions don't seem to be propagating:

All of those should actually be retracted, per the go.mod file:

[zx2c4]

It looks like it's still very goofed up:

It thinks that the most recent commit is https://git.zx2c4.com/wireguard-go/commit/?id=6a128dd which is some random old commit from January, rather than the most recent commit, which is today.

[jba]

Sorry for the delay getting back to you.

Your module is an interesting case. You've retracted all the tagged versions, leaving only pseudo-versions. And some of those pseudo-versions, having been landed on top of tagged versions, are technically "later" than the more recent ones you've published.

Can I suggest tagging the version you want to be latest with a new, higher tag? Tagged versions are definitely the way to go.

[zx2c4]

You've retracted all the tagged versions, leaving only pseudo-versions.

I'm following precisely the suggestion given by the Go team earlier for retracting all versions. Things work perfectly with go get. CC @bcmills

Can I suggest tagging the version you want to be latest with a new, higher tag? Tagged versions are definitely the way to go.

The project is not yet ready to release a stable version. Therefore no tags are possible at the moment. Rather, the logic in pkgsite is clearly borked, as go get does the right thing.

[jba]

Agreed, pkgsite has a bug. We'll work on fixing it.

However, tags don't imply stability. You can tag with a major version of zero.

[jba]

Actually, I'm not sure that the bug is in pkgsite. Or at least, I don't quite know what the bug is.

To go over the logic again:

Since all the tagged versions are retracted, only the pseudo-versions matter.
Pseudo-versions are ordered by semver.
The most recent pseudo-version begins v0.0.0, but an older one has a higher version. The older is in the proxy's index and can be downloaded from the proxy:

> go get golang.zx2c4.com/wireguard@v0.0.20201119-0.20210128142622-6a128dde71d9
go: downloading golang.zx2c4.com/wireguard v0.0.20201119-0.20210128142622-6a128dde71d9
go get: upgraded golang.zx2c4.com/wireguard v0.0.0-20210506092213-60a26371f42f => v0.0.20201119-0.20210128142622-6a128dde71d9

Note that the go command thinks it is "upgrading" the version.

The difference between the go command and pkgsite is that the former knows only what the proxy tells it, and the proxy believes that the v0.0.0 pseudo-version is the latest. I'm not sure why. @heschi @hyangah, is there logic to ignore pseudo-versions derived from retracted tagged versions? More likely, the proxy @latest endpoint is just tracking the main/master branch.

Pkgsite has the full history so it "correctly" serves the highest version.

@jayconrod @julieqiu What's the right fix here? Maybe pkgsite should ignore pseudo-versions derived from retracted versions?

[heschi]

We just report whatever go get @latest says. Based on what you're saying, I presume that retracted versions no longer serve as pseudoversion bases, so after the 20201119 tag was retracted the pseudoversions "downgraded".

As I think we've discussed elsewhere, AFAIK the only way to know for sure what the "latest" version is is to ask the go command each time.

[bcmills]

Based on what you're saying, I presume that retracted versions no longer serve as pseudoversion bases, so after the 20201119 tag was retracted the pseudoversions "downgraded".

That is correct. Ideally, when you retract an erroneous release version you should also retract the range of pseudo-versions derived from that release, although in some sense those pseudo-versions were never “released” to begin with.

[gopherbot]

Change https://golang.org/cl/318069 mentions this issue: internal/postgres: ensure good version not later than cooked

[jba]

This should be fixed. https://pkg.go.dev/golang.zx2c4.com/wireguard shows a version published on May 13, a few days ago. Reopen if there are still problems.

[zx2c4]

Excellent, thank you for fixing this!