cmd/go: go mod download needs to support toolchain upgrades

[bcmills]

In the changes for go mod download for Go 1.21, we chose not to support toolchain upgrades during go mod download, because it was going to be fairly complicated to get right, and it wouldn't affect CI systems or users who run a plain go mod download before running tests:
https://cs.opensource.google/go/go/+/master:src/cmd/go/internal/modcmd/download.go;l=184-198;drc=8035de592fbc782d27eb78e914ca4e04ec991a3f

However, it turns out that it would be very useful for the Go module proxy itself — and probably other proxy implementations as well — to be able to go mod download specific module versions, branches, latest, and so on even if they require a newer toolchain than the one the proxy runs by default.

This is especially important for minor releases, since modules may declare (say) go 1.21.3 as soon as Go 1.21.3 is released, which (in order to produce checksums for such a module) would require the proxy maintainer to either immediately deploy the new release for all modules (which greatly complicates pre-deployment testing), or detect failures and explicitly retry (which is exactly what go mod download itself would do).

[bcmills]

@gopherbot, please backport to Go 1.21. This is a blocker for the Go module proxy to be able to deploy Go 1.21 releases with adequate pre-release testing.

[gopherbot]

Backport issue(s) opened: #62055 (for 1.21).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

[dmitshur]

CC @golang/tools-team (since this involves the Go module mirror). Also CC @golang/release as FYI.