New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
extended forwards compatibility for Go #57001
Comments
Updated link to design doc: https://go.dev/design/57001-gotoolchain. |
I just want to note that the potential for confusion here is high. |
I'm not sure about that. What I hope will be a common mode of usage is that you install some update-aware Go toolchain on your machine and then from that point on just edit go or toolchain lines in your go.mod in various projects and never explicitly update the locally installed Go toolchain ever again. The Go toolchain becomes a detail managed inside go.mod just like all the other inputs to your build. In that mode of usage, this reporting would print on literally every go command that gets run, which is too noisy. |
I think the combination of auto-upgrading and not allowing old toolchains to build new code will change the Go ecosystem to either no longer support multiple versions of Go, or to delay experimenting with new language features. Where previously a Go module could target Go 1.21 but make use of 1.22 features in optional, build-tagged files, it can no longer do so. To be allowed to use Go 1.22 language features, the go.mod has to specify Auto-uprading makes this seem fine at first: anyone who needs Go 1.22 will get it for free, automatically. However, I don't think auto-upgrading can be assumed to be pervasive. For one, most people who hack on Go at least occasionally will be using GOTOOLCHAIN=local, pointed at a git checkout. Furthermore, I predict that some Linux distributions will patch their Go packages to disable the automatic upgrading, as it side-steps their packages and some are allergic to that. This'll lead to users who are left behind even more than they're now due to slow-moving distributions. |
I personally have a few reservations about this proposal. Firstly, I don't feel that the Go version in It's very possible to enable a newer version of Go in This pattern seems common enough; Secondly, the automatic download/execution of binary releases of Go seems really surprising. I feel like it's going to be very awkward for Linux distributions to lose control of the toolchain in use without environment variables (especially if they patch Go). I do wonder how many distros might patch Go entirely to force There are also systems where the binaries downloaded from This part of the proposal is being compared to module dependency management itself. I strongly feel that modules work really, really well in comparison to other languages' package management scenarios (like (I mentioned some of this in #55092 (comment), but my thread seems to have been missed as all of the threads around it were replied to.) |
I spoke to @ianlancetaylor about his concerns. For most commands, you can always run For
I think it should continue to be shorthand for that, which would mean ignoring the In the case where
|
This is true. Older versions of the module will still support Go 1.21, of course. It's just the latest version of the module that only supports Go 1.22. I don't think it's a sure thing that this is a problem. The same happens today for dependencies, of course, and it seems to be fine. I agree that making it easier to update to a new Go toolchain may well result in people updating more quickly.
People who hack on Go will be using I agree that some Linux distributions are likely to patch Go to default to |
@zikaeroh, Alpine should not have problems running standard Go distributions starting in Go 1.20. I am not sure about NixOS. The only thing it should need is a libc.so.6 for the dynamic linker to resolve. Or maybe we should build the distribution cmd/go with -tags netgo and then it wouldn't even need that. I wonder what that would break... |
My (rudimentary) understanding is that |
x/tools/gopls only tests that far back because they want to build with what's on people's machines. If what's on people's machines knew how to fetch a newer toolchain then we'd have stopped needing to support older versions long ago. I think that one reason people are slow to update to new Go versions because it is too difficult. What's difficult is managing Go installations. This proposal removes that difficulty, which in turn should make it easier for people to keep up with newer versions. |
Regarding Linux distributions setting GOTOOLCHAIN=local by default (which I think would be fine), I'm curious whether they apply similar rules to rustup or nvm. Does anyone know? |
I guess I'm confused; my impression was that the hardest problem in upgrading was not obtaining a new version of Go, but making sure that the Go code works for that new version of Go, which is the backwards compatibility proposal (not this one). I would be very interested to know what proportion of Go users obtain Go via a package manager (versus If all of those users are going to end up getting
On Arch, you can either install the Arch doesn't package |
We should describe what happens if we drop an existing port (per https://go.dev/wiki/PortingPolicy). Presumably the go command 1.N will see "go 1.N+1", try to download the binaries for 1.N+1, fail, and then give an error and stop the build. For cases like NixOS I think we would have to expect users on that system to set |
This proposal has been added to the active column of the proposals project |
I filed #57007 for building cmd/go without libc.so.6, which I think would make NixOS happy. I agree that it should be possible to build a toolchain with GOTOOLCHAIN=local as the default, but I don't think most package managers should do this. For example I don't think it makes sense for user-installed package managers like Chocolatey or Homebrew to do this at all. They are not as pedantic about "we are the only way to install software on your machine!" as the operating system-installed package managers are, and they should not be going out of their way to break what will end up being a core feature of the Go experience. I also think a fair number of Go developers still use the installers we provide, and those of course will have the GOTOOLCHAIN=auto default. |
Yes, exactly. And we can give a good error. |
Go's backward compatibility is already very good. To the extent that it needs work, the backwards compatibility proposal will make it even better. That will leave actually getting the upgraded Go toolchain as the hardest problem. My experience maintaining other machines where I build Go programs but don't do Go development has been that I don't upgrade often at all because it is annoying to go download and unpack the right tar files. If all it took was editing a go.mod, I would do that far more often. Another point that I forgot to make in the doc is that setting the Go toolchain in go.mod means that all developers working on a project will agree on the toolchain version, without other special arrangements. This was of course one of the big improvements of modules and moving out of GOPATH, for dependency versions. The same property can be provided for the Go toolchain version. This would also mean that if you are moving back and forth between two projects that have chosen different Go toolchain versions as their standard toolchain, you get the right one for that project automatically by virtue of just being in that project. You don't have to change your PATH each time you switch, or maintain a global symlink in $HOME/bin, or remember to type 'go1.18 build' in one place and 'go1.19 build' in the other, or any other kludge. It just does the right thing. |
In a CI/CD environment, I don't think this feature would be useful. I would expect that job to be configured to use the correct Go version in the first place. Downloading a newer toolchain might not work anyway due to firewall restrictions. And as others have mentioned, using an older compiler with On another note, while the go directive in go.mod controls language features like octal literals and generics, today it has no bearing on the standard library implementation. My expectation is that if I run
We build our own version of the toolchain and distribute it to our developers, rather than using what is on golang.org. So this would not address that problem for us, especially because it sounds like such a toolchain will default to Another use case is if I am writing a library and want to easily test it with multiple go versions on my local machine. It would be convenient if I could just do something like |
I find this to conflict with the idea from the original proposal that it's a minimum version; if my project says "go 1.13" because that's the minimum, I very likely do not want to use that version of Go for development. A newer toolchain will be faster and behave better when called by tooling like gopls or other analyzers (regardless of the version of Go that gopls is compiled with). Or, I will definitely want to publish binaries using the absolute latest version of Go possible. For example, esbuild says "go 1.13", but the actual binaries published to npm are from Go 1.19. If 1.13 this were to be the expected development version, that wouldn't be very optimal. |
To be honest I am fine with: |
I feel like this proposal also doesn't really address this problem. To me, it sounds like both of these features are fundamentally mis-designed. If I am publishing a library on GitHub, then I should be specifying a range of minor versions (e.g, 1.17+), and then it tests with the latest patch release of each of them as part of my merge gate. Since the go.mod file cannot be relied upon to denote the lower or the upper bound, it really has no bearing on this, except possibly as the default lower bound. Separately, if I am publishing an actual binary as a release artifact, then I should be specifying the Go version (for build reproducibility), but the go.mod should not be considered at all.
Existing versions of the Go compiler already do not fail just because the |
I don't think this is a concern here. This proposal does not say that newer Go versions should download an older Go toolchain. It says that older Go versions should download a newer Go toolchain. When a newer Go version sees an older "go" line in go.mod, it will emulate the language features of the older Go language (that is already true today). |
I agree; that was my original interpretation of the proposal. It just seemed like the followups implied otherwise. (That is how pinning tends to work in other languages like node.) |
Change https://go.dev/cl/450916 mentions this issue: |
As others here have explained, this proposal would cause a lot of problems for CI/CD, for package management, for conditionally using newer features in older codebases with conditional compilation, ... The version of go in go.mod is now a minimum, which is fine. Rather than a magic environment variable, I would add a new command, go upgrade, that can upgrade the go compiler if installed locally from the official packages, but gives a helpful message if one should upgrade using the package manager in stead. |
FWIW I only mentioned This also is not exclusive to language features; I recall a series of CLs updating the
Which to me very much supports the idea that (I apparently forgot to actually submit this reply days ago, oops.) |
This seems like a bad idea to me. Other text file formats include a line identifying the tool version that created the file, and do not imply that older versions cannot process the file properly. So the current behavior is not without precedent. (Examples of these other formats aren't springing to mind, but it wasn't a novel concept when I first saw it in go.mod). This change assumes that upgrading the toolchain will always be beneficial and regressions will never happen. That stance doesn't seem much different to me than that of a certain OS, where users can be forced to reboot for an update - with no possibility to defer or skip the update. With a language rather than an OS, this behavior may be less infuriating and less disruptive for most users, but I think it's still unacceptable. If a change is to be made, I would instead deprecate the If I understand correctly, the |
@mark-pictor-csec I think you just described the actual proposal. minimum_go is spelled 'go'. It sets the minimum Go toolchain version that must be used to compile the module, whether used directly (via git clone + cd into it + run go commands) or indirectly (as a module dependency). recommended_go is spelled 'toolchain'. It only applies when code is used directly by cd'ing into the module. It is ignored when the module is used indirectly as a required dependency. It is not true that the proposal "assumes that upgrading the toolchain will always be beneficial and regressions will never happen." In fact I have gone out of my way in the past to explain why that's false. See for example https://research.swtch.com/vgo-mvs. Instead, the proposal, like Go modules as a whole, puts the user in control of which toolchain they use. There are no gremlins going around updating the Go toolchain behind your back. You only get a new go toolchain when you explicitly make a change to your go.mod file to indicate that. This is the same behavior as other modules: you keep getting the one listed in go.mod, even if there are newer ones, until you explicitly upgrade. |
Change https://go.dev/cl/497399 mentions this issue: |
As part of the work for #57179 we moved configurable defaults to GOROOT/go.env, so that packagers don't have to modify source code to change those defaults. Since packagers may want to modify GOTOOLCHAIN's default, move it to go.env too. This CL modifies 'go env' to print GOTOOLCHAIN by default. It also refines CL 496957 from yesterday to recognize any env var in either go.env or the user go/env, not just the user go/env. When I put GOTOOLCHAIN in go.env, but before I added it to the default printing list, 'go env GOTOOLCHAIN' was printing an empty string, and it was incredibly confusing. For #57001. Fixes #60361 while we're here. Also includes a fix for a review comment on CL 497079 that I forgot to mail. Change-Id: I7b904d9202f05af789aaa33aed93f903b515aa28 Reviewed-on: https://go-review.googlesource.com/c/go/+/497437 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com> Run-TryBot: Russ Cox <rsc@golang.org>
Add new toolchain directive to go.mod and go.work parser. Also fix error checking in parsing tests. For golang/go#57001. Change-Id: Ib7603f82cbd667f2152ed6b0c5989f08c28ceb1c Reviewed-on: https://go-review.googlesource.com/c/mod/+/497399 Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com>
Change https://go.dev/cl/497555 mentions this issue: |
Change https://go.dev/cl/497400 mentions this issue: |
This sets up for introducing the 'go' and 'toolchain' modules but should be a no-op by itself. For #57001. Change-Id: I2e02b5d417f1edd4f4653b101e4975fe23093f66 Reviewed-on: https://go-review.googlesource.com/c/go/+/497456 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com> Run-TryBot: Russ Cox <rsc@golang.org>
Also add tests of previous CLs. For golang/go#57001. Change-Id: I755429dd07c0e84910108ce9807d607115329b79 Reviewed-on: https://go-review.googlesource.com/c/mod/+/497400 Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com>
cd src/cmd go get golang.org/x/mod@fc83a8f # CL 497400 go mod vendor go mod tidy For #57001. Change-Id: I46b8584e493934883cc4148a16e287f667dcab7d Reviewed-on: https://go-review.googlesource.com/c/go/+/497295 Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>
We have added a new toolchain directive in go.mod and go.work. This CL adds support in mod edit and work edit for changing the toolchain line. For #57001. Change-Id: I36a960796630a359b8a587877cb9548c299d5c87 Reviewed-on: https://go-review.googlesource.com/c/go/+/497296 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com> Run-TryBot: Russ Cox <rsc@golang.org>
Change https://go.dev/cl/497879 mentions this issue: |
For people following along, it looks like there is a new option // GOTOOLCHAIN=auto looks in PATH and then falls back to download.
// GOTOOLCHAIN=path only looks in PATH. I don't recall if that came up during the proposal discussion, but that seems like a nice improvement. As I understand it, (In comparison, It would be nice if it turns out that distros that want to modify the GOROOT/go.env file to prevent automatic downloads of the Go toolchain will be satisfied with setting |
Change https://go.dev/cl/498076 mentions this issue: |
Change https://go.dev/cl/498077 mentions this issue: |
The actual selection code already worked (except for the x/mod parser not reading the file), so all that is necessary is a test. For the test, move the version check up before the module line presence check. For #57001. Change-Id: Iaa4f9b92d38fcfd99dc1665ec8d3eb0e52007bb4 Reviewed-on: https://go-review.googlesource.com/c/go/+/497555 TryBot-Bypass: Russ Cox <rsc@golang.org> Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com>
Change https://go.dev/cl/497795 mentions this issue: |
Change https://go.dev/cl/498118 mentions this issue: |
Credit to @rittneje for the idea of looking in PATH. #57001 (comment) |
Change https://go.dev/cl/498120 mentions this issue: |
Change https://go.dev/cl/498260 mentions this issue: |
In general an older version of Go does not know how to construct a module written against a newer version of Go: the details may change over time, such as for issues like #42965 (an ignore mechanism). For #57001. Change-Id: Id43fcfb71497375ad2eb5dfd292bad0adca0652e Reviewed-on: https://go-review.googlesource.com/c/go/+/497795 Run-TryBot: Russ Cox <rsc@golang.org> Auto-Submit: Russ Cox <rsc@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>
The original code only handled go install m@v1.0.0 but not queries like go install m@v1 or m@master. Handle those by invoking more of the module machinery. For #57001. Change-Id: I7d54fc3dd65072e5906a17ff95b407b0f7feac69 Reviewed-on: https://go-review.googlesource.com/c/go/+/497879 Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com> Auto-Submit: Russ Cox <rsc@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>
go get go@version and toolchain@version updates the go and toolchain lines in go.mod. If toolchain ends up <= go, it is dropped. When the go version crosses certain version boundaries, it may be necessary to run 'go mod tidy -go=version'. That's left for a followup CL. When the go or toolchain version ends up higher than the current toolchain version, we cannot be sure we know how to write the file out, so we fail with an error message. In GOTOOLCHAIN auto mode, the newer toolchain should be downloaded and reinvoked; that's left for a followup CL too. For #57001. Change-Id: Ibfdcc549b40555a53bdb2d019816d18f1bd16be6 Reviewed-on: https://go-review.googlesource.com/c/go/+/497081 TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Russ Cox <rsc@golang.org> Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com>
Allow both prefix-goVERSION and goVERSION-suffix for custom toolchains. Also make sure that a tie in the toolchain and min version goes to the toolchain line. For #57001. Change-Id: Ibeea5f47db7349cfdf36da188bb43e195e196f5b Reviewed-on: https://go-review.googlesource.com/c/go/+/498076 Auto-Submit: Russ Cox <rsc@golang.org> Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>
Suggested by bcmills in CL 497555 review but easier to do at the top of the stack. For #57001. Change-Id: I6f2cc0f546e8d6b1e03c7335a89f07dcb45cf5b6 Reviewed-on: https://go-review.googlesource.com/c/go/+/498077 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com> Auto-Submit: Russ Cox <rsc@golang.org> Run-TryBot: Russ Cox <rsc@golang.org>
Change https://go.dev/cl/498267 mentions this issue: |
Change https://go.dev/cl/498436 mentions this issue: |
Change https://go.dev/cl/498270 mentions this issue: |
Found by the vet check that runs with 'go test cmd/go/internal/modget'. For #57001. For #60463. Change-Id: I4be94f7156724459a5c47bb9745cbb5651fb972c Reviewed-on: https://go-review.googlesource.com/c/go/+/498270 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> Auto-Submit: Dmitri Shuralyov <dmitshur@google.com>
Many people believe the
go
line in thego.mod
file specifies which Go toolchain to use. This proposal would correct this widely held misunderstanding by making it reality. At the same time, the proposal would improve forward compatibility by making sure that old Go toolchains never try to build newer Go programs.Define the “work module” as the one containing the directory where the go command is run. We sometimes call this the “main module”, but I am using “work module” in this document for clarity.
Updating the
go
line in thego.mod
of the work module, or thego.work
file in the current workspace, would change the minimum Go toolchain used to run go commands. A newtoolchain
line would provide finer-grained control over Go toolchain selection.An environment variable
GOTOOLCHAIN
would control this new behavior. The default,GOTOOLCHAIN=auto
, would use the information ingo.mod
. Setting GOTOOLCHAIN to something else would override thego.mod
.GOTOOLCHAIN=local
would force use of the locally installed toolchain, and other values would choose specific releases. For example, to test the package in the current directory with Go 1.17.2:As part of this change, older Go toolchains would refuse to try to build newer Go code. The new system would arrange that normally this would not come up - the new toolchain would be used automatically. But if forced, such as when using
GOTOOLCHAIN=local
, the older Go toolchain would no longer assume it can build newer Go code. This in turn would make it safe to revisit and fix for loop scoping (discussion #56010).See my talk on this topic at GopherCon for more background.
See the design document for details.
The text was updated successfully, but these errors were encountered: