cmd/go, cmd/compile: record language version to support language transitions

[ianlancetaylor]

As part of moving toward Go 2 as initiated at https://blog.golang.org/toward-go2, I propose some changes to cmd/compile and cmd/go. For background see https://github.com/golang/proposal/blob/master/design/28221-go2-transitions.md.

I propose that we change cmd/compile to take a new -lang option. This option will take an argument which is the name of a Go language version. Go language versions use the same naming scheme as Go release versions and Go release build constraints: "goN.M". For example, -lang=go1.12.

This new option will direct the compiler to compile the code using the specified version of the language. It is an error to specify a future version of the language. All versions of the language from go1.12 forward must be supported.

Note that this may need to be recorded in the export data when describing any inlineable functions.

Since we don't know what language changes, if any, will be in the 1.12 release, it's hard to give a real example. But, suppose we had this feature for the 1.9 release, when type aliases were introduced to the language. Then passing -lang=go1.8 to the compiler would give a syntax error for any code that used a type alias. Passing -lang=go1.9 would permit type aliases as is usual today.

I further propose that we modify cmd/go to add a new directive to the go.mod file: lang goN.M. This directive sets the language version to use when building this module.

When updating a go.mod file, if the file has no lang directive a directive is inserted with the version of the go tool. Otherwise the lang directive is unchanged (unless of course go mod edit is used to change it).

When building a module, if the go.mod file has a lang directive, and it specifies a version N.M that is before or equal to the version of the go tool doing the building, then when invoking the compiler for this module the go tool will pass the option -lang=goN.M. In other words, if the module records that it was built using an older Go release, a newer Go release will build the module using the language version of the older release.

The rationale for these changes is explained in the design document mentioned above.

[gopherbot]

Change https://golang.org/cl/142417 mentions this issue: design: add Go 2 transitions background document

[beoran]

I like this idea.

I read the document and I would like to comment on it as follows:

I feel changes to the Go language that are not backwards compatible should increase the first version number of Go. I am normally not such a fan of semver but in this case it seems appropriate. In practise this will mean that after implementing all planned language changes, we will end up having a Go v5.x or maybe even v27.x, but that is fine, since it makes it very clear that the versions are definitely not backwards compatible.

I don't think redefinitions in the language are as much a problem as you make them out to be. In the case of the size of int changing, this was a backwards compatible change since the size of int is implementation defined, and the users of Go should not write any code that requires it to be of any given size. In this case, and all similar cases where backwards compatibility is retained in a redefinition, increase in the minor version of the is sufficient.

In the case of adding the key word check, this is in essence a backwards-incompatible change. This, as well as any other non backwards compatible changes should come with an increase in the major version number.

So if my proposal here is accepted, then we will certainly have a go version 2.0, soon to be followed by 3.0, maybe even 4.0 and 5.0. I think that the requirement to bump the major version number will give some pause as to whether or not we really want to make an incompatible change.

[mvdan]

I further propose that we modify cmd/go to add a new directive to the go.mod file: lang goN.M. This directive sets the language version to use when building this module.

I worry that this would be confusing, as go.mod already contains a go N.M statement. That isn't used for much yet, though, so perhaps this could piggyback off that statement. Edit: forgot to link to it: https://go-review.googlesource.com/c/125940/

I wonder if this would add lots of complexity over time to cmd/compile, as it would have to keep language support for an arbitrarily long number of Go versions. If a package uses a language feature introduced in 1.15, is a lang go1.15 statement in go.mod much better than the already possible go 1.15 statement?

[beoran]

GCC supports 5 different C++ standards from 98 up to 2a, and 3 different C standards from C89 until C11. Code that actually works is actually valuable, and there are many million line legacy projects that cannot be updated easily any more. Some complexity to stay backward compatible is unavoidable but necessary.

[ianlancetaylor]

@mvdan Unfortunately I haven't been closely tracking go.mod files. What does the go N.M statement mean and when is it updated? It doesn't seem to be documented.

Looking at the CL it doesn't seem to mean exactly what I'm looking for. It seems to be computed as the upper bound of the go N.M versions listed in required packages. Using those semantics would prevent a 1.20 package from depending on a 1.25 package while still using the 1.20 language version. I'm certainly fine with using the existing go N.M line if it will do what we need.

As the background doc says, it's definitely confusing that we use the same names for Go release versions and for Go language versions. I don't know how to avoid that. Fortunately I don't think the names will leak out to regular users very often.

I agree with @beoran that the additional complexity in cmd/compile is manageable.

[DeedleFake]

I really like this idea. A couple of random thoughts:

• Unlike module versions, this should probably allow automatic patch upgrades. For example, if I said lang go2.2, it would always use the latest supported v2.2.x.
• Go 2 and on would need to require a go.mod file to specify the version, or otherwise existing code that hasn't been upgraded to a module would quickly stop working or, depending on what changes were made to the language, silently start working differently.

[magical]

1. go.mod is a convenient dumping ground for this kind of metadata, but it's not obvious to me that module scope is the right granularity for the lang directive. It seems like it would be more useful at package or even file scope, to allow large projects to transition gradually/piecemeal.

2. You mention Perl 6 and Python 3 as examples of what not to do. It might also be worth discussing Python's __future__ imports and Perl 5's use VERSION as more successful examples of how to evolve a language.

[DeedleFake]

I don't think __future__ solves the same problem. The point of __future__ is to allow people to start using features that haven't been released yet and allow testing of those features. The point of this proposal is to allow a Go 2 project to import a Go 3 project which imports a Go 1 project and have it just work. That way legacy code doesn't need updating when backwards incompatible changes get made. This should hopefully allow Go to avoid the migration problem Python has had.

[magical]

It's certainly a different approach, but i do think it tackles the same problem: namely, "how can we add language features without breaking old code?".

Instead of asking old code to tag itself with an old language version, as in the current proposal, Python asks new code to explicitly request the features it needs. There's no reason a priori why this approach couldn't be adopted by Go (although i agree that it probably wouldn't be a good fit).

For one thing, in Python, the transition period during which old code and new code can be mixed is usually only a single release, after which the feature becomes the default; whereas Go would presumably want to extend this for a much longer period of time, possibly indefinitely. In such a case, one can imagine that feature directives would tend pile up in new code, leading to an unergonomic language, which is why i say it probably wouldn't be a good fit for Go.

In any case, it seems worth discussing in the proposal, if only to reject.

[ianlancetaylor]

@DeedleFake I am assuming that we are never going to change the language in a minor version, so lang go2.2 is going to mean exactly the same thing for 2.2.0, 2.2.1, 2.2.2, etc. If that assumption is wrong, then we may need to use three digits in a language version.

I'm not sure it's precisely accurate to say that code without a go.mod file containing a lang directive will quickly stop working. It would only stop working if we removed some language feature that that code relies on. But that is going to be a rare step. It's an important case to consider, but we aren't going to remove features that are widely used. That said, you are correct that using the go.mod file assumes that those will become widely used. If that is not the case, we should record the maximum language in some other way.

Which is, of course, what @magical suggests. I'd prefer to use the go.mod file if possible, because every separate piece of metadata is a significant pain point. Even if go.mod is not ideal, I think it's worth using if we can, just to avoid that pain.

@magical I updated the background document CL to include a section on Python and Perl feature selections. I don't think feature selection is a good fit for Go, but I agree that it's worth mentioning.

[DeedleFake]

Maybe a change to the package statement? Could be something like package main go2, for example, and a lack of a second argument would be the same as saying go1. That would increase the granularity and provide a place to quickly check what version you're dealing with. Editors and plugins could then detect the version of existing files when creating new files, and copy it, the way vim-go does already for the package name. One downside, though, would be that updating could be more awkward, as you'd have to change every file in the package.

Personally, I think go.mod would probably be a better fit, but I figured I'd throw this in there.

[networkimprov]

more useful at package or even file scope

Building on @magical's insight, here's a //go: directive listing only backward-incompatible language features used in the file.

//go:lang handle, xyz

That's more descriptive than min and/or max versions. (Does a max version imply dependence on the bugs or performance of that version?) This directive (or its absence) also informs new versions of the compiler to disable backwards-incompatible features when compiling older code.

One line should be enough, since very few backwards-incompatible features are expected. A //go:lib directive could do the same for the stdlib.

[ianlancetaylor]

Any annotation in the source code itself is troubling because we ideally want the go tool to be able to record the version being used, one way or another, but we certainly do not want the go tool to modify people's source code.