Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go, cmd/compile: record language version to support language transitions #28221

Closed
ianlancetaylor opened this issue Oct 15, 2018 · 54 comments

Comments

Projects
None yet
@ianlancetaylor
Copy link
Contributor

commented Oct 15, 2018

As part of moving toward Go 2 as initiated at https://blog.golang.org/toward-go2, I propose some changes to cmd/compile and cmd/go. For background see https://github.com/golang/proposal/blob/master/design/28221-go2-transitions.md.

I propose that we change cmd/compile to take a new -lang option. This option will take an argument which is the name of a Go language version. Go language versions use the same naming scheme as Go release versions and Go release build constraints: "goN.M". For example, -lang=go1.12.

This new option will direct the compiler to compile the code using the specified version of the language. It is an error to specify a future version of the language. All versions of the language from go1.12 forward must be supported.

Note that this may need to be recorded in the export data when describing any inlineable functions.

Since we don't know what language changes, if any, will be in the 1.12 release, it's hard to give a real example. But, suppose we had this feature for the 1.9 release, when type aliases were introduced to the language. Then passing -lang=go1.8 to the compiler would give a syntax error for any code that used a type alias. Passing -lang=go1.9 would permit type aliases as is usual today.

I further propose that we modify cmd/go to add a new directive to the go.mod file: lang goN.M. This directive sets the language version to use when building this module.

When updating a go.mod file, if the file has no lang directive a directive is inserted with the version of the go tool. Otherwise the lang directive is unchanged (unless of course go mod edit is used to change it).

When building a module, if the go.mod file has a lang directive, and it specifies a version N.M that is before or equal to the version of the go tool doing the building, then when invoking the compiler for this module the go tool will pass the option -lang=goN.M. In other words, if the module records that it was built using an older Go release, a newer Go release will build the module using the language version of the older release.

The rationale for these changes is explained in the design document mentioned above.

@gopherbot gopherbot added this to the Proposal milestone Oct 15, 2018

@gopherbot gopherbot added the Proposal label Oct 15, 2018

@gopherbot

This comment has been minimized.

Copy link

commented Oct 15, 2018

Change https://golang.org/cl/142417 mentions this issue: design: add Go 2 transitions background document

@beoran

This comment has been minimized.

Copy link

commented Oct 16, 2018

I like this idea.

I read the document and I would like to comment on it as follows:

I feel changes to the Go language that are not backwards compatible should increase the first version number of Go. I am normally not such a fan of semver but in this case it seems appropriate. In practise this will mean that after implementing all planned language changes, we will end up having a Go v5.x or maybe even v27.x, but that is fine, since it makes it very clear that the versions are definitely not backwards compatible.

I don't think redefinitions in the language are as much a problem as you make them out to be. In the case of the size of int changing, this was a backwards compatible change since the size of int is implementation defined, and the users of Go should not write any code that requires it to be of any given size. In this case, and all similar cases where backwards compatibility is retained in a redefinition, increase in the minor version of the is sufficient.

In the case of adding the key word check, this is in essence a backwards-incompatible change. This, as well as any other non backwards compatible changes should come with an increase in the major version number.

So if my proposal here is accepted, then we will certainly have a go version 2.0, soon to be followed by 3.0, maybe even 4.0 and 5.0. I think that the requirement to bump the major version number will give some pause as to whether or not we really want to make an incompatible change.

@mvdan

This comment has been minimized.

Copy link
Member

commented Oct 16, 2018

I further propose that we modify cmd/go to add a new directive to the go.mod file: lang goN.M. This directive sets the language version to use when building this module.

I worry that this would be confusing, as go.mod already contains a go N.M statement. That isn't used for much yet, though, so perhaps this could piggyback off that statement. Edit: forgot to link to it: https://go-review.googlesource.com/c/125940/

I wonder if this would add lots of complexity over time to cmd/compile, as it would have to keep language support for an arbitrarily long number of Go versions. If a package uses a language feature introduced in 1.15, is a lang go1.15 statement in go.mod much better than the already possible go 1.15 statement?

@beoran

This comment has been minimized.

Copy link

commented Oct 16, 2018

GCC supports 5 different C++ standards from 98 up to 2a, and 3 different C standards from C89 until C11. Code that actually works is actually valuable, and there are many million line legacy projects that cannot be updated easily any more. Some complexity to stay backward compatible is unavoidable but necessary.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor Author

commented Oct 16, 2018

@mvdan Unfortunately I haven't been closely tracking go.mod files. What does the go N.M statement mean and when is it updated? It doesn't seem to be documented.

Looking at the CL it doesn't seem to mean exactly what I'm looking for. It seems to be computed as the upper bound of the go N.M versions listed in required packages. Using those semantics would prevent a 1.20 package from depending on a 1.25 package while still using the 1.20 language version. I'm certainly fine with using the existing go N.M line if it will do what we need.

As the background doc says, it's definitely confusing that we use the same names for Go release versions and for Go language versions. I don't know how to avoid that. Fortunately I don't think the names will leak out to regular users very often.

I agree with @beoran that the additional complexity in cmd/compile is manageable.

@DeedleFake

This comment has been minimized.

Copy link

commented Oct 16, 2018

I really like this idea. A couple of random thoughts:

  • Unlike module versions, this should probably allow automatic patch upgrades. For example, if I said lang go2.2, it would always use the latest supported v2.2.x.
  • Go 2 and on would need to require a go.mod file to specify the version, or otherwise existing code that hasn't been upgraded to a module would quickly stop working or, depending on what changes were made to the language, silently start working differently.
@magical

This comment has been minimized.

Copy link
Contributor

commented Oct 16, 2018

  1. go.mod is a convenient dumping ground for this kind of metadata, but it's not obvious to me that module scope is the right granularity for the lang directive. It seems like it would be more useful at package or even file scope, to allow large projects to transition gradually/piecemeal.

  2. You mention Perl 6 and Python 3 as examples of what not to do. It might also be worth discussing Python's __future__ imports and Perl 5's use VERSION as more successful examples of how to evolve a language.

@DeedleFake

This comment has been minimized.

Copy link

commented Oct 16, 2018

I don't think __future__ solves the same problem. The point of __future__ is to allow people to start using features that haven't been released yet and allow testing of those features. The point of this proposal is to allow a Go 2 project to import a Go 3 project which imports a Go 1 project and have it just work. That way legacy code doesn't need updating when backwards incompatible changes get made. This should hopefully allow Go to avoid the migration problem Python has had.

@magical

This comment has been minimized.

Copy link
Contributor

commented Oct 16, 2018

It's certainly a different approach, but i do think it tackles the same problem: namely, "how can we add language features without breaking old code?".

Instead of asking old code to tag itself with an old language version, as in the current proposal, Python asks new code to explicitly request the features it needs. There's no reason a priori why this approach couldn't be adopted by Go (although i agree that it probably wouldn't be a good fit).

For one thing, in Python, the transition period during which old code and new code can be mixed is usually only a single release, after which the feature becomes the default; whereas Go would presumably want to extend this for a much longer period of time, possibly indefinitely. In such a case, one can imagine that feature directives would tend pile up in new code, leading to an unergonomic language, which is why i say it probably wouldn't be a good fit for Go.

In any case, it seems worth discussing in the proposal, if only to reject.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor Author

commented Oct 16, 2018

@DeedleFake I am assuming that we are never going to change the language in a minor version, so lang go2.2 is going to mean exactly the same thing for 2.2.0, 2.2.1, 2.2.2, etc. If that assumption is wrong, then we may need to use three digits in a language version.

I'm not sure it's precisely accurate to say that code without a go.mod file containing a lang directive will quickly stop working. It would only stop working if we removed some language feature that that code relies on. But that is going to be a rare step. It's an important case to consider, but we aren't going to remove features that are widely used. That said, you are correct that using the go.mod file assumes that those will become widely used. If that is not the case, we should record the maximum language in some other way.

Which is, of course, what @magical suggests. I'd prefer to use the go.mod file if possible, because every separate piece of metadata is a significant pain point. Even if go.mod is not ideal, I think it's worth using if we can, just to avoid that pain.

@magical I updated the background document CL to include a section on Python and Perl feature selections. I don't think feature selection is a good fit for Go, but I agree that it's worth mentioning.

@DeedleFake

This comment has been minimized.

Copy link

commented Oct 16, 2018

Maybe a change to the package statement? Could be something like package main go2, for example, and a lack of a second argument would be the same as saying go1. That would increase the granularity and provide a place to quickly check what version you're dealing with. Editors and plugins could then detect the version of existing files when creating new files, and copy it, the way vim-go does already for the package name. One downside, though, would be that updating could be more awkward, as you'd have to change every file in the package.

Personally, I think go.mod would probably be a better fit, but I figured I'd throw this in there.

@networkimprov

This comment has been minimized.

Copy link

commented Oct 16, 2018

more useful at package or even file scope

Building on @magical's insight, here's a //go: directive listing only backward-incompatible language features used in the file.

//go:lang handle, xyz

That's more descriptive than min and/or max versions. (Does a max version imply dependence on the bugs or performance of that version?) This directive (or its absence) also informs new versions of the compiler to disable backwards-incompatible features when compiling older code.

One line should be enough, since very few backwards-incompatible features are expected. A //go:lib directive could do the same for the stdlib.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor Author

commented Oct 16, 2018

Any annotation in the source code itself is troubling because we ideally want the go tool to be able to record the version being used, one way or another, but we certainly do not want the go tool to modify people's source code.

@networkimprov

This comment has been minimized.

Copy link

commented Oct 16, 2018

go.mod could store the tool version that was selected via a feature directive or the module author. That is build metadata, whereas a list of backwards-incompatible features required by a specific file is code metadata.

EDIT:
I think the main drawback to a //go:lang directive is new boilerplate for Go2 code.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Oct 17, 2018

go.mod already recognizes a 'go 1.8' etc line. I think we should use that instead of 'lang go1.8', and have the language apply to the whole module.

I'm on board with cmd/compile accepting -lang and whatever changes are needed to make compilation with one language version be able to import compilations from another language version. (I would hope that exported information would just be written in some 'go superset' instead of having to tag individual packages during export with language versions.)

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor Author

commented Oct 17, 2018

@rsc using the existing go line in go.mod files may be OK. (I didn't know about it because it isn't documented.)

The current go line does seem to have a somewhat different meaning than I was envisioning. Right now it seems that if a go.mod file says go 1.21, and I am building with Go version 1.20, I will get an error saying "module requires Go 1.21". What I am proposing here is that that case should just compile without incident. I think it's important that the language version be a maximum required language version, not a minimum. Otherwise, everyone is forced up to the version of Go used by whoever built the newest library that they depend on. That's a fast pull to tip, which is not going to work for organizations that need to be conservative. Pulling to tip is not required in the general case, and we shouldn't require it here.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Oct 17, 2018

Right, we need to figure that out - we've started that conversation before of course. :-)

gopherbot pushed a commit to golang/proposal that referenced this issue Oct 23, 2018

design: add Go 2 transitions background document
Updates golang/go#28221

Change-Id: I16cd63ed1ae8e816a0a991524c6192223506d9b6
Reviewed-on: https://go-review.googlesource.com/c/142417
Reviewed-by: Robert Griesemer <gri@golang.org>
@rsc

This comment has been minimized.

Copy link
Contributor

commented Oct 24, 2018

Talked to @ianlancetaylor, @griesemer, and others at proposal review.

The current go line in the go.mod file means "use this exact version of the language semantics".

For a dependency module with a go.mod file with no go line, or no go.mod at all, we will assume the last version of go before we start making breaking changes. (Let's call that Go 1.L.) That way, imported old code keeps using the older version and does not break.

For the current module, once we start making breaking changes, if you run a go command in that module tree and there is no go line, the go command will add a go 1.X line where X is the current version of go. That way go.mod is "initialized" correctly and people don't have to go out of their way to get new language features.

Having decided for a given module which go version to use, the go command would pass -lang=xxx to the compiler when compiling a package. (For alternate build systems that don't yet know about -lang, we'd default to the compiler assuming the latest version of the language. But the go command would simply always pass -lang.) Note that this means all packages in one module get a consistent version of go, which might help for being sure it's OK to move code around inside one module.

For a dependency asking for Go 1.(N+1) or later, when the current go version is Go 1.N, one option is to refuse to build. Another option is to try compiling as Go 1.N and see if the build succeeds. If we never redefine the meaning of existing syntax, then it should be safe to conclude from a successful build that everything is OK. And if the build fails, then it can give the error and say 'by the way, this was supposed to use a newer version of Go, so maybe that's the problem.' @ianlancetaylor has been advocating for this behavior, which would make as much code build as possible, even if people accidentally set the go version line too new. It sounds like this should work - in part because we've all agreed not to redefine the meaning of any existing syntax - so we should probably do it and only roll it back if there is a major problem.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Oct 24, 2018

Leaving open for additional comments.

@gopherbot

This comment has been minimized.

Copy link

commented Oct 24, 2018

Change https://golang.org/cl/144340 mentions this issue: cmd/compile: add -lang flag to specify language version

@wsc1

This comment has been minimized.

Copy link

commented Oct 25, 2018

+1: reclassifying core std lib vs penumbra
+1: -lang compiler flag
-1: per file directives

I think more thought needs to be put into how to deal with mixed -lang= versions. Neither solution "refuse to build" or "issue a warning" seems right because: "refuse to build" means collecting packages/modules for a product could become quite involved and difficult for trivial reasons. "issue a warning" is against the go practice of failing to build unless we're sure it works. One option may be to explore introducing binary compatibility at the same time -lang= is introduced. So if -lang= is first available in go1.12, then from go1.12 onwards, a commitment to binary compatibility of packages/modules could be made. In this case, -lang=x in one module and lang=y in another with x!=y could run and a warning could be issued, but only if the caller requests uniformity of lang= or some other constraints on it.

question: lang meaning changes: #24543 changes the meaning of a loop without function calls to include the possibility of pre-emption by the Go scheduler/runtime. How does this proposal relate?

@wsc1

This comment has been minimized.

Copy link

commented Oct 25, 2018

I think it's important that the language version be a maximum required language version, not a minimum. Otherwise, everyone is forced up to the version of Go used by whoever built the newest library that they depend on.

This point also relates to language meaning changes. As long as the language meaning/semantics is not precise, it doesn't make much sense to declare the meaning of go 1.N compatible with go 1.N-1 (which is equivalent to specifying a minimum version). Case in point is #24543, which (while solving a myriad problems, also) changes the semantics to something less expressive and has restrictive implications for lower level systems programming. If the language semantics (memory model, scheduling, relation to system calls) were precise, then backwards compatibility (minimum version specification) would be reasonable. But it's just not the case.

@networkimprov

This comment has been minimized.

Copy link

commented Oct 25, 2018

@wsc1 maybe propose runtime.Get/SetPreempt() in a new issue, with milestone 1.12?

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor Author

commented Oct 25, 2018

@wsc1 There is no intention of supporting binary compatibility. That's not what the -lang option means. If you are building your program with Go 1.N, then you have to build everything with Go 1.N. We've only ever provided source-level compatibility, and that is all we ever expect to provide.

The -lang option doesn't mean "build with Go 1.X", it means "build with the language semantics of Go 1.X". Just like the -std option of GCC or clang.

What @rsc outlined above is: if the module requests a newer language version, try to build it with the current language version. If that works, fine. If it fails, report the error, and also say that it may be due to the fact that the module requires a newer language version. I think that is consistent with what you are suggesting.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor Author

commented Oct 25, 2018

@wsc1 You are correct that source compatibility is not going to handle all possible cases. So it goes.

@wsc1

This comment has been minimized.

Copy link

commented Oct 25, 2018

@wsc1 maybe propose runtime.Get/SetPreempt() in a new issue, with milestone 1.12?

Already proposed (as something that would work providing it isn't implemented by means of a pre-empting sys call) in that issue.

Also #21314 I proposed a user compiler directive to replace go:nosplit.

No responses (except a thumbs down :), no idea where the pre-emption stuff is or whether it will be implemented, so timeline is awfully tight for figuring out a solution for 1.12, and I'm not a compiler or runtime expert so...

@wsc1

This comment has been minimized.

Copy link

commented Oct 25, 2018

The -lang option doesn't mean "build with Go 1.X", it means "build with the language semantics of Go 1.X". Just like the -std option of GCC or clang.

One potential difference: I think I can link -std=C11 with std=C99, at least in many cases, provided the exported .h includes both or an intersection.

gopherbot pushed a commit that referenced this issue Nov 8, 2018

cmd/go: add goversion environment variable to testing script language
Updates #28221

Change-Id: I8a1e352cd9122bce200d45c6b19955cb50308d71
Reviewed-on: https://go-review.googlesource.com/c/147280
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>

gopherbot pushed a commit that referenced this issue Nov 10, 2018

cmd/go: add go statement when initializing go.mod
When creating a go.mod file, add a go statement mentioning the current
Go version. We can be reasonably confident that the current version is
able to build the module. This is as described in the language
transition proposal at https://golang.org/issue/28221.

Updates #28221

Change-Id: I70a99b3a53f4b6c0288da07473c5a71bb28cd86f
Reviewed-on: https://go-review.googlesource.com/c/147281
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>

gopherbot pushed a commit that referenced this issue Nov 10, 2018

cmd/go: add go mod edit -go flag
It can be used to set the Go language version used by the module.

RELNOTES=yes

Updates #28221

Change-Id: Ief0dd185c01195a17be20dff8627c80943c436e7
Reviewed-on: https://go-review.googlesource.com/c/147282
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@zephyr

This comment has been minimized.

Copy link

commented Nov 10, 2018

@rsc wrote:

For the current module, once we start making breaking changes, if you run a go command in that module tree and there is no go line, the go command will add a go 1.X line where X is the current version of go.

What will it add if one is using a tip/master version of go (like go version devel +fe2c588b1c)?

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor Author

commented Nov 10, 2018

What will it add if one is using a tip/master version of go (like go version devel +fe2c588b1c)?

The next release. More precisely, the highest numbered release tag that is treated as a matching build constraint as described at https://golang.org/pkg/go/build.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor Author

commented Nov 14, 2018

This is implemented.

@gopherbot

This comment has been minimized.

Copy link

commented Nov 27, 2018

Change https://golang.org/cl/151358 mentions this issue: [release-branch.go1.11] cmd/go: don't fail if requested Go version is later than current one

@tmornini

This comment has been minimized.

Copy link

commented Dec 1, 2018

Please don’t use any form of “language” to describe this.

The language is Go. We’re all clear on that, right?

How about grammar instead?

gopherbot pushed a commit that referenced this issue Dec 3, 2018

[release-branch.go1.11] cmd/go: don't fail if requested Go version is…
… later than current one

This is a partial backport of CL 147278 from tip to the Go 1.11 branch.

Change the behavior when the go.mod file requests a Go version that is
later than the current one. Previously cmd/go would give a fatal error
in this situation. With this change it attempts the compilation, and
if (and only if) the compilation fails it adds a note saying that the
requested Go version is newer than the known version.  This is as
described in https://golang.org/issue/28221.

Updates #28221

Change-Id: Iea03ca574b6b1a046655f2bb2e554126f877fb66
Reviewed-on: https://go-review.googlesource.com/c/151358
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@ianlancetaylor

This comment has been minimized.

Copy link
Contributor Author

commented Dec 3, 2018

@tmornini When we speak of Go 1.8 and Go 1.9, we are always speaking of the Go language, but at the same time there is a different between the Go 1.8 language and the Go 1.9 language, in that the latter accepts type aliases.

@tmornini

This comment has been minimized.

Copy link

commented Dec 4, 2018

@ianlancetaylor Yes?!?

I’m suggesting a small change from

-lang=go1.12

to:

-grammar=go1.12

Or, simpler:

-grammar=1.12

The point being: it’s all go...

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor Author

commented Dec 4, 2018

@tmornini Thanks, I think I understand your point, and I disagree. Go 1.8 and Go 1.9 are (slightly) different languages.

@gopherbot

This comment has been minimized.

Copy link

commented Mar 4, 2019

Change https://golang.org/cl/164878 mentions this issue: cmd/go: document GoVersion field in Module struct

gopherbot pushed a commit that referenced this issue Mar 6, 2019

cmd/go: document GoVersion field in Module struct
The 'go version' statement was added during Go 1.11 development in
CL 125940. That CL added the GoVersion field to modinfo.ModulePublic
struct, but did not document it in cmd/go documentation. This was
consistent with the CL description, which stated "We aren't planning
to use this or advertise it much yet".

CL 147281, applied during Go 1.12 development, was a change to start
adding the 'go version' statement when initializing go.mod. The 'go
version' statement is now being used, and it has been documented in
the Go 1.12 release notes at https://golang.org/doc/go1.12#modules.
It's now due time to documement the GoVersion field in cmd/go as well.

Keep the Error field bottom-most, both because it makes sense not to
place it in the middle of other fields, and for consistency with the
field order in struct Package, where the Error information is located
at the very bottom.

Regenerate alldocs.go by running mkalldocs.sh.

Updates #28221
Updates #23969

Change-Id: Iaf43a0da4f6a2489d861092a1d4e002a532952cb
Reviewed-on: https://go-review.googlesource.com/c/go/+/164878
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.