RFC: Semantic versioning for the language

[nikomatsakis]

Define what kinds of breaking changes are permitted in a minor release (bugs, semantic fixes, clarifications) and introduce an opt-in mechanism to permit other kinds of minor additions.

Rendered view.

[sfackler]

We could avoid some of this feedback weirdness by specifying the language version on the command line like rustc --version 1.3, analagous to e.g. gcc --std=c11. The language version could be tracked in Cargo.toml then, which would also be nice since it could be displayed with the rest of the crate's info.

[seanmonstar]

I prefer that as well.

[nikomatsakis]

@sfackler @seanmonstar ah, interesting thought. @alexcrichton @brson @wycats -- thoughts? seems to touch on general cargo/rustc interaction.

[alexcrichton]

Most options like this tend to actually show up in two places, either on the command line or in the source itself. The reason for this is rooted in our desire to be able to run the compiler independently of Cargo in an ergonomic fashion. For example a Cargo project does not specify #![crate_type] in each file, but a non-Cargo project would to avoid passing the flags to the compiler.

Along those lines I may expect that we accept both forms here. The #![rust_version] attribute would probably not be used much in Cargo, but anyone driving the compiler manually would likely leverage it to avoid passing flags everywhere. Additionally Cargo would probably grow a "Rust version" field and it would pass this argument to the crate automatically, but it would be Cargo configuration instead of configuration in the source.

All in all, I'd be fine putting this responsibility on Cargo itself, but I will personally desire the ability to put it in both places. The only snag I can think of for Cargo is that Cargo will need to detect pre---version compilers and not pass the --version flag possibly.

[nikomatsakis]

@alexcrichton

I am sort of inclined to limit this version to the command line because it is SO much more convenient to know what version you are "parsing as" before you begin parsing. (We could always add a #![rust_version] attribute later.)

One question is what the defaults ought to be: Presumably cargo will supply 1.0.0 by default, but I would sort of prefer if rustc used the most recent version by default. This is driven by the fact that I only run rustc by hand for small scripts etc and I imagine they will want the most recent version.

[ruuda]

Do you mean those with many dependants (i.e. crates that are a dependency of many others)?

[liaoarden]

👍 on crates that have many dependencies rather than crates w/ that dependent feature.

[nikomatsakis]

I meant crates are frequently depended on, yes, thanks.

[nagisa]

Sounds like a highway to code bloat. How long do we want to keep supporting a certain rust_version? Indefinitely?

[tshepang]

My guess is the bloat would be removed in 2.0.

[nrc]

Presumably how long is up to the next major version.

[nrc]

Using feature names rather than version numbers seems better to me. For any given Rust version there will be some (hopefully documented) list of features which are supported in some way, but also some (non-opt-in) features which are not. Furthermore, a crate might want to use feature A which was removed in version 1.4 (say) but also want to use unrelated feature B which was introduced in 1.5. Will the compiler support this? It is at best confusing for the user. Also confusing is that presumably a library compiled on version 1.8 of the compiler but using #![rust_version="1.7"] will be ABI compatible with a binary generated using the same compiler but no version attribute, but not with a binary from the actual 1.7 compiler.

[nikomatsakis]

@nrc

I prefer version numbers primarily, I think, because it's a clearer model for end-users: you are using Rust version X.Y, not Rust with feature X and feature Z. That is, I like to think of Rust as a cohesive whole, not a kind of fragmented landscape.

What prior art I could find in this area seems to focus on version numbers. I am thinking of Java and C (--lang=ansi, etc) primarily, and secondarily of XSLT (a language that I happen to have used a lot, but I imagine relatively few others). I don't know how C# handles things and would be curious to know.

Python has the import from future mechanism which is keyword based, but it works rather differently. It is not intended as a long-term mechanism for allowing older code to continue working, but rather a way to signal that you'd prefer to opt-in to a change that will later become universal.

That said, other languages release less frequently, so maybe version numbers work better there (i.e., the new features from Java 1.8 vs Java 1.7 are going to be easier to remember than Rust 1.22 vs 1.23).

Furthermore, a crate might want to use feature A which was removed in version 1.4 (say) but also want to use unrelated feature B which was introduced in 1.5.

I don't anticipate removing features at all, so I don't imagine this scenario would arise. At most, I would think we would just deprecate a feature, but it would remain unavailable. In any case, this kind of "pick and choose" scenario also just feels like the kind of thing I want to avoid as much as possible. Knowing that you're ensuring compatibility to an older Rust seems ok. Knowing that you're ensuring compatibility to a random "variant" on Rust consisting of some features but not others seems less good.

Also confusing is that presumably a library compiled on version 1.8 of the compiler but using #![rust_version="1.7"] will be ABI compatible with a binary generated using the same compiler but no version attribute, but not with a binary from the actual 1.7 compiler.

That's an interesting point, but seems like a non-sequitor to me. In other words, you're correct that someone could be confused, but it seems like, ultimately, we should just fix the ABI problems so that, in fact, the binaries that get produced ARE compatible (and perhaps have some mechanism for emitting binaries in older formats, if necessary). Given that we are talking about "additive" features here that should always be possible, I would expect, though it will take effort to design things properly.

[nrc]

Presumably this means until we get to 2.0, you don't mean to preclude the possibility of ever having 2.0?

[liaoarden]

Also, would this essentially mean that Rust 1.0 would put semantic versioning, as specified by semvar.org, on hold until 2.0?

[sfackler]

I don't see why we wouldn't fix soundness issues in 2.0 and beyond.

[nikomatsakis]

@nrc Right, I mean that until 2.0, we limit breaking changes to bugs/soundness issues. I can clarify the language here.

@ardentsonata I think this same policy would apply to version 2.0 and beyond, though of course we could change it. That is, we will continue to fix bugs and soundness issues. I am basically trying to define more precisely what can go into a "minor change" in Rust (whether it be from 1.1 to 1.2, or 2.1 to 2.2).

[nrc]

It might be worth commenting on changes which we think are non-breaking and then when released into the wild actually are not. I expect that will be rare, and extremely so while many of our users are on nightly and beta releases. But in the future where (hopefully) most users are on the release channel, it seems possible (though hopefully still rare). Presumably for us to miss the breaking change, it must be small and subtle.

[nikomatsakis]

Yes, this is a good question -- what policy do we have in such cases? Do we rollback? Stick with it? Note that this has come up already in the runup to 1.0, and we opted to stick with the change (on the grounds that some people had adapted, so changing back was even worse than status quo), but the scenario was of course different.

[nrc]

Is this going to cause any bump in the semver version?

[nikomatsakis]

@nrc Apart from a minor version bump, no. That's the point of the RFC, defining what can be done in a "minor version bump".

[nrc]

Sorry I wasn't clear, I didn't mean a major version bump, but that leaves three options - a minor version bump, a minor minor version (whatever they are really called) bump, or no bump at all.

I assume from your reply that this would cause a minor version bump every time?

Actually perhaps I am confused - are we using the minor version bumps for the 6-weekly release? In which case are you saying the processes in this RFC will only happen as part of that normal cycle? And/or that an out-of-cycle release will also cause a minor version bump?

[nrc]

I feel like we should make some attempt at defining a soundness bug. Since we have no formal model, we are not using the formal definition here. Are soundness bugs limited to formal type soundness? Only bugs which affect Rust's major invariants. What about technically sound but unexpected behaviour from some language feature? A linguistic equivalent of the recent leak stuff?

[nikomatsakis]

@nrc

I feel like we should make some attempt at defining a soundness bug.

yeah, I mentioned this in the unresolved questions. I'm not sure precisely how to define it. There are many things that are only unsound when unsafe code gets involved, for example, so this really comes down to deciding what unsafe code should or should not be able to do. That is a work-in-progress -- so I think we should approach this on a case-by-case basis, but use the debates over proposed breaking changes to feed back into the evolving document clarifying what is legal in unsafe code.

So essentially a breaking change is warranted if any of the following results:

1. Ability to produce undefined behavior purely from safe code.
2. Ability to produce undefined behavior using standard library APIs or other unsafe code that "should work" (which will be partly defined by the changes we choose to make or not make, as I said above).

[nikomatsakis]

Ah, I thought of something I forgot here -- I think we want to include other kinds of type safety holes, such as coherence, which may not directly cause undefined behavior, but break other invariants we are trying to maintain (in the case of coherence, that crates can always be linked together).

[nrc]

This assumes that the fix is to introduce an error rather than a silent change in behaviour. That is obviously preferable, but perhaps there will be situations where it is not possible. How will we handle those?

[nikomatsakis]

@nrc

This assumes that the fix is to introduce an error rather than a silent change in behaviour. That is obviously preferable, but perhaps there will be situations where it is not possible. How will we handle those?

this is described in this section https://github.com/nikomatsakis/rfcs/blob/language-semver/text/0000-contingency-plan.md#changes-that-alter-dynamic-semantics-versus-typing-rules

[nrc]

We should attempt to supply a 'rustfix' tool specifically to fix these changes too. Given that this might be possible only in certain circumstances, we should specify how it affects the above steps.

[nikomatsakis]

@nrc

We should attempt to supply a 'rustfix' tool specifically to fix these changes too.

Interesting. Given that we have no such tool, it feels a bit premature, but I I think it falls under a strong version of point 2: rustfix would be equivalent to a very clear error message, and then a bit more. I'll edit something in.

[nrc]

I was thinking that we could write a specific tool (rather than a general purpose rustfix) which fixes specific fallout from a language change and we could make that available at the time of the announcement. Obviously that would not be feasible in many cases, but I imagine such a think could be knocked together in a few days in many cases.

[nrc]

This feels like it might be really hard and sometimes impossible in some cases. We should not guarantee it.

[nikomatsakis]

@nrc

We should not guarantee it.

I did not mean to imply that we would guarantee any such thing. I'm just discussing possible measures we can employ, not all of which will be applicable to or necessary in any given scenario. I'll edit the text to make that clearer.

[nrc]

Feels like we're missing something here on the magnitude of the affects of the error. Presumably an error which means Rust is totally memory unsafe would be treated more urgently than one which allowed public access to a private variable or something.

[nikomatsakis]

@nrc

Feels like we're missing something here on the magnitude of the affects of the error.

Heh, good point.

[nrc]

A "permanently available" soundness hole sounds very bad to me. I can't imagine why we'd allow it to remain. The more severe the hole, the more it should be fixed. And the more motivation for tool support to help fix it.

[nikomatsakis]

@nrc

A "permanently available" soundness hole sounds very bad to me.

I agree, I was envisioning primarily "less dangerous" kinds of soundness holds, such as bugs in coherence that could lead to overlapping impls. I'll rework the text though to make it clear that "permanently available" changes would be the exception.

[nrc]

This does not feel underspecifed to me. The specification is precise and complete - if you're not using #[repr(C)] the compiler can do what it likes with data layout and you cannot rely on it.

[nrc]

Would be good to specify how these will affect semver

[nrc]

Again, presumably this is just covering what happens until we bump the major version number?

[nrc]

This is getting scary - in particular, how will this interact with syntax extensions? I mean we need to figure out what the compatibility story will be with syntax extensions before we stabilise them. But this seems like saying there is no guarantee at all about what the token stream/AST will contain.

[nikomatsakis]

@nrc

Well, I guess that would depend a lot on the interface to syntax extensions, right? But I imagine that syntax extensions take as input token trees that do not differentiate keywords (after all, macro-rules definitions can insert arbitrary keywords, for example). If you were to use a $t:expr (or $t:ty etc) fragment, then of course the current Rust version would be relevant, but for raw token trees it is not. But this of course raises the question of the best syntax extension interface, which I know has somewhat debated :)

[nrc]

+1 for this alternative as mentioned above

[nrc]

Why do we want to encourage this thinking? If we thought that some set of features was coherent and major enough to always use together, we should issue a new major version

[nikomatsakis]

@nrc

If we thought that some set of features was coherent and major enough to always use together, we should issue a new major version

I think that the set of features should always be consistent and major enough to use together at all times? I'm not really sure what "to use together" means, I guess, I'm not sure why you would not use them together.

I feel like major versions are a way of signalling that we removed deprecating things, or we made other "lateral" changes that are not additive nor bug fixes (certainly that's the intention of this RFC).

[nrc]

I hope we don't have so many opt-in features between major versions that this becomes a problem

[nikomatsakis]

@nrc

I hope we don't have so many opt-in features between major versions that this becomes a problem

I guess that will depend on how frequently we issue major versions.

[nikomatsakis]

@nrc

I hope we don't have so many opt-in features between major versions that this becomes a problem

To expound a bit here, I do think that if we start to accumulate a lot of opt-in changes, it's probably a sign that it's time for a new major version. Using feature names rather than version numbers makes that pressure more obvious, which could be seen as an argument either for or against it.

[nrc]

The fact that a feature can be used on beta/release branches means that it is stable.

[nikomatsakis]

@nrc

The fact that a feature can be used on beta/release branches means that it is stable.

Sure, but I can see that people might want to use nightly simply to get acccess to bug fixes and so forth faster, but wouldn't want to rely on unstable features. Knowing that "#[feature] means unstable" makes it easier to audit. Admittedly, probably not too hard either way, but that's what I meant by "mental separation". It's not about what branch you're on.

[nrc]

This seems the most important point and hard to argue against. I guess I can only hope that opt-in features are rare enough that this won't be a problem. How many are you envisaging between major versions?

[nrc]

Yes, we should :-) I don't think it needs to be super formal, but some guidance would be useful.

[nrc]

This seems so easy to fix with a refactoring tool that it should be unnecessary.

[nikomatsakis]

@nrc

This seems so easy to fix with a refactoring tool that it should be unnecessary.

I don't follow. Point is, if I update my compiler and rename my methods, I break my clients and force them to refactor. Maybe your point is: that's acceptable, because we can have a tool that renames functions from match to match_ or something? (we may itself introduce conflicts...)

[nrc]

I was thinking a combination of: it is not so important for downstream libs to break there clients because they can bump their semver major version easily without breaking their downstream deps thanks to Cargo. And it is easy for their downstream libs to adjust to the change thanks to the refactoring tool (I'm envisaging having some kind of script for a refactoring tool and the commit message or new version announcment or something includes a refactoring script for the name change).

Conflicts would make this harder, but if we have good refactoring support, then not too much harder.

[nikomatsakis]

well, I'm just presuming that downstream libs will not necessarily want to bump their major version number just to gain access to a new keyword internally, but I agree that we could possibly just not provide a way to escape and live with it.

[nrc]

I've long wanted to make cfg token tree based.

[nrc]

We've talked about scoped attributes before, it seems like we could take advantage of that here. It does have the backwards compatibility issue, but only once, not for every attribute.

[nrc]

Is this file meant to be included? It seems like an early version of the other one.

[nikomatsakis]

Probably not. :)

[seanmonstar]

In the nodejs world, npm allows properties like these in the package.json. Some are engine.node, engine.npm, and engine.os. Unfortunately, some use them to try to force their preference that everyone upgrade immediately, such as stating engine.node = 0.12, even though those using nodejs in production would recommend waiting for several more point releases.

So, eventually npm stopped treating this as an error, and simply a warning.

[nikomatsakis]

@seanmonstar

So, eventually npm stopped treating this as an error, and simply a warning.

Interesting, thanks! I think I wrote that the equivalent scenario (specifying a version of Rust newer than the current compiler) would be a warning here as well...

[DanielKeep]

Happy to see this again! :)

I'm a little leery of version numbers being used to opt into old behaviour, but I can't think of any reasonable alternatives.

One minor suggestion: I'd like to propose that the #[rust_version] attribute be brought back, but only as a guard mechanism. That is, the presence or non-presence of the attribute cannot change the way code is parsed or compiled, it merely asserts that code is being compiled with an "appropriate" compiler version. Given the version number's use in reverting to old behaviours, "appropriate" here probably equates to "exactly as specified".

The reason I'd like to see this is for code examples. One of the justifications for my PR #457 (Rust language version attribute) was to document the language version of code snippets in documentation, articles and on sites like Stack Overflow.

In very large code bases, it might also help keep track of which parts of the code have been migrated to new versions of the compiler (especially if we're talking about changes to semantics, not syntax or types). (Of course, in that case, some sort of --ignore-version-attrs switch would probably be necessary...)

[comex]

Just a quick note in response to the meeting minutes about a hypothetical 2.0 release schedule... since the text of this RFC itself is somewhat ambiguous about how the proposed rust-version attribute would apply to new major releases. (At least I interpret that way.)

Today I can run clang and compile unchanged a C program written over 25 years ago. (Not C++, though - older C++ compilers were too permissive and so old programs break now. This is a PITA. Still, the standard itself is largely backwards compatible.) Perl 5 was first released in 1994, and I think modern Perl 5 has kept backwards compatibility from there(?). JavaScript appeared in 1995 and it - at least with standards-track APIs - has never broken backwards compatibility except in very minor ways. Python 2 is 15 years old, and has now spent almost half of its life deprecated, while new development using it continues. Look how their attempt to break backwards compatibility is going... migrating the wall of shame is/was one thing, but the long tail is another. I'd say the long tail of old, unmaintained codebases in the wild tends to last an order of magnitude longer than the popular, public, churned stuff. And I don't think that's such a bad thing.

I think Rust should live up to that standard: I want to be able to run rustc in 2040 and compile a program written for Rust 1.0. It's okay if I have to specify --rust-version=1.0, but I should be able to link to it from crates written for newer language versions. Alternately, it's okay if I have to run a migrator tool, but only if it's 100% guaranteed to make the code work (even if in an ugly way) rather than bailing out in tricky cases.

That is, all breaking changes should be opt-in changes.

Of course, I am not the one who would have to do the work for this, but I think it should be done. shrug

[glaebhoerl]

Today I can run clang and compile unchanged a C program written over 25 years ago.

Though it might eat your laundry due to the compiler more aggressively exploiting latent UB in the code than the compilers of 25 years earlier. ;)

[nikomatsakis]

I've been thinking about this RFC a fair amount. It seems to me that most everybody agrees we should fix soundness problems. What is a bit less obvious is what long-term strategy we want to use for evolving and tweaking the language.

This RFC is generally oriented around allowing us to expand the language without issuing new major versions. An alternative philosophy might be that we should be making major version releases on a regular basis, and thus backwards-incompatible changes and extensions can hitch a ride on one of these major versions instead.

There are good arguments to be made for both sides. I think the best strategy and timing for major version updates is not yet entirely clear. On the one hand, some people argue (for good reasons) that we should never issue a major release. Others that we can use major versions more regularly, at least now while Rust is young. (I think one thing that is relatively clear is that whenever we DO issue a major version, we should ensure that there is an incremental path to adopting it.)

In any case, for the time being, I think I will just remove the language on "opt-in" changes from this RFC and focus on what kinds of breaking changes are permitted.

One interesting question is whether we will accept other sorts of breaking changes that are not strictly tied to soundness, but just cases where we seem to have gotten something slightly wrong. I have exactly one example in mind at the moment (some details of trait object lifetime defaults), but I can easily imagine that this scenario will arise. Probably best is to just address such things on a case-by-case basis: every good rule needs an exception, after all.

[nikomatsakis]

Just for the record, some of the hazards with the strategy that the RFC currently espouses (version numbers):

1. The more we evolve the language, the more that online content such as tutorials and so forth bitrots. If we issue major versions, it is easier to think about (ah, that tutorial is for 1.x, not 2.x).
2. I am worried about a proliferation of "random" version numbers -- you can't tell if the version a package declares is because it really needs it, or just happens to be the version, or what. Major versions do help here, since they accumulate things.

[nikomatsakis]

Hear ye, hear ye. This RFC is entering the final comment period.

[llogiq]

Regarding "opting out", I would humbly request the RFC states that we should make the targeted rust version a crate attribute and also strive to ensure interoperability between crates targeting different rust versions (as much as possible).

To enact this will restrict us a bit (as we should refrain from making changes that break compatibility between crates of different rust versions unless we have very good reasons) but give users a clean update path that minimizes breakage.

[nikomatsakis]

@llogiq I will add your comment to the unresolved question section. I also think I should add back some form of the text I cut, in particular mentioning that we should probably standardize some form opt out in advance, so it's ready when we need it. I don't think I like the idea of using a crate-wide attribute though -- typically I want the opt-out to be much more targeted (e.g., opt out this particular impl from the coherence rule, but not others).

[nikomatsakis]

I made a few minor amendments: first, I incorporated the suggestion by @llogiq that crates which employ opt out should be compatible with those that don't, and I also added two unresolved questions (What should "opt-out" look like? Can we ever make breaking changes that are not soundness related?). These changes seem uncontroversial (though the answer to the final question may be, but that can be discussed in other venues, such as #1156.

[nikomatsakis]

It's official... the language subteam has decided to accept this RFC.