Add the experimental_keywords ability

[Lokathor]

rendered

I rushed a little bit, so we probably need extra words towards the end for the drawbacks / alternatives / etc., but it should overall be ready for serious feedback.

[joshtriplett]

Thank you for writing this up, @Lokathor!

[scottmcm]

I think this needs a justification for $ in particular. Why is r#$ better than r#@, say? (Not that I think that one's necessarily superior, just that it's not clearly worse either.)

Historically macros have zealously guarded the use of $ for themselves, and it's not impossible to imagine someone trying to do ($mystring:literal) => { r#$mystring }...

[Lokathor]

Speaking as the RFC author: the current text simply reflects the "first thing that came to mind" during the discussion that lead to this proposal, with not much bike-shedding during that discussion.

Speaking as a user: either way is fine and I have no opinion.

So I'll happily put in whatever people want to decide on for this.

[kennytm]

r#$mystring doesn't work today, and I don't think it will ever work in the future. You can't just stick a r# next to a lexed token to turn it into a raw string. Macro's use of $ shouldn't really matter here.

[Pauan]

@kennytm Sure, but macro authors use $ so commonly that it can cause confusion (they might think that r#$foo can work). So why not reduce that confusion by choosing a different character?

[kennytm]

well for sure there are plenty of alternatives we could choose.

spellings which currently are tokenization error:

• r##keyword
• r#[keyword]
• r#@keyword / r#!keyword / etc
• r#805209
• br#haha_lang_design_go

[cuviper]

FWIW, br#keyword was also mentioned in #2151.

[petrochenkov]

The only question I have while reading this thread is "why?".
Why come up with these contrived monstrosities messing with lexer if pretty much any combination of two existing keywords would work (do catch style)?

If you want a generic tool for this (which is, again, excessive in my opinion), make __experimental a keyword and use __experimental other_keyword as the combination.

[comex]

The Ember framework for javascript places experimental framework abilities into a special module that is clearly experimental so that users can try new things and become familiar with new features even before the feature is fully stable.

Isn't that just the same idea as #[feature]? As in, experimental features could be changed or dropped before being stabilized, and that's why they have to be opted into. (I could be wrong, ,as I'm only going off your description and a quick Google; I haven't used Ember myself.)

That's pretty different from stabilizing an experimental form of a feature, which is what you're proposing here.

Really, this RFC seems like it's asking for a much broader shift in direction than a mere change in keyword policy. Has it ever actually happened that we're ready to commit to the exact final design of a feature, but need to wait for the next edition to give it its final name? None of the features you mentioned satisfy that condition:

• do catch: The final syntax (try) has been reserved since the 2018 edition, but the final semantics are still undecided, specifically whether implicit Ok-wrapping should occur.
• raw references: There's been some discussion about whether, instead of having the syntax return existing raw pointers, we should have a brand new raw pointer type instead.
• try!: ? is more general than try!, with the Try trait allowing non-Result types to use the operator. This is almost a backwards-compatible extension, but not quite, as it weakens type inference.
• asm: "Ugly keyword that exists solely to have a proc macro expand to it" is a different use case that's not related to experimentation.

So I guess what you're really asking for is openness to stabilizing experimental designs for features, to be maintained indefinitely in parallel with the final design, which might be slightly different.

Which sounds great to me. I think Rust has worked itself into a corner with the experiments-only-on-nightly policy. (But then I'm not the one who'd be maintaining old features in the compiler.)

Still, if we're going to move in that direction, the RFC should be more clear about what it's proposing.

For example: would stabilizing experimental keywords be appropriate:
...even when we already have a final keyword waiting, but we're unsure about the exact final semantics? Example: try blocks.
...even when the final syntax would not be a breaking change at all? Example: async fn before it was stabilized.

I think the answer should be yes to both, since such features are not less deserving of experimentation than ones with keyword availability issues. But I'm not sure what the RFC intends.

[Lokathor]

To be absolutely clear, and perhaps this needs big bold text in the RFC, actually ever stabilizing an experimental keyword is on a feature by feature basis only.

I'm writing this as part of a proposal from earlier today, so others can have their own opinions on this, but to me the primary benefit is to put off syntax bikeshed until after the feature is in nightly and able to be used, instead of having to settle it before the rfc for the feature can even be accepted to even start the work.

[RalfJung]

raw references: There's been some discussion about whether, instead of having the syntax return existing raw pointers, we should have a brand new raw pointer type instead.

I don't think there is any doubt that there should be a way to turn places into raw pointers without going through a reference.

There is doubt whether &raw is the right syntax for that, and/or whether that syntax maybe should return a different, yet-to-be-introduced kind of unsafe pointer -- but that makes raw references fall exactly into the motivation for this RFC: an operation with clear semantics and clear need, but no good syntax, and the one "okay" syntax we have might be better used for another operation that we don't know any details of yet.

[Ixrec]

Honestly, I'm extremely confused as to what's even being proposed here.

to me the primary benefit is to put off syntax bikeshed until after the feature is in nightly and able to be used, instead of having to settle it before the rfc for the feature can even be accepted to even start the work.

Isn't this exactly what already happens for... well, every feature with syntax? Because that's just what "nightly" is? This is exactly what async/await did, right? (in fact, I can't think of a single feature besides await! where we knew the exact behavior before we committed to a syntax) How does a new kind of stable syntax affect this in any way?

[RalfJung]

(in fact, I can't think of a single feature besides await! where we knew the exact behavior before we committed to a syntax)

Another example mentioned in the RFC is rust-lang/rust#64490.

[comex]

There is doubt whether &raw is the right syntax for that, and/or whether that syntax maybe should return a different, yet-to-be-introduced kind of unsafe pointer -- but that makes raw references fall exactly into the motivation for this RFC: an operation with clear semantics and clear need, but no good syntax, and the one "okay" syntax we have might be better used for another operation that we don't know any details of yet.

This is just a difference in terminology.

You're describing the experimental-keyword iteration and the future final iteration as different "operations"; I'd call them two versions of the same feature, or in the terminology of the RFC, two "forms" of an "ability".

Regardless, my point is that we would be stabilizing one thing with the intent to quickly deprecate it in favor of another thing, which wouldn't just have different syntax but also possibly different semantics. If it did have different semantics, the compiler would have to support two sets of semantics forever.

In the case of &raw, the risk of having to do that might be pretty easy to justify. I'd expect the implementation burden of maintaining another variant of the & operator to be pretty small. And unlike most features, &raw is not just a nice-to-have, but sort of a necessary soundness fix.

But for other features it might be harder to justify.

Again, I'm not against this RFC! I'm just saying it's more than an issue of syntax. Talking about keywords makes it sound like the long-term implementation burden is just having the lexer support two names for the same keyword. The reality is a risk of something more than that.

[RalfJung]

You're describing the experimental-keyword iteration and the future final iteration as different "operations"; I'd call them two versions of the same feature, or in the terminology of the RFC, two "forms" of an "ability".

Raw pointers are not going to go away no matter what other pointer types we add, so having &raw makes total sense even if we later also get &unsafe. The feature at hand, turning a place into a raw pointer, is entirely unaffected by adding new pointer types that we might also want to be able to turn places into. Arguably we should have such an operation for every pointer type we have.

If we end up also getting another pointer type, then the address-of operator for that, while solving similar problems to &raw, is semantically speaking (in terms of language spec, Miri implementation, etc) an entirely independent operation. Just because two different features solve the same problem doesn't make them "two versions of the same feature".

[petrochenkov]

So r#$keyword is not a new kind of token, even if it looks like one, but just a sequence of existing tokens r # $ keyword.

This resolves the primary concern I had when looking at this RFC for the first time.
(I didn't look into any possible ambiguity issues caused by these tokens looking too similar to raw literals or other constructions, but they may exist. It would be nice to select a sequence that doesn't affect lexer at all.)

Note that it turns r into a context-sensitive keyword, something the lang team was against of, the last time it was discussed.

---

I still don't see much motivation for this though.
If this is a generic tool for problems like expansion of things like asm! into some tokens (this is what was discussed recently), then it's strange to have generic tools for problems that arise ~once in five years.

[kennytm]

@petrochenkov No r#$keyword is a new kind of token, and that's the whole point why this RFC chooses such a funny sequence. If r#$keyword and r # $ keyword are equivalent it may break existing macros (however unlikely). I don't see how this is a concern.

macro_rules! check {
    ($($x:tt)*) => {}
}

check!(r # $ abc);
check!(r#$abc); // this line errors (expected)

[Lokathor]

huh, i honestly thought that it was a sequence of tokens too, just an invalid one. that's why i spaced it out in one spot.

So that's worth writing down either way once we agree on what's what.

[kennytm]

If a sequence is acceptable we would have used k#keyword 🤷

[petrochenkov]

Introduction of new token kinds will require extending proc macro APIs as well, which makes it even more of an overkill for something that I'm not sure is a problem at all.

[comex]

Raw pointers are not going to go away no matter what other pointer types we add, so having &raw makes total sense even if we later also get &unsafe. The feature at hand, turning a place into a raw pointer, is entirely unaffected by adding new pointer types that we might also want to be able to turn places into. Arguably we should have such an operation for every pointer type we have.

Wait, so assume we hypothetically stabilized &r#$raw to create "old" raw pointers, and then hypothetically added and stabilized a "new" raw pointer type with an accompanying operator.

Are you envisioning that we would then move to give the old-raw-pointer operator its own permanent name? In which case, the purpose of stabilizing it as &r#$raw instead of &raw would just be to leave open the option of using the syntax &raw for "new" raw pointers (and give the old-raw-pointer operator some other name)?

Or are you envisioning that the old-raw-pointer operator would stay as &r#$raw forever?

[RalfJung]

Or are you envisioning that the old-raw-pointer operator would stay as &r#$raw forever?

I am not sure, honestly. I feel it could go either way, depending on other aspects of the new raw/unsafe pointer type. If it is truly a replacement for raw pointers and we'd basically deprecated *const and *mut, then I imagine we wouldn't really need pretty syntax for creating those deprecated pointer types.

[Diggsey]

It would also be possible to do this with an attribute: make the new keywords contextual keywords based on the presence of that attribute.

eg.

#[keyword(feature_name = foo)]
mod a {
    // foo is a keyword
}

mod b {
    // foo is not a keyword
}

Then if the keyword changes, the attribute can be updated to remap the keyword appropriately so the code continues to work.

[kennytm]

@Diggsey This proposal is supposed to be usable on Stable (no #![feature] required). So for your proposal, the #[keyword] attribute must be available on stable as well, and this means the keyword name (I expect this to be a plain name, and less ugly than r#$abcdef) chosen by that feature is forever stabilized inside the #[keyword] mod. This doesn't feel correct for me.

[Diggsey]

@kennytm the keyword name would be whatever was specified in the attribute. You could think of it as being a procedural macro that just substitutes all instances of "foo" (because we used feature_name = foo) with the "real" keyword (which initially is unnameable, but might eventually get a stable name).

The only thing that would be stabilised is the name of the feature itself, although there's nothing stopping us from adding additional names for it.

[kennytm]

I see.

But let's say we stabilized feature_name to use the keyword foo in a new edition. Inside a #[keyword(feature_name = bar)], what happens if we use foo? (keyword? plain identifier? compilation error due because feature_name is stable in this edition?)

Can we nest #[keyword] blocks? Can macros generate #[keyword] blocks? Does #[keyword] affect macro invocations?

macro_rules! foo1 {
    () => { fn bar() {} }
}
#[keyword(feature_name = bar)]
mod a {
    foo1!(); // ?
    pub macro_rules! foo2() {
         () => { /* code which involve the keyword */ bar; }
    }
}
a::foo2!(); // ?

[Diggsey]

@kennytm there are certainly several questions to answer, but of the ones you posed it doesn't seem to really matter what answers we choose as long as it's consistent.

I don't see any back-compat problems because if a new keyword is indeed stabilised it will necessitate a new edition anyway: we already have the problem of macros being used across editions so we can go ahead and used the exact same rules here in terms of macro hygiene.

[kornelski]

Experience with vendor prefixes shows that this doesn't work, because:

• experimental features may change semantics, but this scheme doesn't have a natural way of versioning or detecting a compatible version of the feature.

• the cost of breaking users and the cost of migration from experimental version to stable version is a disincentive for experimentation and iteration on new features, especially if there could be subtle breakage if someone just find'n'replaced r#$ with nothing.

• If there are enough users of an experimental feature, it won't matter that it was called experimental. It will become de-facto stable and necessary: Hyrum's Law

[Lokathor]

Closing this since it's gone stagnant.

If people want to continue the idea later they should open an issue for a major change proposal.