Review proc macro API 1.2

[petrochenkov]

cc #38356

Summary of applied changes:

• Documentation for proc macro API 1.2 is expanded.
• Renamed APIs: Term -> Ident, TokenTree::Term -> TokenTree::Ident, Op -> Punct, TokenTree::Op -> TokenTree::Punct, Op::op -> Punct::as_char.
• Removed APIs: Ident::as_str, use Display impl for Ident instead.
• New APIs (not stabilized in 1.2): Ident::new_raw for creating a raw identifier (I'm not sure new_x it's a very idiomatic name though).
• Runtime changes:
  • Punct::new now ensures that the input char is a valid punctuation character in Rust.
  • Ident::new ensures that the input str is a valid identifier in Rust.
  • Lifetimes in proc macros are now represented as two joint tokens - Punct('\'', Spacing::Joint) and Ident("lifetime_name_without_quote") similarly to multi-character operators.
• Stabilized APIs: None yet.

A bit of motivation for renaming (although it was already stated in the review comments):

• With my compiler frontend glasses on Ident is the single most appropriate name for this thing, especially if we are doing input validation on construction. TokenTree::Ident effectively wraps token::Ident or ast::Ident + is_raw, its meaning is "identifier" and it's already named ident in declarative macros.
• Regarding Punct, the motivation is that Op is actively misleading. The thing doesn't mean an operator, it's neither a subset of operators (there is non-operator punctuation in the language), nor superset (operators can be multicharacter while this thing is always a single character). So I named it Punct (first proposed in the original RFC, then by @SimonSapin) , together with input validation it's now a subset of ASCII punctuation character category (u8::is_ascii_punctuation).

[rust-highfive]

r? @alexcrichton

(rust_highfive has picked a reviewer for you, use r? to override)

[petrochenkov]

@dtolnay @alexcrichton @eddyb @nrc
Could you go through the comments marked with "REVIEW" and tell what you think?

[bors]

☔️ The latest upstream changes (presumably #50453) made this pull request unmergeable. Please resolve the merge conflicts.

[eddyb]

Also, Word is now Term, right?

[eddyb]

Oh I didn't expect these changes. I should leave most of this up to @alexcrichton (I do not have a strong preference either way).

[eddyb]

Validation seems really nice, I think previously it's been deferred until conversion to the compiler-internal forms is needed.

[eddyb]

I can confirm that this would not cause issues with #49219.

[eddyb]

Can this become unreachable!() now? As only a finite set of characters can be used.

[eddyb]

If we do keep it Copy, that means we can never soundly offer an &self -> &str API. See #38356 (comment).

[dtolnay]

I personally don't mind if we remove the Copy here.

[alexcrichton]

All the changes here look great to me, thanks @petrochenkov! While we've already FCP'd for stability in #38356 I think the Ident name is probably better than Term and I'm personally ambivalent on Op or Punct, but am happy to go with Punct.

I'd prefer to land this all at once if possible to reduce churn in the ecosystem, so I'd be mostly in favor of going ahead and stabilizing everything in this PR (if you're ok with that). I don't mind doing the work myself for the APIs here and pushing those up (annotation-wise). Along those lines, want to remove as_str as well from this?

[alexcrichton]

I think this doesn't match the original implementation, when we first started we were sure to return an error I thought...

In any case I'd think it's a bug that this panics, it should definitely be returning an error (but we can fix that later)

[alexcrichton]

Yeah this is currently the main way to go from a TokenTree to a TokenStream, and I think we also accidentally stabilized it already so we're required to keep it.

[alexcrichton]

Yeah I think we're likely to always want Span to be Copy

[alexcrichton]

I suspect impls like these will grow and come to exist over time yeah, that's definitely the direction that libstd has taken.

[alexcrichton]

I think this is less critical than Span to be copy, and wouldn't mind going either way here. Making this not-Copy is probably the most conservative thing to do

[dtolnay]

I agree, let's go with no Copy.

[alexcrichton]

Yes let's stick to char for maximal future flexibility.

[alexcrichton]

Yes for future flexibility I think char's the way to go here

[dtolnay]

The reason we decided against TokenTree::Ident was because the matcher $i:ident does not accept lifetimes. I would be opposed to calling it Ident here if it can hold lifetimes and $i:ident cannot.

[dtolnay]

The reason we decided against TokenTree::Ident was because the matcher $i:ident does not accept lifetimes. I would be opposed to calling it Ident here if it can hold lifetimes and $i:ident cannot.

[petrochenkov]

@dtolnay
I'd be totally okay with accepting lifetimes in $i: ident (with or without dropping $l: lifetime) if we are ready to cement lifetimes as a "lexical class inside identifiers" as opposed to "' plus an identifier" (which proc macro API already does).
There's internals thread about this https://internals.rust-lang.org/t/pre-rfc-splitting-lifetime-into-two-tokens/6716/10. I don't think the original suggestion is a good idea anymore, but see https://internals.rust-lang.org/t/pre-rfc-splitting-lifetime-into-two-tokens/6716/11 in particular about the "lexical class inside identifiers" alternative.

The key questions here are probably:

• Is it important to discern between identifiers and lifetimes in procedural macros? Why?
  • We want to minimize API surface. It's always possible to detect lifetimes using the first character.
• Is it important to discern between identifiers and lifetimes in declarative macros? Why?
  • I'd personally want to minimize the language surface (and simplify compiler as well if we drop $l: lifetime). It's possible to detect lifetimes in macros left sides if we keep lifetime as a subset of ident (kinda like path is a subset of ty/expr/pat) or allow lifetime idents in macro "variables" ($'var).

[petrochenkov]

@alexcrichton

I'd prefer to land this all at once if possible to reduce churn in the ecosystem, so I'd be mostly in favor of going ahead and stabilizing everything in this PR (if you're ok with that). I don't mind doing the work myself for the APIs here and pushing those up (annotation-wise). Along those lines, want to remove as_str as well from this?

Sure, I'll add stable annotations and remove as_str.

[eddyb]

Is it important to discern between identifiers and lifetimes in declarative macros? Why?

Probably to parse anything looking like a list of generics, if you want to tell apart 'a and T.

[dtolnay]

It's possible to detect lifetimes in macros left sides if we keep lifetime as a subset of ident

I would need to play with this in nightly before I would feel comfortable committing to it. What I wouldn't want is:

• We stabilize TokenTree::Ident under the expectation that we will make $:ident accept lifetimes.
• We change $:ident in decl macros to accept lifetimes, while keeping $:lifetime.
• After some use we discover this is worse this way than having $:ident and $:lifetime disjoint.
• We revert $:ident to reject lifetimes.
• Now proc macro and decl macro are inconsistent in an obnoxious way.

[dtolnay]

Probably to parse anything looking like a list of generics, if you want to tell apart 'a and T.

Hmm, can we imagine any macros that would need to handle lifetime parameters and type parameters differently? There is some appeal in being able to say the distinction never matters until much later in the compile process.

[dtolnay]

One place I can imagine running into trouble is if a macro somehow needs to handle optional lifetimes on a reference.

macro_rules! m {
    (& $($p:lifetime)? $i:ident) => {}
}

m!(&foo);
m!(&'a foo);

error: local ambiguity: multiple parsing options: built-in NTs lifetime ('p') or ident ('i').
 --> src/main.rs:4:5
  |
4 | m!(&foo);
  |     ^^^

[petrochenkov]

@dtolnay

local ambiguity: multiple parsing options: built-in NTs lifetime ('p') or ident ('i').

This is a weird error. I never investigated how our macro matching works in detail, but I expected it to be modeled after regular expressions (greedy by default) and work in both cases, whether lifetime is a subset of ident or not.

extern crate regex;
use regex::Regex;

fn main() {
    let not_subset = Regex::new(r"a?b").unwrap();
    assert!(not_subset.is_match("ab"));
    assert!(not_subset.is_match("b"));
    let subset = Regex::new(r"a?\w").unwrap();
    assert!(subset.is_match("ab"));
    assert!(subset.is_match("aa"));
    assert!(subset.is_match("b"));
    assert!(subset.is_match("a"));
}

http://play.rust-lang.org/?gist=372d5a5f25219670dabc89a1dfb536ba&version=nightly&mode=debug

[petrochenkov]

We have 6 weeks if we want to stabilize the API in 1.28.

@eddyb
Can the lifetime question get a slot in the next lang team meeting?

• Q1. Treating lifetimes as a lexical subset of identifiers (identifier = '? XID_Start XID_Continue*), closing https://internals.rust-lang.org/t/pre-rfc-splitting-lifetime-into-two-tokens/6716/11. Procedural macros currently treat lifetimes like this.
• Q2. Accepting lifetimes in $i: ident matcher in declarative macros, delaying stabilization of $l: lifetime (#34303) until a good use case not covered by ident is found.
• Q3. How to refer to both identifiers and lifetime identifiers if they are kept separate. Is "term" previously used by proc macro API a good name?

[petrochenkov]

@bors r=alexcrichton

Stabilization commit for later cherry-picking: petrochenkov@3e71e49

[bors]

📌 Commit dab8c0a has been approved by alexcrichton

[kennytm]

@bors p=1

Unblocks #49219

[bors]

⌛️ Testing commit dab8c0a with merge 2a3f536...

[bors]

☀️ Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing 2a3f536 to master...