proc_macro: implement `TokenTree`, `TokenKind`, hygienic `quote!`, and other API #40939

jseyfried · Mar 31, 2017

All new API is gated behind #![feature(proc_macro)] and may be used with #[proc_macro], #[proc_macro_attribute], and #[proc_macro_derive] procedural macros.

More specifically, this PR adds the following in proc_macro:

// `TokenStream` constructors:
impl TokenStream { fn empty() -> TokenStream { ... } }
impl From<TokenTree> for TokenStream { ... }
impl From<TokenKind> for TokenStream { ... }
impl<T: Into<TokenStream>> FromIterator<T> for TokenStream { ... } 
macro quote($($t:tt)*) { ... } // A hygienic `TokenStream` quoter

// `TokenStream` destructuring: 
impl TokenStream { fn is_empty(&self) -> bool { ... } }
impl IntoIterator for TokenStream { type Item = TokenTree; ... }

struct TokenTree { span: Span, kind: TokenKind }
impl From<TokenKind> for TokenTree { ... }
impl Display for TokenTree { ... }

struct Span { ... } // a region of source code along with expansion/hygiene information
impl Default for Span { ... } // a span from the current procedural macro definition
impl Span { fn call_site() -> Span { ... } } // the call site of the current expansion
fn quote_span(span: Span) -> TokenStream;

enum TokenKind {
    Group(Delimiter, TokenStream), // A delimited sequence, e.g. `( ... )`
    Term(Term), // a unicode identifier, lifetime ('a), or underscore
    Op(char, Spacing), // a punctuation character (`+`, `,`, `$`, etc.).
    Literal(Literal), // a literal character (`'a'`), string (`"hello"`), or number (`2.3`)
}

enum Delimiter {
    Parenthesis, // `( ... )`
    Brace, // `[ ... ]`
    Bracket, // `{ ... }`
    None, // an implicit delimiter, e.g. `$var`, where $var is  `...`.
}

struct Term { ... } // An interned string
impl Term {
    fn intern(string: &str) -> Symbol { ... }
    fn as_str(&self) -> &str { ... }
}

enum Spacing {
    Alone, // not immediately followed by another `Op`, e.g. `+` in `+ =`.
    Joint, // immediately followed by another `Op`, e.g. `+` in `+=`
}

struct Literal { ... }
impl Display for Literal { ... }
impl Literal {
    fn integer(n: i128) -> Literal { .. } // unsuffixed integer literal
    fn float(n: f64) -> Literal { .. } // unsuffixed floating point literal
    fn u8(n: u8) -> Literal { ... } // similarly: i8, u16, i16, u32, i32, u64, i64, f32, f64
    fn string(string: &str) -> Literal { ... }
    fn character(ch: char) -> Literal { ... }
    fn byte_string(bytes: &[u8]) -> Literal { ... }
}

For details on quote! hygiene, see this example and declarative macros 2.0.

r? @nrc

jseyfried · Mar 31, 2017

cc @rust-lang/compiler @abonander @dtolnay @Ericson2314 @SergioBenitez @SimonSapin @sgrif @tikue @wycats

eddyb · Mar 31, 2017

Can't we call TokenTree just Token? I'm a bit disappointed in how I fooled myself we could get rid of the "tree" model but flattening seems to not be in the cards.
However, I still think we can keep simpler names in the API.

SimonSapin · Mar 31, 2017

Nice! I like the overall approach, some comments on the details:

impl IntoIterator for TokenStream { type Item = TokenTree; ... }

Would impls for &TokenStream and &mut TokenStream also be useful?

struct Symbol { ... } // An interned string

To me the word "symbol" evokes punctuation, excluding letters. Could this be called something else? Identifier, Ident, Word, … ? (Are the first two not appropriate because "identifier" excludes keywords?)

Joint, // immediately followed by another `Op`, e.g. `+` in `+=`

So the = in += would be a separate token? &&, .., or :: would similarly be two tokens each? (>>= or ... three tokens?) Please expand the doc-comments in the PR to explain this.

impl Display for Literal { ... }
impl Literal {

I’d also add:

fn byte_string(bytes: &[u8]) -> Literal, much more convenient that building a Sequence of u8 literals for a slice literal. (And more efficient?)
Maybe fn byte(byte: u8) -> Literal similar to Literal::u8 but with byte literal syntax, though it only makes a difference when looking at generated code for debugging.
fn as_str(&self) -> Option<&str> and other fn as_*(&self) -> Option<*> methods to extract the value. (Though number literals without a type suffix are tricky.) The Display impl technically provides the same information, but writing a Rust parser (even just for literals) is a complex undertaking.

In src/libproc_macro/lib.rs

#[unstable(feature = "proc_macro", issue = "38356")]
#[macro_export]
macro_rules! quote { () => {} }

In src/libproc_macro/quote.rs

macro_rules! quote {
    () => { TokenStream::empty() };
    ($($t:tt)*) => { [ $( quote_tree!($t), )* ].iter().cloned().collect::<TokenStream>() };
}

Why are there two quote macros, with one empty, and only the empty one exported?

SimonSapin · Mar 31, 2017

I think this PR addresses all of my previous comments in #38356 (comment) 👍

bors · Mar 31, 2017

☔️ The latest upstream changes (presumably #40950) made this pull request unmergeable. Please resolve the merge conflicts.

jseyfried · Apr 1, 2017

@eddyb

I'm a bit disappointed in how I fooled myself we could get rid of the "tree" model but flattening seems to not be in the cards.

Do you mean w.r.t. the internals? I believe we can still use the internal flattened representation you proposed with this API (imo, would be nice but not high priority).

@SimonSapin

Would impls for &TokenStream and &mut TokenStream also be useful?

impl IntoIterator for &TokenStream { type Item = TokenTree; ... } and/or
impl TokenStream { fn iter(&self) -> TokenIter { ... } } could be useful -- both would be equivalent to tokens.clone().into_iter().

impl IntoIterator for &TokenStream { type Item = &TokenTree; ... } isn't possible (yet, perhaps), and would make an internal "flattened" internal representation more complex / less efficient.

TokenStream uses shared, ref-counted memory, so I don't think impl IntoIterator for &mut TokenStream makes sense.

To me the word "symbol" evokes punctuation, excluding letters. Could this be called something else?

Yeah. Currently, Symbol is just an interned string, so Word/Ident don't seem appropriate. If we restrict the type to valid TokenKind::Words, I'd prefer Word for symmetry with TokenKind. Otherwise, perhaps Str/InternedStr? cc @nrc

So the = in += would be a separate token? &&, .., or :: would similarly be two tokens each?

Yeah, it's TokenKind::Op(char, OpKind), so a just single character per Op -- I'll mention that in the doc comment.

I’d also add [more methods for Literal]

Yeah, I agree that we should have Literal::{byte, byte_string, as_*}. This API isn't intended to be exhaustive, e.g. we also want API for emitting "first class" diagnostics (@sergio is working on this).

Why are there two quote macros, with one empty, and only the empty one exported?

The non-empty one is incomplete (by design) and only used to bootstrap the internal implementation of proc_macro::quote. The exported one is a placeholder for the internal implementation (à la lang items). After we support stability hygiene and stability-checking macro resolutions, we should be able to replace the internal implementation with a #[proc_macro].

bors · Apr 5, 2017

☔️ The latest upstream changes (presumably #40811) made this pull request unmergeable. Please resolve the merge conflicts.

nrc · Apr 10, 2017

nrc reviewed Apr 10, 2017

View changes

There's a bunch of changes requested, but most are just about docs, and I don't think there is anything major. Assuming all the fixes are straightforward, then r=me

nrc · Apr 10, 2017

src/libproc_macro/lib.rs

-pub struct TokenStream {
-    inner: TokenStream_,
-}
+pub struct TokenStream(tokenstream::TokenStream);


Not this PR (and pretty nitty), but just pointing out that it should either token_stream or Tokenstream, currently the naming is inconsistent.

nrc · Apr 10, 2017

src/libproc_macro/lib.rs

+
+/// An iterator over `TokenTree`s.
+#[unstable(feature = "proc_macro", issue = "38356")]
+pub struct TokenIter {


Should we call this a TokenTreeIter? That leaves room for a flattening TokenIter and means the name and target correspond

@nrc Yeah, TokenTreeIter makes more sense.

On a similar note, @eddyb proposed using TokenNode instead of TokenKind for better correspondence to TokenTree and to leave room for Token/TokenKind -- what do you think?

nrc · Apr 10, 2017

src/libproc_macro/lib.rs

+}
+
+impl TokenTree {
+    fn from_raw(stream: tokenstream::TokenStream, next: &mut Option<tokenstream::TokenStream>)


I don't like from_raw as a name - 'raw' can mean so many different things, and perhaps the most likely here is as in raw string which is wrong. How about from_internal or from_rustc_internal or something like that?

I'll change to from_internal.

nrc · Apr 10, 2017

src/libproc_macro/lib.rs

+        TokenTree { span: Span(span), kind: kind }
+    }
+
+    fn to_raw(self) -> tokenstream::TokenStream {


likewise here.

And somewhat connected, is there a way to mark the API which we expect macro authors to use? Is it all the pub API? Or is some of that transitional or only intended for the compiler to use?

It's all the pub API not in __internal (which is unusable without #![feature(proc_macro_internals)]).

nrc · Apr 10, 2017

src/libproc_macro/lib.rs

@@ -80,7 +524,11 @@ pub struct LexError {
 /// all of the contents.
 #[unstable(feature = "proc_macro_internals", issue = "27812")]
 #[doc(hidden)]
+#[path = ""]


what is this for? I'm not sure what the effect is

mod quote; in __internal; this allows the file to be libproc_macro/quote.rs. Thinking about this some more, I'll move mod quote into the crate root and remove the #[path = ""].

nrc · Apr 10, 2017

src/libproc_macro/quote.rs

+
+//! # Quasiquoter
+//! This file contains the implementation internals of the quasiquoter provided by `qquote!`.
+


Could you add an overview of how the quasi-quoter works please?

nrc · Apr 10, 2017

src/libsyntax/tokenstream.rs

@@ -196,6 +201,10 @@ impl TokenStream {
        }
    }

+    pub fn builder() -> TokenStreamBuilder {


I think this would be better as a free function rather than a method on TokenStream

Or it could be a new method on TokenStreamBuilder

I'll change to TokenStreamBuilder::new() (motivation for TokenStream::builder() was to avoid making people use anything else into their scopes).

nrc · Apr 10, 2017

src/libsyntax/tokenstream.rs

@@ -225,13 +234,107 @@ impl TokenStream {
        }
        true
    }
+
+    pub fn as_tree(self) -> (TokenTree, bool /* joint? */) {


Could these public methods get docs please? (In particular here, some more explanation of what joint means would be good).

Also, since this takes self by value, should this be named into_tree instead? Or are the conventions different for reference counted things

@nrc will do.
@lfairy Perhaps, I don't think it matters much.

nrc · Apr 10, 2017

src/test/run-pass-fulldeps/auxiliary/proc_macro_def.rs

@@ -8,47 +8,37 @@
 // option. This file may not be copied, modified, or distributed
 // except according to those terms.

-#![feature(plugin, plugin_registrar, rustc_private)]


It looks like anyone with an existing procedural macro is going to have to make some significant changes. Could you write up either here or on internals or somewhere, a guide to the changes needed to make a proc macro work after this PR please?

Existing procedural macros will lose the syntax::tokenstream::TokenStream quoter from proc_macro_internals, AFAIK other than that they won't have to make any changes. I could include a syntax::tokenstream::TokenStream quoter for back-compat.

How many people are actually using "existing procedural macros" though? This PR doesn't effect the widely used legacy plugin system; I didn't think many people were using SyntaxExtension::ProcMacro from #36154.

nrc · Apr 10, 2017

src/libsyntax/parse/token.rs

@@ -455,3 +461,38 @@ pub fn is_op(tok: &Token) -> bool {
        _ => true,
    }
 }
+
+#[derive(Clone, Eq, PartialEq, Debug)]
+pub struct LazyTokenStream(RefCell<Option<TokenStream>>);


Is this meant to be used by macro authors? If so could you document with why and when.

No, nothing in libsyntax can be used by proc macro authors.

lfairy · Apr 10, 2017

lfairy reviewed Apr 10, 2017

View changes

lfairy · Apr 10, 2017

src/libsyntax/parse/token.rs

+}
+
+#[derive(Clone, Eq, PartialEq, Debug)]
+pub struct LazyTokenStream(RefCell<Option<TokenStream>>);


As an aside: if the dynamic borrow checks end up too inefficient, then this could be re-implemented on top of MoveCell.

std::cell::Cell is now like MoveCell :) rust-lang/rfcs#1651

lfairy · Apr 10, 2017

lfairy reviewed Apr 10, 2017

View changes

lfairy · Apr 10, 2017

src/libsyntax/parse/token.rs

+        let mut opt_stream = self.0.borrow_mut();
+        if opt_stream.is_none() {
+            *opt_stream = Some(f());
+        };


Nit: extra semicolon

carols10cents · Apr 17, 2017

Friendly ping that there are merge conflicts and some comments for you @jseyfried! ❤️

abonander · Apr 23, 2017

Should we use a different feature gate for the unstable APIs? I'm under the impression that language feature gates and library feature gates are meant to be distinct from each other, and proc_macro currently falls under the language side.

nikomatsakis · Apr 24, 2017

Hmm, is there a strict separation between language/lib feature-gates? I wouldn't think so.

abonander · Apr 24, 2017

It's kind of implied by the structure of the Unstable Book but that might be more of an issue for #41476 where this came up originally.

arielb1 · Apr 25, 2017

status: still waiting for #40847.

arielb1 · May 2, 2017

status: still waiting for #40847.

carols10cents · May 8, 2017

status: still waiting for #40847.

jseyfried · Jul 6, 2017

@alexcrichton Awesome, thanks!

For the record, the failing test was because tokens from a macros 1.1-style proc-macro (i.e. via FromStr for TokenStream) now are given the proc-macro's call site span, as opposed to the proc-macros's call site span with extra expansion information before this PR.

Since we no longer have the extra expansion information, @nrc's save-analysis code tries to re-parses the call site span and expects it to match the proc-macro generated tokens.

I had to get rid of the expansion information for backwards compatibility -- if the expansion information showed that the span came from a procedural macro 2.0, then the underlying tokens would resolve hygienically (as per hygiene 2.0). This would be a backwards-incompatible change, and would mean that stringifying, reparsing and returning the input of a proc-macro would change its semantics, even if the input were entirely unexpanded.

If we want to fix this, we could instead mark tokens from FromStr for TokenStream specially so that save-analysis would know that they are "weird", but so that they would still behave like unexpanded tokens w.r.t. hygiene. I don't have a strong opinion here.

oli-obk · Jul 9, 2017

The lack of expansion info broke clippy together with proc macros, since code inside derives will now trigger lints. Is there any new way to detect expanded code? If not, what needs to be done in order to create expansion info that doesn't influence save analysis.

See rust-lang/rust-clippy#1877

nrc · Jul 13, 2017

Since we no longer have the extra expansion information, @nrc's save-analysis code tries to re-parses the call site span and expects it to match the proc-macro generated tokens.

Ah, I wondered why I was seeing so many errors in the RLS :-(

I think we must fix this. This feels like another reason not to conflate macro hygiene with expansion history. That might be too hard a way to fix this though.

If we want to fix this, we could instead mark tokens from FromStr for TokenStream specially so that save-analysis would know that they are "weird", but so that they would still behave like unexpanded tokens w.r.t. hygiene. I don't have a strong opinion here.

Not using spans for hygiene seems a much nicer fix here - the span should reflect the expansion history and the hygiene should be altered (to erase hygiene, essentially). I expect we want the correct expansion info for errors as well as save-analysis.

oli-obk · Jul 13, 2017

I already have a fix in #43179

jseyfried · Jul 13, 2017

@nrc
We could instead just include hygiene bending information along with expansion information so that span expansion info and hygiene info remain unified (@oli-obk's solution is fine for impl FromStr for TokenStream, so we're OK for now).

That is, I don't think think hygiene bending requires us to split up span/hygiene info or construct an "fake" expansion history just for hygiene. I think it'd be nicer to have a single source of truth for expansion info that is expressive enough for hygiene bending.

nrc · Jul 25, 2017

So #43179 did not fix the expansion info issue (at least not completely) - I'm still seeing errors in the RLS where we are treating macro-generated names as hand-written ones (and I think this is causing #43371 too).

rust-highfive assigned nrc Mar 31, 2017

jseyfried force-pushed the jseyfried:proc_macro_api branch 2 times, most recently from 30f3956 to 40f81aa Mar 31, 2017

This was referenced Mar 31, 2017

Merged

macros: improve `Span`'s expansion information #40597

Closed

Tracking issue for RFC 1566: Procedural macros #38356

jseyfried referenced this pull request Mar 31, 2017
Merged
Initial implementation of declarative macros 2.0 #40847

jseyfried force-pushed the jseyfried:proc_macro_api branch from 9999e58 to 6151672 Apr 3, 2017

Mark-Simulacrum added the S-waiting-on-author label Apr 14, 2017

lfairy referenced this pull request Apr 22, 2017
Closed
Build broken since rustc nightly-2017-03-04 #80

abonander referenced this pull request Apr 23, 2017
Merged
Document the `proc_macro` feature in the Unstable Book #41476

jseyfried referenced this pull request Apr 24, 2017
Closed
warning: Quasi-quoting might make incremental compilation very inefficient: NtIdent(..) #40946

abonander referenced this pull request Apr 30, 2017
Open
Improve diagnostics for parse errors in proc macro output #41663

dtolnay referenced this pull request May 10, 2017
Closed
Misleading "not found in this scope" error #923

mystor referenced this pull request Jul 6, 2017
Closed
Actually test the `proc_macro` integration #3

kennytm referenced this pull request Jul 7, 2017
Merged
Only match a fragment specifier if it starts with certain tokens. #42913

lfairy referenced this pull request Jul 7, 2017
Closed
Port to new proc-macro interface #92

alexcrichton referenced this pull request Jul 7, 2017
Closed
What is proc-macro2 #1005

lfairy referenced this pull request Jul 16, 2017
Merged
Enable proc_macro support on Travis #4

Mark-Simulacrum referenced this pull request Jul 18, 2017
Closed
Change in behavior of `include!` in doctests on nightly #43153

jdm referenced this pull request Jul 20, 2017
Closed
Source is missing from docs for crates that use a procedural macro #43371

nrc referenced this pull request Jul 29, 2017
Merged
Three small fixes for save-analysis #43533

aidanhs referenced this pull request Aug 10, 2017
Closed
Mis-calculated spans #43796

dtolnay referenced this pull request May 10, 2018
Merged
Review proc macro API 1.2 #50473

petrochenkov referenced this pull request May 20, 2018
Closed
TokenStream::parse does not resolve with Span::call_site() #50050

nnethercote referenced this pull request Feb 15, 2019
Merged
Remove `LazyTokenStream`. #58476

rust-lang/rust

Join GitHub today

proc_macro: implement `TokenTree`, `TokenKind`, hygienic `quote!`, and other API #40939

Conversation

jseyfried commented Mar 31, 2017 • edited

rust-highfive assigned nrc Mar 31, 2017

jseyfried force-pushed the jseyfried:proc_macro_api branch 2 times, most recently from 30f3956 to 40f81aa Mar 31, 2017

This was referenced Mar 31, 2017

This comment has been minimized.

jseyfried commented Mar 31, 2017

jseyfried referenced this pull request Mar 31, 2017

This comment has been minimized.

eddyb commented Mar 31, 2017

This comment has been minimized.

SimonSapin commented Mar 31, 2017

This comment has been minimized.

SimonSapin commented Mar 31, 2017

This comment has been minimized.

bors commented Mar 31, 2017

This comment has been minimized.

jseyfried commented Apr 1, 2017

jseyfried force-pushed the jseyfried:proc_macro_api branch from 9999e58 to 6151672 Apr 3, 2017

This comment has been minimized.

bors commented Apr 5, 2017

nrc reviewed Apr 10, 2017 View changes

nrc left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

lfairy reviewed Apr 10, 2017 View changes

This comment has been minimized.

This comment has been minimized.

lfairy reviewed Apr 10, 2017 View changes

This comment has been minimized.

Mark-Simulacrum added the S-waiting-on-author label Apr 14, 2017

This comment has been minimized.

carols10cents commented Apr 17, 2017

lfairy referenced this pull request Apr 22, 2017

This comment has been minimized.

abonander commented Apr 23, 2017 • edited

abonander referenced this pull request Apr 23, 2017

This comment has been minimized.

nikomatsakis commented Apr 24, 2017

This comment has been minimized.

abonander commented Apr 24, 2017

jseyfried referenced this pull request Apr 24, 2017

This comment has been minimized.

arielb1 commented Apr 25, 2017

abonander referenced this pull request Apr 30, 2017

This comment has been minimized.

arielb1 commented May 2, 2017

This comment has been minimized.

carols10cents commented May 8, 2017

dtolnay referenced this pull request May 10, 2017

This comment has been minimized.

jseyfried commented Jul 6, 2017

mystor referenced this pull request Jul 6, 2017

kennytm referenced this pull request Jul 7, 2017

lfairy referenced this pull request Jul 7, 2017

alexcrichton referenced this pull request Jul 7, 2017

This comment has been minimized.

oli-obk commented Jul 9, 2017 • edited

jseyfried commented Mar 31, 2017 •

edited

jseyfried force-pushed the jseyfried:proc_macro_api branch 2 times, most recently from `30f3956` to `40f81aa` Mar 31, 2017

jseyfried force-pushed the jseyfried:proc_macro_api branch from `9999e58` to `6151672` Apr 3, 2017

nrc reviewed Apr 10, 2017

View changes

lfairy reviewed Apr 10, 2017

View changes

lfairy reviewed Apr 10, 2017

View changes

abonander commented Apr 23, 2017 •

edited

oli-obk commented Jul 9, 2017 •

edited

jseyfried commented Jul 13, 2017 •

edited