Skip to content

Commit

Permalink
proc_macro: Reorganize public API
Browse files Browse the repository at this point in the history
This commit is a reorganization of the `proc_macro` crate's public user-facing
API. This is the result of a number of discussions at the recent Rust All-Hands
where we're hoping to get the `proc_macro` crate into ship shape for
stabilization of a subset of its functionality in the Rust 2018 release.

The reorganization here is motivated by experiences from the `proc-macro2`,
`quote`, and `syn` crates on crates.io (and other crates which depend on them).
The main focus is future flexibility along with making a few more operations
consistent and/or fixing bugs. A summary of the changes made from today's
`proc_macro` API is:

* The `TokenNode` enum has been removed and the public fields of `TokenTree`
  have also been removed. Instead the `TokenTree` type is now a public enum
  (what `TokenNode` was) and each variant is an opaque struct which internally
  contains `Span` information. This makes the various tokens a bit more
  consistent, require fewer wrappers, and otherwise provides good
  future-compatibility as opaque structs are easy to modify later on.

* `Literal` integer constructors have been expanded to be unambiguous as to what
  they're doing and also allow for more future flexibility. Previously
  constructors like `Literal::float` and `Literal::integer` were used to create
  unsuffixed literals and the concrete methods like `Literal::i32` would create
  a suffixed token. This wasn't immediately clear to all users (the
  suffixed/unsuffixed aspect) and having *one* constructor for unsuffixed
  literals required us to pick a largest type which may not always be true. To
  fix these issues all constructors are now of the form
  `Literal::i32_unsuffixed` or `Literal::i32_suffixed` (for all integral types).
  This should allow future compatibility as well as being immediately clear
  what's suffixed and what isn't.

* Each variant of `TokenTree` internally contains a `Span` which can also be
  configured via `set_span`. For example `Literal` and `Term` now both
  internally contain a `Span` rather than having it stored in an auxiliary
  location.

* Constructors of all tokens are called `new` now (aka `Term::intern` is gone)
  and most do not take spans. Manufactured tokens typically don't have a fresh
  span to go with them and the span is purely used for error-reporting
  **except** the span for `Term`, which currently affects hygiene. The default
  spans for all these constructed tokens is `Span::call_site()` for now.

  The `Term` type's constructor explicitly requires passing in a `Span` to
  provide future-proofing against possible hygiene changes. It's intended that a
  first pass of stabilization will likely only stabilize `Span::call_site()`
  which is an explicit opt-in for "I would like no hygiene here please". The
  intention here is to make this explicit in procedural macros to be
  forwards-compatible with a hygiene-specifying solution.

* Some of the conversions for `TokenStream` have been simplified a little.

* The `TokenTreeIter` iterator was renamed to `token_stream::IntoIter`.

Overall the hope is that this is the "final pass" at the API of `TokenStream`
and most of `TokenTree` before stabilization. Explicitly left out here is any
changes to `Span`'s API which will likely need to be re-evaluated before
stabilization.

All changes in this PR have already been reflected to the [`proc-macro2`],
`quote`, and `syn` crates. New versions of all these crates have also been
published to crates.io.

Once this lands in nightly I plan on making an internals post again summarizing
the changes made here and also calling on all macro authors to give the APIs a
spin and see how they work. Hopefully pending no major issues we can then have
an FCP to stabilize later this cycle!

[`proc-macro2`]: https://docs.rs/proc-macro2/0.3.1/proc_macro2/
  • Loading branch information
alexcrichton committed Apr 2, 2018
1 parent 097efa9 commit 553c04d
Show file tree
Hide file tree
Showing 10 changed files with 726 additions and 314 deletions.
747 changes: 562 additions & 185 deletions src/libproc_macro/lib.rs

Large diffs are not rendered by default.

138 changes: 83 additions & 55 deletions src/libproc_macro/quote.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
//! This quasiquoter uses macros 2.0 hygiene to reliably access
//! items from `proc_macro`, to build a `proc_macro::TokenStream`.

use {Delimiter, Literal, Spacing, Span, Term, TokenNode, TokenStream, TokenTree};
use {Delimiter, Literal, Spacing, Span, Term, Op, Group, TokenStream, TokenTree};

use syntax::ext::base::{ExtCtxt, ProcMacro};
use syntax::parse::token;
Expand All @@ -23,47 +23,59 @@ use syntax::tokenstream;
pub struct Quoter;

pub fn unquote<T: Into<TokenStream> + Clone>(tokens: &T) -> TokenStream {
T::into(tokens.clone())
tokens.clone().into()
}

pub trait Quote {
fn quote(self) -> TokenStream;
}

macro_rules! tt2ts {
($e:expr) => (TokenStream::from(TokenTree::from($e)))
}

macro_rules! quote_tok {
(,) => { TokenNode::Op(',', Spacing::Alone) };
(.) => { TokenNode::Op('.', Spacing::Alone) };
(:) => { TokenNode::Op(':', Spacing::Alone) };
(,) => { tt2ts!(Op::new(',', Spacing::Alone)) };
(.) => { tt2ts!(Op::new('.', Spacing::Alone)) };
(:) => { tt2ts!(Op::new(':', Spacing::Alone)) };
(|) => { tt2ts!(Op::new('|', Spacing::Alone)) };
(::) => {
[
TokenNode::Op(':', Spacing::Joint),
TokenNode::Op(':', Spacing::Alone)
].iter().cloned().collect::<TokenStream>()
TokenTree::from(Op::new(':', Spacing::Joint)),
TokenTree::from(Op::new(':', Spacing::Alone)),
].iter()
.cloned()
.map(|mut x| {
x.set_span(Span::def_site());
x
})
.collect::<TokenStream>()
};
(!) => { TokenNode::Op('!', Spacing::Alone) };
(<) => { TokenNode::Op('<', Spacing::Alone) };
(>) => { TokenNode::Op('>', Spacing::Alone) };
(_) => { TokenNode::Op('_', Spacing::Alone) };
(0) => { TokenNode::Literal(::Literal::integer(0)) };
(&) => { TokenNode::Op('&', Spacing::Alone) };
($i:ident) => { TokenNode::Term(Term::intern(stringify!($i))) };
(!) => { tt2ts!(Op::new('!', Spacing::Alone)) };
(<) => { tt2ts!(Op::new('<', Spacing::Alone)) };
(>) => { tt2ts!(Op::new('>', Spacing::Alone)) };
(_) => { tt2ts!(Op::new('_', Spacing::Alone)) };
(0) => { tt2ts!(Literal::i8_unsuffixed(0)) };
(&) => { tt2ts!(Op::new('&', Spacing::Alone)) };
($i:ident) => { tt2ts!(Term::new(stringify!($i), Span::def_site())) };
}

macro_rules! quote_tree {
((unquote $($t:tt)*)) => { $($t)* };
((quote $($t:tt)*)) => { ($($t)*).quote() };
(($($t:tt)*)) => { TokenNode::Group(Delimiter::Parenthesis, quote!($($t)*)) };
([$($t:tt)*]) => { TokenNode::Group(Delimiter::Bracket, quote!($($t)*)) };
({$($t:tt)*}) => { TokenNode::Group(Delimiter::Brace, quote!($($t)*)) };
(($($t:tt)*)) => { tt2ts!(Group::new(Delimiter::Parenthesis, quote!($($t)*))) };
([$($t:tt)*]) => { tt2ts!(Group::new(Delimiter::Bracket, quote!($($t)*))) };
({$($t:tt)*}) => { tt2ts!(Group::new(Delimiter::Brace, quote!($($t)*))) };
($t:tt) => { quote_tok!($t) };
}

macro_rules! quote {
() => { TokenStream::empty() };
($($t:tt)*) => {
[
$(TokenStream::from(quote_tree!($t)),)*
].iter().cloned().collect::<TokenStream>()
[$(quote_tree!($t),)*].iter()
.cloned()
.flat_map(|x| x.into_iter())
.collect::<TokenStream>()
};
}

Expand Down Expand Up @@ -97,72 +109,81 @@ impl Quote for TokenStream {
let tokens = self.into_iter().filter_map(|tree| {
if after_dollar {
after_dollar = false;
match tree.kind {
TokenNode::Term(_) => {
match tree {
TokenTree::Term(_) => {
let tree = TokenStream::from(tree);
return Some(quote!(::__internal::unquote(&(unquote tree)),));
}
TokenNode::Op('$', _) => {}
TokenTree::Op(ref tt) if tt.op() == '$' => {}
_ => panic!("`$` must be followed by an ident or `$` in `quote!`"),
}
} else if let TokenNode::Op('$', _) = tree.kind {
after_dollar = true;
return None;
} else if let TokenTree::Op(tt) = tree {
if tt.op() == '$' {
after_dollar = true;
return None;
}
}

Some(quote!(::TokenStream::from((quote tree)),))
}).collect::<TokenStream>();
}).flat_map(|t| t.into_iter()).collect::<TokenStream>();

if after_dollar {
panic!("unexpected trailing `$` in `quote!`");
}

quote!([(unquote tokens)].iter().cloned().collect::<::TokenStream>())
quote!(
[(unquote tokens)].iter()
.cloned()
.flat_map(|x| x.into_iter())
.collect::<::TokenStream>()
)
}
}

impl Quote for TokenTree {
fn quote(self) -> TokenStream {
quote!(::TokenTree { span: (quote self.span), kind: (quote self.kind) })
match self {
TokenTree::Op(tt) => quote!(::TokenTree::Op( (quote tt) )),
TokenTree::Group(tt) => quote!(::TokenTree::Group( (quote tt) )),
TokenTree::Term(tt) => quote!(::TokenTree::Term( (quote tt) )),
TokenTree::Literal(tt) => quote!(::TokenTree::Literal( (quote tt) )),
}
}
}

impl Quote for TokenNode {
impl Quote for char {
fn quote(self) -> TokenStream {
macro_rules! gen_match {
($($i:ident($($arg:ident),+)),*) => {
match self {
$(TokenNode::$i($($arg),+) => quote! {
::TokenNode::$i($((quote $arg)),+)
},)*
}
}
}
TokenTree::from(Literal::character(self)).into()
}
}

gen_match! { Op(op, kind), Group(delim, tokens), Term(term), Literal(lit) }
impl<'a> Quote for &'a str {
fn quote(self) -> TokenStream {
TokenTree::from(Literal::string(self)).into()
}
}

impl Quote for char {
impl Quote for usize {
fn quote(self) -> TokenStream {
TokenNode::Literal(Literal::character(self)).into()
TokenTree::from(Literal::usize_unsuffixed(self)).into()
}
}

impl<'a> Quote for &'a str {
impl Quote for Group {
fn quote(self) -> TokenStream {
TokenNode::Literal(Literal::string(self)).into()
quote!(::Group::new((quote self.delimiter()), (quote self.stream())))
}
}

impl Quote for usize {
impl Quote for Op {
fn quote(self) -> TokenStream {
TokenNode::Literal(Literal::integer(self as i128)).into()
quote!(::Op::new((quote self.op()), (quote self.spacing())))
}
}

impl Quote for Term {
fn quote(self) -> TokenStream {
quote!(::Term::intern((quote self.as_str())))
quote!(::Term::new((quote self.as_str()), (quote self.span())))
}
}

Expand All @@ -182,31 +203,38 @@ macro_rules! literals {
impl LiteralKind {
pub fn with_contents_and_suffix(self, contents: Term, suffix: Option<Term>)
-> Literal {
let contents = contents.0;
let suffix = suffix.map(|t| t.0);
let sym = contents.sym;
let suffix = suffix.map(|t| t.sym);
match self {
$(LiteralKind::$i => {
Literal(token::Literal(token::Lit::$i(contents), suffix))
Literal {
token: token::Literal(token::Lit::$i(sym), suffix),
span: contents.span,
}
})*
$(LiteralKind::$raw(n) => {
Literal(token::Literal(token::Lit::$raw(contents, n), suffix))
Literal {
token: token::Literal(token::Lit::$raw(sym, n), suffix),
span: contents.span,
}
})*
}
}
}

impl Literal {
fn kind_contents_and_suffix(self) -> (LiteralKind, Term, Option<Term>) {
let (lit, suffix) = match self.0 {
let (lit, suffix) = match self.token {
token::Literal(lit, suffix) => (lit, suffix),
_ => panic!("unsupported literal {:?}", self.0),
_ => panic!("unsupported literal {:?}", self.token),
};

let (kind, contents) = match lit {
$(token::Lit::$i(contents) => (LiteralKind::$i, contents),)*
$(token::Lit::$raw(contents, n) => (LiteralKind::$raw(n), contents),)*
};
(kind, Term(contents), suffix.map(Term))
let suffix = suffix.map(|sym| Term::new(&sym.as_str(), self.span()));
(kind, Term::new(&contents.as_str(), self.span()), suffix)
}
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@

extern crate proc_macro;

use proc_macro::{TokenStream, TokenTree, TokenNode, Delimiter, Literal, Spacing};
use proc_macro::{TokenStream, TokenTree, Delimiter, Literal, Spacing, Group};

#[proc_macro_attribute]
pub fn foo(attr: TokenStream, input: TokenStream) -> TokenStream {
Expand Down Expand Up @@ -52,24 +52,30 @@ pub fn bar(attr: TokenStream, input: TokenStream) -> TokenStream {
}

fn assert_inline(slice: &mut &[TokenTree]) {
match slice[0].kind {
TokenNode::Op('#', _) => {}
match &slice[0] {
TokenTree::Op(tt) => assert_eq!(tt.op(), '#'),
_ => panic!("expected '#' char"),
}
match slice[1].kind {
TokenNode::Group(Delimiter::Bracket, _) => {}
match &slice[1] {
TokenTree::Group(tt) => assert_eq!(tt.delimiter(), Delimiter::Bracket),
_ => panic!("expected brackets"),
}
*slice = &slice[2..];
}

fn assert_doc(slice: &mut &[TokenTree]) {
match slice[0].kind {
TokenNode::Op('#', Spacing::Alone) => {}
match &slice[0] {
TokenTree::Op(tt) => {
assert_eq!(tt.op(), '#');
assert_eq!(tt.spacing(), Spacing::Alone);
}
_ => panic!("expected #"),
}
let inner = match slice[1].kind {
TokenNode::Group(Delimiter::Bracket, ref s) => s.clone(),
let inner = match &slice[1] {
TokenTree::Group(tt) => {
assert_eq!(tt.delimiter(), Delimiter::Bracket);
tt.stream()
}
_ => panic!("expected brackets"),
};
let tokens = inner.into_iter().collect::<Vec<_>>();
Expand All @@ -79,49 +85,55 @@ fn assert_doc(slice: &mut &[TokenTree]) {
panic!("expected three tokens in doc")
}

match tokens[0].kind {
TokenNode::Term(ref t) => assert_eq!("doc", t.as_str()),
match &tokens[0] {
TokenTree::Term(tt) => assert_eq!("doc", tt.as_str()),
_ => panic!("expected `doc`"),
}
match tokens[1].kind {
TokenNode::Op('=', Spacing::Alone) => {}
match &tokens[1] {
TokenTree::Op(tt) => {
assert_eq!(tt.op(), '=');
assert_eq!(tt.spacing(), Spacing::Alone);
}
_ => panic!("expected equals"),
}
match tokens[2].kind {
TokenNode::Literal(_) => {}
match tokens[2] {
TokenTree::Literal(_) => {}
_ => panic!("expected literal"),
}

*slice = &slice[2..];
}

fn assert_invoc(slice: &mut &[TokenTree]) {
match slice[0].kind {
TokenNode::Op('#', _) => {}
match &slice[0] {
TokenTree::Op(tt) => assert_eq!(tt.op(), '#'),
_ => panic!("expected '#' char"),
}
match slice[1].kind {
TokenNode::Group(Delimiter::Bracket, _) => {}
match &slice[1] {
TokenTree::Group(tt) => assert_eq!(tt.delimiter(), Delimiter::Bracket),
_ => panic!("expected brackets"),
}
*slice = &slice[2..];
}

fn assert_foo(slice: &mut &[TokenTree]) {
match slice[0].kind {
TokenNode::Term(ref name) => assert_eq!(name.as_str(), "fn"),
match &slice[0] {
TokenTree::Term(tt) => assert_eq!(tt.as_str(), "fn"),
_ => panic!("expected fn"),
}
match slice[1].kind {
TokenNode::Term(ref name) => assert_eq!(name.as_str(), "foo"),
match &slice[1] {
TokenTree::Term(tt) => assert_eq!(tt.as_str(), "foo"),
_ => panic!("expected foo"),
}
match slice[2].kind {
TokenNode::Group(Delimiter::Parenthesis, ref s) => assert!(s.is_empty()),
match &slice[2] {
TokenTree::Group(tt) => {
assert_eq!(tt.delimiter(), Delimiter::Parenthesis);
assert!(tt.stream().is_empty());
}
_ => panic!("expected parens"),
}
match slice[3].kind {
TokenNode::Group(Delimiter::Brace, _) => {}
match &slice[3] {
TokenTree::Group(tt) => assert_eq!(tt.delimiter(), Delimiter::Brace),
_ => panic!("expected braces"),
}
*slice = &slice[4..];
Expand All @@ -132,22 +144,17 @@ fn fold_stream(input: TokenStream) -> TokenStream {
}

fn fold_tree(input: TokenTree) -> TokenTree {
TokenTree {
span: input.span,
kind: fold_node(input.kind),
}
}

fn fold_node(input: TokenNode) -> TokenNode {
match input {
TokenNode::Group(a, b) => TokenNode::Group(a, fold_stream(b)),
TokenNode::Op(a, b) => TokenNode::Op(a, b),
TokenNode::Term(a) => TokenNode::Term(a),
TokenNode::Literal(a) => {
TokenTree::Group(b) => {
TokenTree::Group(Group::new(b.delimiter(), fold_stream(b.stream())))
}
TokenTree::Op(b) => TokenTree::Op(b),
TokenTree::Term(a) => TokenTree::Term(a),
TokenTree::Literal(a) => {
if a.to_string() != "\"foo\"" {
TokenNode::Literal(a)
TokenTree::Literal(a)
} else {
TokenNode::Literal(Literal::integer(3))
TokenTree::Literal(Literal::i32_unsuffixed(3))
}
}
}
Expand Down
Loading

0 comments on commit 553c04d

Please sign in to comment.