proc_macro: Reorganize public API

[alexcrichton]

This commit is a reorganization of the proc_macro crate's public user-facing
API. This is the result of a number of discussions at the recent Rust All-Hands
where we're hoping to get the proc_macro crate into ship shape for
stabilization of a subset of its functionality in the Rust 2018 release.

The reorganization here is motivated by experiences from the proc-macro2,
quote, and syn crates on crates.io (and other crates which depend on them).
The main focus is future flexibility along with making a few more operations
consistent and/or fixing bugs. A summary of the changes made from today's
proc_macro API is:

• The TokenNode enum has been removed and the public fields of TokenTree
  have also been removed. Instead the TokenTree type is now a public enum
  (what TokenNode was) and each variant is an opaque struct which internally
  contains Span information. This makes the various tokens a bit more
  consistent, require fewer wrappers, and otherwise provides good
  future-compatibility as opaque structs are easy to modify later on.

• Literal integer constructors have been expanded to be unambiguous as to what
  they're doing and also allow for more future flexibility. Previously
  constructors like Literal::float and Literal::integer were used to create
  unsuffixed literals and the concrete methods like Literal::i32 would create
  a suffixed token. This wasn't immediately clear to all users (the
  suffixed/unsuffixed aspect) and having one constructor for unsuffixed
  literals required us to pick a largest type which may not always be true. To
  fix these issues all constructors are now of the form
  Literal::i32_unsuffixed or Literal::i32_suffixed (for all integral types).
  This should allow future compatibility as well as being immediately clear
  what's suffixed and what isn't.

• Each variant of TokenTree internally contains a Span which can also be
  configured via set_span. For example Literal and Term now both
  internally contain a Span rather than having it stored in an auxiliary
  location.

• Constructors of all tokens are called new now (aka Term::intern is gone)
  and most do not take spans. Manufactured tokens typically don't have a fresh
  span to go with them and the span is purely used for error-reporting
  except the span for Term, which currently affects hygiene. The default
  spans for all these constructed tokens is Span::call_site() for now.

  The Term type's constructor explicitly requires passing in a Span to
  provide future-proofing against possible hygiene changes. It's intended that a
  first pass of stabilization will likely only stabilize Span::call_site()
  which is an explicit opt-in for "I would like no hygiene here please". The
  intention here is to make this explicit in procedural macros to be
  forwards-compatible with a hygiene-specifying solution.

• Some of the conversions for TokenStream have been simplified a little.

• The TokenTreeIter iterator was renamed to token_stream::IntoIter.

Overall the hope is that this is the "final pass" at the API of TokenStream
and most of TokenTree before stabilization. Explicitly left out here is any
changes to Span's API which will likely need to be re-evaluated before
stabilization.

All changes in this PR have already been reflected to the proc-macro2,
quote, and syn crates. New versions of all these crates have also been
published to crates.io.

Once this lands in nightly I plan on making an internals post again summarizing
the changes made here and also calling on all macro authors to give the APIs a
spin and see how they work. Hopefully pending no major issues we can then have
an FCP to stabilize later this cycle!

Closes #49596

[alexcrichton]

cc @dtolnay, @nrc, @jseyfried

[alexcrichton]

r? @nrc

[sgrif]

Explicitly left out here is any changes to Span's API which will likely need to be re-evaluated before
stabilization.

Are there concrete plans on changes there? Has this been documented anywhere?

[alexcrichton]

@sgrif AFAIK there are no concrete plans yet, so no documentation either

[TimNN]

Your PR failed on Travis. Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

[00:00:45] configure: rust.quiet-tests     := True
---
[00:41:49] .........................................................................i..........................
[00:41:55] ................i...................................................................................
---
[00:42:31] ............................................................................................i.......
[00:42:38] ................................................................i...................................
---
[00:43:33] .............................................i......................................................
---
[00:47:28] .............................i......................................................................
[00:47:43] ..............................................................i.....................................
[00:47:59] ...............................................i....................................................
[00:48:19] ....................................................................................................
[00:48:41] ....................................................................................................
[00:49:02] ....................................................................................................
[00:49:27] .i................................................................................................i.
[00:49:54] ..................................................................................test [run-pass] run-pass/mir_heavy_promoted.rs has been running for over 60 seconds
[00:50:04] ..................
[00:50:34] ....................................................................................................
[00:51:10] ...............................................................ii...................................
[00:51:55] ..........................i....................................................i.ii....test [run-pass] run-pass/saturating-float-casts.rs has been running for over 60 seconds
[00:52:00] .............
[00:52:40] .......................................................................................iiiiiii......
---
[00:54:42] ....................................i...............................................................
[00:54:49] ....................................................................................................
[00:54:57] ..................i............................................................ii.iii...............
[00:55:04] ....................................................................................................
[00:55:12] ........i..............................i............................................................
[00:55:20] ....................................................................................................
[00:55:27] .....................i..............................................................................
[00:55:35] ....................................................................................................
[00:55:45] ....................................................................................................
[00:55:55] ....................................................................................................
[00:56:06] ....................................................................................................
[00:56:20] ....................................................................................................
[00:56:29] ..............i.....................................................................................
[00:56:39] .................i..ii..............................................................................
[00:56:48] ....................................................................................................
[00:56:59] ....................................................................................................
[00:57:08] ....................................................................................i...............
[00:57:19] ..............................i.....................................................................
---
[00:57:56] ...........................i........................................................................
[00:57:57] ....................................................................i...............................
[00:57:59] ................i.......................................................
---
[00:58:13] ...........i........................
---
[00:58:43] i...i..ii....i.............ii........iii......i..i...i...ii..i..i..ii.....
---
[00:58:46] i.......i......................i......
---
[00:59:24] iiii.......i..i........i..i.i.............i..........iiii...........i...i..........ii.i.i.......ii..
[00:59:25] ....ii...
---
acro/auxiliary/parent-source-spans.rs","byte_start":740,"byte_end":745,"line_start":22,"line_end":22,"column_start":23,"column_end":28,"is_primary":true,"text":[{"text":"        Literal(..) | Group(..) => (tt.span, tt.to_string().trim().into()),","highlight_start":23,"highlight_end":28}],"label":"not found in this scope","suggested_replacement":null,"expansion":null}],"children":[{"message":"possible candidate is found in another module, you can import it into scope","code":null,"level":"help","spans":[{"file_name":"/checkout/src/test/ui-fulldeps/proc-macro/auxiliary/parent-source-spans.rs","byte_start":569,"byte_end":569,"line_start":17,"line_end":17,"column_start":1,"column_end":1,"is_primary":true,"text":[{"text":"use proc_macro::{TokenStream, TokenTree, TokenNode, Span};","highlight_start":1,"highlight_end":1}],"label":null,"suggested_replacement":"use proc_macro::TokenTree::Group;\n","expansion":null}],"children":[],"rendered":null}],"rendered":"error[E0531]: cannot find tuple struct/variant `Group` in this scope\n  --> /checkout/src/test/ui-fulldeps/proc-macro/auxiliary/parent-source-spans.rs:22:23\n   |\nLL |         Literal(..) | Group(..) => (tt.span, tt.to_string().trim().into()),\n   |                       ^^^^^ not found in this scope\nhelp: possible candidate is found in another module, you can import it into scope\n   |\nLL | use proc_macro::TokenTree::Group;\n   |\n\n"}
[00:59:34] {"message":"aborting due to 3 previous errors","code":null,"level":"error","spans":[],"children":[],"rendered":"error: aborting due to 3 previous errors\n\n"}
[00:59:34] {"message":"Some errors occurred: E0432, E0531.","code":null,"level":"","spans":[],"children":[],"rendered":"Some errors occurred: E0432, E0531.\n"}
[00:59:34] {"message":"For more information about an error, try `rustc --explain E0432`.","code":null,"level":"","spans":[],"children":[],"rendered":"For more information about an error, try `rustc --explain E0432`.\n"}
---
[00:59:34] error: auxiliary build of "/checkout/src/test/ui-fulldeps/proc-macro/auxiliary/three-equals.rs" failed to compile:
[00:59:34] status: exit code: 101
[00:59:34] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui-fulldeps/proc-macro/auxiliary/three-equals.rs" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui-fulldeps" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "--out-dir" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui-fulldeps/proc-macro/three-equals.stage2-x86_64-unknown-linux-gnu.aux" "-Crpath" "-O" "-Zmiri" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui-fulldeps/proc-macro/three-equals.stage2-x86_64-unknown-linux-gnu.aux"
[00:59:34] stdout:
[00:59:34] ------------------------------------------
[00:59:34]
[00:59:34] -----iary/three-equals.rs:17:31\n   |\nLL | use proc_macro::{TokenStream, TokenNode, Span, Diagnostic};\n   |                               ^^^^^^^^^ no `TokenNode` in the root. Did you mean to use `TokenTree`?\n\n"}
[00:59:34] {"message":"attempted to take value of method `span` on type `proc_macro::TokenTree`","code":{"code":"E0615","explanation":"\nAttempted to access a method like a field.\n\nErroneous code example:\n\n```compile_fail,E0615\nstruct Foo {\n    x: u32,\n}\n\nimpl Foo {\n    fn method(&self) {}\n}\n\nlet f = Foo { x: 0 };\nf.method; // error: attempted to take value of method `method` on type `Foo`\n```\n\nIf you want to use a method, add `()` after it:\n\n```\n# struct Foo { x: u32 }\n# impl Foo { fn method(&self) {} }\n# let f = Foo { x: 0 };\nf.method();\n```\n\nHowever, if you wanted to access a field of a struct check that the field name\nis spelled correctly. Example:\n\n```\n# struct Foo { x: u32 }\n# impl Foo { fn method(&self) {} }\n# let f = Foo { x: 0 };\nprintln!(\"{}\", f.x);\n```\n"},"level":"error","spans":[{"file_name":"/checkout/src/test/ui-fulldeps/proc-macro/auxiliary/three-equals.rs","byte_start":800,"byte_end":804,"line_start":23,"line_end":23,"column_start":25,"column_end":29,"is_primary":true,"text":[{"text":"        let span = tree.span;","highlight_start":25,"highlight_end":29}],"label":null,"suggested_replacement":null,"expansion":null}],"children":[{"message":"maybe a `()` to call it is missing?","code":null,"level":"help","spans":[],"children":[],"rendered":null}],"rendered":"error[E0615]: attempted to take value of method `span` on type `proc_macro::TokenTree`\n  --> /checkout/src/tenknown-linux-gnu/native
---
56668 ./obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/incremental/syntax-2bjr8lg1wd06d/s-ezr2bofnbx-1tjk4f8-3oe29bqr0b31g
---
$ ls -lat $HOME/Library/Logs/DiagnosticReports/
ls: cannot access /home/travis/Library/Logs/DiagnosticReports/: No such file or directory
travis_time:end:1c3df970:start=1522700286854383955,finish=1522700286862465213,duration=8081258
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:0e66bd0a
$ find $HOME/Library/Logs/DiagnosticReports -type f -name '*.crash' -not -name '*.stage2-*.crash' -not -name 'com.apple.CoreSimulator.CoreSimulatorService-*.crash' -exec printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" {} \; -exec head -750 {} \; -exec echo travis_fold":"end:crashlog \; || true
find: `/home/travis/Library/Logs/DiagnosticReports': No such file or directory
travis_time:end:0e66bd0a:start=1522700286870059867,finish=1522700286878255196,duration=8195329
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:07b54980
$ dmesg | grep -i kill
[   10.986113] init: failsafe main process (1093) killed by TERM signal

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN.

[sgrif]

@alexcrichton LMK if there's any discussion. Diesel is using pretty much all the methods on Span and Diagnostic (and the removal of resolved_at from proc-macro2 makes upgrading a pain)

[alexcrichton]

@sgrif will do!

[petrochenkov]

Ok, maybe it's time to ask before it's stabilized.
Where's the name "token stream" actually came from?

I always found it actively confusing.
What's "streamy" about it? It's not streaming anywhere, it's just a data structure sitting there.
It's not a stream, it's kind of... tree? Or rather forest. Or Group of trees, using terminology from former TokenNode and now TokenTree. Or "token-based AST".

[petrochenkov]

Maybe TokenStream can even be eliminated?
Isn't top-level token stream equivalent to TokenTree::Group with empty delimiter?

[alexcrichton]

@petrochenkov I think TokenStream is the one stable name already have though, right?

[petrochenkov]

I think TokenStream is the one stable name already have

Sigh, macros 1.1 strike again.
The question about origin of the term stays though. Quick googling gives some links like this but this is not what we are working with.

Maybe TokenStream can even be eliminated?

As I see it:

enum TokenTree {
    Leaf(LeafAuxData /* leaf kind, span, ... */),
    Subtree(Vec<TokenTree>, SubTreeAuxData /* delimiter kind, span, ... */),
}

In this sense TokenStream == TokenTree and there's no need for an extra entity. A single leaf is a token stream, a subtree with non-empty delimiters is a token stream, a subtree with empty delimiters is a token stream as well.

The current library, however introduces one more layer of abstraction than strictly necessary - TokenStream is an interface that hides that it's actually a TokenTree and provides only iterator-based methods, kinda like

type TokenStream = impl Iterator<TokenTree> + Aux /* printing, parsing, ...*/;

From this perspective the name "stream" actually starts making sense.

I'm not 100% sure this additional layer is necessary, but it probably is, at least it's certainly better to put something like impl Iterator<TokenTree> in public TokenTree::Subtree(...) instead of literally Vec or other concrete type.

I also can guess where this scheme comes from historically - tt matcher in macros doesn't accept token trees with empty delimiter, so it kinda forces to introduce a separate entity for "tt minus delimiter" aka "tt*" aka Vec<TokenTree> without realizing it's actually the same exact token tree.

[petrochenkov]

The only thing I'm not entirely happy about is naming for Group.
Perhaps Subtree would be more clear.

[alexcrichton]

Ok thanks for the review! I talked about this pretty extensively with @nrc at the recent all-hands so I'm going to go ahead and...

@bors: r=petrochenkov

@dtolnay do you have thoughts on @petrochenkov's thoughts?

I'd personally agree that TokenStream is necessary today as an abstraction boundary to let us tweak how we implement stream of tokens (e.g. rc slices today), but I'd be fine renaming Group!

[bors]

📌 Commit 553c04d has been approved by petrochenkov

[alexcrichton]

Additionally with #49655

@bors: r=petrochenkov

[bors]

📌 Commit a57b1fb has been approved by petrochenkov

[nrc]

lgtm, r+

[SimonSapin]

I think this has regressed a stable impl: #49725