New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove `TokenStream::JointTree`. #56964

Merged
merged 1 commit into from Dec 24, 2018

Conversation

Projects
None yet
6 participants
@nnethercote
Copy link
Contributor

nnethercote commented Dec 18, 2018

This is done by adding a new IsJoint field to TokenStream::Tree,
which simplifies a lot of match statements. And likewise for
CursorKind.

The commit also adds a new method TokenTree:stream() which can replace
a choice between .into() and .joint().

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

rust-highfive commented Dec 18, 2018

r? @nikomatsakis

(rust_highfive has picked a reviewer for you, use r? to override)

@nnethercote

This comment has been minimized.

Copy link
Contributor

nnethercote commented Dec 18, 2018

After these commits, TokenStream looks like this:

pub enum TokenStream {
    Empty,
    Tree(TokenTree, IsJoint),
    Stream(Lrc<Vec<TokenStream>>),
}

I wonder if the recursion via the Stream variant is necessary. Perhaps it could instead be this:

    Stream(Lrc<Vec<TokenTree>>),

To break it down further, perhaps TokenStream is conceptually just this:

Vec<(TokenTree, IsJoint)>

(Possibly with the use of Lrc and Empty/singleton variants for performance
reasons.)

There's a comment at the top of tokenstream.rs that suggests this is true:

TokenStreams represent syntactic objects before they are converted into
ASTs. A TokenStream is, roughly speaking, a sequence (eg stream) of
TokenTrees, which are themselves a single Token or a Delimited
subsequence of tokens.

@petrochenkov

This comment has been minimized.

Copy link
Contributor

petrochenkov commented Dec 18, 2018

perhaps TokenStream is conceptually just this: Vec<(TokenTree, IsJoint)>

More that that, IsJoint is not even a property of TokenTree, but only punctuation tokens, so TokenStream is conceptually just Vec<TokenTree>.
See stable public interfaces in proc_macro for what "canonical" token streams look like in terms of interface.
In general, it would be nice to reuse types from proc_macro in the compiler if possible, or at least align with them.

@nnethercote

This comment has been minimized.

Copy link
Contributor

nnethercote commented Dec 18, 2018

@petrochenkov: I tried looking at those public interfaces but I'm having trouble understanding them, particularly the bridge and define_handles! stuff. Can you give me some hints? Thanks.

@petrochenkov

This comment has been minimized.

Copy link
Contributor

petrochenkov commented Dec 18, 2018

Oh, there's the bridge thing now.
Then my first hint would be to look at the snapshot of libproc_macro preceding merge of #49219.

@petrochenkov

This comment has been minimized.

Copy link
Contributor

petrochenkov commented Dec 18, 2018

(libproc_macro was much more straightforward and closer to its "conceptual" form before all that machinery for client-server communication was added.)

@eddyb

This comment has been minimized.

Copy link
Member

eddyb commented Dec 18, 2018

You can still look at the public API, IMO, it's not like the old code was self-contained.

Group, for example, used syntax::tokenstream::TokenStream before (back then wrapped by proc_macro::TokenStream), and it still uses that today, albeit the actual definition of Group now lives in syntax_ext::proc_macro_server (as does the rest of the compiler side of the implementation).

Show resolved Hide resolved src/libsyntax/tokenstream.rs Outdated
@petrochenkov

This comment has been minimized.

Copy link
Contributor

petrochenkov commented Dec 19, 2018

r=me with #56964 (comment) addressed.
@bors delegate+

@bors

This comment has been minimized.

Copy link
Contributor

bors commented Dec 19, 2018

✌️ @nnethercote can now approve this pull request

Remove `TokenStream::JointTree`.
This is done by adding a new `IsJoint` field to `TokenStream::Tree`,
which simplifies a lot of `match` statements. And likewise for
`CursorKind`.

The commit also adds a new method `TokenTree:stream()` which can replace
a choice between `.into()` and `.joint()`.

@nnethercote nnethercote force-pushed the nnethercote:TokenStream-IsJoint branch from 50eff25 to e7c5146 Dec 19, 2018

@nnethercote

This comment has been minimized.

Copy link
Contributor

nnethercote commented Dec 19, 2018

@bors r=petrochenkov

@bors

This comment has been minimized.

Copy link
Contributor

bors commented Dec 19, 2018

📌 Commit e7c5146 has been approved by petrochenkov

@nnethercote

This comment has been minimized.

Copy link
Contributor

nnethercote commented Dec 20, 2018

perhaps TokenStream is conceptually just this: Vec<(TokenTree, IsJoint)>

More that that, IsJoint is not even a property of TokenTree, but only punctuation tokens, so TokenStream is conceptually just Vec<TokenTree>.

Perhaps it's more like this: (Vec<TokenTree>, IsJoint). The IsJoint applies only to the final TokenTree, and should be used if the Vec gets extended. (All the TokenTrees prior to the last one should have already been joined if necessary.)

At least, that's the case for any TokenStream that may still be extended. A TokenStream that is known to be complete can drop the IsJoint.

pietroalbini added a commit to pietroalbini/rust that referenced this pull request Dec 21, 2018

Rollup merge of rust-lang#56964 - nnethercote:TokenStream-IsJoint, r=…
…petrochenkov

Remove `TokenStream::JointTree`.

This is done by adding a new `IsJoint` field to `TokenStream::Tree`,
which simplifies a lot of `match` statements. And likewise for
`CursorKind`.

The commit also adds a new method `TokenTree:stream()` which can replace
a choice between `.into()` and `.joint()`.

bors added a commit that referenced this pull request Dec 21, 2018

Auto merge of #57025 - pietroalbini:rollup, r=pietroalbini
Rollup of 21 pull requests

Successful merges:

 - #56802 (Add DoubleEndedIterator::nth_back)
 - #56842 (Add unstable VecDeque::rotate_{left|right})
 - #56869 (Reduce search-index.js size)
 - #56887 (Disable field reordering for repr(int).)
 - #56892 (rustc: Update Clang used to build LLVM on Linux)
 - #56909 (static eval: Do not ICE on layout size overflow)
 - #56914 (Ignore ui/target-feature-gate on sparc, sparc64, powerpc, powerpc64 and powerpc64le)
 - #56917 (Simplify MIR generation for logical operations)
 - #56919 (Remove a wrong multiplier on relocation offset computation)
 - #56933 (Add --progress to git submodule commands in x.py)
 - #56941 (deny intra-doc link resolution failures in libstd)
 - #56954 (Add dist builder for Armv8-M Mainline)
 - #56964 (Remove `TokenStream::JointTree`.)
 - #56970 (Mem uninit doc ptr drop)
 - #56973 (make basic CTFE tracing available on release builds)
 - #56979 (Adding unwinding support for x86_64_fortanix_unknown_sgx target.)
 - #56981 (miri: allocation is infallible)
 - #56984 (A few tweaks to dropck_outlives)
 - #56989 (Fix compiletest `trim` deprecation warnings)
 - #56992 (suggest similar lint names for unknown lints)
 - #57002 (Stabilize Vec(Deque)::resize_with)

Failed merges:

r? @ghost

pietroalbini added a commit to pietroalbini/rust that referenced this pull request Dec 21, 2018

Rollup merge of rust-lang#56964 - nnethercote:TokenStream-IsJoint, r=…
…petrochenkov

Remove `TokenStream::JointTree`.

This is done by adding a new `IsJoint` field to `TokenStream::Tree`,
which simplifies a lot of `match` statements. And likewise for
`CursorKind`.

The commit also adds a new method `TokenTree:stream()` which can replace
a choice between `.into()` and `.joint()`.

bors added a commit that referenced this pull request Dec 21, 2018

Auto merge of #57025 - pietroalbini:rollup, r=pietroalbini
Rollup of 20 pull requests

Successful merges:

 - #56802 (Add DoubleEndedIterator::nth_back)
 - #56842 (Add unstable VecDeque::rotate_{left|right})
 - #56869 (Reduce search-index.js size)
 - #56887 (Disable field reordering for repr(int).)
 - #56892 (rustc: Update Clang used to build LLVM on Linux)
 - #56909 (static eval: Do not ICE on layout size overflow)
 - #56914 (Ignore ui/target-feature-gate on sparc, sparc64, powerpc, powerpc64 and powerpc64le)
 - #56917 (Simplify MIR generation for logical operations)
 - #56919 (Remove a wrong multiplier on relocation offset computation)
 - #56933 (Add --progress to git submodule commands in x.py)
 - #56941 (deny intra-doc link resolution failures in libstd)
 - #56964 (Remove `TokenStream::JointTree`.)
 - #56970 (Mem uninit doc ptr drop)
 - #56973 (make basic CTFE tracing available on release builds)
 - #56979 (Adding unwinding support for x86_64_fortanix_unknown_sgx target.)
 - #56981 (miri: allocation is infallible)
 - #56984 (A few tweaks to dropck_outlives)
 - #56989 (Fix compiletest `trim` deprecation warnings)
 - #56992 (suggest similar lint names for unknown lints)
 - #57002 (Stabilize Vec(Deque)::resize_with)

Failed merges:

r? @ghost

bors added a commit that referenced this pull request Dec 21, 2018

Auto merge of #57025 - pietroalbini:rollup, r=pietroalbini
Rollup of 20 pull requests

Successful merges:

 - #56802 (Add DoubleEndedIterator::nth_back)
 - #56842 (Add unstable VecDeque::rotate_{left|right})
 - #56869 (Reduce search-index.js size)
 - #56887 (Disable field reordering for repr(int).)
 - #56892 (rustc: Update Clang used to build LLVM on Linux)
 - #56909 (static eval: Do not ICE on layout size overflow)
 - #56914 (Ignore ui/target-feature-gate on sparc, sparc64, powerpc, powerpc64 and powerpc64le)
 - #56917 (Simplify MIR generation for logical operations)
 - #56919 (Remove a wrong multiplier on relocation offset computation)
 - #56933 (Add --progress to git submodule commands in x.py)
 - #56941 (deny intra-doc link resolution failures in libstd)
 - #56964 (Remove `TokenStream::JointTree`.)
 - #56970 (Mem uninit doc ptr drop)
 - #56973 (make basic CTFE tracing available on release builds)
 - #56979 (Adding unwinding support for x86_64_fortanix_unknown_sgx target.)
 - #56981 (miri: allocation is infallible)
 - #56984 (A few tweaks to dropck_outlives)
 - #56989 (Fix compiletest `trim` deprecation warnings)
 - #56992 (suggest similar lint names for unknown lints)
 - #57002 (Stabilize Vec(Deque)::resize_with)

Failed merges:

r? @ghost

Centril added a commit to Centril/rust that referenced this pull request Dec 22, 2018

Rollup merge of rust-lang#56964 - nnethercote:TokenStream-IsJoint, r=…
…petrochenkov

Remove `TokenStream::JointTree`.

This is done by adding a new `IsJoint` field to `TokenStream::Tree`,
which simplifies a lot of `match` statements. And likewise for
`CursorKind`.

The commit also adds a new method `TokenTree:stream()` which can replace
a choice between `.into()` and `.joint()`.

Centril added a commit to Centril/rust that referenced this pull request Dec 23, 2018

Rollup merge of rust-lang#56964 - nnethercote:TokenStream-IsJoint, r=…
…petrochenkov

Remove `TokenStream::JointTree`.

This is done by adding a new `IsJoint` field to `TokenStream::Tree`,
which simplifies a lot of `match` statements. And likewise for
`CursorKind`.

The commit also adds a new method `TokenTree:stream()` which can replace
a choice between `.into()` and `.joint()`.

bors added a commit that referenced this pull request Dec 23, 2018

Auto merge of #57075 - Centril:rollup, r=Centril
Rollup of 10 pull requests

Successful merges:

 - #56188 (enum type instead of variant suggestion unification )
 - #56916 (Fix mutable references in `static mut`)
 - #56917 (Simplify MIR generation for logical operations)
 - #56953 (Mark tuple structs as live if their constructors are used)
 - #56964 (Remove `TokenStream::JointTree`.)
 - #56966 (Correct strings for raw pointer deref and array access suggestions)
 - #56999 (AST/HIR: Introduce `ExprKind::Err` for better error recovery in the front-end)
 - #57020 (Point to cause of `fn` expected return type)
 - #57032 (fix deprecation warnings in liballoc benches)
 - #57053 (Fix alignment for array indexing)

Failed merges:

r? @ghost

Centril added a commit to Centril/rust that referenced this pull request Dec 23, 2018

Rollup merge of rust-lang#56964 - nnethercote:TokenStream-IsJoint, r=…
…petrochenkov

Remove `TokenStream::JointTree`.

This is done by adding a new `IsJoint` field to `TokenStream::Tree`,
which simplifies a lot of `match` statements. And likewise for
`CursorKind`.

The commit also adds a new method `TokenTree:stream()` which can replace
a choice between `.into()` and `.joint()`.

Centril added a commit to Centril/rust that referenced this pull request Dec 23, 2018

Rollup merge of rust-lang#56964 - nnethercote:TokenStream-IsJoint, r=…
…petrochenkov

Remove `TokenStream::JointTree`.

This is done by adding a new `IsJoint` field to `TokenStream::Tree`,
which simplifies a lot of `match` statements. And likewise for
`CursorKind`.

The commit also adds a new method `TokenTree:stream()` which can replace
a choice between `.into()` and `.joint()`.

bors added a commit that referenced this pull request Dec 23, 2018

Auto merge of #57083 - Centril:rollup, r=Centril
Rollup of 10 pull requests

Successful merges:

 - #56188 (enum type instead of variant suggestion unification )
 - #56916 (Fix mutable references in `static mut`)
 - #56917 (Simplify MIR generation for logical operations)
 - #56953 (Mark tuple structs as live if their constructors are used)
 - #56964 (Remove `TokenStream::JointTree`.)
 - #56966 (Correct strings for raw pointer deref and array access suggestions)
 - #57020 (Point to cause of `fn` expected return type)
 - #57032 (fix deprecation warnings in liballoc benches)
 - #57053 (Fix alignment for array indexing)
 - #57067 (Stabilize min_const_unsafe_fn in 1.33)

Failed merges:

r? @ghost

bors added a commit that referenced this pull request Dec 23, 2018

Auto merge of #57083 - Centril:rollup, r=Centril
Rollup of 10 pull requests

Successful merges:

 - #56188 (enum type instead of variant suggestion unification )
 - #56916 (Fix mutable references in `static mut`)
 - #56917 (Simplify MIR generation for logical operations)
 - #56953 (Mark tuple structs as live if their constructors are used)
 - #56964 (Remove `TokenStream::JointTree`.)
 - #56966 (Correct strings for raw pointer deref and array access suggestions)
 - #57020 (Point to cause of `fn` expected return type)
 - #57032 (fix deprecation warnings in liballoc benches)
 - #57053 (Fix alignment for array indexing)
 - #57067 (Stabilize min_const_unsafe_fn in 1.33)

Failed merges:

r? @ghost

Centril added a commit to Centril/rust that referenced this pull request Dec 23, 2018

Rollup merge of rust-lang#56964 - nnethercote:TokenStream-IsJoint, r=…
…petrochenkov

Remove `TokenStream::JointTree`.

This is done by adding a new `IsJoint` field to `TokenStream::Tree`,
which simplifies a lot of `match` statements. And likewise for
`CursorKind`.

The commit also adds a new method `TokenTree:stream()` which can replace
a choice between `.into()` and `.joint()`.

bors added a commit that referenced this pull request Dec 24, 2018

Auto merge of #57087 - Centril:rollup, r=Centril
Rollup of 14 pull requests

Successful merges:

 - #56188 (enum type instead of variant suggestion unification )
 - #56342 (Improve docs for collecting into `Option`s)
 - #56916 (Fix mutable references in `static mut`)
 - #56917 (Simplify MIR generation for logical operations)
 - #56939 (Pin stabilization)
 - #56953 (Mark tuple structs as live if their constructors are used)
 - #56964 (Remove `TokenStream::JointTree`.)
 - #56966 (Correct strings for raw pointer deref and array access suggestions)
 - #57020 (Point to cause of `fn` expected return type)
 - #57032 (fix deprecation warnings in liballoc benches)
 - #57053 (Fix alignment for array indexing)
 - #57062 (Fix a comment)
 - #57067 (Stabilize min_const_unsafe_fn in 1.33)
 - #57078 (Ignore two tests on s390x)

Failed merges:

r? @ghost

bors added a commit that referenced this pull request Dec 24, 2018

Auto merge of #57087 - Centril:rollup, r=Centril
Rollup of 14 pull requests

Successful merges:

 - #56188 (enum type instead of variant suggestion unification )
 - #56342 (Improve docs for collecting into `Option`s)
 - #56916 (Fix mutable references in `static mut`)
 - #56917 (Simplify MIR generation for logical operations)
 - #56939 (Pin stabilization)
 - #56953 (Mark tuple structs as live if their constructors are used)
 - #56964 (Remove `TokenStream::JointTree`.)
 - #56966 (Correct strings for raw pointer deref and array access suggestions)
 - #57020 (Point to cause of `fn` expected return type)
 - #57032 (fix deprecation warnings in liballoc benches)
 - #57053 (Fix alignment for array indexing)
 - #57062 (Fix a comment)
 - #57067 (Stabilize min_const_unsafe_fn in 1.33)
 - #57078 (Ignore two tests on s390x)

Failed merges:

r? @ghost

@bors bors merged commit e7c5146 into rust-lang:master Dec 24, 2018

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment