Simplify `TokenStream` some more #57486

nnethercote · 2019-01-10T03:42:14Z

These commits simplify TokenStream, remove ThinTokenStream, and avoid some clones. The end result is simpler code and a slight perf win on some benchmarks.

r? @petrochenkov

nnethercote · 2019-01-10T03:42:32Z

(Note that the first commit is from #57004, and will be removed before landing.)

src/libsyntax/tokenstream.rs

src/libsyntax/parse/parser.rs

nnethercote · 2019-01-11T00:56:06Z

I added a new commit that changes TokenStream to uses Option.

petrochenkov · 2019-01-11T09:41:42Z

src/libsyntax/tokenstream.rs

+            None => true,
+            Some(ref stream) => {
+                // An empty stream should be represented as `None`, not as an empty `Vec`.
+                debug_assert!(stream.len() > 0);


Hmm, do we have operations that remove trees from the stream's vector?
It's correct that we should not create new empty streams as Some, but perhaps previously non-empty streams can turn into empty?

Not that I know of. But if it does happen, the old code for this function would silently do the wrong thing, while the new code will assert. I could change it to check for a length of zero in the Some case.

Yes, could you use the correct Some(ref stream) => stream.is_empty(), that would be more future-proof.

petrochenkov · 2019-01-11T09:41:56Z

@bors r+

bors · 2019-01-11T09:41:57Z

📌 Commit ea2784b8701bfd63dada95ea12bd854479b88e45 has been approved by petrochenkov

petrochenkov · 2019-01-13T11:08:52Z

@bors r-

petrochenkov · 2019-01-13T11:10:13Z

Also, it would be nice to make a perf run once #57004 is merged and this PR contains only the following diff.

bors · 2019-01-13T19:59:02Z

☔ The latest upstream changes (presumably #57577) made this pull request unmergeable. Please resolve the merge conflicts.

`TokenStream::Stream` can represent a token stream containing any number of token trees. `TokenStream::Tree` is the special case representing a single token tree. The latter doesn't occur all that often dynamically, so this commit removes it, which simplifies the code quite a bit. This change has mixed performance effects. - The size of `TokenStream` drops from 32 bytes to 8 bytes, and there is one less case for all the match statements. - The conversion of a `TokenTree` to a `TokenStream` now requires two allocations, for the creation of a single element Lrc<Vec<_>>. (But a subsequent commit in this PR will reduce the main source of such conversions.)

`TokenStream` is now almost identical to `ThinTokenStream`. This commit removes the latter, replacing it with the former.

This avoids some allocations.

Because that's the more typical way of representing an all-or-nothing type.

nnethercote · 2019-01-14T00:06:46Z

I updated is_empty and added a comment about empty streams not necessarily being represented with Empty.

nnethercote · 2019-01-14T00:06:51Z

@bors try

bors · 2019-01-14T00:07:07Z

⌛ Trying commit 7285724 with merge ad694da89a85d9e7fa5f5eee94b1159f186c2778...

petrochenkov · 2019-01-14T09:50:58Z

Travis says the build completed successfully, but this wasn't reported to this thread for some reason.
Let's see if rust-timer works with it or not.

@rust-timer build ad694da89a85d9e7fa5f5eee94b1159f186c2778

rust-timer · 2019-01-14T09:50:59Z

Insufficient permissions to issue commands to rust-timer.

petrochenkov · 2019-01-14T17:30:28Z

@bors r-

petrochenkov · 2019-01-14T17:32:40Z

(The "comparison URL" didn't work for me previously and showed "missing commit" or something like that, but now it seems ok.)

@bors r+

bors · 2019-01-14T17:32:41Z

📌 Commit 7285724 has been approved by petrochenkov

@petrochenkov

…re, r=petrochenkov Simplify `TokenStream` some more These commits simplify `TokenStream`, remove `ThinTokenStream`, and avoid some clones. The end result is simpler code and a slight perf win on some benchmarks. r? @petrochenkov

Centril · 2019-01-15T09:03:21Z

Failed in rollup #57624 (comment) due to toolstate change.

@bors r-

petrochenkov · 2019-01-15T09:18:57Z

Marking as blocked, we'll be able to merge this in few days when 1.32 is released and the pre-release toolstate restriction is lifted.

nnethercote · 2019-01-15T09:22:03Z

What is toolstate?

petrochenkov · 2019-01-15T09:24:18Z

@nnethercote
Build status of things like clippy/rustfmt/... that depend on compiler internals.
The restriction is that compiler changes must not break tools during the week before release.

nnethercote · 2019-01-17T23:04:18Z

Rust 1.32 has been released.

@bors r=petrochenkov

bors · 2019-01-17T23:04:19Z

📌 Commit 7285724 has been approved by petrochenkov

@petrochenkov

…re, r=petrochenkov Simplify `TokenStream` some more These commits simplify `TokenStream`, remove `ThinTokenStream`, and avoid some clones. The end result is simpler code and a slight perf win on some benchmarks. r? @petrochenkov

@ghost

Rollup of 7 pull requests Successful merges: - #57486 (Simplify `TokenStream` some more) - #57502 (make trait-aliases work across crates) - #57598 (Add missing unpretty option help message) - #57649 (privacy: Account for associated existential types) - #57659 (Fix release manifest generation) - #57699 (add applicability to remaining suggestions) - #57719 (Tweak `expand_node`) Failed merges: r? @ghost

rust-highfive assigned petrochenkov Jan 10, 2019

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jan 10, 2019

petrochenkov reviewed Jan 10, 2019

View reviewed changes

src/libsyntax/tokenstream.rs Outdated Show resolved Hide resolved

petrochenkov reviewed Jan 10, 2019

View reviewed changes

src/libsyntax/parse/parser.rs Show resolved Hide resolved

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 10, 2019

petrochenkov reviewed Jan 11, 2019

View reviewed changes

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jan 11, 2019

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jan 13, 2019

nnethercote added 4 commits January 14, 2019 09:10

Remove ThinTokenStream.

ce0d994

`TokenStream` is now almost identical to `ThinTokenStream`. This commit removes the latter, replacing it with the former.

Avoid some TokenTree-to-TokenStream conversions.

ba31d83

This avoids some allocations.

Make TokenStream use Option.

7285724

Because that's the more typical way of representing an all-or-nothing type.

nnethercote force-pushed the simplify-TokenStream-more branch from ea2784b to 7285724 Compare January 14, 2019 00:06

petrochenkov added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jan 14, 2019

petrochenkov closed this Jan 14, 2019

petrochenkov reopened this Jan 14, 2019

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jan 14, 2019

Centril mentioned this pull request Jan 15, 2019

Rollup of 8 pull requests #57624

Closed

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jan 15, 2019

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-blocked Status: Marked as blocked ❌ on something else such as an RFC or other implementation work. labels Jan 17, 2019

Centril mentioned this pull request Jan 19, 2019

Rollup of 7 pull requests #57755

Merged

bors merged commit 7285724 into rust-lang:master Jan 19, 2019

nnethercote deleted the simplify-TokenStream-more branch January 21, 2019 20:41

nnethercote mentioned this pull request Jan 30, 2019

proc_macro: make TokenStream::from_streams pre-allocate its vector. #58001

Merged

nnethercote mentioned this pull request Feb 15, 2019

Remove LazyTokenStream. #58476

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Simplify `TokenStream` some more #57486

Simplify `TokenStream` some more #57486

nnethercote commented Jan 10, 2019

nnethercote commented Jan 10, 2019

nnethercote commented Jan 11, 2019

petrochenkov Jan 11, 2019

nnethercote Jan 11, 2019

petrochenkov Jan 13, 2019

petrochenkov commented Jan 11, 2019

bors commented Jan 11, 2019

petrochenkov commented Jan 13, 2019

petrochenkov commented Jan 13, 2019

bors commented Jan 13, 2019

nnethercote commented Jan 14, 2019

nnethercote commented Jan 14, 2019

bors commented Jan 14, 2019

petrochenkov commented Jan 14, 2019

rust-timer commented Jan 14, 2019

petrochenkov commented Jan 14, 2019

petrochenkov commented Jan 14, 2019

bors commented Jan 14, 2019

Centril commented Jan 15, 2019

petrochenkov commented Jan 15, 2019 •

edited

nnethercote commented Jan 15, 2019

petrochenkov commented Jan 15, 2019

nnethercote commented Jan 17, 2019

bors commented Jan 17, 2019

Simplify TokenStream some more #57486

Simplify TokenStream some more #57486

Conversation

nnethercote commented Jan 10, 2019

nnethercote commented Jan 10, 2019

nnethercote commented Jan 11, 2019

petrochenkov Jan 11, 2019

Choose a reason for hiding this comment

nnethercote Jan 11, 2019

Choose a reason for hiding this comment

petrochenkov Jan 13, 2019

Choose a reason for hiding this comment

petrochenkov commented Jan 11, 2019

bors commented Jan 11, 2019

petrochenkov commented Jan 13, 2019

petrochenkov commented Jan 13, 2019

bors commented Jan 13, 2019

nnethercote commented Jan 14, 2019

nnethercote commented Jan 14, 2019

bors commented Jan 14, 2019

petrochenkov commented Jan 14, 2019

rust-timer commented Jan 14, 2019

petrochenkov commented Jan 14, 2019

petrochenkov commented Jan 14, 2019

bors commented Jan 14, 2019

Centril commented Jan 15, 2019

petrochenkov commented Jan 15, 2019 • edited

nnethercote commented Jan 15, 2019

petrochenkov commented Jan 15, 2019

nnethercote commented Jan 17, 2019

bors commented Jan 17, 2019

Simplify `TokenStream` some more #57486

Simplify `TokenStream` some more #57486

petrochenkov commented Jan 15, 2019 •

edited