Make `TokenStream` less recursive. #57004

nnethercote · 2018-12-20T09:12:34Z

TokenStream is currently recursive in two ways:

the TokenTree variant contains a ThinTokenStream, which can
contain a TokenStream;
the TokenStream variant contains a Vec<TokenStream>.

The latter is not necessary and causes significant complexity. This
commit replaces it with the simpler Vec<(TokenTree, IsJoint)>.

This reduces complexity significantly. In particular, StreamCursor is
eliminated, and Cursor becomes much simpler, consisting now of just a
TokenStream and an index.

The commit also removes the Extend impl for TokenStream, because it
is only used in tests. (The commit also removes those tests.)

Overall, the commit reduces the number of lines of code by almost 200.

rust-highfive · 2018-12-20T10:53:58Z

The job mingw-check of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_time:end:054da1d2:start=1545297245717990325,finish=1545297303072910640,duration=57354920315
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=mingw-check
---
[00:05:51] configure: build.locked-deps    := True
[00:05:51] configure: llvm.ccache          := sccache
[00:05:51] configure: build.cargo-native-static := True
[00:05:51] configure: dist.missing-tools   := True
[00:05:51] configure: build.configure-args := ['--enable-sccache', '--disable-manage-submodu ...
[00:05:51] configure: writing `config.toml` in current directory
[00:05:51] configure: 
[00:05:51] configure: run `python /checkout/x.py --help`
[00:05:51] configure: 
---
[00:08:15]     Checking rustc_errors v0.0.0 (/checkout/src/librustc_errors)
[00:08:18] error: unused import: `mem`
[00:08:18]   --> src/libsyntax/tokenstream.rs:35:22
[00:08:18]    |
[00:08:18] 35 | use std::{fmt, iter, mem};
[00:08:18]    |
[00:08:18]    = note: `-D unused-imports` implied by `-D warnings`
[00:08:18] 
[00:08:23] error: aborting due to previous error
[00:08:23] error: aborting due to previous error
[00:08:23] 
[00:08:23] error: Could not compile `syntax`.
[00:08:23] 
[00:08:23] To learn more, run the command again with --verbose.
[00:08:23] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "check" "--target" "i686-pc-windows-gnu" "-j" "4" "--release" "--locked" "--color" "always" "--features" "" "--manifest-path" "/checkout/src/rustc/Cargo.toml" "--message-format" "json"
[00:08:23] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap check --target=i686-pc-windows-gnu --host=i686-pc-windows-gnu
[00:08:23] Build completed unsuccessfully in 0:01:21
travis_time:end:0015db30:start=1545297311720268660,finish=1545297815425529278,duration=503705260618
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 1.
---
travis_time:end:0bb6472a:start=1545297815908039571,finish=1545297815914791154,duration=6751583
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:111da833
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:1a7ae108
travis_time:start:1a7ae108
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:062b8bd3
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

bors · 2018-12-24T10:11:25Z

☔ The latest upstream changes (presumably #57087) made this pull request unmergeable. Please resolve the merge conflicts.

`TokenStream` is currently recursive in *two* ways: - the `TokenTree` variant contains a `ThinTokenStream`, which can contain a `TokenStream`; - the `TokenStream` variant contains a `Vec<TokenStream>`. The latter is not necessary and causes significant complexity. This commit replaces it with the simpler `Vec<(TokenTree, IsJoint)>`. This reduces complexity significantly. In particular, `StreamCursor` is eliminated, and `Cursor` becomes much simpler, consisting now of just a `TokenStream` and an index. The commit also removes the `Extend` impl for `TokenStream`, because it is only used in tests. (The commit also removes those tests.) Overall, the commit reduces the number of lines of code by almost 200.

nnethercote · 2019-01-08T04:10:13Z

New code is ready for review.

r? @eddyb

petrochenkov · 2019-01-08T10:49:32Z

(eddyb is absent, reassigning to me.)

petrochenkov · 2019-01-08T12:33:50Z

A bit of a history:

Originally token streams used simple Vec<TokenTree> (Introducing TokenStreams and TokenSlices for procedural macros #34575).
Then they were reimplemented as ropes as an optimization (Reimplement TokenStreams using ropes #35018), apparently to increase sharing, but this also introduced recursive token streams and complexity. It looks like no benchmarking was done for that change.
Then ropes were replaced with RcSlice<TokenStream> (Refactor TokenStream #39173), simplifying the code, but keeping the recursive token streams. Looks like no benchmarking was done for that change either.
Then RcSlice turned into RcVec (TokenStream::extend #53304), and then into Lrc<Vec> (TokenStream improvements #56737).
Now this PR completes the cycle and returns to the original implementation. So, we shouldn't repeat the past errors and at least benchmark it 😄

petrochenkov · 2019-01-08T12:49:00Z

@bors try

bors · 2019-01-08T12:49:14Z

⌛ Trying commit e80a930 with merge 7346647...

Make `TokenStream` less recursive. `TokenStream` is currently recursive in *two* ways: - the `TokenTree` variant contains a `ThinTokenStream`, which can contain a `TokenStream`; - the `TokenStream` variant contains a `Vec<TokenStream>`. The latter is not necessary and causes significant complexity. This commit replaces it with the simpler `Vec<(TokenTree, IsJoint)>`. This reduces complexity significantly. In particular, `StreamCursor` is eliminated, and `Cursor` becomes much simpler, consisting now of just a `TokenStream` and an index. The commit also removes the `Extend` impl for `TokenStream`, because it is only used in tests. (The commit also removes those tests.) Overall, the commit reduces the number of lines of code by almost 200.

bors · 2019-01-08T15:01:47Z

☀️ Test successful - status-travis
State: approved= try=True

petrochenkov · 2019-01-08T15:18:54Z

@rust-timer build 7346647

rust-timer · 2019-01-08T15:18:57Z

Insufficient permissions to issue commands to rust-timer.

pietroalbini · 2019-01-08T15:26:52Z

@rust-timer build 7346647

rust-timer · 2019-01-08T15:26:54Z

Success: Queued 7346647 with parent 7ad470c, comparison URL.

rust-timer · 2019-01-08T18:19:02Z

Finished benchmarking try commit 7346647

nnethercote · 2019-01-08T21:16:44Z

The performance results are basically a wash: some minor improvements, some minor regressions. (Which matches what I saw locally.)

But note that the regressions are almost entirely within the short-running (and less interesting) benchmarks: issue-46449, regression-31157, unify-linearly, deeply-nested, unused-warnings, helloworld. The "real world" benchmarks were largely unaffected or slightly improved.

petrochenkov · 2019-01-08T21:22:39Z

I'd be entirely happy to merge this simplification if performance is unaffected.

Since @cgswords is no longer active, pinging @nrc who approved #35018, please r- if you have concerns.

@bors r+

bors · 2019-01-08T21:22:40Z

📌 Commit e80a930 has been approved by petrochenkov

bors · 2019-01-09T01:40:24Z

⌛ Testing commit e80a930 with merge bd981cc...

Make `TokenStream` less recursive. `TokenStream` is currently recursive in *two* ways: - the `TokenTree` variant contains a `ThinTokenStream`, which can contain a `TokenStream`; - the `TokenStream` variant contains a `Vec<TokenStream>`. The latter is not necessary and causes significant complexity. This commit replaces it with the simpler `Vec<(TokenTree, IsJoint)>`. This reduces complexity significantly. In particular, `StreamCursor` is eliminated, and `Cursor` becomes much simpler, consisting now of just a `TokenStream` and an index. The commit also removes the `Extend` impl for `TokenStream`, because it is only used in tests. (The commit also removes those tests.) Overall, the commit reduces the number of lines of code by almost 200.

bors · 2019-01-09T05:40:24Z

💥 Test timed out

bors · 2019-01-10T06:01:10Z

⌛ Testing commit e80a930 with merge 8090a9c09bf365105ef844d61a2d69124e2c56fa...

bors · 2019-01-10T06:01:29Z

💔 Test failed - status-appveyor

nnethercote · 2019-01-10T06:05:23Z

broken_heart Test failed - status-appveyor

No idea about that, but it seems spurious, so...

@bors retry

bors · 2019-01-10T06:05:54Z

⌛ Testing commit e80a930 with merge 7062e23180480c2f15a1f27a6d444b76b0419bd9...

bors · 2019-01-10T06:06:06Z

💔 Test failed - status-appveyor

pietroalbini · 2019-01-10T08:17:16Z

@bors retry
AppVeyor... what's wrong with you today?

…rochenkov Make `TokenStream` less recursive. `TokenStream` is currently recursive in *two* ways: - the `TokenTree` variant contains a `ThinTokenStream`, which can contain a `TokenStream`; - the `TokenStream` variant contains a `Vec<TokenStream>`. The latter is not necessary and causes significant complexity. This commit replaces it with the simpler `Vec<(TokenTree, IsJoint)>`. This reduces complexity significantly. In particular, `StreamCursor` is eliminated, and `Cursor` becomes much simpler, consisting now of just a `TokenStream` and an index. The commit also removes the `Extend` impl for `TokenStream`, because it is only used in tests. (The commit also removes those tests.) Overall, the commit reduces the number of lines of code by almost 200.

@ghost

Rollup of 4 pull requests Successful merges: - #57004 (Make `TokenStream` less recursive.) - #57102 (NLL: Add union justifications to conflicting borrows.) - #57337 (rustc: Place wasm linker args first instead of last) - #57549 (Add #[must_use] message to Iterator and Future) Failed merges: r? @ghost

rust-highfive assigned eddyb Dec 20, 2018

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Dec 20, 2018

nnethercote force-pushed the TS-change-Stream branch from e2e57bb to facc038 Compare January 8, 2019 04:06

nnethercote force-pushed the TS-change-Stream branch from facc038 to e80a930 Compare January 8, 2019 04:09

nnethercote changed the title ~~De-recursify TokenStream~~ Make TokenStream less recursive. Jan 8, 2019

petrochenkov assigned petrochenkov and unassigned eddyb Jan 8, 2019

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 8, 2019

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jan 10, 2019

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 10, 2019

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jan 10, 2019

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 10, 2019

Centril mentioned this pull request Jan 13, 2019

Rollup of 7 pull requests #57574

Closed

Centril mentioned this pull request Jan 13, 2019

Rollup of 6 pull requests #57575

Closed

Centril mentioned this pull request Jan 13, 2019

Rollup of 5 pull requests #57576

Closed

Centril mentioned this pull request Jan 13, 2019

Rollup of 4 pull requests #57577

Merged

bors merged commit e80a930 into rust-lang:master Jan 13, 2019

nnethercote deleted the TS-change-Stream branch January 13, 2019 22:28

pnkfelix mentioned this pull request Jan 28, 2019

Project fails (or takes long time) to compile on latest stable and nightly (hang/stackoverflow) #57735

Closed

nnethercote mentioned this pull request Jan 30, 2019

proc_macro: make TokenStream::from_streams pre-allocate its vector. #58001

Merged

petrochenkov mentioned this pull request Mar 23, 2020

Store tokens alongside more AST expressions #70091

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make `TokenStream` less recursive. #57004

Make `TokenStream` less recursive. #57004

nnethercote commented Dec 20, 2018 •

edited

Loading

rust-highfive commented Dec 20, 2018

bors commented Dec 24, 2018

nnethercote commented Jan 8, 2019

petrochenkov commented Jan 8, 2019

petrochenkov commented Jan 8, 2019

petrochenkov commented Jan 8, 2019

bors commented Jan 8, 2019

bors commented Jan 8, 2019

petrochenkov commented Jan 8, 2019

rust-timer commented Jan 8, 2019

pietroalbini commented Jan 8, 2019

rust-timer commented Jan 8, 2019

rust-timer commented Jan 8, 2019

nnethercote commented Jan 8, 2019

petrochenkov commented Jan 8, 2019

bors commented Jan 8, 2019

bors commented Jan 9, 2019

bors commented Jan 9, 2019

bors commented Jan 10, 2019

bors commented Jan 10, 2019

nnethercote commented Jan 10, 2019

bors commented Jan 10, 2019

bors commented Jan 10, 2019

pietroalbini commented Jan 10, 2019

Make TokenStream less recursive. #57004

Make TokenStream less recursive. #57004

Conversation

nnethercote commented Dec 20, 2018 • edited Loading

rust-highfive commented Dec 20, 2018

bors commented Dec 24, 2018

nnethercote commented Jan 8, 2019

petrochenkov commented Jan 8, 2019

petrochenkov commented Jan 8, 2019

petrochenkov commented Jan 8, 2019

bors commented Jan 8, 2019

bors commented Jan 8, 2019

petrochenkov commented Jan 8, 2019

rust-timer commented Jan 8, 2019

pietroalbini commented Jan 8, 2019

rust-timer commented Jan 8, 2019

rust-timer commented Jan 8, 2019

nnethercote commented Jan 8, 2019

petrochenkov commented Jan 8, 2019

bors commented Jan 8, 2019

bors commented Jan 9, 2019

bors commented Jan 9, 2019

bors commented Jan 10, 2019

bors commented Jan 10, 2019

nnethercote commented Jan 10, 2019

bors commented Jan 10, 2019

bors commented Jan 10, 2019

pietroalbini commented Jan 10, 2019

Make `TokenStream` less recursive. #57004

Make `TokenStream` less recursive. #57004

nnethercote commented Dec 20, 2018 •

edited

Loading