Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move token tree related lexer state to a separate struct #60763

Merged
merged 3 commits into from May 16, 2019

Conversation

Projects
None yet
5 participants
@matklad
Copy link
Member

commented May 12, 2019

Just a types-based refactoring.

We only used a bunch of fields when tokenizing into a token tree, so let's move them out of the base lexer

@matklad

This comment has been minimized.

Copy link
Member Author

commented May 12, 2019

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented May 12, 2019

r? @pnkfelix

(rust_highfive has picked a reviewer for you, use r? to override)

@matklad matklad force-pushed the matklad:tt-parser branch 3 times, most recently from 248b837 to 64d1066 May 13, 2019

@matklad

This comment has been minimized.

Copy link
Member Author

commented May 13, 2019

Turns out it is possible to move a whole bunch of stuff to tt-specifc code.

Previously, StringReader played a dual role of both producer and consumer of tokens. Now consumption and peeking logic is fully within TokenTreesReader.

If the logic for joint-ness checking looks strange it is because it was, in fact, pretty non-obvious. See https://github.com/rust-lang/rust/pull/50838/files#r283296243

@matklad matklad force-pushed the matklad:tt-parser branch from 64d1066 to ab90da9 May 13, 2019

matklad added some commits May 12, 2019

Move token tree related lexer state to a separate struct
We only used a bunch of fields when tokenizing into a token tree,
so let's move them out of the base lexer
move raw span to tt reader
See https://github.com/rust-lang/rust/pull/50838/files#r283296243 for
explanation how jointness checking works with *next* pair

@matklad matklad force-pushed the matklad:tt-parser branch from ab90da9 to e249f2e May 13, 2019

@petrochenkov

This comment has been minimized.

Copy link
Contributor

commented May 14, 2019

@bors r+

@bors

This comment has been minimized.

Copy link
Contributor

commented May 14, 2019

📌 Commit e249f2e has been approved by petrochenkov

Centril added a commit to Centril/rust that referenced this pull request May 15, 2019

Rollup merge of rust-lang#60763 - matklad:tt-parser, r=petrochenkov
Move token tree related lexer state to a separate struct

Just a types-based refactoring.

We only used a bunch of fields when tokenizing into a token tree, so let's move them out of the base lexer

Centril added a commit to Centril/rust that referenced this pull request May 15, 2019

Rollup merge of rust-lang#60763 - matklad:tt-parser, r=petrochenkov
Move token tree related lexer state to a separate struct

Just a types-based refactoring.

We only used a bunch of fields when tokenizing into a token tree, so let's move them out of the base lexer

bors added a commit that referenced this pull request May 15, 2019

Auto merge of #60844 - Centril:rollup-1g9brpm, r=Centril
Rollup of 6 pull requests

Successful merges:

 - #59276 (Cleanup (pretty) printing of `ty::Const`)
 - #60174 (Add match arm scopes and other scope fixes)
 - #60693 (refactor some `FnType` stuff to `rustc::ty::layout`)
 - #60763 (Move token tree related lexer state to a separate struct)
 - #60811 (Bump measureme dependency to 0.3)
 - #60832 (CMake: Do not print installation messages for up-to-date files)

Failed merges:

 - #60827 (Use `Symbol` more in lint APIs)

r? @ghost

Centril added a commit to Centril/rust that referenced this pull request May 15, 2019

Rollup merge of rust-lang#60763 - matklad:tt-parser, r=petrochenkov
Move token tree related lexer state to a separate struct

Just a types-based refactoring.

We only used a bunch of fields when tokenizing into a token tree, so let's move them out of the base lexer

bors added a commit that referenced this pull request May 15, 2019

Auto merge of #60859 - Centril:rollup-x7rcz9c, r=Centril
Rollup of 9 pull requests

Successful merges:

 - #59825 (string: implement From<&String> for String)
 - #59923 (Fix convert module's documentation links)
 - #60691 (Include expression to wait for to the span of Await)
 - #60763 (Move token tree related lexer state to a separate struct)
 - #60769 (Update rustc book CLI docs.)
 - #60811 (Bump measureme dependency to 0.3)
 - #60816 (README.md: Mention MSVC 2017+, not 2013(!))
 - #60827 (Use `Symbol` more in lint APIs)
 - #60851 (Move `box` from the stable keyword to unstable keywords list)

Failed merges:

r? @ghost

Centril added a commit to Centril/rust that referenced this pull request May 15, 2019

Rollup merge of rust-lang#60763 - matklad:tt-parser, r=petrochenkov
Move token tree related lexer state to a separate struct

Just a types-based refactoring.

We only used a bunch of fields when tokenizing into a token tree, so let's move them out of the base lexer

bors added a commit that referenced this pull request May 15, 2019

Auto merge of #60864 - Centril:rollup-l9jnv2h, r=Centril
Rollup of 8 pull requests

Successful merges:

 - #59825 (string: implement From<&String> for String)
 - #59923 (Fix convert module's documentation links)
 - #60691 (Include expression to wait for to the span of Await)
 - #60763 (Move token tree related lexer state to a separate struct)
 - #60769 (Update rustc book CLI docs.)
 - #60811 (Bump measureme dependency to 0.3)
 - #60816 (README.md: Mention MSVC 2017+, not 2013(!))
 - #60851 (Move `box` from the stable keyword to unstable keywords list)

Failed merges:

r? @ghost
@bors

This comment has been minimized.

Copy link
Contributor

commented May 16, 2019

⌛️ Testing commit e249f2e with merge 024c25d...

bors added a commit that referenced this pull request May 16, 2019

Auto merge of #60763 - matklad:tt-parser, r=petrochenkov
Move token tree related lexer state to a separate struct

Just a types-based refactoring.

We only used a bunch of fields when tokenizing into a token tree, so let's move them out of the base lexer
@bors

This comment has been minimized.

Copy link
Contributor

commented May 16, 2019

☀️ Test successful - checks-travis, status-appveyor
Approved by: petrochenkov
Pushing 024c25d to master...

@bors bors added the merged-by-bors label May 16, 2019

@bors bors merged commit e249f2e into rust-lang:master May 16, 2019

2 checks passed

Travis CI - Pull Request Build Passed
Details
homu Test successful
Details

@matklad matklad deleted the matklad:tt-parser branch May 16, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.