Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
Move token tree related lexer state to a separate struct #60763
Turns out it is possible to move a whole bunch of stuff to tt-specifc code.
Previously, StringReader played a dual role of both producer and consumer of tokens. Now consumption and peeking logic is fully within
If the logic for joint-ness checking looks strange it is because it was, in fact, pretty non-obvious. See https://github.com/rust-lang/rust/pull/50838/files#r283296243