Remove support for 1-token lookahead from the lexer #62329
Conversation
|
cc @GuillaumeGomez for highlight.rs change I guess (first commit) |
| self.real_token(); | ||
| let is_joint = raw.hi() == self.string_reader.peek_span_src_raw.lo() | ||
| && self.token.is_op(); | ||
| let is_joint = self.joint_to_prev == Joint && self.token.is_op(); |
There was a problem hiding this comment.
self.token.is_op() makes me think that perhaps we should check that the previous token is also an op?
That is, we currently can say Join for an identifier followed by (, for example.
There was a problem hiding this comment.
All the observable behavior is represented by the Spacing enum returned by Punct::spacing.
It has pretty specific documentation - we are "revealing" the jointness knowledge only for Punct + Punct pairs and additionally for Punct(') + Ident (for lifetimes).
Any other jointness is not revealed, Ident + ( in particular is not revealed because ( is not is_op, so the implementation looks correct.
Perhaps we should do this hiding entirely at the proc macro interface border though, and keep the full knowledge internally.
There was a problem hiding this comment.
in particular is not revealed because ( is not is_op, so the implementation looks correct.
Ah, indeed, it needs to be Ident-, for example.
Yeah, I think we should either handle this completely on proc-macro layer, or completely in the TokenTreesReader, but this is unrelated to PR at hand
matklad
force-pushed
the
no-peeking
branch
3 times, most recently
from
8b2cfd2
to
f1f8def
Compare
|
@bors r+ |
|
📌 Commit f1f8def5e9f1ee88222a2dce3d4da007e0350556 has been approved by |
bors
added
the
S-waiting-on-bors
|
@bors r=petrochenkov |
|
📌 Commit 07a9e4dcd2a518f7916a9db0487a1b950fa50e01 has been approved by |
|
☔ The latest upstream changes (presumably #62355) made this pull request unmergeable. Please resolve the merge conflicts. |
bors
added
S-waiting-on-author
The reader itself doesn't need ability to peek tokens, so it's better if clients implement this functionality. This hopefully becomes especially easy once we use iterator interface for lexer, but this is not too easy at the moment, because of buffered errors.
|
@bors r=petrochenkov |
|
📌 Commit 3e362a4 has been approved by |
bors
added
S-waiting-on-bors
Remove support for 1-token lookahead from the lexer `StringReader` maintained `peek_token` and `peek_span_src_raw` for look ahead. `peek_token` was used only by rustdoc syntax coloring. After moving peeking logic into highlighter, I was able to remove `peek_token` from the lexer. I tried to use `iter::Peekable`, but that wasn't as pretty as I hoped, due to buffered fatal errors. So I went with hand-rolled peeking. After that I've noticed that the only peeking behavior left was for raw tokens to test tt jointness. I've rewritten it in terms of trivia tokens, and not just spans. After that it became possible to simplify the awkward constructor of the lexer, which could return `Err` if the first peeked token contained error.
|
A bit late but looks good to me as well (for the rustdoc part at least). |
Remove support for 1-token lookahead from the lexer `StringReader` maintained `peek_token` and `peek_span_src_raw` for look ahead. `peek_token` was used only by rustdoc syntax coloring. After moving peeking logic into highlighter, I was able to remove `peek_token` from the lexer. I tried to use `iter::Peekable`, but that wasn't as pretty as I hoped, due to buffered fatal errors. So I went with hand-rolled peeking. After that I've noticed that the only peeking behavior left was for raw tokens to test tt jointness. I've rewritten it in terms of trivia tokens, and not just spans. After that it became possible to simplify the awkward constructor of the lexer, which could return `Err` if the first peeked token contained error.
Rollup of 8 pull requests Successful merges: - #60260 (Add support for UWP targets) - #62151 (Update linked OpenSSL version) - #62245 (Miri engine: support extra function (pointer) values) - #62257 (forward read_c_str method from Memory to Alloc) - #62264 (Fix perf regression from Miri Machine trait changes) - #62296 (request at least ptr-size alignment from posix_memalign) - #62329 (Remove support for 1-token lookahead from the lexer) - #62377 (Add test for ICE #62375) Failed merges: r? @ghost
Remove support for 1-token lookahead from the lexer `StringReader` maintained `peek_token` and `peek_span_src_raw` for look ahead. `peek_token` was used only by rustdoc syntax coloring. After moving peeking logic into highlighter, I was able to remove `peek_token` from the lexer. I tried to use `iter::Peekable`, but that wasn't as pretty as I hoped, due to buffered fatal errors. So I went with hand-rolled peeking. After that I've noticed that the only peeking behavior left was for raw tokens to test tt jointness. I've rewritten it in terms of trivia tokens, and not just spans. After that it became possible to simplify the awkward constructor of the lexer, which could return `Err` if the first peeked token contained error.
Rollup of 7 pull requests Successful merges: - #62151 (Update linked OpenSSL version) - #62245 (Miri engine: support extra function (pointer) values) - #62257 (forward read_c_str method from Memory to Alloc) - #62264 (Fix perf regression from Miri Machine trait changes) - #62296 (request at least ptr-size alignment from posix_memalign) - #62329 (Remove support for 1-token lookahead from the lexer) - #62377 (Add test for ICE #62375) Failed merges: r? @ghost
StringReadermaintainedpeek_tokenandpeek_span_src_rawfor look ahead.peek_tokenwas used only by rustdoc syntax coloring. After moving peeking logic into highlighter, I was able to removepeek_tokenfrom the lexer. I tried to useiter::Peekable, but that wasn't as pretty as I hoped, due to buffered fatal errors. So I went with hand-rolled peeking.After that I've noticed that the only peeking behavior left was for raw tokens to test tt jointness. I've rewritten it in terms of trivia tokens, and not just spans.
After that it became possible to simplify the awkward constructor of the lexer, which could return
Errif the first peeked token contained error.