Stop the lexer reading past the end of a byte_range - #3040
Open
ksss wants to merge 1 commit into
Open
Conversation
4 tasks
`rbs_next_char` ends the input when `byte_pos == end_pos`. `rbs_skip` advances by a whole character, so a multibyte character starting before `end_pos` and ending after it steps over the boundary and equality never holds again — the lexer then reads to the end of the string. The realistic way to land inside a character is to pass a character offset where a byte offset is expected, the mistake ruby#2945 fixed in `parse_inline_*_annotation`. `"日本語"` is 5 characters but 11 bytes, so offset 5 falls inside `本`: Parser.parse_type('"日本語" | Integer', byte_range: 0...5) #=> Types::Union spanning the whole input, rather than an error `require_eof: true` does not catch it, because the lexer really is at EOF by then. It needs a character to straddle the boundary, so ASCII-only input never hits it. Compare with `>=` so stepping over the boundary still ends the input.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
rbs_next_charends the input whenbyte_pos == end_pos.rbs_skipadvances by a whole character, so a multibyte character starting beforeend_posand ending after it steps over the boundary — equality never holds again, and the lexer reads to the end of the string.The realistic way to land inside a character is to pass a character offset where a byte offset is expected — the mistake #2945 fixed in
parse_inline_*_annotation."日本語"is 5 characters but 11 bytes, so offset 5 falls inside本, which spans bytes 4...7:require_eof: truedoes not catch it, because by then the lexer really is at EOF.It takes a character straddling the boundary, so ASCII-only input never hits it. On
mastertoday it also needs that character to sit somewhere non-ASCII is already accepted — a string literal or a comment — since a non-ASCII byte in an identifier stops at anErrorTokenfirst. #3028 lets identifiers hold non-ASCII, which widens the set of inputs that reach it, but the bug is independent of that PR.Fix
Compare with
>=so stepping over the boundary still ends the input.That leaves the straddling character itself read whole, one character past the range. On
masterthat is not observable — every case I could construct still ends in a parse error — so nothing more is done here. It becomes observable once an identifier can hold non-ASCII, since an identifier is a complete type on its own; #3028 tightens the bound there, where a test can cover it.Test plan
rake test— 968 tests, 0 failures. Two errors also present onmaster, unrelated.test_parse__byte_range_ending_mid_character, confirmed to fail with the==comparison restored. Covers the character-offset mistake above, the byte offset the caller meant, and anend_pospast the end of the buffer.