parser.h Allow up to 256 characters in a token #23779
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This value is the maximum number of bytes required to represent in UTF-8 any code point in the legal Unicode range of 0 .. 0x10FFFF
This is already the claimed allowed length. But that is a lie, until this commit. Instead, the buffer has been 256 bytes long, which means, we can have 256 1-byte characters in an identifier; but only 128 2-byte ones, etc. Unicode can have 4-byte identifier characters, so our limit has really been just 64 for those. The direction perl is supposed to be going, according to perldiag, is to eliminate any identifier length limit. I don't feel the urge to do that now, but simply increasing the buffer size to accommodate any 256 Unicode identifier characters causes us to meet our claim. The trickiest part of this by far was to get parser.t to pass, which contrary to perldiag, tests very specifically about identifiers just shy of 256. One thing it does is to create a long string. I just replaced every character in it by 4 repeats, and then split into shorter lines.
tonycoz
approved these changes
Sep 30, 2025
khwilliamson
added a commit
to khwilliamson/perl5
that referenced
this pull request
Sep 30, 2025
Merged
khwilliamson
added a commit
to khwilliamson/perl5
that referenced
this pull request
Oct 1, 2025
khwilliamson
added a commit
to khwilliamson/perl5
that referenced
this pull request
Oct 1, 2025
khwilliamson
added a commit
that referenced
this pull request
Oct 1, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is already the claimed allowed length. But that is a lie, until this commit. Instead, a limiting buffer has been 256 bytes long, which means, we can have 256 1-byte characters in an identifier; but only 128 2-byte ones, etc. Unicode can have 4-byte identifier characters, so our limit has really been just 64 for those.
The direction perl is supposed to be going, according to perldiag, is to eliminate any identifier length limit. I don't feel the urge to do that now, but simply increasing the buffer size to accommodate any 256 Unicode identifier characters causes us to meet our current claim.
The trickiest part of this by far was to get parser.t to pass, which contrary to perldiag, tests very specifically about identifiers just shy of 256.