Always parse as UTF-8. #270
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dvander
force-pushed
the
split-headers-p2
branch
2 times, most recently
from
ee26bb3
to
9caa869
Compare
This is a ~13% compile time improvement on SourceMod plugins. Mysteriously, it wasn't enough to parse three times: the compiler also pre-scans the entire text of source/include files to determine if each uses UTF-8. This is particularly mysterious as the result of pre-scanning for UTF-8 is that it will allow parsing more UTF-8 characters. Why bother scanning in the first place? One reason is to remove the byte order mark (BOM), which this patch retains by scanning the first three bytes. Otherwise, it seems this code can be removed. It's possible upstream had more dependent logic and we removed it long ago. This patch also fixes a long-standing bug where only include files had BOM correctly skipped, and an additional buffer overflow when parsing BOMs.
This argument doesn't really make any sense. Anything at the top of source files intended to be ignored, and not a comment, would be non-standard.
dvander
force-pushed
the
split-headers-p2
branch
from
9caa869
to
5a0bd34
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a ~13% compile time improvement on SourceMod plugins.
Mysteriously, it wasn't enough to parse three times: the compiler
also pre-scans the entire text of source/include files to determine
if each uses UTF-8. This is particularly mysterious as the result
of pre-scanning for UTF-8 is that it will allow parsing more UTF-8
characters. Why bother scanning in the first place?
One reason is to remove the byte order mark (BOM), which this patch
retains by scanning the first three bytes. Otherwise, it seems this code
can be removed. It's possible upstream had more dependent logic and we
removed it long ago.