buffer: search utf16le at every byte offset - #64917
Open
inoway46 wants to merge 1 commit into
Open
Conversation
inoway46
commented
Aug 1, 2026
|
|
||
| { | ||
| // Test usc2 and utf16le encoding | ||
| // Test ucs2 and utf16le encodings. |
Contributor
Author
There was a problem hiding this comment.
Fix typo (usc2 → ucs2) and wording.
inoway46
force-pushed
the
fix-utf16le-indexof
branch
from
August 1, 2026 16:57
a9308a3 to
dbe8930
Compare
inoway46
marked this pull request as ready for review
August 1, 2026 17:02
UTF-16LE searches treated Buffer contents as 16-bit values. This rounded odd offsets down and ignored matches at odd byte positions, even though a Buffer can contain UTF-16LE data beginning at any byte offset. Search encoded string needles byte-by-byte at every offset. Buffer and Uint8Array needles are now compared byte-for-byte in their entirety. This is a breaking change because a match spanning UTF-16 code unit boundaries can be returned before an aligned match. UCS2 searches using Buffer or Uint8Array needles can also return matches that were previously ignored or truncated. Signed-off-by: inoway46 <inoueyuya416@gmail.com>
inoway46
force-pushed
the
fix-utf16le-indexof
branch
from
August 1, 2026 17:11
dbe8930 to
d033ea6
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #64917 +/- ##
=======================================
Coverage 90.29% 90.30%
=======================================
Files 760 760
Lines 247061 247046 -15
Branches 46585 46580 -5
=======================================
- Hits 223092 223088 -4
+ Misses 15451 15447 -4
+ Partials 8518 8511 -7
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes: #26448
Refs: #26594
Background
UTF-16LE/UCS2 searches currently treat the
Bufferas an array of 16-bitvalues. This rounds odd byte offsets down and only searches positions aligned
to the start of the
Buffer.For example:
This change encodes string needles using the requested encoding and compares
the resulting bytes at every byte offset.
BufferandUint8Arrayneedles arecompared byte-for-byte in their entirety. It applies to the shared
implementation used by
indexOf(),lastIndexOf(), andincludes().The
Buffer#indexOf()documentationdefines
encodingas determining the binary representation of string needles,while
BufferandUint8Arrayneedles are used in their entirety. This changealso makes their implementation consistent with that contract.
Compatibility
This is a semver-major behavior change.
A UTF-16LE byte sequence can now match at an unaligned byte offset, crossing
code unit boundaries even when the search starts at an even offset. Such a
match may be returned before an aligned match.
BufferandUint8Arrayneedles are now compared as complete byte sequences.This changes the previous behavior for one-byte and odd-length needles when a
UTF-16LE/UCS2 encoding argument is supplied.
Byte-wise UTF-16LE searching may affect performance compared to the previous
16-bit implementation. The big-endian decoding path should also be confirmed
by CI.
Testing