Skip to content

buffer: search utf16le at every byte offset - #64917

Open
inoway46 wants to merge 1 commit into
nodejs:mainfrom
inoway46:fix-utf16le-indexof
Open

buffer: search utf16le at every byte offset#64917
inoway46 wants to merge 1 commit into
nodejs:mainfrom
inoway46:fix-utf16le-indexof

Conversation

@inoway46

@inoway46 inoway46 commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Fixes: #26448
Refs: #26594

Background

UTF-16LE/UCS2 searches currently treat the Buffer as an array of 16-bit
values. This rounds odd byte offsets down and only searches positions aligned
to the start of the Buffer.

For example:

const buf = Buffer.from('\u6881\u6882\u6881', 'utf16le');

buf.indexOf('\u6881', 1, 'utf16le');
// Current: 0
// Expected: 4

This change encodes string needles using the requested encoding and compares
the resulting bytes at every byte offset. Buffer and Uint8Array needles are
compared byte-for-byte in their entirety. It applies to the shared
implementation used by indexOf(), lastIndexOf(), and includes().

The Buffer#indexOf() documentation
defines encoding as determining the binary representation of string needles,
while Buffer and Uint8Array needles are used in their entirety. This change
also makes their implementation consistent with that contract.

Compatibility

This is a semver-major behavior change.

A UTF-16LE byte sequence can now match at an unaligned byte offset, crossing
code unit boundaries even when the search starts at an even offset. Such a
match may be returned before an aligned match.

Buffer and Uint8Array needles are now compared as complete byte sequences.
This changes the previous behavior for one-byte and odd-length needles when a
UTF-16LE/UCS2 encoding argument is supplied.

Byte-wise UTF-16LE searching may affect performance compared to the previous
16-bit implementation. The big-endian decoding path should also be confirmed
by CI.

Testing

ninja -C out/Release node
./tools/test.py -p tap parallel/test-buffer-indexof parallel/test-buffer-includes
./tools/test.py -p tap parallel/test-buffer-generic-methods
make test-doc

@nodejs-github-bot nodejs-github-bot added buffer Issues and PRs related to the buffer subsystem. c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. labels Aug 1, 2026

{
// Test usc2 and utf16le encoding
// Test ucs2 and utf16le encodings.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix typo (usc2 → ucs2) and wording.

@inoway46
inoway46 force-pushed the fix-utf16le-indexof branch from a9308a3 to dbe8930 Compare August 1, 2026 16:57
@inoway46
inoway46 marked this pull request as ready for review August 1, 2026 17:02
UTF-16LE searches treated Buffer contents as 16-bit values. This
rounded odd offsets down and ignored matches at odd byte positions,
even though a Buffer can contain UTF-16LE data beginning at any byte
offset.

Search encoded string needles byte-by-byte at every offset.
Buffer and Uint8Array needles are now compared byte-for-byte in their
entirety.

This is a breaking change because a match spanning UTF-16 code unit
boundaries can be returned before an aligned match.
UCS2 searches using Buffer or Uint8Array needles can also return matches
that were previously ignored or truncated.

Signed-off-by: inoway46 <inoueyuya416@gmail.com>
@inoway46
inoway46 force-pushed the fix-utf16le-indexof branch from dbe8930 to d033ea6 Compare August 1, 2026 17:11
@codecov

codecov Bot commented Aug 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.30%. Comparing base (b9dacd4) to head (d033ea6).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main   #64917   +/-   ##
=======================================
  Coverage   90.29%   90.30%           
=======================================
  Files         760      760           
  Lines      247061   247046   -15     
  Branches    46585    46580    -5     
=======================================
- Hits       223092   223088    -4     
+ Misses      15451    15447    -4     
+ Partials     8518     8511    -7     
Files with missing lines Coverage Δ
src/node_buffer.cc 69.08% <100.00%> (-0.33%) ⬇️

... and 28 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

buffer Issues and PRs related to the buffer subsystem. c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

buffer.indexOf is incorrect in utf16le encoding for odd byteOffset

2 participants