fix(runtime): derive each split part's metadata from its own bytes (#6085 follow-up)

[proggeramlug]

Follow-up to #6286 (#6085). That PR was merged at 3525a31dd, one commit before this fix landed on the branch, so this review finding is not on main. It is a real correctness bug and is filed here on its own.

Root cause

js_string_split_n's ASCII fast path gated on is_ascii_string(s), which only compares byte_len == utf16_len over the whole source. Malformed bytes can satisfy that aggregate while the individual parts cannot — so the fast path stamped wrong utf16_len (and hardcoded flags = 0) onto every part.

This lands squarely in #6085's own threat model: a payload that is not well-formed UTF-8 is the entire reason that issue exists (FFI / Buffer blobs split on an ASCII delimiter).

Verified, not inferred

Instrumented js_string_split_n with [0x80, '|', 0xF0]:

SOURCE byte_len=3 utf16_len=3 is_ascii_string=true    <- the aggregate lies
  part[0] bytes=[80] byte_len=1 utf16_len(RECORDED)=1 utf16_len(CORRECT)=0
  part[1] bytes=[F0] byte_len=1 utf16_len(RECORDED)=1 utf16_len(CORRECT)=2

A stray continuation byte is 0 UTF-16 units; a 4-byte lead is 2. Both were recorded as 1. That wrong .length propagates into every downstream index/length operation.

Lone-surrogate metadata was dropped too. With "\uD800|B" (ED A0 80 | B):

SRC flags=1 utf16_len=3 byte_len=5
  PART[0] bytes=[ED, A0, 80] flags=0  (isWellFormed=true)   <- wrong

Note js_string_from_bytes also hardcodes flags = 0 (it only derives utf16_len), so the flag has to be derived from each part's own bytes, not mirrored from that constructor.

Fix

• Replace the aggregate is_ascii_string(s) gate with an actual byte scan of the source payload. A genuinely all-ASCII source provably has all-ASCII parts, so that shortcut is sound per-part — it is kept (common case, hot path), but now gated on something true.
• Otherwise derive utf16_len per part with the bounded compute_utf16_len over that part's own bytes.
• Derive STRING_FLAG_HAS_LONE_SURROGATES per part via a new bounds-driven bytes_have_lone_surrogate() helper, on both the non-empty and empty-delimiter paths. A part carries the flag only if the surrogate actually landed in it.

Tests

• split_parts_get_metadata_from_their_own_bytes — asserts utf16_len 0 and 2 for the two parts, and asserts the precondition that is_ascii_string really does misfire, so a regression back to the aggregate check fails the test.
• split_parts_preserve_lone_surrogate_flag — flagged part → isWellFormed() == false; the clean "B" part stays unflagged; covers the split("") path too.

cargo test -p perry-runtime -- --test-threads=1: 1257 passed, 0 failed. cargo fmt --all -- --check clean. Node parity repro (per-frame split('\n') → split('|') → parseFloat, plus every scanner touched): byte-identical to a stock main build and identical to node --experimental-strip-types.

Summary by CodeRabbit

• Bug Fixes

  • Improved string splitting for malformed UTF-8 and lone-surrogate content.
  • Split results now retain accurate character lengths and encoding metadata.
  • Corrected handling for both delimiter-based and character-by-character splitting.
  • Prevented incorrect metadata from being copied from the original string to each result.

• Tests

  • Added regression coverage for split-result metadata and lone-surrogate preservation.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 4535e6a7-87e5-439e-94e8-d44c72105f55

📥 Commits

Reviewing files that changed from the base of the PR and between 05d3dbd and fbc9cba.

📒 Files selected for processing (3)

• crates/perry-runtime/src/string/mod.rs
• crates/perry-runtime/src/string/split.rs
• crates/perry-runtime/src/string/tests_guard_page.rs

---

📝 Walkthrough

Walkthrough

js_string_split_n now derives each output part’s UTF-16 length and flags from its own bytes, preserving lone-surrogate metadata for malformed UTF-8 and WTF-8 inputs. A bounded surrogate detector and regression tests support the change.

Changes

String split metadata

Layer / File(s)	Summary
Lone-surrogate detection and source metadata crates/perry-runtime/src/string/mod.rs, crates/perry-runtime/src/string/split.rs	Adds bounded WTF-8 lone-surrogate detection and computes source-wide metadata for split decisions.
Per-part metadata propagation and validation crates/perry-runtime/src/string/split.rs, crates/perry-runtime/src/string/tests_guard_page.rs	Derives output metadata from each part’s bytes for both delimiter paths and tests malformed-byte and surrogate-flag preservation.
Estimated code review effort: 3 (Moderate)	~20 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and accurately summarizes the metadata-from-own-bytes fix in runtime split handling.
Description check	✅ Passed	The description covers the bug, fix, related issue, and verification, though it doesn't strictly follow the template's sectioned checklist.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/6085-split-part-metadata

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}