perf: eliminate hot-path allocations in chunker core

[jonathanong]

Summary

• default_length_counter rewritten (tokens.rs): single-pass char walk with in_ws/started state flags — zero allocations, no regex, one scan instead of regex-replace + char-count.
• set_length allocation eliminated (utils.rs): counts breadcrumb and text independently; the \n\n separator contributes exactly 1 under whitespace-normalization, so no format! needed.
• Phase 3 deep-clone eliminated (merge.rs): result is now consumed by value via mem::take + into_iter().peekable() — removes the 6×N full Chunk clones that occurred even on no-op passes.
• format! → push_str in merge bodies (merge.rs): reuses existing String capacity instead of allocating a fresh one per merge; "#".repeat(level) replaced with a const-sliced "######".
• split.rs hot-path cleanup: collect only (start, end, &str) triples from HEADER_REGEX instead of full Captures; use bytes() for the # level count; drop intermediate Vec<&str> from paragraph-split loops.
• N-API safety (package/src/lib.rs): replace assert! (which aborts the Node process) with Err(...) so an oversized chunk surfaces as a Promise rejection.

None of these change the public API or algorithm.

Test plan

• cargo test — 101 tests pass
• cargo llvm-cov --package breadchunks --show-missing-lines --fail-under-lines 99 — 99.14% line coverage, exit 0
• cargo check in package/ — compiles clean

🤖 Generated with Claude Code

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

package/src/lib.rs (1)

50-54: ⚡ Quick win

Consider including the actual length value in the error message for better debugging.

Including c.length would help users understand how far they exceeded the limit, consistent with the error message pattern on line 84.

📊 Proposed enhancement to include actual length

                 .map(|c| {
                     if c.length > u32::MAX as usize {
                         return Err(Error::from_reason(
-                            "chunk length exceeds u32::MAX; docs >4 GiB unsupported on Node binding",
+                            format!(
+                                "chunk length {} exceeds u32::MAX; docs >4 GiB unsupported on Node binding",
+                                c.length
+                            ),
                         ));
                     }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@package/src/lib.rs` around lines 50 - 54, The error message returned from
Error::from_reason when checking if c.length > u32::MAX should include the
actual c.length value to aid debugging; update the check in package/src/lib.rs
(the branch using c.length and Error::from_reason) to format the message with
both c.length and the u32::MAX limit (or 4 GiB) so callers see how large the
chunk was in the error text.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crate/src/utils.rs`:
- Around line 11-13: The length calculation overcounts when chunk.text
normalizes to empty: instead of always adding a separator with b + 1 + t, only
add the separator when both breadcrumb and text are non-empty. Update the
assignment that uses default_length_counter(&chunk.breadcrumb) and
default_length_counter(&chunk.text) so chunk.length = if b == 0 { t } else if t
== 0 { b } else { b + 1 + t }, ensuring the separator is only counted when t >
0.

---

Nitpick comments:
In `@package/src/lib.rs`:
- Around line 50-54: The error message returned from Error::from_reason when
checking if c.length > u32::MAX should include the actual c.length value to aid
debugging; update the check in package/src/lib.rs (the branch using c.length and
Error::from_reason) to format the message with both c.length and the u32::MAX
limit (or 4 GiB) so callers see how large the chunk was in the error text.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: b7f25ee6-258c-486a-a47b-ef12a244104b

📥 Commits

Reviewing files that changed from the base of the PR and between a10156a and 81e31a5.

📒 Files selected for processing (5)

• crate/src/merge.rs
• crate/src/split.rs
• crate/src/tokens.rs
• crate/src/utils.rs
• package/src/lib.rs

Fix already applied in commit 089c4ff — CodeRabbit approved in subsequent review.