feat(chunking): list-aware break point scanner

[galligan]

Pseudo-stack

Cross-fork PRs can't use GitHub's stacked-PR mechanism, so all four PRs in this series target main. The logical stack:

main
└── #538  fix: code fence pairing               (foundation)
    └── #539  refactor: rename to ProtectedRegion
        ├── #540  feat: list-aware chunking        ← you are here (parallel with #541)
        └── #541  feat: XML tag regions             (parallel with #540)

This PR: #540. Stacked on #539 which is stacked on #538. The diff below shows commits from both of those PRs in addition to the list scanner — they'll collapse once the lower PRs land and I rebase. Parallel to #541 (XML tag regions); they branch off #539 independently and don't depend on each other.

Full context: qmd chunker improvements — four-PR series overview

---

Summary

Replaces the two naive list patterns in BREAK_PATTERNS with a stack-based scanner that tracks nested list frames and emits depth-weighted break points plus a list-end transition signal. Long lists now split cleanly at item boundaries instead of mid-item.

The problem

The existing list detection was two flat regex patterns:

[/
[-*]\s/g, 5, 'list'],        // unordered
[/
\d+\.\s/g, 5, 'numlist'],    // ordered

Two consequences:

1. Score 5 is so low it almost always loses. Any nearby blank line (20), hr (60), heading (50–100), or codeblock boundary (80) outranks it. On long prose-heavy lists, the chunker happily splits mid-item because a `

` scored higher than the list-item boundary.

2. Zero structural awareness. The regex doesn't know about indentation, so nested sublists are invisible. - sub-item inside a - top-item list gets no break point at all. The ordered 1) form isn't detected either. End-of-list transitions (the single most valuable break point in a document with lists) get no special treatment.

Net effect: lists were the worst-handled structure in the chunker.

The fix

findListBreakPoints(text: string): BreakPoint[] — a line-by-line state machine that maintains a stack of ListFrame { indent, contentCol } entries and emits:

Break point	Score	Meaning
list-end	75	Transition from list back to non-list flow. Third-highest score after h1/h2 — list endings are excellent split points.
list-item-0	70	Top-level list item start. Matches h4; items at the top of a list are as good a break as a level-4 heading.
list-item-1	45	Second-level (first sublist) item. Above blank (20), below h6 (50).
list-item-2	25	Third-level and deeper. Just above blank — used only when nothing better is nearby.

These scores sit in gaps in the existing score table so comparisons stay unambiguous.

What it handles

• Unordered markers - and * (matches existing behavior; + intentionally not supported)
• Ordered markers 1. and 1) (the parenthesis form was previously missed entirely)
• Mixed marker characters at the same indent are treated as one list. CommonMark says - foo → * bar at the same indent ends one list and starts another; we don't follow that rule because it would insert a high-priority break where the user visually sees no break at all. For chunking, "same indent = same list" produces better results.
• Nested sublists with proper depth tracking. `- foo
  • bar
    • baz` correctly produces depth 0/1/2 break points.
• Mixed nesting (unordered top with ordered sublist, or vice versa) works with the same stack logic.
• Blank lines inside items don't terminate the list — state is preserved until a non-indented non-list line appears.
• Column-0 non-list lines terminate the list and emit list-end at the preceding newline.
• List at end of document emits list-end at text.length.

What it deliberately doesn't handle

Each of these was evaluated against a strict "don't degrade existing behavior" bar and deferred:

• Loose vs tight list distinction — affects rendering, not chunking.
• Lazy continuation — a column-0 non-list line that CommonMark folds back into the preceding item. Treated as list-end. The wrong answer is a slightly degraded chunk, not a broken one.
• 4-space indented code blocks inside items — ambiguous with sublist continuation. Modern docs use fenced code.
• Tab indentation — neither the old regex nor the new one handles `
  • item. The only tab pattern the old regex did match was - ` (dash followed by literal tab as separator), which is a typing pattern nobody uses in practice. Space-separated markers only.
• Marker-type transitions at same indent — see "mixed marker characters" above; intentional deviation from CommonMark.

A block comment on the scanner documents these limitations so the next person doesn't chase them.

Integration

chunkDocument and chunkDocumentAsync now merge findListBreakPoints output with scanBreakPoints before passing to chunkDocumentWithBreakPoints. mergeBreakPoints already handles the "higher score wins at same position" case, so the merge is trivial:

const regexPoints = scanBreakPoints(content);
const listPoints = findListBreakPoints(content);
const breakPoints = mergeBreakPoints(regexPoints, listPoints);

In the async path, AST points continue to layer on top via a second mergeBreakPoints call when chunkStrategy === 'auto'.

Removed

The two old patterns are deleted from BREAK_PATTERNS:

[/
[-*]\s/g, 5, 'list'],
[/
\d+\.\s/g, 5, 'numlist'],

An existing test in the scanBreakPoints block that asserted list detection was updated to assert non-detection (with a pointer to findListBreakPoints).

Regression analysis

The only pattern that used to score and no longer does is - (dash followed by a literal tab character as the marker separator). This is not a typing pattern that occurs in real markdown — every editor inserts spaces, and every style guide specifies spaces. The "tab-indented list" concern is a non-issue because the old regex never matched tab-indented items in the first place ( - foo was invisible to it).

Everything the old code detected, the new code detects and scores higher. Patterns the old code missed (nested sublists, 1) form, list-end transitions) are now handled properly.

Tests

16 new tests in test/store.test.ts under describe("findListBreakPoints", ...):

1. Empty input → no break points
2. Pure prose → no break points
3. Single unordered list → item + list-end break points, correct scores
4. Single ordered list with 1.
5. Single ordered list with 1)
6. Mixed marker characters at same indent → one list
7. Nested unordered list → depth 0 and depth 1 scores
8. Three-deep nesting → depth 0, 1, 2 scores
9. Mixed nesting (unordered top + ordered sublist)
10. List followed by prose → list-end at correct position
11. List at end of document → list-end at text end
12. Single blank line between items → does not terminate
13. Blank then non-list prose → terminates list
14. + markers → not detected (decision 4)
15. Position convention → pos is the before the line, matching scanBreakPoints
16. Integration test: chunkDocument on a 200-item list → splits land on item boundaries

Test plan

• npx vitest run test/store.test.ts passes (219/219, was 203 + 16 new)
• npx vitest run test/ast-chunking.test.ts passes (12/12)
• npx tsc -p tsconfig.build.json --noEmit clean
• CI green