feat(parser): detect DOCX choice token facts

[thewrz]

Why

Bracketed and angle-bracketed option text is common in master specs, but the parser did not surface those source cues for downstream tooling.

What

• Add a DOCX parser scanner for adjacent angle and bracket choice-token groups.
• Emit meta.sourceFacts.choiceTokens with kind, options, and flattened-text span.
• Preserve lone bracketed segments as single-option keep/delete candidates.
• Pin ambiguous nested-bracket and section-reference-like bracket behavior in tests.

Testing

• pnpm vitest run --project unit src/parser/docx/index.test.ts
• pnpm test
• pnpm lint
• CI green

Design decisions

• Adjacent same-kind segments form one candidate; lone bracketed segments are retained as single-option candidates.
• Nested same-delimiter brackets are skipped rather than guessed.
• [Section NN NN NN]-style brackets are skipped to avoid treating likely CSI references as choices.

Co-authored by Codex noreply@openai.com.

Closes #130

Summary by CodeRabbit

• New Features

  • Added support for extracting and tracking color and highlight metadata from document content and runs
  • Added automatic detection of choice tokens representing inline delimited options for document selections

• Tests

  • Expanded test coverage for document color extraction with coverage metrics and choice token detection across various delimiter patterns

[coderabbitai]

Warning

Review limit reached

@thewrz, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 1 hour, 39 minutes, and 23 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more credits in the billing tab to continue.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: af6451dd-829c-4fd6-8ef9-3c9ae0babede

📥 Commits

Reviewing files that changed from the base of the PR and between 598995c and 7ae1dea.

📒 Files selected for processing (4)

• src/ast/types.ts
• src/parser/docx/choice-tokens.ts
• src/parser/docx/index.test.ts
• src/parser/docx/source-facts.ts

📝 Walkthrough

Walkthrough

Adds SourceColorFact and SourceChoiceTokenFact to the AST types and extends SourceFacts to carry them. Introduces choice-tokens.ts for scanning bracket/angle choice candidates and a new source-facts.ts module that centralizes all per-paragraph XML extraction (comments, colors, choice tokens). Refactors document.ts to delegate to parseParagraphSources. Adds Vitest coverage for both fact types.

Changes

DOCX source facts pipeline

Layer / File(s)	Summary
AST fact type contracts src/ast/types.ts	Adds SourceColorFact and SourceChoiceTokenFact exported interfaces and extends SourceFacts with optional colors and choiceTokens arrays.
Choice token scanner src/parser/docx/choice-tokens.ts	New module implementing scanChoiceTokens: delimiter config for angle/bracket tokens, section-reference ambiguity regex, nested-delimiter rejection, and adjacency grouping of segments into SourceChoiceTokenFact entries.
source-facts.ts: XML setup, types, and inline extraction src/parser/docx/source-facts.ts	Configures XMLParser for ordered traversal, defines internal state types and ParagraphSource export, implements XML utilities and comment-marker mapping, extracts run-level color/highlight tokens and inline text, and converts inline state into InlineParagraph objects.
source-facts.ts: comment, color, and choice fact assembly src/parser/docx/source-facts.ts	Builds SourceCommentFact entries from cross-paragraph markers, merges contiguous color spans and computes coverage ratios into SourceColorFact entries, assembles conditional SourceFacts per paragraph, and exports parseParagraphSources wrapping XML parse failures in ParserError.
document.ts delegation refactor src/parser/docx/document.ts	Removes local ordered-XML parse, comment-marker scaffolding, and source-fact construction pipeline; updates parseParagraph to accept a single ParagraphSource parameter; updates parseDocument to call parseParagraphSources and pass indexed results to parseParagraph.
Test helper updates and new source-facts suites src/parser/docx/index.test.ts	Adds sourceColors/sourceChoiceTokens helpers with generalized TestSourceFacts typing; adds run-colors tests (spans, coverage, black/auto omission, distinct facts); adds choice-tokens tests (angle/bracket grouping, unclosed/nested/section-reference no-output).

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• wrzonance/SpecR#181: Introduced the sourceFacts/comment-facts pipeline in document.ts and src/ast/types.ts that this PR directly refactors and extends.
• wrzonance/SpecR#183: Extended SourceFacts with SourceColorFact and modified DOCX parsing and tests for run color facts, which this PR consolidates into source-facts.ts.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'feat(parser): detect DOCX choice token facts' accurately describes the main feature added—detecting choice token facts in DOCX documents, which is the primary objective of this PR.
Linked Issues check	✅ Passed	The implementation meets all coding requirements from issue #130: creates a text scanner for bracketed/angle-bracketed segments (scanChoiceTokens function), emits choice token facts with kind/options/span, groups adjacent segments as pick-one candidates, captures lone segments as single-option candidates, and includes KNOWN AMBIGUITY tests.
Out of Scope Changes check	✅ Passed	All changes align with the linked issue scope: new type definitions (SourceColorFact, SourceChoiceTokenFact), choice token scanner, paragraph source parsing, and tests. The addition of color facts support extends infrastructure but directly supports the choice tokens feature and falls within the in-memory parser scope.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/issue-130

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[thewrz]

:)
@coderabbitai full review

[coderabbitai]

✅ Action performed

Full review finished.

[coderabbitai]

@thewrz 😄 Kicking off a fresh full review for you!

✅ Action performed

Full review finished.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/parser/docx/choice-tokens.ts`:
- Around line 84-92: The groupToFact function should not emit a fact for single
angle-delimited segments, as only bracket-delimited singles are valid exceptions
to the pick-one rule. Add a check in the groupToFact function that returns null
when the group contains only a single segment that is angle-delimited (check the
segment's kind or delimiter property to distinguish between angle and bracket
delimiters). Place this check after verifying that first and last exist but
before constructing the return object.
- Around line 51-55: The nested delimiter detection in the choice-tokens parser
advances to the first close delimiter found when nesting is detected, which is
insufficient and allows false candidates to be emitted. In the
hasNestedDelimiter check within the choice-tokens.ts file, when nested
delimiters are detected, instead of returning nextIndex as closeIndex + 1, you
must skip the entire nested structure by advancing past all nested content until
finding the actual matching close delimiter. Additionally, add a regression test
with a symptom-based name in the corresponding .test.ts file that validates the
parser correctly rejects or handles ambiguous nested constructs like [[a][b]],
and add a // KNOWN AMBIGUITY: comment documenting why this pattern presents
challenges for the parser.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 6e24e7c8-7077-4492-9122-73e638c8e0ae

📥 Commits

Reviewing files that changed from the base of the PR and between c4bae72 and 598995c.

📒 Files selected for processing (5)

• src/ast/types.ts
• src/parser/docx/choice-tokens.ts
• src/parser/docx/document.ts
• src/parser/docx/index.test.ts
• src/parser/docx/source-facts.ts