feat(core,chunker): structure-aware chunker (Step 1.5)

[officialCodeWork]

Summary

Step 1.5 introduces the chunking stage of the ingest pipeline. New `Chunker` SPI in rag-core, default `HeadingAwareChunker` in the new `rag-chunker` package, OCR adapter so Step 1.4 output flows through the same chunker, and a `ragctl chunk` smoke command.

Chunker SPI — rag-core 0.9.0 → 0.10.0

• `rag_core.spi.chunker.Chunker` ABC: `async chunk(ctx, parsed) -> list[Chunk]`.
• `NoopChunker` in `rag_core.spi.noop` — one chunk per non-empty block, no parent links.
• 7 contract tests enforce: monotonic 0-based `position`; `parent_id` references earlier chunks only (no forward refs); `tenant_id` / `document_id` propagated from `ctx` and `parsed`; whitespace-only blocks dropped; content or content_ref always set.
• SPI signature linter (`tests/contract/spi_signature.py`) extended so any future Chunker method must take `ctx: RequestContext` first.

HeadingAwareChunker — packages/chunker/ v0.1.0

Walks `ParsedDocument.blocks` and:

• Maintains a heading-level stack so each emitted heading chunk's `parent_id` is the closest enclosing higher-level heading (h2 under h1 → parent=h1; new h2 reparents under the same h1).
• Accumulates body blocks under each heading up to `max_tokens` (default 512), flushes when adding the next block would exceed the budget.
• Within-section overlap (default 64 tokens) seeds the next buffer with the trailing text of the just-flushed chunk; overlap is suppressed across heading boundaries to avoid bleeding content.
• Oversized blocks split on sentence boundaries first (pure-Python regex via `rag_chunker.sentence`); sentences still too large hard-split by token count so no body chunk ever exceeds the budget.
• `Chunk.metadata` carries `{"kind": "heading", "level": N}` or `{"kind": "body"}` so downstream consumers can distinguish.
• `acl_labels` propagated from `ctx.principal`; `tenant_id` / `document_id` / `corpus_id` from `ctx` and `parsed.document`.

Pluggable tokenizer

`TokenCounter` Protocol + `TiktokenCounter` (default `cl100k_base`). Swap encodings (`o200k_base` for GPT-4o-style budgets) via constructor, or plug in a HuggingFace / Cohere tokenizer by implementing the two-method Protocol. `tiktoken` is a default dependency — pure-extension wheels for every platform in the CI matrix, no extras needed.

OCR adapter

`ocr_result_to_parsed_document(ctx, result, …)` turns Step 1.4's `OCRResult` into a synthetic `ParsedDocument` so OCR'd image-only pages flow through `HeadingAwareChunker` identically to native parser output. Each `OCRRegion` becomes one paragraph-typed `Block` whose `attributes` preserve `engine`, `ocr_confidence`, `bbox` coordinates, and optional `page` for downstream consumers (GUI overlay, faithfulness scoring).

ragctl chunk

```bash
ragctl chunk doc.md # default budget (512/64)
ragctl chunk doc.txt --max-tokens 256 --overlap-tokens 32
ragctl chunk doc.md -n 10 # show first 10 chunks
```

Pipes the Step 1.3 parser into the Step 1.5 chunker and prints block/chunk counts, per-kind tally, and first-N chunks with parent-link arrows. 5 CLI tests.

Policy boundary

The chunker is upstream of PolicyEngine in the ingest direction (no `Chunk`s exist for the PDP to ACL-check yet). The Step 1.7 PII detector and Step 1.10 ingest pipeline are the enforcement points. `rag-chunker` depends only on `rag-core`, not on `rag-policy`. PolicyEngine coverage linter passes.

Cross-platform

Pure-Python except the `tiktoken` extension, which ships wheels for ubuntu / macos / windows-latest. Unit tests use a deterministic `_WordCounter` stand-in so token-boundary assertions don't depend on tiktoken's BPE specifics.

Tests

• `tests/chunker/test_sentence.py` (5) — boundary detection
• `tests/chunker/test_tokens.py` (4) — TiktokenCounter contract + special-token safety
• `tests/chunker/test_heading_aware.py` (16) — hierarchy, budget flush, overlap, oversized splits, ACL/tenant propagation, error wrapping, helpers
• `tests/chunker/test_ocr_adapter.py` (3) — adapter shape
• `tests/contract/test_chunker.py` (7) — SPI conformance
• `packages/ragctl/tests/test_chunk.py` (5) — CLI

Full local sweep clean:
```
uv run pytest tests/ packages/ -x -q # 40 new tests, no regressions
uv run ruff check . # clean
uv run ruff format --check . # clean
uv run mypy packages/ apps/gateway/ # 131 source files, no issues
uv run python scripts/check_logging.py # RAG001 clean
```

Documentation

• docs/reference/chunker.md — public API, `HeadingAwareChunker` semantics, `TokenCounter` protocol, OCR adapter, CLI, errors, extension points.
• docs/architecture/chunker.md — why structure-aware, boundary detection strategy, parent-link semantics, overlap policy, token counting rationale, OCR-region handling, pipeline placement, policy boundary, hot-path discipline, cross-platform notes, ADR backlinks.
• docs/README.md — both new entries added.

Test plan

• `uv run pytest tests/ packages/ -x -q` — full suite passes (40 new tests, no regressions)
• `uv run pytest tests/chunker/ tests/contract/test_chunker.py -x -q` — chunker tests pass
• `uv run pytest packages/ragctl/tests/test_chunk.py -x -q` — CLI tests pass
• `uv run ruff check .` / `ruff format --check .` — clean
• `uv run mypy packages/ apps/gateway/` — Success: no issues found in 131 source files
• `uv run python -m rag_core.gen_schemas dist/schemas/` — no drift
• CI matrix will exercise ubuntu-22.04 + macos-14 + windows-latest

🤖 Generated with Claude Code