Add source-preserving tokenizer

[aryamantodkar]

Summary

• add a deterministic Unicode-aware tokenize() interface
• return immutable WordToken values with original [start, end) character offsets
• preserve original casing and internal straight or curly apostrophes
• separate punctuation while ignoring whitespace as tokens
• cover common ASR, numeric, email-like, hyphenated, and time-like inputs

Why

The contextual classifier needs word-level inputs, while every public span and later edit must continue to refer to exact coordinates in the original transcript. This tokenizer establishes that source-preserving seam before model subword tokenization is introduced.

Invariant

For every returned token, text[token.start:token.end] == token.text. Empty and whitespace-only inputs return no tokens.

Without this invariant, model predictions could be mapped back to the wrong transcript evidence, breaking provenance and potentially rewriting text outside the accepted span. The tests independently pin exact tokens and offsets across whitespace, punctuation, apostrophes, leading zeros, alphanumeric values, email-like text, hyphens, and time punctuation.

Validation

• uv run pytest — 44 passed
• uv run ruff check .
• uv build
• git diff --check