fix(tokenization): replace substring matching with character offsets from annotations

[hanneshapke]

Summary

• Preserve character offsets from Label Studio annotations through the preprocessing pipeline (preprocessing.py) — start/end fields are now included in privacy_mask entries instead of being discarded
• Use offsets directly in _find_privacy_mask_positions (tokenization.py) instead of re-discovering positions via text.find() substring search
• Validate offset integrity by checking text[start:end] == value and logging a warning on mismatch (entry is skipped to avoid corrupt labels)
• Error on missing offsets — if a privacy mask item lacks start/end, a ValueError is raised instead of silently falling back to fragile substring matching

Problem

The previous text.find(value) approach caused three classes of silent data quality bugs:

1. Partial matches: "Alex" matched inside "Alexandria", "Park" inside "Parking" — non-PII words got labeled as entities
2. Duplicate over-matching: If "John" appeared twice but only one was annotated, both got labeled — injecting noise into training data
3. Overlapping collisions: Short entity values (e.g. "Dan") matched inside longer ones (e.g. "DanTheMan"), creating conflicting annotations

These bugs silently degraded training signal quality across the entire dataset.

Changes

File	Change
model/src/preprocessing.py	Include start and end in both privacy_mask.append() calls (coreference entities and standalone entities)
model/dataset/tokenization.py	Rewrite _find_privacy_mask_positions to use annotation offsets directly, with offset validation and ValueError on missing offsets

Test plan

• Run training pipeline on existing Label Studio dataset and verify no ValueError is raised (confirms all annotations have offsets)
• Compare token-level labels before and after on a sample with duplicate PII values (e.g. same first name appearing twice) to confirm only annotated instances are labeled
• Verify offset mismatch warning fires correctly by testing with a synthetically corrupted annotation

Closes #264

🤖 Generated with Claude Code