Add deterministic dataset candidate selection

[aryamantodkar]

Summary

• Add a generic TrainingRecord and TrainingSpan contract for Google and future enrichment sources.
• Update the Google TN compiler and pipeline to emit the generic record type.
• Add a streaming, bounded-memory candidate selector using content-based SHA-256 ranking.
• Deduplicate selected content without giving repeated provenance extra sampling chances.
• Allocate rare classes first, credit all spans from selected multi-span records, and select context-only records from a separate quota.
• Report selected counts, shortfalls, multi-span overshoot, source counts, and bounded-reservoir duplicate metrics.

Invariants

• Memory grows with configured quotas, not with the 71M-record accepted corpus.
• Record content determines sampling rank; provenance does not.
• Exact duplicate content has one sampling identity and keeps the lexicographically smallest provenance representative.
• A record is selected only while it contributes to an underfilled span target, or while the context-only quota is underfilled.
• Multi-span records remain intact and credit every contained span.
• retained_unique_candidates describes only unique candidates surviving the bounded reservoirs.

Without these rules, duplicate Wikipedia records could receive extra lottery tickets, common DATE/CARDINAL records could crowd out rare classes, and enrichment would need to construct a misleading Google-specific record type.

Validation

• uv run pytest -q — 209 passed
• uv run ruff check src tests
• Ruff format checks on all changed files
• git diff --check

This PR does not add the Google shard adapter, enrichment, train/validation splitting, or final dataset writing.