Select enrichment and assemble Dataset V1

[aryamantodkar]

Summary

• add deterministic, quota-based selection from the immutable enrichment reservoir
• select only truly span-free SGD negatives and preserve collision examples as intact semantic pairs
• add final Dataset V1 assembly with artifact verification, shared-record recompilation, canonical deduplication, and Google-over-enrichment source preference
• preserve primary and duplicate provenance outside the generic training record
• add immutable artifacts with manifests, SHA-256 checksums, no-overwrite publication, and rollback
• expose premove-select-enrichment and premove-assemble-dataset-v1 CLIs

Why

The compiled enrichment reservoir is intentionally large and skewed. Dataset V1 needs a smaller, explicit enrichment contribution before it is combined with frozen Google data. This change makes that selection reproducible and keeps controlled PHONE/TIME versus DIGIT_SEQUENCE contrasts together.

Assembly then establishes one validated, deduplicated candidate artifact before grouped train/validation splitting. Without this boundary, duplicate or related records could be split independently, and source-specific compilation paths could diverge.

Invariants

• selection is deterministic from the input artifacts, source data, quotas, and seed
• selected records are not modified
• SGD context-only records come only from source turns with no original SGD spans
• collision records are validated and retained as complete pairs
• every input TrainingRecord is recompiled with the shared compiler
• exact compiled duplicates collapse to one record
• organic Google provenance wins an identical cross-source duplicate
• output is immutable, checksummed, and published manifest-last

Frozen V1 audit

Selected enrichment:

• 15,401 records
• 2,779 SGD positives
• 2,500 clean conversational negatives
• 3,500 PHONE purpose-built records
• 3,500 ELECTRONIC purpose-built records
• 1,000 DIGIT_SEQUENCE purpose-built records
• 1,000 PHONE/DIGIT_SEQUENCE collision pairs
• 61 TIME/DIGIT_SEQUENCE collision pairs
• 728 multi-span records
• 98 selected PHONE collision pairs with the same realization

Combined Dataset V1:

• 40,339 records
• 24,938 Google records
• 15,401 enrichment records
• 10,500 context-only records
• 10,016 multi-span records
• no within-source or cross-source exact duplicates
• artifact SHA-256: 5ef45f4d3792cbbe68aec57c8ac63013f30478e5849c32140c6ba11bc92a3e83

Generated corpus artifacts remain untracked.

Validation

• uv run ruff check .
• uv run pytest — 47 passed
• uv run --package premove-itn-data pytest tools/data_pipeline/tests — 260 passed
• cargo test --manifest-path rust/Cargo.toml — 21 passed
• uv build --package premove-itn
• uv build --package premove-itn-data