Materialize Google TN accepted corpus shards

[aryamantodkar]

Summary

• add premove-write-google-tn-shard to materialize accepted Google TN records as immutable JSONL corpus shards
• preserve GoogleTnTrainingRecord literally inside a provenance wrapper
• write source-qualified manifests and audit summaries with zero-based half-open source ranges
• publish through temporary files without overwrite, with the manifest last as the completion marker
• reuse the existing audit outcome taxonomy while streaming accepted records
• accept narrow, class-specific formatting equivalence for CARDINAL, DECIMAL, MONEY, and TIME

Why

This creates the durable accepted-corpus pool between trusted Google ingestion and later dataset selection. These shards are not final train/validation data. They preserve every passing compiler record so balancing, deduplication, and stable split assignment can happen later without repeatedly scanning the raw corpus.

The final equivalence pass removes only presentation differences that preserve semantic value. It does not broaden unsupported classes or accept arbitrary punctuation changes.

Invariants

• source ranges are zero-based and half-open
• source_end_exclusive records the actual processed end, including short final ranges
• artifact names and manifests include a safe source identity
• the compiled training record is unchanged; provenance remains generation metadata
• existing shard, summary, or manifest files are never overwritten
• a processing failure leaves no completed artifacts
• the manifest is published last and acts as the completion marker
• known rejections continue through the audit; unexpected parser, policy, compiler, and I/O errors abort generation
• canonical acceptance always uses the Rust realization as the training replacement
• whole-sentence rejection and all quarantine classes remain unchanged

Without these rules, partial files could look complete, multiple source files could collide, generation metadata could leak into the model record contract, or unsafe formatter disagreements could enter training labels.

Narrow equivalence policies

• CARDINAL: valid thousands separators only (11,331 == 11331)
• DECIMAL: valid thousands separators in the integer field (5,661.38 == 5661.38)
• MONEY: valid grouping only when the currency wrapper and numeric value are identical ($7,000 == $7000)
• TIME: approved forms parse to the same (hour_24, minute), including hour-leading zeroes, 12/24-hour conversion, and AM/PM case/space/period variants

Explicit regressions keep these rejected:

• malformed grouping (11,33 vs 1133)
• semantic MONEY disagreement ($62,500 vs $6200500)
• currency identity change ($7,000 vs USD 7000)
• midnight/noon mismatch (00:12 vs 12:12)
• minute changes or missing minute zeroes (04:12 vs 04:13, 4:02 vs 4:2)
• timezone-bearing and unapproved TIME punctuation

Two-batch result

The same final policy was rerun against both independent 10k batches:

Source range	Before	After	Gain
[0, 10,000)	7,835	7,924	+89
[10,000, 20,000)	7,892	7,960	+68
Total accepted pool	15,727	15,884	+157

The dangerous MONEY example remains rejected. Unsupported classes and the established rejection taxonomy remain unchanged. Generated shards and audits stay ignored by Git.

Validation

• changed Python files pass uv run ruff format --check
• uv run ruff check .
• uv run pytest — 158 passed
• cargo test --manifest-path rust/Cargo.toml — 17 passed
• git diff --check
• uv run ruff format --check . — existing unrelated formatting failures remain in tests/test_golden.py and tests/test_types.py