Add Google TN ingestion audit milestone - #18
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
premove-audit-google-tncommand for reproducible Google TN ingestion auditspremove_itn._rustthrough Maturin and test the Python/Rust boundaryPUNCTrows as context with punctuation-aware sentence assemblyWhy
This freezes the first 10,000 Google TN sentences as the development milestone before evaluating a different 10,000-sentence generalization batch.
The objective is not to eliminate every rejection. It is to extract a large subset where every training label is trustworthy. Unsupported or semantically incompatible rows must continue to reject the whole sentence.
Invariants
LETTERS,VERBATIM,FRACTION, andADDRESSremain quarantined.Olabel.Without these rules, the importer could accept semantically different written forms or train unsupported rows as context. The tests prove punctuation offsets remain exact, canonical DATE matches use the Rust target, DATE order/value mismatches fail closed, and the packaged Rust extension is the production realizer.
First 10k audit
The next 10k must be run without policy changes first. It is a generalization batch used to compare acceptance rate, quarantine distribution, DATE matching, and rejection taxonomy. New equivalence policies should be narrow, class-specific, semantically lossless, and verified against both batches.
Validation
uv run ruff check .uv run pytest— 126 passedcargo test --manifest-path rust/Cargo.toml— 17 passedgit diff --checkuv run ruff format --checkuv run ruff format --check .— blocked by two pre-existing unformatted files not modified by this PR:tests/test_golden.pyandtests/test_types.py