Skip to content

Add deterministic Rust realization adapter - #5

Merged
aryamantodkar merged 1 commit into
mainfrom
codex/add-rust-manifest
Aug 22, 2026
Merged

Add deterministic Rust realization adapter#5
aryamantodkar merged 1 commit into
mainfrom
codex/add-rust-manifest

Conversation

@aryamantodkar

Copy link
Copy Markdown
Contributor

Summary

  • add the premove-itn-core Rust crate and pin text-processing-rs v0.3.0
  • expose PyO3 functions for forced span realization, the upstream sentence baseline, and TN
  • route configured English ITN kinds directly to their upstream parsers
  • add the strict local DIGIT_SEQUENCE extension with leading-zero and repetition support
  • keep Rust tests in a separate test file and document the deterministic routes

Why

The contextual model must select spans and classes without generating replacement text. This adapter establishes the deterministic boundary that converts an accepted source span through one explicitly selected realizer.

Invariant

A supported class calls only its configured deterministic parser. Unknown classes return an error, rejected spans return no replacement, and the local digit-sequence parser rejects incomplete or unknown input.

Without this boundary, the later classifier could depend on the upstream fixed-priority sentence dispatcher or construct normalized values itself, which would hide class errors and weaken fail-closed behavior.

Supported routes

  • upstream: CARDINAL, DATE, DECIMAL, ELECTRONIC, MONEY, MEASUREMENT, ORDINAL, PUNCTUATION, PHONE, TIME, WHITELIST, WORD
  • local extension: DIGIT_SEQUENCE

FRACTION is not routed because the pinned crate exposes it under tn::en, not itn::en; that parser runs in the opposite direction.

Validation

  • cargo fmt --check
  • cargo test --locked — 15 passed
  • cargo clippy --locked --all-targets -- -D warnings
  • uv run pytest — 1 passed
  • uv run ruff check .
  • uv build

@aryamantodkar
aryamantodkar merged commit 8bc1b4f into main Aug 22, 2026
@aryamantodkar
aryamantodkar deleted the codex/add-rust-manifest branch August 22, 2026 07:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant