Skip to content

Mege staging to main - Hi TN v3 - #464

Merged
tbartley94 merged 11 commits into
mainfrom
staging/hi_tn_v3
Jul 29, 2026
Merged

Mege staging to main - Hi TN v3#464
tbartley94 merged 11 commits into
mainfrom
staging/hi_tn_v3

Conversation

@mgrafu

@mgrafu mgrafu commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do ?

  • Adds coverage for Address, Roman and Serial classes
  • Fixes and improvements for Electronic, Cardinal, Fraction, Date, Money, Measure

Before your PR is "Ready for review"

Pre checks:

  • Have you signed your commits? Use git commit -s to sign.
  • Do all unittests finish successfully before sending PR?
    1. pytest or (if your machine does not have GPU) pytest --cpu from the root folder (given you marked your test cases accordingly @pytest.mark.run_only_on('CPU')).
    2. Sparrowhawk tests bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...
  • If you are adding a new feature: Have you added test cases for both pytest and Sparrowhawk here.
  • Have you added __init__.py for every folder and subfolder, including data folder which has .TSV files?
  • Have you followed codeQL results and removed unused variables and imports (report is at the bottom of the PR in github review box) ?
  • Have you added the correct license header Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. to all newly added Python files?
  • If you copied nemo_text_processing/text_normalization/en/graph_utils.py your header's second line should be Copyright 2015 and onwards Google, Inc.. See an example here.
  • Remove import guards (try import: ... except: ...) if not already done.
  • If you added a new language or a new feature please update the NeMo documentation (lives in different repo).
  • Have you added your language support to tools/text_processing_deployment/pynini_export.py.

PR Type:

  • New Feature
  • Bugfix
  • Documentation
  • Test

If you haven't finished some of the above items you can still open "Draft" PR.

shrpawar-alt and others added 10 commits May 13, 2026 16:29
@shrpawar-alt @pre-commit-ci
Hi TN: fraction class improvements with lexical mappings for special …
cce8879 
…cases (#420)

* Fix: fraction class improvements with lexical mappings

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* move common fraction mappings to TSV and add ASCII digit support

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* Jenkins date conflict resolved

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
@shrpawar-alt @pre-commit-ci
Hi TN date class accuracy improvement (#418)
0daa8a0 
* date class accuracy improvement

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* Jenkins file date update for Hi TN

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Address Reviewer feedback regarding date tagger, tsv files and inclusion of more test cases

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* hi-tn-date: minor formatting fix

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* date tagger modification according to feedback removed support for d-mm-yyyy, dd-m-yyyy and mm-yyyy

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* replace open()+union() with string_map for suffix and prefix unions

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
@shrpawar-alt @pre-commit-ci
Hi TN money class bug fix (#424)
42bf4aa 
* Hi TN money class bug fix

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* added test cases with decimal portion having more than two digits for Indian Currency

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
@shrpawar-alt @pre-commit-ci
Hi TN Electronic: Fix partial TSV matching, chemical formula logic, a…
205fff6 
…nd relative file paths (#425)

* Fix partial TSV matching, added support for chemical formulas, and relative file paths

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Refactor electronic class with dynamic symbol matrix and chemical rules. Replaced hardcoded TSVs and Python symbol rules with a data-driven symbol_classes.tsv and dynamic elements.tsv. Fixed greedy over-tagging of standard English words and added Unicode support for complex chemical ions without breaking URL hyphen logic.

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
@shrpawar-alt
Remove single capital letter units from Measure class (#429)
88c2421 
Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
@shrpawar-alt @pre-commit-ci
Hi TN: Implement Serial tagger (#440)
317783e 
* Hi TN Serial: Implement SerialFst tagger for Devanagari-numeric mixtures, number chains, and mathematical powers

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Address feedback for Hindi Serial Tagger PR

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
@shrpawar-alt @pre-commit-ci
Hi TN: Implement Roman semiotic class (#442)
9203a96 
* Hi TN Roman class implementation

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* Refactor Roman FST to utilize shared serial Devanagari chars

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* Add __init__.py for serial and roman data folders

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update roman_graph weight to 1.07 and other minor changes

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* Feedback changes

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
@shrpawar-alt
reducing uncommon entries in electronic class tsv data files (#458)
8205ab2 
Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
@shrpawar-alt @pre-commit-ci
Hi TN: Address, Electronic, Serial, and Cardinal FST Optimization an…
7b4356c 
…d Bug Fix (#461)

* Hi TN Address, Electronic, Serial, and Cardinal FST Optimizations and Bug Fixes

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* review feedback changes

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* feedback review changes for address and electronic

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
@mgrafu
Merge branch 'main' into staging/hi_tn_v3
f131c50 
Signed-off-by: Mariana <47233618+mgrafu@users.noreply.github.com>
github-advanced-security[bot]
github-advanced-security AI found potential problems Jul 28, 2026
View reviewed changes
Comment thread nemo_text_processing/text_normalization/hi/taggers/measure.py Fixed
@shrpawar-alt
Hi tn v3: staging to main error fixed (#466)
a1147ed 
Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
@tbartley94
tbartley94 self-requested a review July 29, 2026 17:43
tbartley94
tbartley94 approved these changes Jul 29, 2026
View reviewed changes

@tbartley94 tbartley94 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tbartley94
tbartley94 merged commit e4d943b into main Jul 29, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tbartley94 tbartley94 tbartley94 approved these changes
+1 more reviewer
@github-advanced-security github-advanced-security[bot] github-advanced-security[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@mgrafu @github-advanced-security @tbartley94 @shrpawar-alt