Skip to content

Hi TN Electronic: Fix partial TSV matching, chemical formula logic, and relative file paths - #425

Merged
mgrafu merged 6 commits into
NVIDIA:staging/hi_tn_v3from
shrpawar-alt:hi-tn-electronic-v2
Jun 3, 2026
Merged

Hi TN Electronic: Fix partial TSV matching, chemical formula logic, and relative file paths#425
mgrafu merged 6 commits into
NVIDIA:staging/hi_tn_v3from
shrpawar-alt:hi-tn-electronic-v2

Conversation

@shrpawar-alt

@shrpawar-alt shrpawar-alt commented May 26, 2026

Copy link
Copy Markdown
Contributor

What does this PR do ?

  • Fix partial TSV matching to enforce full-word literal fallback (e.g., "robinson")
  • Add chemical_names.tsv and resolve unmatched chemical formula logic
  • Support relative Unix file paths without a leading slash (e.g., home/desktop)

Before your PR is "Ready for review"

Pre checks:

  • Have you signed your commits? Use git commit -s to sign.
  • Do all unittests finish successfully before sending PR?
    1. pytest or (if your machine does not have GPU) pytest --cpu from the root folder (given you marked your test cases accordingly @pytest.mark.run_only_on('CPU')).
    2. Sparrowhawk tests bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...
  • If you are adding a new feature: Have you added test cases for both pytest and Sparrowhawk here.
  • Have you added __init__.py for every folder and subfolder, including data folder which has .TSV files?
  • Have you followed codeQL results and removed unused variables and imports (report is at the bottom of the PR in github review box) ?
  • Have you added the correct license header Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. to all newly added Python files?
  • If you copied nemo_text_processing/text_normalization/en/graph_utils.py your header's second line should be Copyright 2015 and onwards Google, Inc.. See an example here.
  • Remove import guards (try import: ... except: ...) if not already done.
  • If you added a new language or a new feature please update the NeMo documentation (lives in different repo).
  • Have you added your language support to tools/text_processing_deployment/pynini_export.py.

PR Type:

  • New Feature
  • Bugfix
  • Documentation
  • Test

If you haven't finished some of the above items you can still open "Draft" PR.

shrpawar-alt and others added 3 commits May 26, 2026 09:59
@shrpawar-alt
Fix partial TSV matching, added support for chemical formulas, and re…
270602a 
…lative file paths

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
@shrpawar-alt
Merge branch 'staging/hi_tn_v3' into hi-tn-electronic-v2
05ef398 
Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
384ebd7 
for more information, see https://pre-commit.ci
@shrpawar-alt
shrpawar-alt marked this pull request as ready for review May 26, 2026 10:56
mgrafu
mgrafu reviewed May 28, 2026
View reviewed changes
Comment thread nemo_text_processing/text_normalization/hi/data/electronic/chemical_names.tsv Outdated
mgrafu
mgrafu reviewed May 28, 2026
View reviewed changes
Comment thread nemo_text_processing/text_normalization/hi/taggers/electronic.py Outdated
shrpawar-alt and others added 2 commits June 2, 2026 08:28
@shrpawar-alt
Refactor electronic class with dynamic symbol matrix and chemical rul…
27cac48 
…es. Replaced hardcoded TSVs and Python symbol rules with a data-driven symbol_classes.tsv and dynamic elements.tsv. Fixed greedy over-tagging of standard English words and added Unicode support for complex chemical ions without breaking URL hyphen logic.

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
148d752 
for more information, see https://pre-commit.ci
mgrafu
mgrafu reviewed Jun 2, 2026
View reviewed changes
Comment thread tests/nemo_text_processing/hi/data_text_normalization/test_cases_electronic.txt
@shrpawar-alt
Merge branch 'staging/hi_tn_v3' into hi-tn-electronic-v2
cb93067 
Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
@mgrafu
mgrafu merged commit 205fff6 into NVIDIA:staging/hi_tn_v3 Jun 3, 2026
2 checks passed
tbartley94 pushed a commit that referenced this pull request Jul 29, 2026
@mgrafu @shrpawar-alt @pre-commit-ci
Mege staging to main - Hi TN v3 (#464)
e4d943b 
* Hi TN: fraction class improvements with lexical mappings for special cases (#420)

* Fix: fraction class improvements with lexical mappings

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* move common fraction mappings to TSV and add ASCII digit support

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* Jenkins date conflict resolved

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Hi TN date class accuracy improvement (#418)

* date class accuracy improvement

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* Jenkins file date update for Hi TN

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Address Reviewer feedback regarding date tagger, tsv files and inclusion of more test cases

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* hi-tn-date: minor formatting fix

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* date tagger modification according to feedback removed support for d-mm-yyyy, dd-m-yyyy and mm-yyyy

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* replace open()+union() with string_map for suffix and prefix unions

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Hi TN money class bug fix (#424)

* Hi TN money class bug fix

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* added test cases with decimal portion having more than two digits for Indian Currency

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Hi TN Electronic: Fix partial TSV matching, chemical formula logic, and relative file paths (#425)

* Fix partial TSV matching, added support for chemical formulas, and relative file paths

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Refactor electronic class with dynamic symbol matrix and chemical rules. Replaced hardcoded TSVs and Python symbol rules with a data-driven symbol_classes.tsv and dynamic elements.tsv. Fixed greedy over-tagging of standard English words and added Unicode support for complex chemical ions without breaking URL hyphen logic.

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Remove single capital letter units from Measure class (#429)

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* Hi TN: Implement Serial tagger (#440)

* Hi TN Serial: Implement SerialFst tagger for Devanagari-numeric mixtures, number chains, and mathematical powers

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Address feedback for Hindi Serial Tagger PR

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Hi TN: Implement Roman semiotic class (#442)

* Hi TN Roman class implementation

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* Refactor Roman FST to utilize shared serial Devanagari chars

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* Add __init__.py for serial and roman data folders

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update roman_graph weight to 1.07 and other minor changes

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* Feedback changes

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* reducing uncommon entries in electronic class tsv data files (#458)

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* Hi TN: Address, Electronic, Serial, and Cardinal FST Optimization and Bug Fix (#461)

* Hi TN Address, Electronic, Serial, and Cardinal FST Optimizations and Bug Fixes

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* review feedback changes

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* feedback review changes for address and electronic

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Hi tn v3: staging to main error fixed (#466)

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Signed-off-by: Mariana <47233618+mgrafu@users.noreply.github.com>
Co-authored-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mgrafu mgrafu mgrafu left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@shrpawar-alt @mgrafu