Update text processing in Magpietts eval

[rlangman]

What does this PR do ?

Adds an interface for doing language-specific text processing for TTS.

Collection: [TTS]

Changelog

• Adds a interface TextProcessor which can be used fornormalization of training/eval data, and for processing text for WER calculation.
• Adds English implementation of TextProcessor to Magpie evaluation code. This reduces false positives in TTS WER caused by ASR models producing text in written (instead of spoken) form.
• Updated punctuation removal to use isalnum(). Previous logic using string.punctuation only removes a limited set of English punctuation.
• Added datasets_base_path to allow doing inference and evaluation using a config file with relative paths.
• Modified Longform CI to use a manifest with a "normalized_text" field. Otherwise, tests had higher WER because it was feeding Magpie raw text with several digits in its input.

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

• New Feature
• Bugfix
• Documentation

[rfejgin]

Overall looks good to me. Left some minor comments.

[rfejgin]

All looks good. One question: have we tested the updated code at all on non-English? At least to the extent that we'd expect WERs to not be higher than before the change, or even to be lower. One possible way to test that while removing variability due to the TTS model would be to run evaluation on a pre-populated directory of Magpie outputs (if the code allows doing that easily), once before the changes and once after.

[blisc]

This function and https://github.com/NVIDIA-NeMo/NeMo/blob/main/nemo/collections/tts/parts/utils/helpers.py#L810 needs to be de-duplicated before we merge. The functionalities of process_text_for_cer() should be merged in and GRPO should use this text processor.

[rlangman]

That method is used by 3 different recipes: Magpie GPRO, Easy Magpie, and Magpie GRPO, and the refactoring needed for each of those is complicated both to implement and test. It is too large and dangerous of a change to couple with this PR, which is relatively small and required for us to proceed with updating our evaluation datasets.

I could implement something and rely on CI tests to validate each recipe still runs, but that is not safe because it is changing the GRPO criteria itself.

[rlangman]

All looks good. One question: have we tested the updated code at all on non-English? At least to the extent that we'd expect WERs to not be higher than before the change, or even to be lower. One possible way to test that while removing variability due to the TTS model would be to run evaluation on a pre-populated directory of Magpie outputs (if the code allows doing that easily), once before the changes and once after.

Using the same Magpie outputs I ran evaluation on CML test (french, german, italian, spanish) and all metrics were identical, except CER on french dropped from 0.022 to 0.019 because the updated logic with isalnum() removes french quotation marks before CER calculation.

Unblock

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[rlangman]

/ok to test 418f0d3

[github-actions]

[🤖]: Hi @rlangman 👋,

We wanted to let you know that a CICD pipeline for this PR just finished successfully.

So it might be time to merge this PR or get some approvals.

[rlangman]

/ok to test d85344d

[github-actions]

[🤖]: Hi @rlangman 👋,

We wanted to let you know that a CICD pipeline for this PR just finished successfully.

So it might be time to merge this PR or get some approvals.