onnx export for G2P models #8177

ppisljar · 2024-01-16T06:19:05Z

What does this PR do ?

Adds support for ONNX export to G2P models (CTC/byT5)

Collection: tts

Jenkins CI

To run Jenkins, a NeMo User with write access must comment jenkins on the PR.

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines

PR Type:

New Feature

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

ppisljar · 2024-01-16T06:21:08Z

Hello,

i am trying to make G2P models exportable, but i am running into issues with ONNX export.
I hope someone here can guide me to the correct solution.

ppisljar · 2024-01-16T06:50:07Z

both (t5 and ctc) return this error:
i am guessing i am doing something wrong with the output types ?

Traceback (most recent call last):
  File "/workspace/6_export.py", line 38, in <module>
    model.export('mymodel.onnx')
  File "/usr/local/lib/python3.10/dist-packages/nemo/core/classes/exportable.py", line 113, in export
    out, descr, out_example = model._export(
  File "/usr/local/lib/python3.10/dist-packages/nemo/core/classes/exportable.py", line 220, in _export
    torch.onnx.export(
  File "/usr/local/lib/python3.10/dist-packages/torch/onnx/utils.py", line 516, in export
    _export(
  File "/usr/local/lib/python3.10/dist-packages/torch/onnx/utils.py", line 1582, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "/usr/local/lib/python3.10/dist-packages/torch/onnx/utils.py", line 1182, in _model_to_graph
    _C._jit_pass_onnx_assign_output_shape(
RuntimeError: ArrayRef: invalid index Index = 0; Length = 0

makes G2P ctc model exportable Signed-off-by: Peter Pisljar <peter.pisljar@gmail.com> Signed-off-by: ppisljar <peter.pisljar@gmail.com>

Signed-off-by: ppisljar <peter.pisljar@gmail.com>

for more information, see https://pre-commit.ci Signed-off-by: ppisljar <peter.pisljar@gmail.com>

ppisljar · 2024-01-16T08:09:06Z

i was able to get onnx export to finish (with a lot of warnings) for CTC model, if i changed the last lines to :

        preds_str, _ = self.decoding.ctc_decoder_predictions_tensor(
            log_probs, decoder_lengths=encoded_len, return_hypotheses=True
        )
        results = [h.y_sequence for h in preds_str]
        return tuple(results)

as it seems that returning a string was an issue ?

however when i try to use this onnx model, it tells me it has 0 inputs:

ort_session = onnxruntime.InferenceSession("mymodel.onnx", providers=["CPUExecutionProvider"])
print(ort_session.get_inputs())

returns []

ppisljar · 2024-01-16T12:14:06Z

in the latest commit e93ffec ctc decoding is skipped, in this way export works succesfully for CTC model as well.

is it not possible to make ctc decoding part of the onnx model ?

Signed-off-by: ppisljar <peter.pisljar@gmail.com>

for more information, see https://pre-commit.ci Signed-off-by: ppisljar <peter.pisljar@gmail.com>

Signed-off-by: ppisljar <peter.pisljar@gmail.com>

for more information, see https://pre-commit.ci Signed-off-by: ppisljar <peter.pisljar@gmail.com>

Signed-off-by: ppisljar <peter.pisljar@gmail.com>

for more information, see https://pre-commit.ci Signed-off-by: ppisljar <peter.pisljar@gmail.com>

Signed-off-by: ppisljar <peter.pisljar@gmail.com>

for more information, see https://pre-commit.ci Signed-off-by: ppisljar <peter.pisljar@gmail.com>

Signed-off-by: ppisljar <peter.pisljar@gmail.com>

for more information, see https://pre-commit.ci Signed-off-by: ppisljar <peter.pisljar@gmail.com>

ppisljar · 2024-01-16T13:22:06Z

i verified the generated onnx models and they seem to work correctly.

still i would really appreciate if someone can guide to towards making ctc decoding part of the nemo model as well as without it its quite hard to consume it.

ppisljar · 2024-01-29T05:13:24Z

@rlangman can you take a look ?

mgrafu · 2024-01-29T19:46:08Z

@borisfom could you take a look at this?

rlangman

I left a few questions, but it would be nice to hear from someone who is more familiar with ONNX and these particular models.

nemo/collections/tts/g2p/models/ctc.py

rlangman · 2024-01-30T18:21:01Z

nemo/collections/tts/g2p/models/ctc.py

+    def output_types(self):
+        return self._output_types
+
+    def input_example(self, max_batch=1, max_dim=44):


It looks like max_batch and max_dim are not used. Are they required by the export code?

nemo/collections/tts/g2p/models/t5.py

borisfom · 2024-02-01T00:29:23Z

@ppisljar @rlangman : For ASR models. it's typical to exclude CTC decoder and tokenizer from ONNX export - as they normally could not be exported. So only DNN part (EncDecModel) is usually represented by ONNX graph.

nemo/collections/tts/g2p/models/ctc.py

mgrafu · 2024-02-05T20:19:48Z

nemo/collections/tts/g2p/models/t5.py

+
+    def forward_for_export(self, input_ids):
+        outputs = self.model.generate(
+            input_ids, output_scores=True, return_dict_in_generate=True, max_length=model_max_target_len


would max_length=model_max_target_len be max_length=self.max_target_len ?

github-actions · 2024-02-20T01:43:36Z

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

Signed-off-by: Peter Pisljar <peter.pisljar@gmail.com>

ekmb · 2024-03-05T00:41:14Z

jenkins

ekmb · 2024-03-05T00:41:51Z

@mgrafu , @borisfom are you OK with the changes?

ekmb · 2024-03-12T16:47:48Z

jenkins

ppisljar · 2024-03-18T06:55:57Z

ping

ekmb · 2024-03-18T08:55:19Z

jenkins

ekmb · 2024-03-18T12:14:23Z

jenkins

ekmb · 2024-03-20T06:27:52Z

jenkins

github-actions bot added the TTS label Jan 16, 2024

ppisljar and others added 3 commits January 16, 2024 07:55

G2P transforer CTC - exportable

7b19c31

makes G2P ctc model exportable Signed-off-by: Peter Pisljar <peter.pisljar@gmail.com> Signed-off-by: ppisljar <peter.pisljar@gmail.com>

g2p onnx export

1dabb91

Signed-off-by: ppisljar <peter.pisljar@gmail.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

396a6c5

for more information, see https://pre-commit.ci Signed-off-by: ppisljar <peter.pisljar@gmail.com>

ppisljar force-pushed the g2p/onnx branch from 01822db to 43bf0af Compare January 16, 2024 06:56

github-actions bot added the NLP label Jan 16, 2024

ppisljar force-pushed the g2p/onnx branch from 43bf0af to 064d422 Compare January 16, 2024 06:59

github-actions bot removed the NLP label Jan 16, 2024

ppisljar and others added 10 commits January 16, 2024 13:15

fixes

464b1ca

Signed-off-by: ppisljar <peter.pisljar@gmail.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

7c184d3

for more information, see https://pre-commit.ci Signed-off-by: ppisljar <peter.pisljar@gmail.com>

fixes

faa141e

Signed-off-by: ppisljar <peter.pisljar@gmail.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

c9ff2a8

for more information, see https://pre-commit.ci Signed-off-by: ppisljar <peter.pisljar@gmail.com>

fixes

40a1f56

Signed-off-by: ppisljar <peter.pisljar@gmail.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

fdbe352

for more information, see https://pre-commit.ci Signed-off-by: ppisljar <peter.pisljar@gmail.com>

fixes

69a27f6

Signed-off-by: ppisljar <peter.pisljar@gmail.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

edc1d21

for more information, see https://pre-commit.ci Signed-off-by: ppisljar <peter.pisljar@gmail.com>

fixes

301027f

Signed-off-by: ppisljar <peter.pisljar@gmail.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

6be6f49

for more information, see https://pre-commit.ci Signed-off-by: ppisljar <peter.pisljar@gmail.com>

ppisljar force-pushed the g2p/onnx branch from 26ec97a to 6be6f49 Compare January 16, 2024 12:16

Merge branch 'main' into g2p/onnx

a2f5948

ppisljar marked this pull request as ready for review January 16, 2024 13:21

ekmb requested a review from mgrafu January 16, 2024 23:17

mgrafu requested a review from borisfom January 29, 2024 19:46

rlangman reviewed Jan 30, 2024

View reviewed changes

borisfom previously approved these changes Feb 1, 2024

View reviewed changes

github-advanced-security bot found potential problems Feb 1, 2024

View reviewed changes

nemo/collections/tts/g2p/models/ctc.py Fixed Show fixed Hide fixed

mgrafu reviewed Feb 5, 2024

View reviewed changes

github-actions bot added the stale label Feb 20, 2024

Update ctc.py

78df720

Signed-off-by: Peter Pisljar <peter.pisljar@gmail.com>

ppisljar dismissed borisfom’s stale review via 78df720 February 26, 2024 10:17

Update t5.py

31dd338

Signed-off-by: Peter Pisljar <peter.pisljar@gmail.com>

github-actions bot removed the stale label Feb 27, 2024

Merge branch 'main' into g2p/onnx

1c0350e

Merge branch 'main' into g2p/onnx

cb6c63d

Merge branch 'main' into g2p/onnx

1bdb8a1

Merge branch 'main' into g2p/onnx

415a67d

ekmb approved these changes Mar 20, 2024

View reviewed changes

ekmb merged commit 3818e42 into NVIDIA:main Mar 20, 2024
10 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

onnx export for G2P models #8177

onnx export for G2P models #8177

ppisljar commented Jan 16, 2024 •

edited

ppisljar commented Jan 16, 2024

ppisljar commented Jan 16, 2024

ppisljar commented Jan 16, 2024

ppisljar commented Jan 16, 2024

ppisljar commented Jan 16, 2024

ppisljar commented Jan 29, 2024

mgrafu commented Jan 29, 2024

rlangman left a comment

rlangman Jan 30, 2024

borisfom commented Feb 1, 2024

mgrafu Feb 5, 2024

github-actions bot commented Feb 20, 2024

ekmb commented Mar 5, 2024

ekmb commented Mar 5, 2024

ekmb commented Mar 12, 2024

ppisljar commented Mar 18, 2024

ekmb commented Mar 18, 2024

ekmb commented Mar 18, 2024

ekmb commented Mar 20, 2024

onnx export for G2P models #8177

onnx export for G2P models #8177

Conversation

ppisljar commented Jan 16, 2024 • edited

What does this PR do ?

Jenkins CI

Before your PR is "Ready for review"

Who can review?

Additional Information

ppisljar commented Jan 16, 2024

ppisljar commented Jan 16, 2024

ppisljar commented Jan 16, 2024

ppisljar commented Jan 16, 2024

ppisljar commented Jan 16, 2024

ppisljar commented Jan 29, 2024

mgrafu commented Jan 29, 2024

rlangman left a comment

Choose a reason for hiding this comment

rlangman Jan 30, 2024

Choose a reason for hiding this comment

borisfom commented Feb 1, 2024

mgrafu Feb 5, 2024

Choose a reason for hiding this comment

github-actions bot commented Feb 20, 2024

ekmb commented Mar 5, 2024

ekmb commented Mar 5, 2024

ekmb commented Mar 12, 2024

ppisljar commented Mar 18, 2024

ekmb commented Mar 18, 2024

ekmb commented Mar 18, 2024

ekmb commented Mar 20, 2024

ppisljar commented Jan 16, 2024 •

edited