How to do inference for ONNX model?

[neso613]

Hi
Please share steps for onnx model inferecing and also share pretrain weight to convert glowTTS to onnx.

Thanks

[neso613]

@synesthesiam looking forward to hearing from you

[synesthesiam]

Hi @neso613,

Onnx export happens here: https://github.com/rhasspy/glow-tts-train/blob/master/glow_tts_train/export_onnx.py

The most important code for export starts here:

glow-tts-train/glow_tts_train/export_onnx.py

Line 65 in 3e16fee

model.eval()

Inference is here: https://github.com/rhasspy/glow-tts-train/blob/master/glow_tts_train/infer_onnx.py

This commit contains many of the changes I made to get Onnx export working: 96452c8

[neso613]

Hi @synesthesiam
I have tried the apporoach you suggested but still no success for inference
I am adding error below, pls do needful

And I think this issue is because of this dummy_input link

[neso613]

@synesthesiam converted onnx model takes dummy input as fixed shape and do not work on dynamic input shape. Your inputs required.

microsoft/onnxruntime#10657

[neso613]

@synesthesiam
If I used this code for exporting model
`
sequences = torch.randn(1,21, requires_grad=True).cpu().long()
sequence_lengths = torch.randn([21],requires_grad=True).cpu().long()
scales = torch.FloatTensor([0.667, 1.0])

dummy_input = (sequences, sequence_lengths, scales)

# Export
torch.onnx.export(
    model,
    dummy_input,
    str(args.output / "generator.onnx"),
    opset_version=OPSET_VERSION,
    do_constant_folding=True,
    export_params=True,
    input_names=["input", "input_lengths", "scales"],
    output_names=["output"],
    dynamic_axes={
        "input": {0: "batch_size", 1: "phonemes"},
        "input_lengths": {0: "batch_size"},
        "output": {0: "batch_size", 1: "time"},
    },
)

`

Then, I get error
Traceback (most recent call last): File "/home/cogknit/anaconda3/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/cogknit/anaconda3/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/cogknit/DGX/TTS/glow-tts-train/glow_tts_train/export_onnx.py", line 136, in <module> main() File "/home/cogknit/DGX/TTS/glow-tts-train/glow_tts_train/export_onnx.py", line 114, in main torch.onnx.export( File "/home/cogknit/DGX/TTS/glow-tts-train/.venv/lib/python3.8/site-packages/torch/onnx/__init__.py", line 316, in export return utils.export(model, args, f, export_params, verbose, training, File "/home/cogknit/DGX/TTS/glow-tts-train/.venv/lib/python3.8/site-packages/torch/onnx/utils.py", line 107, in export _export(model, args, f, export_params, verbose, training, input_names, output_names, File "/home/cogknit/DGX/TTS/glow-tts-train/.venv/lib/python3.8/site-packages/torch/onnx/utils.py", line 724, in _export _model_to_graph(model, args, verbose, input_names, File "/home/cogknit/DGX/TTS/glow-tts-train/.venv/lib/python3.8/site-packages/torch/onnx/utils.py", line 493, in _model_to_graph graph, params, torch_out, module = _create_jit_graph(model, args) File "/home/cogknit/DGX/TTS/glow-tts-train/.venv/lib/python3.8/site-packages/torch/onnx/utils.py", line 437, in _create_jit_graph graph, torch_out = _trace_and_get_graph_from_model(model, args) File "/home/cogknit/DGX/TTS/glow-tts-train/.venv/lib/python3.8/site-packages/torch/onnx/utils.py", line 388, in _trace_and_get_graph_from_model torch.jit._get_trace_graph(model, args, strict=False, _force_outplace=False, _return_inputs_states=True) File "/home/cogknit/DGX/TTS/glow-tts-train/.venv/lib/python3.8/site-packages/torch/jit/_trace.py", line 1166, in _get_trace_graph outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs) File "/home/cogknit/DGX/TTS/glow-tts-train/.venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "/home/cogknit/DGX/TTS/glow-tts-train/.venv/lib/python3.8/site-packages/torch/jit/_trace.py", line 127, in forward graph, out = torch._C._create_graph_by_tracing( File "/home/cogknit/DGX/TTS/glow-tts-train/.venv/lib/python3.8/site-packages/torch/jit/_trace.py", line 118, in wrapper outs.append(self.inner(*trace_inputs)) File "/home/cogknit/DGX/TTS/glow-tts-train/.venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "/home/cogknit/DGX/TTS/glow-tts-train/.venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1090, in _slow_forward result = self.forward(*input, **kwargs) File "/home/cogknit/DGX/TTS/glow-tts-train/glow_tts_train/export_onnx.py", line 76, in infer_forward (mel, mel_lengths, *_), _, _ = old_forward( File "/home/cogknit/DGX/TTS/glow-tts-train/glow_tts_train/models.py", line 324, in forward x_m, x_logs, logw, x_mask = self.encoder(x, x_lengths, g=g) File "/home/cogknit/DGX/TTS/glow-tts-train/.venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "/home/cogknit/DGX/TTS/glow-tts-train/.venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1090, in _slow_forward result = self.forward(*input, **kwargs) File "/home/cogknit/DGX/TTS/glow-tts-train/glow_tts_train/models.py", line 121, in forward x = self.emb(x) * math.sqrt(self.hidden_channels) # [b, t, h] File "/home/cogknit/DGX/TTS/glow-tts-train/.venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "/home/cogknit/DGX/TTS/glow-tts-train/.venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1090, in _slow_forward result = self.forward(*input, **kwargs) File "/home/cogknit/DGX/TTS/glow-tts-train/.venv/lib/python3.8/site-packages/torch/nn/modules/sparse.py", line 158, in forward return F.embedding( File "/home/cogknit/DGX/TTS/glow-tts-train/.venv/lib/python3.8/site-packages/torch/nn/functional.py", line 2044, in embedding return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse) IndexError: index out of range in self

[neso613]

This issue is basically - while exportin torch model to onnx, we have specified dummy_input and that dummy_input shape is somehow taken as fixed parameter to onnx model. Thats why it hold that dummy_shape as fixed and cannot taken dynamic input shape.
Model Input graph -

Model Output graph -

This is the code-

[neso613]

While analysing model graph, I found -

Here is the issue at this node -

Please suggest some way for this error or this is a BUG.

[synesthesiam]

@neso613 I started seeing similar problems after upgrading my setup. I've since moved on to a different TTS model, so I'm not sure where the problem actually was 🙁

[neso613]

Thanks @synesthesiam for confirming the error.