Make T5 compatible with ONNX

[abelriboulot]

This is a small PR to make T5 exportable to ONNX with any op>9. It addresses an issue outlined in #5075 where T5 would not export to ONNX. In order to make it exportable, 2 changes are made:

• A torch.einsum is replaced with a tensor multiplication in 96d0ec7 since onnx does not currently support this notation
• Decoder inputs / embeddings are defaulted to the encoder's inputs / embeddings if they are not declared. I believe this is clearer as most of the examples right now include something along the lines of model(input_ids=input_ids, decoder_input_ids=input_ids). It also allows t5 to be executed with the more common paradigm of calls like model(inputs)

[codecov]

Codecov Report

Merging #5518 into master will decrease coverage by 1.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #5518      +/-   ##
==========================================
- Coverage   77.83%   76.81%   -1.03%     
==========================================
  Files         141      141              
  Lines       24634    24637       +3     
==========================================
- Hits        19175    18925     -250     
- Misses       5459     5712     +253     

Impacted Files	Coverage Δ
src/transformers/modeling_t5.py	84.44% <100.00%> (+0.09%)	⬆️
src/transformers/modeling_tf_mobilebert.py	23.62% <0.00%> (-73.11%)	⬇️
src/transformers/modeling_tf_electra.py	26.92% <0.00%> (-68.47%)	⬇️
src/transformers/modeling_openai.py	81.09% <0.00%> (+1.37%)	⬆️
src/transformers/generation_tf_utils.py	86.71% <0.00%> (+1.50%)	⬆️
src/transformers/modeling_tf_distilbert.py	98.76% <0.00%> (+32.51%)	⬆️
src/transformers/modeling_tf_openai.py	94.98% <0.00%> (+74.19%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 58cca47...904fa94. Read the comment docs.

[mfuntowicz]

Thanks for investigating, your PR looks very cool.

Just one comment so far, which is more related to the coding style we use in the repo.

Happy to merge right after 👍.

[abelriboulot]

Thanks a lot for the review @mfuntowicz! I adjusted it to the coding style you outlined. Feel free to merge if you're happy with it.

[mfuntowicz]

LGTM! Thanks @abelriboulot, great addition 👍

[ConProgramming]

Hey, did you happen to make a colab which shows this off? I was trying to figure out exporting T5 as ONNX a week ago, but got stuck. It seems you've fixed it though?

[abelriboulot]

@ConProgramming sure thing, I’ll share something this weekend!

[ConProgramming]

@abelriboulot Did you ever get around to making that colab? It'd help a lot. 😅

[abelriboulot]

Hey @ConProgramming, I had a very ad-hoc solution for this, therefore I worked on a PR to make the huggingface conversion compatible with all models with a compatible graph. You can take a look at it there: #5687
If you pull this version you should be able to export T5 with the following line:
python convert_graph_to_onnx.py --framework pt --model t5-base ~/test-t5/t5.onnx --check-loading --opset 12

I checked and it seems to work well! Let me know if it works for you.

[ConProgramming]

Thanks @abelriboulot, but I'm still having some issues with it... it works with t5-base, but depending on how I provide the path to my own model I get two different errors:

• !python transformers/src/transformers/convert_graph_to_onnx.py --framework pt --model "drive/My Drive/paraphraser/t5_paraphrase/pytorch_model.bin" onnx/paraphraser.onnx --check-loading --opset 12 : Error while converting the model: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
• drive/My Drive/paraphraser/t5_paraphrase : Error while converting the model: Model name 'drive/My Drive/paraphraser/t5_paraphrase' was not found in tokenizers model name list (t5-small, t5-base, t5-large, t5-3b, t5-11b). We assumed 'drive/My Drive/paraphraser/t5_paraphrase' was a path, a model identifier, or url to a directory containing vocabulary files named ['spiece.model'] but couldn't find such vocabulary files at this path or url.

Is it designed to work with finetuned models?

[abelriboulot]

Hey @ConProgramming, it should work on fine tuned models, you can have a look at the test_onnx file as an example of this. The model path should be to the directory that contains the model (and the tokenizer in case you do not specify it). It looks like the second error relates to not being able to find a tokenizer, is it present in your directory? If you are using another directory / pretrained model you can specify it with --tokenizer
If you still have issues and it's something you can share, I'm happy to have a look and help you with this.

[ConProgramming]

@abelriboulot Adding --tokenizer t5-base fixed the issue and exported a model without any errors... looks like it worked, thanks again!!

[abelriboulot]

Oh awesome! Great to hear it! I might add a message to make it more obvious to the user.

[spookypineapple]

@abelriboulot

I tried this (for cpu):
convert_graph_to_onnx.py --framework=pt --tokenizer=t5-base --model=t5-base onnx\t5.onnx --check-loading --opset=12

but getting error:

ONNX opset version set to: 12
Loading pipeline (model: t5-base, tokenizer: t5-base)
Some weights of T5Model were not initialized from the model checkpoint at t5-base and are newly initialized: ['encoder.embed_tokens.weight', 'decoder.embed_tokens.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Using framework PyTorch: 1.5.1+cpu
Found input input_ids with shape: {0: 'batch', 1: 'sequence'}
Found input attention_mask with shape: {0: 'batch', 1: 'sequence'}
Found output output_0 with shape: {0: 'batch', 1: 'sequence'}
Error while converting the model: 'BaseModelOutputWithPast' object has no attribute 'shape'

Am I doing something wrong here?

[abelriboulot]

Hey @oliversms, are you using the specific fork or master? I can confirm the command you submitted works on my side.

[spookypineapple]

Apologies for the delayed reply; Im actually using the fork. I beleive it may have been an env related issue. However after getting past that issue Im now running into a new issue:
Specifically on this line:
tokens = nlp.tokenizer("This is a sample output", return_tensors=framework)
getting this error:
ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length.

Attempting set padding & truncation to True doesnt fix the issue.

[abelriboulot]

Hey @oliversms! It looks like you are not using the right branch. You need this specific branch for it to work. Hope it works for you!

Abel

[abelriboulot]

If anyone needs, I created a small package (onnxt5) which lets you easily and efficiently export T5 and serve it! Feel free to raise issues, it's an alpha at the moment.

[patrickvonplaten]

This actually does not make much sense - if no decoder_input_ids are provided they should not just be set to the encoder_input_ids IMO. @mfuntowicz - can we revert this change? or will it break onnx support?

[TanAidan]

@abelriboulot Hi, I pulled your branch and tried to convert a t5-base with
python ../transformers/src/convert_graph_to_onnx.py --framework pt --model t5-base t5-base.onnx --check-loading --opset 12

and still got the "Error while converting the model: You have to specify either decoder_input_ids or decoder_inputs_embeds" Any ideas?