I rewrite the demo of Huggingface t5 to bart, And I inference with method of 'generate' ,and paramters with top_p sampling and return_num_sequences=10

[zzj-otw]

like the screenshot, when return_num_sequences=1, the speed of tensorrt engine is faster 2x than pytorch model. But when i increase 'return_num_sequences', trt engine is gradually slower than pytoch. when return_num_sequences=10 , trt engine is obviously slower than pytorch model. Who konws about it, and how to resolve it? And the batch_size of trt bart_decoder equal to return_num_sequences when export to trt bart decoder

TensorRT Version: 8.2.2
GPU: T4

[zzj-otw]

The main problem is that when batch_size is large, tensorRT is slow and slower than pytorch.
Does anyone know the impact of batch_size on tensorRT?
Any one knows about it? Tanks!

[zzj-otw]

@ttyio @lukeyeager @pbridger @aaronp24 Sorry for spending your time, I don't know how to accelerate tensorRT when batch_size is large. Do you know about that? Thanks for reply!

[zzj-otw]

there is the script filehttps://pan.baidu.com/s/1C_N17uLBbLhJRTCqc3GnRA,password:v27t
in the directory of HuggingFace_hc, with command 'python run.py run Bart trt --variant bart-base --working-dir temp_dir' to run the trt, and with command 'python run.py run Bart frameworks --variant bart-base --working-dir temp_dir' to run frameworks, like t5 and gpt2.

[ttyio]

Hi @yuanhuachao , did you use dynamic shape in your script? It's worth try:

• either batch or sequence is dynamic (only one, not both)
• use static shape

Thanks!

[zzj-otw]

Hi @ttyio, In my script, the sequence is dynamic and batch is static.
And the length of input text is not fixed during generation.
When testing different batches, I need to export different trt engines.
And did you try t5 or gpt2 with batch > 1, for example batch=10？

Thanks!

[pommedeterresautee]

I did for seq len from 6 to 256 on GPT2, TensorRT is significantly faster than Pytorch, but requires to manage carefully IO
Check https://github.com/ELS-RD/transformer-deploy/blob/main/demo/generative-model/gpt2.ipynb for more info (merge to master coming very soon I hope)

@yuanhuachao may I ask you if you have met any difficulty to build Bart tensorRT engine? (I plan to work on T5 and bart soon, not yet really tried, and wondering if it requires some modif to model code)
@vblagoje

[zzj-otw]

@pommedeterresautee Did you try batch size from 1 to 10 on GPT2, is TensorRT faster than Pytorch?
To build Bart tensorRT engine, you just need change the name of 'T5' to 'Bart', and try to run the scripts.

[pommedeterresautee]

I ran a modified version https://github.com/ELS-RD/transformer-deploy/blob/main/demo/generative-model/gpt2.ipynb where tensorrt is built with batch size 8 (instead of 1) and each inference call is modified to perform a beam search .generate(inputs.input_ids, max_length=256, num_beams=8) (note num_beams=8). No other modification.

Latencies are:

• Pytorch: 5.15s/sequence
• ONNX Runtime (standard API): 30.81s/sequence
• ONNX Runtime (binding IO API): 2.62/sequence
• TensorRT: 3.10/sequence

So you are kind of right, TensorRT is less performant than ONNX Runtime and its optimizations but still better than Pytorch.

[zzj-otw]

@pommedeterresautee I want to run your scripts, but i cant't pull the docker with command
'docker run -it --rm --gpus all -v $PWD:/project ghcr.io/els-rd/transformer-deploy:0.4.0 bash -c "cd /project && convert_model -m gpt2--backend tensorrt onnx --seq-len 6 256 256 --task text-generation"'.
And i have downloaded the code from github, i want to know how to run scripts to test the speed of gpt2 on pytorch, onnx and tensorrt.

[pommedeterresautee]

What issue do you have with docker?
One thing, the image is very big, if there are some checksum issues don't hesitate to delete the related cache.

If you run locally, and have installed all Nvidia dependencies and the package like described in the documentation, just launch the Jupyter notebook? (I may miss something)