When using libtorch, gpu decoding is slower than cpu.

[hms1205]

When using gpu to decode, gpu memory gets allocated but gpu-util rises after a lot of time.
For example, if you proceed with decoding 600 voices, it progresses very slowly until about the 100th, and then speeds up from the point when gpu-util rises.
Increasing the number of threads in decoder_main.cc makes it faster, but I'd like to fix the problem when it's single-threaded.
What should I do?

cpu = 24 cores
gpu = rtx a5000(24gb) x 2
ubuntu 20.04.4

[hms1205]

Same symptoms when tested on cpu.
(Speed ​​increases after a certain number of progress)
There is a big difference in speed in the 'forward_attention_decoder' api part, but I don't know if it's an attention problem or a libtorch problem.

[robin1001]

The slow decoding at the begging maybe caused by libtorch, libtorch model requires warmup.

[hms1205]

The slow decoding at the begging maybe caused by libtorch, libtorch model requires warmup.

What number should I specify when using warmup?
Should I do a test?

[robin1001]

Seems 100 is okay in your testing.

[hms1205]

Seems 100 is okay in your testing.

Thank you. I'll give it a try.

[hms1205]

The slow decoding at the begging maybe caused by libtorch, libtorch model requires warmup.

Sorry, it's an additional question. Why does libtorch need a warmup?

[xingchensong]

The slow decoding at the begging maybe caused by libtorch, libtorch model requires warmup.

Sorry, it's an additional question. Why does libtorch need a warmup?

search the best path to execute forward calculation.