How to speed up inference?

[Tareqox]

Currently it takes around 30s-40s on a big machine (with serveral tesla v100) to generate a wav file. How can we speed this process up? On your fish audio page generation audio is much quicker.

[AnyaCoder]

Add command flag --compile

[Tareqox]

--compile is activated and it is still quiet slow.

[AnyaCoder]

Please refer to the speed of the second inference after compilation

[farzadsbmila]

hi @AnyaCoder
Thanks for the hint, the speed of the second inference is pretty fast.
Is compilation a one-time thing? ie. it happens only in the beginning, and not every time a new prompt is given?