Prediction runs slowly

[matnshrn]

Hi! Thank you so much for this project.
I'm trying to speed up the running time of the predictions, but even when running on GPU, it still takes a long time.
Is there any way to speed up the prediction? (I have tried to increase the data loader workers to 10 and batch size to 1000, but it didn't help).
Thank you!

[gabrielmittag]

How many files do you have and how long does it take? With a GPU the inference time should be really quick but the best settings for number of workers and batch size depend on your system. Maybe you could try just 4 workers and a batch size of 150. Also, how long are the durations of your files?

[matnshrn]

Thanks for your help!
I have about 100K files, each of length ~10 seconds, and from what I've seen so far it takes about 10 minutes for 100 files.
I didn't see a different between using GPU and CPU, the speed remain approximately the same.

[gabrielmittag]

That is too long, even on CPU 100 files should just take around 10-20 seconds. Not sure what is wrong here. Which OS are you running it on? Did you use the env.yml for installing a new conda environment? Which model weights are you using?

[gabrielmittag]

Your problem might actually be the large batch size. You may want to try something smaller and then see how high you can go. Maybe start with 8.