GPU usage limited to <10%

[jagadeeshmadinni]

Hi
I cloned the AdvTrajectoryPrediction repo and completed the set up following the instructions. I am running the test.py on a high performance compute cluster with access to a Tesla V100 GPU but noticed that multi_frame mode tests do not complete even after 12 hours. When I check GPU utilization it's in the range of 0-7%.

I know that this code was originally written for an RTX 2080 with CUDA 10.2 and the Trajectron model that I am using for testing also optimized their code for GPU but I am unable to locate the source of delay even after going through all the files.

Is it supposed to take this long or am I missing something? How can I make the testing faster?

Below is the command I am running:

python test.py --dataset apolloscape --model trajectron --mode single_frame --overwrite

Thanks in advance!