CUDA memory usage varies

[yifanlu0227]

Hi runsheng,

Thanks for your framework again! One thing bothers me that I found that the CUDA memory usage varies a lot at runtime. Is there any solution to fix the memory usage? Then my program would not be preempted by others :).

Thank you in advance.

[DerrickXuNu]

Hi Yifan, is it during training or inference?

[yifanlu0227]

during the training phase

[DerrickXuNu]

I see. This is probably caused by different frames have different number of cavs. Sometime there are 3 agents but sometime 5.