Processing on long video with high resolution

[devidlatkin]

Hello!
Thank you for the amazing framework!

I have an issue while processing on long video with high resolution. I ran out of GPU memory.
As I understand, mivos tries to upload all images directly to GPU and if the video is too long or in high-resolution mivos can't handle such cases.
Is there is a way to fix this issue? Maybe modify code to work with data chunks?

Thank you in advance!

[hkchengrex]

That is correct. Perhaps you can try a different memory profile (e.g. 2, then the images are not cached on GPU):

MiVOS/interactive_gui.py

Line 990 in 3d92044

parser.add_argument('--mem_profile', default=0, type=int, help='0 - Faster and more memory intensive; 2 - Slower and less memory intensive. Default: 0.')

I think it will still struggle if the video is really high-resolution and long... You can also increase mem_freq:

MiVOS/interactive_gui.py

Line 989 in 3d92044

parser.add_argument('--mem_freq', default=5, type=int)

[devidlatkin]

Yeah, it works on CPU (--mem_profile 2) with my video but it takes a lot of time to propagate so it is almost impossible to annotate in real-time =(
It can work on GPU only if I use small chunks of 30 frames but it is really inconvenient to annotate the whole video by such small chunks...

I'm thinking about modifying interactive_gui.py to do propagation by chunks inside the framework. Is it possible to implement such logic? Does it violate some method restrictions?

[hkchengrex]

With mem_profile=2 the algorithm still runs on the GPU. There are just a lot more CPU-to-GPU communications (images are stored on CPU, send to GPU for encoding and propagation, and the results are sent back to CPU) which are still unavoidable even if you do them in chunks... You can perhaps try half precision.

Doing them in chunks does have the benefit of batching the CPU-GPU-CPU transfer so it might still be worth doing. The performance would not be the same, but you can try to implement a behavior where the algorithm discards most frames (except maybe last 5 for memory features) after 30 frames and loads the next 30 frames into the GPU for propagation and so on.

[devidlatkin]

Ok, got it.
Thank you!