How to run inference on larger frames e.g. 360p?

[machinelearnear]

Hola! Thanks for the great work with VRT. I wanted to know if you have any tips and recommendations to how we can run your evaluation code against our own higher resolution frames. It seems from my tests that anything above 180p just runs OOM in a K80 (12G) and a T4 (16G) regardless of the tile size that I use for all models (REDS, Vimeo, etc.). Do you have any advice? Thanks!

[JingyunLiang]

If out-of-memory, try to reduce --tile at the expense of slightly decreased performance. For example, --tile 40 128 128 means dividing the video into 40x128x128 (40 frames, height 128, width 128) clips in testing.

[machinelearnear]

Hi @JingyunLiang ! Thanks for the quick reply. Unfortunately, regardless of the tile size that I use, I'm still getting an OOM error. Here's some sample code to replicate, first to download a video from YT and extract frames, and then to run main_test_vrt.py against that set of frames, with different tile sizes. I used the lowest value == args.window_size as an example. When I run this against anything 180p or below, it works no problem. Any advice would be appreciated, thanks!

from pathlib import Path
import shutil, os

if not os.path.exists("data"): os.makedirs("data")
YouTubeID = 'kgCbG0q4jmc' # random 360p video
OutputFile = 'data/video.mp4'
!youtube-dl -o $OutputFile $YouTubeID --restrict-filenames -f 'best[filesize<50M]'
!rm -r testsets/*
!rm -r results/*
inputFolder = 'testsets/uploaded/000'
Path(inputFolder).mkdir(parents=True, exist_ok=True)
print(f'extracting the video as frames to `{str(inputFolder)}`')
os.system(f'ffmpeg -i {OutputFile} -qscale:v 1 -qmin 1 -qmax 1 -vsync 0  {inputFolder}/frame%08d.png')

Then running your test code

!python main_test_vrt.py --task 001_VRT_videosr_bi_REDS_6frames --folder_lq testsets/uploaded --tile 6 8 8 --tile_overlap 2 20 20

[JingyunLiang]

Maybe loading the whole video into GPU consumes too much GPU memory?

[machinelearnear]

You were right. Loading up more than 5 seconds clips was killing the GPU. Using --tile 32 64 64, I can now process 125 frames (5 seconds) with size 480x360 in a little bit over 1hr on an NVIDIA T4. Is that expected?

[JingyunLiang]

Yes, it is slow is if you test it patch by patch. Testing different patches in parallel can help.