Out of memory error when refacing

[ooofest]

This was happening in a similar manner yesterday, but as of the main branch at commit 63d501c , I am getting this error when attempting to run Refacer on a 380MB MP4 (h264):

Trying FFMPEG h264_nvenc encoder
FFMPEG h264_nvenc encoder works
Video codec for FFMPEG: h264_nvenc
CUDA mode with providers ['CUDAExecutionProvider', 'CPUExecutionProvider']
inswapper-shape: [1, 3, 128, 128]
Running on local URL: http://127.0.0.1:7860

To create a public link, set share=True in launch().
Total frames: 52567
Extracting frames: 21%|████████████ | 11276/52567 [00:20<01:14, 557.87it/s]
Traceback (most recent call last):
File "D:\refacer-main\venv\lib\site-packages\gradio\routes.py", line 427, in run_predict
output = await app.get_blocks().process_api(
File "D:\refacer-main\venv\lib\site-packages\gradio\blocks.py", line 1323, in process_api
result = await self.call_function(
File "D:\refacer-main\venv\lib\site-packages\gradio\blocks.py", line 1051, in call_function
prediction = await anyio.to_thread.run_sync(
File "D:\refacer-main\venv\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "D:\refacer-main\venv\lib\site-packages\anyio_backends_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
File "D:\refacer-main\venv\lib\site-packages\anyio_backends_asyncio.py", line 807, in run
result = context.run(func, *args)
File "D:\refacer-main\app.py", line 31, in run
return refacer.reface(video_path,faces)
File "D:\refacer-main\refacer.py", line 196, in reface
flag, frame = cap.read()
cv2.error: OpenCV(4.7.0) D:\a\opencv-python\opencv-python\opencv\modules\core\src\alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 2764800 bytes in function 'cv::OutOfMemoryError'

Although I have configured to use GPU (3090 with 24GB), the system RAM shot up from 17GB used to over 53GB (out of 64GB total) and I did not see GPU memory being allocated in the monitor.

As mentioned, I have experienced an out of memory error since the new (faster) codebase was pushed in the past day or so. While some files can process fine, I notice that smaller files are still hit-or-miss, so I can't supply a definite criteria of when this might occur.

[xaviviro]

That makes sense, as one of the features that enables the program to run so quickly is by first loading all the frames into memory and then processing them in parallel threads. This works well as it is incredibly fast, as long as you have enough memory. A potential solution could be to work in batches of frames and then merge them. Sharing this kind of error helps me a lot to improve the software. Thank you.

[ooofest]

Thanks for looking into it.

It does seem odd, because in this case I'm wondering why the extracted images would require so much memory when the original video was only 330MB. Even if we generously offer that each of the extracted 52K frames was 2MB in size, that doesn't add up to 32GB that Refacer apparently allocated from system RAM before it hit logic which said that it ran out of headroom.

And at the point where it stopped processing, it stopped due to a reported lack of 3MB - but that was still easily available from both system RAM and GPU VRAM.

So, I feel that looking into the memory management here - preloading or not - will only be a good thing.

It might be a consideration to write the extracted frames to a temporary folder and then work on them in parallel from there. Not as fast as keeping everything in memory, but as you implied loading every video's full set of frames into memory will have issues on systems without lots of memory (and in my case, 64GB RAM and 24GB of VRAM would seem to be plenty for processing a 33OMB h264 file,)

[xaviviro]

Could you please post the result of running:

ffprobe -show_streams -select_streams v YOUR_VIDEO.mp4

[rmhckrkz]

Nvidia RTX 2070

C:\Python\REFACE\refacer>python app.py
Trying FFMPEG h264_nvenc encoder
FFMPEG h264_nvenc encoder works
Video codec for FFMPEG: h264_nvenc
CUDA mode with providers ['CUDAExecutionProvider', 'CPUExecutionProvider']
inswapper-shape: [1, 3, 128, 128]
Running on local URL: http://127.0.0.1:7860

Total frames: 2757 link, set share=True in launch().
Extracting frames: 41%|████████████████▎ | 1126/2757 [00:02<00:03, 472.68it/s]
Traceback (most recent call last):
File "C:\Users\RMuratuly\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\gradio\routes.py", line 427, in run_predict
output = await app.get_blocks().process_api(
File "C:\Users\RMuratuly\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\gradio\blocks.py", line 1323, in process_api
result = await self.call_function(
File "C:\Users\RMuratuly\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\gradio\blocks.py", line 1051, in call_function
prediction = await anyio.to_thread.run_sync(
File "C:\Users\RMuratuly\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\anyio\to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "C:\Users\RMuratuly\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\anyio_backends_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
File "C:\Users\RMuratuly\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\anyio_backends_asyncio.py", line 807, in run
result = context.run(func, *args)
File "C:\Python\REFACE\refacer\app.py", line 31, in run
return refacer.reface(video_path,faces)
File "C:\Python\REFACE\refacer\refacer.py", line 196, in reface
flag, frame = cap.read()
cv2.error: OpenCV(4.7.0) D:\a\opencv-python\opencv-python\opencv\modules\core\src\alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 2764800 bytes in function 'cv::OutOfMemoryError'
ffmpeg.txt

[ooofest]

ffprobe -show_streams -select_streams v "D:\test_refacer.mp4"

ffprobe version 6.0-full_build-www.gyan.dev Copyright (c) 2007-2023 the FFmpeg developers
built with gcc 12.2.0 (Rev10, Built by MSYS2 project)
configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-bzlib --enable-lzma --enable-libsnappy --enable-zlib --enable-librist --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-sdl2 --enable-libaribb24 --enable-libdav1d --enable-libdavs2 --enable-libuavs3d --enable-libzvbi --enable-librav1e --enable-libsvtav1 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libaom --enable-libjxl --enable-libopenjpeg --enable-libvpx --enable-mediafoundation --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-liblensfun --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libvpl --enable-libshaderc --enable-vulkan --enable-libplacebo --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libilbc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chromaprint
libavutil 58. 2.100 / 58. 2.100
libavcodec 60. 3.100 / 60. 3.100
libavformat 60. 3.100 / 60. 3.100
libavdevice 60. 1.100 / 60. 1.100
libavfilter 9. 3.100 / 9. 3.100
libswscale 7. 1.100 / 7. 1.100
libswresample 4. 10.100 / 4. 10.100
libpostproc 57. 1.100 / 57. 1.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'D:\test_refacer.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf55.33.100
Duration: 00:29:12.29, start: 0.000000, bitrate: 1805 kb/s
Stream #0:00x1: Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 1280x720 [SAR 1:1 DAR 16:9], 1672 kb/s, 30 fps, 30 tbr, 15360 tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
Stream #0:10x2: Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 124 kb/s (default)
Metadata:
handler_name : SoundHandler
vendor_id : [0][0][0][0]
[STREAM]
index=0
codec_name=h264
codec_long_name=H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
profile=High
codec_type=video
codec_tag_string=avc1
codec_tag=0x31637661
width=1280
height=720
coded_width=1280
coded_height=720
closed_captions=0
film_grain=0
has_b_frames=2
sample_aspect_ratio=1:1
display_aspect_ratio=16:9
pix_fmt=yuv420p
level=31
color_range=unknown
color_space=unknown
color_transfer=unknown
color_primaries=unknown
chroma_location=left
field_order=progressive
refs=1
is_avc=true
nal_length_size=4
id=0x1
r_frame_rate=30/1
avg_frame_rate=30/1
time_base=1/15360
start_pts=0
start_time=0.000000
duration_ts=26914304
duration=1752.233333
bit_rate=1672563
max_bit_rate=N/A
bits_per_raw_sample=8
nb_frames=52567
nb_read_frames=N/A
nb_read_packets=N/A
extradata_size=42
DISPOSITION:default=1
DISPOSITION:dub=0
DISPOSITION:original=0
DISPOSITION:comment=0
DISPOSITION:lyrics=0
DISPOSITION:karaoke=0
DISPOSITION:forced=0
DISPOSITION:hearing_impaired=0
DISPOSITION:visual_impaired=0
DISPOSITION:clean_effects=0
DISPOSITION:attached_pic=0
DISPOSITION:timed_thumbnails=0
DISPOSITION:captions=0
DISPOSITION:descriptions=0
DISPOSITION:metadata=0
DISPOSITION:dependent=0
DISPOSITION:still_image=0
TAG:language=und
TAG:handler_name=VideoHandler
TAG:vendor_id=[0][0][0][0]
[/STREAM]

[rmhckrkz]

@xaviviro @ooofest @alunit3 @fAIseh00d any progress, gentleman?

[fAIseh00d]

@rmhckrkz @ooofest
Might I suggest a quick workaround, so you don't have to wait for a "proper" fix?
You can split your videos without re-encoding them using this.
720p video will result in ~(1280x720x3)bytes=2.63Mb per frame depending on the format, so 16Gb can hold roughly 6200 frames.
The video in question is 52567 frames long, so it requires ~135Gb RAM. Splitting the video into 3 parts probably will suffice for 64Gb RAM.

[ooofest]

Thanks, I was using a different splitter program (paid) for the same effect, but felt this bug was useful to offer from both usability and long-term memory management perspectives, as well.

And you helped clarify that I was accidentally off in my quickie calculations for space needed to process this file, carried the decimal point in the wrong position - whoops.

Which still doesn't explain why even a 1280x720 mp4 with 10.6K frames would run into memory allocation errors about 1/3 of the way into face processing, considering there remains plenty of RAM and VRAM in this system. It doesn't seem that raw size of the file's frames is the only factor in this issue.

[shyt47]

Has this been resolved yet? I also get this error.