-
Notifications
You must be signed in to change notification settings - Fork 123
Out of memory error when refacing #18
Comments
That makes sense, as one of the features that enables the program to run so quickly is by first loading all the frames into memory and then processing them in parallel threads. This works well as it is incredibly fast, as long as you have enough memory. A potential solution could be to work in batches of frames and then merge them. Sharing this kind of error helps me a lot to improve the software. Thank you. |
Thanks for looking into it. It does seem odd, because in this case I'm wondering why the extracted images would require so much memory when the original video was only 330MB. Even if we generously offer that each of the extracted 52K frames was 2MB in size, that doesn't add up to 32GB that Refacer apparently allocated from system RAM before it hit logic which said that it ran out of headroom. And at the point where it stopped processing, it stopped due to a reported lack of 3MB - but that was still easily available from both system RAM and GPU VRAM. So, I feel that looking into the memory management here - preloading or not - will only be a good thing. It might be a consideration to write the extracted frames to a temporary folder and then work on them in parallel from there. Not as fast as keeping everything in memory, but as you implied loading every video's full set of frames into memory will have issues on systems without lots of memory (and in my case, 64GB RAM and 24GB of VRAM would seem to be plenty for processing a 33OMB h264 file,) |
Could you please post the result of running: ffprobe -show_streams -select_streams v YOUR_VIDEO.mp4 |
Nvidia RTX 2070 C:\Python\REFACE\refacer>python app.py Total frames: 2757 link, set |
ffprobe -show_streams -select_streams v "D:\test_refacer.mp4" ffprobe version 6.0-full_build-www.gyan.dev Copyright (c) 2007-2023 the FFmpeg developers |
@xaviviro @ooofest @alunit3 @fAIseh00d any progress, gentleman? |
@rmhckrkz @ooofest |
Thanks, I was using a different splitter program (paid) for the same effect, but felt this bug was useful to offer from both usability and long-term memory management perspectives, as well. And you helped clarify that I was accidentally off in my quickie calculations for space needed to process this file, carried the decimal point in the wrong position - whoops. Which still doesn't explain why even a 1280x720 mp4 with 10.6K frames would run into memory allocation errors about 1/3 of the way into face processing, considering there remains plenty of RAM and VRAM in this system. It doesn't seem that raw size of the file's frames is the only factor in this issue. |
Has this been resolved yet? I also get this error. |
This was happening in a similar manner yesterday, but as of the main branch at commit 63d501c , I am getting this error when attempting to run Refacer on a 380MB MP4 (h264):
Trying FFMPEG h264_nvenc encoder
FFMPEG h264_nvenc encoder works
Video codec for FFMPEG: h264_nvenc
CUDA mode with providers ['CUDAExecutionProvider', 'CPUExecutionProvider']
inswapper-shape: [1, 3, 128, 128]
Running on local URL: http://127.0.0.1:7860
To create a public link, set
share=True
inlaunch()
.Total frames: 52567
Extracting frames: 21%|████████████ | 11276/52567 [00:20<01:14, 557.87it/s]
Traceback (most recent call last):
File "D:\refacer-main\venv\lib\site-packages\gradio\routes.py", line 427, in run_predict
output = await app.get_blocks().process_api(
File "D:\refacer-main\venv\lib\site-packages\gradio\blocks.py", line 1323, in process_api
result = await self.call_function(
File "D:\refacer-main\venv\lib\site-packages\gradio\blocks.py", line 1051, in call_function
prediction = await anyio.to_thread.run_sync(
File "D:\refacer-main\venv\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "D:\refacer-main\venv\lib\site-packages\anyio_backends_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
File "D:\refacer-main\venv\lib\site-packages\anyio_backends_asyncio.py", line 807, in run
result = context.run(func, *args)
File "D:\refacer-main\app.py", line 31, in run
return refacer.reface(video_path,faces)
File "D:\refacer-main\refacer.py", line 196, in reface
flag, frame = cap.read()
cv2.error: OpenCV(4.7.0) D:\a\opencv-python\opencv-python\opencv\modules\core\src\alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 2764800 bytes in function 'cv::OutOfMemoryError'
Although I have configured to use GPU (3090 with 24GB), the system RAM shot up from 17GB used to over 53GB (out of 64GB total) and I did not see GPU memory being allocated in the monitor.
As mentioned, I have experienced an out of memory error since the new (faster) codebase was pushed in the past day or so. While some files can process fine, I notice that smaller files are still hit-or-miss, so I can't supply a definite criteria of when this might occur.
The text was updated successfully, but these errors were encountered: