Iterating 4K GoPro video using python API, results in huge memory leaks and process killed

[amorul-artei]

I am doing basic usage: instantiate reader, read frames, skip frames. 5 minutes into the video, process memory allocation is already at 5GB. By minute 10 it goes up to 14GB. ImageIO needs less than 100MB for the same file (with 30% slower performance)

class DecordVideoIterator:
def init( self, videoPathName ):
self.videoReader = VideoReader( videoPathName, ctx = cpu(0) )
self.currentIndex = 0

def ReadNextFrame( self ):
    gc.collect()
    nextFrame = self.videoReader.next().asnumpy()
    self.currentIndex += 1
    return nextFrame
    
def SkipFrames( self, count ):
    self.videoReader.skip_frames( count )
    self.currentIndex += count

def CurrentIndex( self ):
    return self.currentIndex

Substituting this iterator with ImageIO python library, everything works fine. Object tracker and tracemalloc shows no Python leaks that I coudl identify. Leaks must be in the native code.

[Yves33]

Same here. if my app does not consume frames, the memory usage increases till segfault or eof. The problem appears with ctx=cpu(0) but not ctx=gpu(0)

steps to reproduce:
wget https://www.libde265.org/hevc-bitstreams/tos-4096x1720-tiles.mkv

from decord import VideoReader, cpu,gpu
vr=VideoReader('./tos-4096x1720-tiles.mkv',ctx=cpu(0))
f=vr.next()
res=input("press key to interrupt: ")

on a 64GB system
(GNU/Linux Fedora 35). decord compiled from latest git with ctx=cpu(0) ->crash with ctx=cpu(0), not gpu(0)
(Windows 10/anaconda). decord installed from pip->no crash, but memory usage peaks at >59GB

[TomRaz]

Having the same issue here with the pytorch bridge.

[leotac]

Is that with the latest version? I've been using an older version (0.4.1) for a long time with no issues, but my videos are typically rather short (<30 seconds).

[Yves33]

On windows, version 0.6.0 from pypi (no crash but eats up to 64GB memory - surelly would eat more if possible!).
On linux, compiled from latest github (which has not changed within the last 9 months).

How does your memory usage increase if you do not consume the decoded frames, or consume them too slowly?

[leotac]

Ah that's a big video :)
Not sure, I have never tried with such a big one.

[Yves33]

it seems that the problem can be solved by setting environnement variable
export DECORD_EOF_RETRY_MAX=128
or any low value (default is 10240)
although video_reader.cc mentions that this variable is only for corrupted streams (well, maybe most of my streams are corrupted)!

[ashwhall]

I have the same issue. If I leave a VideoReader instance idle, it just continues to consume memory, I assume as it is pre-reading and caching. If I seek elsewhere in the video (in particular, backwards) the memory drops quickly.

I tried setting the above flag and it didn't change anything - it just consumes memory until segfault.

[ruancomelli]

Building up on @ashwhall's idea, I'm getting consistent results by always doing video_reader.seek(0) after any operations. I eventually wrote a wrapper class that looks like this:

import decord

class VideoReaderWrapper(decord.VideoReader):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.seek(0)

    def __getitem__(self, key):
        frames = super().__getitem__(key)
        self.seek(0)
        return frames

    # and similarly for the other methods I need

The only thing I still have to be careful with is when running time-consuming methods like VideoReader.get_batch. When trying to get a batch of e.g. 1000 frames, since this operation takes some seconds to execute, memory consumption grows until all frames are collected and the self.seek(0) line gets executed - at this point, memory usage drops back to almost zero.

[sueskind]

I have the same issue of memory filling up when doing debugging in PyCharm. Furthermore, the issue also persists when pausing via time.sleep instead of using a breakpoint in the debugger.

However, doing seek(0) works for me as well.

[dukecyto]

I warn everyone here that the seek(0) workaround, while avoiding the memory issue, may end up reading in the incorrect frames.

I confirmed this with a script to dump out individual frames from a video like:

    video_reader = VideoReader(video_file)
    for f in range(len(video_reader)):
        frame = video_reader[f].asnumpy()
        video_reader.seek(0)
        image = Image.fromarray(frame)
        image.save(f'{f:04d}.jpg')

Then, I put the image sequence back together with ffmpeg for visual inspection.
My test videos ended up with lots of jumpy frames.

Unfortunately, the memory issue and the side effect of the seek(0) workaround make decord unusable for me.

[busbaby]

I was just bit by this bug. I see it's been over a year since originally reported. Is anyone looking at this? It's easy to reproduce, but if more information is needed, please let me know. Happy to help.

[chunyu-li]

I warn everyone here that the seek(0) workaround, while avoiding the memory issue, may end up reading in the incorrect frames.

I confirmed this with a script to dump out individual frames from a video like:

    video_reader = VideoReader(video_file)
    for f in range(len(video_reader)):
        frame = video_reader[f].asnumpy()
        video_reader.seek(0)
        image = Image.fromarray(frame)
        image.save(f'{f:04d}.jpg')

Then, I put the image sequence back together with ffmpeg for visual inspection. My test videos ended up with lots of jumpy frames.

Unfortunately, the memory issue and the side effect of the seek(0) workaround make decord unusable for me.

What's your decord verson? I tried this code on decord 0.6.0 and it works well, no jumpy frame appears.