Possible memory leak with repeated use of VideoCapture (in python)

[opencv-pushbot]

Transferred from http://code.opencv.org/issues/4253

|| Jean-Olivie Irisson on 2015-03-25 22:20
|| Priority: Normal
|| Affected: branch 'master' (3.0-dev)
|| Category: python bindings
|| Tracker: Bug
|| Difficulty: 
|| PR: 
|| Platform: x86 / Linux

Possible memory leak with repeated use of VideoCapture (in python)

Each creation of a @VideoCapture@ object consumes memory that do not get freed when @release()@ is called on the object or even when the object is deleted. On my system, about 50MB are allocated (no matter the size of the video file).

I am new to OpenCV and I have only used it through the python bindings. I don't know C++ enough to provide a reproducible example/test in that language. However, I prepared a minimal example in python with code + some test videos + a memory analysis code (in R, but the log file is probably easily processed with python or another language):
https://www.dropbox.com/sh/bpxowl0f9g7n8xd/AABieVtvsCBu3w-w_gI44Xvsa?dl=0

The gist is:
<pre>
list all avi files
open each file in a loop
   for each file
      get each frame
      when the last frame is reached, cleanup all objects (in the attempt of freeing memory)
</pre>

As the plot shows, every time a new @VideoCapture@ is created, rss memory jumps by ~40-50MB. Total virtual memory size increased less regularly but never stops increasing. In my real life application, I have several thousands video files to read; memory use goes all the way up to my full 16GB and then swapping begins, killing performance for everything on the machine.

I've noticed this with 2.4.10, 2.4.11 and the latest master branch (I cannot get python bindings to install with 3.0.0_beta).
This is on linux, Ubuntu 14.04. Video support comes from the gstreamer backend (ffmpeg segfaults on these codec-less videos on linux).
OpenCV is compiled with:

<pre>
cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local -D WITH_V4L=OFF \
      -D WITH_FFMPEG=OFF -D WITH_GSTREAMER=ON ..
make -j 6
sudo make install
</pre>

and the configure script gives the following (selected) info

<pre>
--   Version control:               3.0.0-beta-880-g97bdc92
--     Host:                        Linux 3.13.0-45-generic x86_64
--     CMake:                       2.8.12.2
--     C++ Compiler:                /usr/bin/c++  (ver 4.8.2)
--     C Compiler:                  /usr/bin/cc
--   OpenCV modules:
--     To be built:                 cudev core cudaarithm flann imgproc ml video cudabgsegm cudafilters cudaimgproc
cudawarping imgcodecs photo shape videoio cudacodec highgui objdetect ts features2d calib3d cudafeatures2d
cudalegacy cudaobjdetect cudaoptflow cudastereo stitching superres videostab python2
-- 
--   Video I/O:
--     DC1394 1.x:                  NO
--     DC1394 2.x:                  YES (ver 2.2.1)
--     FFMPEG:                      NO
--       codec:                     NO
--       format:                    NO
--       util:                      NO
--       swscale:                   NO
--       gentoo-style:              NO
--     GStreamer:                   
--       base:                      YES (ver 1.2.4)
--       video:                     YES (ver 1.2.4)
--       app:                       YES (ver 1.2.4)
--       riff:                      YES (ver 1.2.4)
--       pbutils:                   YES (ver 1.2.4)
--     OpenNI:                      NO
--     OpenNI PrimeSensor Modules:  NO
--     OpenNI2:                     NO
--     PvAPI:                       NO
--     GigEVisionSDK:               NO
--     UniCap:                      NO
--     UniCap ucil:                 NO
--     V4L/V4L2:                    NO/NO
--     XIMEA:                       NO
--     Xine:                        NO
--   Other third-party libraries:
--     Use IPP:                     8.2.1 [8.2.1]
--          at:                     /home/jiho/_config/opencv-git/3rdparty/ippicv/unpack/ippicv_lnx
--     Use IPP Async:               NO
--     Use Eigen:                   NO
--     Use TBB:                     NO
--     Use OpenMP:                  NO
--     Use GCD                      NO
--     Use Concurrency              NO
--     Use C=:                      NO
--     Use Cuda:                    YES (ver 5.5)
--     Use OpenCL:                  YES
-- 
--   NVIDIA CUDA
--     Use CUFFT:                   YES
--     Use CUBLAS:                  NO
--     USE NVCUVID:                 NO
--     NVIDIA GPU arch:             11 12 13 20 21 30 35
--     NVIDIA PTX archs:            30
--     Use fast math:               NO
-- 
--   Python 2:
--     Interpreter:                 /usr/bin/python2.7 (ver 2.7.6)
</pre>

It may be related to http://code.opencv.org/issues/1481 but I am not sure.

Thanks in advance for your help. I am happy to provide further tests to the best of my abilities.

History

Eray HANGUL on 2015-03-28 12:18

Hi Vadim,

This problem is also occurring in Java for version 2.4.9. 
I am also sure about that the VideoCapture constructor causes a memory leak. I use an IP camera for motion tracking. If camera goes down, my program tries to release the current VideoCapture instance set it to null and create new instance and use it. 

Although i release the VideoCapture instance and invoke the Garbage Collector nothing is changed. (Some folks said that invoking System.gc will solve the problem of unused memory references caused by Native library, but this also does not work. And using gc is the worst i know!)

The exception is like that. Some people said that some of the leak problems were solved through targeting 2.4.10 and 2.4.11. I will also try on 2.4.11 and inform you from this thread. 

CvException [org.opencv.core.CvException: cv::Exception: ..\..\..\..\opencv\modules\core\src\alloc.cpp:52: error: (-4) Failed to allocate 2764804 bytes in function cv::OutOfMemoryError]
        at org.opencv.highgui.VideoCapture.read_0(Native Method)
        at org.opencv.highgui.VideoCapture.read(VideoCapture.java:341)

Regards,
Eray

Eray HANGUL on 2015-04-02 09:27

Hi,

I changed the JVM from JDK1.6 32 bit to JDK1.8 64 bit and give 2 Gigs for Heap Size (wt. 1280x720 resolution motion tracking consumes 500 - 900 MBs at day and 1.1 - 1.5 Gigs at night because of blurry image from camera) to my program and the out of memory allocation problems lost.
I also optimized the code through the approach listed below. After more Heap Size usage (32 bit limits us @ 1.4 Gigs) and optimizing the code i tested it for 3 days non-stop and no memory problems rised :

* Setup the VideoCapture reference in a singleton constructor
* When the ip camera goes down because of its own reset or any network error, the scheduled link controller must be run as synchronized. I mean concurrent VideoCapture reference creation and Video Stream binding must not be done!
* Initialize the Mat objects at constructor side. Avoid from too much Mat object re-creation or cloning. If you must use new Mat references, try the clone mechanism since cloned references are deleted from memory more quick. 
* At the end of infinite loop, release all the Mat references.
* If you work with high resolutions like 1152x864, 1280x720, 1390x1040, 1920x1080... etc. set max memory limit to 2 Gigs at least. Especially at night, if there is no lighting in the area of camera view; too much contours are examined.

So, Sample code is like that. (It is Java code but i guess usage logic can be understood) :

<pre>
Environment information :
* JDK 1.8 64 bit
* Win7 64 bit
* OpenCV 2.4.11
</pre>

<pre>
// Video capture reference initializer
// Used at first call or any time when the Video Signal is lost
private void setup() {
        Logger.getLogger(getClass()).log(Level.DEBUG, "Binding camera stream to panel");
        Logger.getLogger(getClass()).log(Level.DEBUG, "Setup started " + Calendar.getInstance().getTime().toString());
        theStreamPath = "";
        try {
            if (capture != null) {
                capture.release();
                capture = null;
            }
            Thread.sleep(500);
        } catch (InterruptedException e) {
            Logger.getLogger(getClass()).log(Level.WARN, "Exception occurred while releasing pregenerated capture instance! > " + e.getMessage());
        }
        if (Constants.cameraType.trim().equals("local")) { 
            capture = new VideoCapture(Constants.cameraIndex);
        } else {
            // IP
            if (!GenericValidator.isBlankOrNull(Constants.streamOtherUrl)) {
                theStreamPath = Constants.streamOtherUrl;
            } else if (!GenericValidator.isBlankOrNull(Constants.streamFullUrl)) {
                theStreamPath = Constants.streamFullUrl;
            } else {
                theStreamPath = Constants.streamProtocol + "://" + Constants.streamAuthUser + ":" + Constants.streamAuthPswd + "@" + Constants.streamHost + ":" + Constants.streamPort + Constants.streamPath;
            }
            capture = new VideoCapture(theStreamPath);
        }
        // Accessing to media device...
        if (capture != null && capture.isOpened()) {
            videoWidth = (int) capture.get(CV_CAP_PROP_FRAME_WIDTH);
            videoHeight = (int) capture.get(CV_CAP_PROP_FRAME_HEIGHT);
            Logger.getLogger(getClass()).log(Level.DEBUG, "Video is opened with " + videoWidth + " X " + videoHeight);
            videoSignalOkay = true;
            if (initialRun) {
                setVideoSignalWorklog();
            }
        } else {
            videoWidth = Constants.SCREEN_WIDTH;
            videoHeight = Constants.SCREEN_HEIGHT;
            Logger.getLogger(getClass()).log(Level.DEBUG, "Video could no be opened!");
            videoSignalOkay = false;
            if (initialRun) {
                setVideoSignalWorklog();
            }
        }
        // Allow initializing...
        int sleepTime = 500;
        try {
            Thread.sleep(sleepTime);  
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        Logger.getLogger(getClass()).log(Level.DEBUG, Constants.streamPath + " is opened @ " + Calendar.getInstance().getTime().toString());
        // Initializing tracking calculation objects...
        if (image != null) {
            image.release();
            image = null;
        }
        image = new Mat();
        size = new Size(videoWidth, videoHeight);
        buf = new Mat[Constants.fps];
        for (int i = 0; i < Constants.fps; i++) {
            buf[i] = Mat.zeros(size, CvType.CV_8UC1);
        }
        initializeTrackingDataStructures();
        startTime = System.nanoTime();
        isPaused = false;
        Logger.getLogger(getClass()).log(Level.DEBUG, Constants.streamPath + " setup ok. " + Calendar.getInstance().getTime().toString());
    }
</pre>

<pre>
// Method which is invoked by RenderThread in infinite loop (isPaused = false)
public void startStream() throws Exception {
        try {
            if (videoSignalOkay) {
                if (capture == null || !capture.isOpened() || !capture.read(image) || image == null || image.empty()) {
                    if (capture != null && !capture.isOpened()) {
                        Logger.getLogger(getClass()).log(Level.WARN, "Capture is not null but closed!");
                        videoSignalOkay = false; // Link control timer captures this signal lost flag and do its job!
                    }
                    return;
                } else {
                    if (Constants.allowTracking && Constants.autoTracking) {
                        doTracking();
                    } else {
                        startTime = System.nanoTime();
                    }
                    // Control if any zoom is needed or toggled!
                    if (!Constants.autoZoom && zoomLevel > 1.0d) {
                        zoomLevel = 1.0d;
                        zoomImage = new Mat((int)(image.rows() * zoomLevel), (int)(image.cols() * zoomLevel), image.type());  
                        Imgproc.resize(image, zoomImage, zoomImage.size(), zoomLevel, zoomLevel, Imgproc.INTER_NEAREST);
                    } else if (Constants.autoZoom) {
                        zoomImage = new Mat((int)(image.rows() * zoomLevel), (int)(image.cols() * zoomLevel), image.type());  
                        Imgproc.resize(image, zoomImage, zoomImage.size(), zoomLevel, zoomLevel, Imgproc.INTER_NEAREST);
                    } else {
                        // No zoom! Render normal!
                        zoomImage = image;
                    }
                    renderWidth = zoomImage.width();
                    renderHeight = zoomImage.height();
                    MatToPImage(zoomImage);
                    image.release();
                }
            } else {
                return;
            }
        } catch (Exception e) {
            Logger.getLogger(getClass()).log(Level.ERROR, "Error while stream rendering! -> Detail : " + e.getMessage());
            e.printStackTrace();
            throw e;
        }
    }
</pre>

<pre>
// Method for Mat object to BufferedRenderImage conversion...
private static void MatToPImage(Mat mat) {
        if (videoSignalOkay) {
            byte[] byteArray = getTheByteArrayFormForMat(mat);
            try {
                if (byteArray != null) {
                    InputStream in = new ByteArrayInputStream(byteArray);
                    renderImage = ImageIO.read(in);
                    in.close();
                    mat.release();
                    mat = null;
                }
            } catch (Exception e) {
                e.printStackTrace();
            } finally {
                byteArray = null;
            }
        }
    }
</pre>

<pre>
// Method used for controlling the video signal
// Called by first run of initial class and periodically called by a timer if any TCP connection is down for ip camera!
private sychronized void setVideoSignalWorklog() {
        if (!isControllingSignal) {
            isControllingSignal = true;
            final Date now = Calendar.getInstance().getTime();
            DataAccessManager dam = DataAccessManager.getTheInstance();
            AtomicDataPojo aPojo = new AtomicDataPojo();
            aPojo.setSource(Constants.WORKLOG_BOOT_REF_FILE_PATH);
            aPojo.setSection(Constants.WORKLOG_SECTION_NAME_GENERAL);
            if (!videoSignalOkay) {
                // Write the lost time...
                aPojo.setKey(Constants.WORKLOG_SECTION_GENERAL_VIDEO_SIGNAL_LOST);
                aPojo.setValue(GeneralUtils.getTheInstance().getFormattedDate(now, DateConversionEnum.YYYYMMDDHH24MISS));
                dam.setDataToFile(aPojo);
                // Updating rendered image as lost signal...
                try {
                    while (capture.isOpened()) {
                        capture.release();
                        Thread.sleep(500);
                    }
                } catch (Exception e) {
                    Logger.getLogger(getClass()).log(Level.WARN, "Exception occurred while trying to releasing old capture reference ["+capture.toString()+"] > " + e.getMessage());
                    e.printStackTrace();
                }
                byte[] noSignalData;
                try {
                    noSignalData = IOUtils.toByteArray(new FileInputStream(new File("img/no_signal.jpg")));
                    InputStream in = new ByteArrayInputStream(noSignalData);
                    ImageIO.setUseCache(false);
                    renderImage = ImageIO.read(in);
                    in.close();
                } catch (Exception e) {
                    Logger.getLogger(getClass()).log(Level.WARN, "Exception occurred while trying to render lost signal image! > " + e.getMessage());
                }
            } else if (capture == null || !capture.isOpened()) { // May be it is a little stream lost and rework again! So no need to initialize. Rendering can continue. This is the last control point!
                if (!initialRun) {
                    setup();
                }
                Date nowAfterSetup = Calendar.getInstance().getTime();
                // Write the initial signal time...
                aPojo.setKey(Constants.WORKLOG_SECTION_GENERAL_VIDEO_SIGNAL_SINCE);
                aPojo.setValue(GeneralUtils.getTheInstance().getFormattedDate(nowAfterSetup, DateConversionEnum.YYYYMMDDHH24MISS));
                dam.setDataToFile(aPojo);
                // Reset the lost time...
            }
            isControllingSignal = false;
        }
      }
</pre>
-   % Done changed from 0 to 30
-   Status changed from New to Open

Jean-Olivie Irisson on 2015-05-05 20:39

Hi again,

I would be happy to run more tests if needed to debug this. I really need the memory link gone but I am not able to look into the C++ code.

Eray, what I gather from your method is that you setup the VideoCapture object in a separate "function" which you then call upon when needed (I do not see an equivalent to the "singleton constructor" in python). Is this sufficient in your case to get the memory released when the capture is released? Is there something else to do?

Thanks in advance for any hint.

Jean-Olivie Irisson on 2015-05-06 00:51

In case that helps, the problem does not seem to be present on OS X with opencv 2.4.9 or 2.4.11.

Jean-Olivie Irisson on 2015-05-06 01:00

Other interesting info: the bug seems gone when using ffmpeg instead of gstreamer. ffmpeg can handle my files thanks to the patch posted to http://code.opencv.org/issues/1915. I've tested this on 2.4.9. I am now testing it with 2.4.11. I will report when testing is more complete.

[ilya-lavrenov]

I'm not sure about V4L camera capture, but GStreamer-based one actually has leaks.

[alalek]

Looks like it is fixed in #5202