memory leak / fragmentation issue

[dgel]

System information (version)

• OpenCV => 4.1.1
• Operating System / Platform => Ubuntu 18.04
• Compiler => GCC 7.4.0

Detailed description

When creating large Mat objects followed by smaller allocations that are kept in a loop, resident memory keeps growing. This does not occur when the matrices are smaller or an allocation does not happen in between. If left to keep growing, the system will run out of memory and lock up, even though no more than 2000 small strings and 2 matrices should be in memory.

I've tried with OpenCV 3.4 as well, and the same thing happens

Steps to reproduce

compile the following, run it and keep an eye on memory.

#include <thread>
#include <utility>
#include <vector>
#include <opencv2/core/mat.hpp>

int main() {
  
    // minimum size for leak on my machine: 558995 x 1
    cv::Size size(600000, 5);
    cv::Mat tmp(size, CV_8UC3, cv::Scalar(0, 0, 0));

    std::vector<std::string> strings;

    for (size_t i = 0; i < 2000; ++i)
    {
      cv::Mat tmp2(size, CV_8UC3, cv::Scalar(0, 0, 0));
      tmp = tmp2;

      // needs to be suitably large, so it gets allocated on the heap
      std::string val("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa");
      strings.push_back(val);

      // gives you time to kill the process before it runs out of memory
      std::this_thread::sleep_for(std::chrono::milliseconds(10));
    }
}

update:

this appears to be an issue with the usage of posix_memalign: if I remove the checks for it in the CMakeLists.txt and remove the checks for it then recompile, the leak disappears. Strangely, just allocating with posix_memalign instead of using cv::Mat does not create the issue on its own, there seems to be some interaction with the other operations cv::Mat does when creating it.

[mshabunin]

When you tried to reproduce the issue with bare posix_memalign did you actually initialized and copied the memory?

[mshabunin]

I think it is normal behavior due to memory fragmentation and stricter posix_memalign requirements. I've been able to reproduce it with OpenCV and corresponding raw posix_memalign calls: https://gist.github.com/mshabunin/8f6d0d4d1ad26b8fdec878ab650a0df2

[dgel]

Thanks for this, The issue with reproducing it was that I was just using malloc'ed data instead of std::string and allocated 24 bytes where 25 is needed (forgot about the null terminator)

I didn't think this is due to fragmentation however, we're allocating and deallocating several megabytes, so the strings should all fit in the deallocated space from the first posix_memalign. I'm not too familiar with posix_memalign however, what extra requirements are you referring to?

[mshabunin]

Related issues:

• https://sourceware.org/bugzilla/show_bug.cgi?id=14581 (there is nice attached code example: https://sourceware.org/bugzilla/attachment.cgi?id=11763)
• https://bugzilla.redhat.com/show_bug.cgi?id=843478

There could be a problem in glibc. My reproducer: https://gist.github.com/mshabunin/048b97c7293b6d89d030554fa00d2a61

Interesting thing is that if we call mallopt to change TRIM_THRESHOLD parameter to the default value (128*1024), the problem does not reproduce. You can try to use it as a workaround.

[dgel]

Thanks for this! I hadn't found those issues before, but I have opened a similar issue for ubuntu as well. It's quite frustrating to see this was known for so long and nothing has been done