Mat::create() fails to allocate memory in 32bits code (and plenty of free available memory)

[LorenaGdL]

Please state the information for your system

• OpenCV version: 2.4.12 / 3.1
• Host OS: Windows 7 / Windows 10
• Compiler & CMake: VS2013 & CMake 3.4+ (several versions depending on testing computer)

In which part of the OpenCV library you got the issue?

• core, matrix.cpp, Mat::create() function

Expected behaviour

When using code in a 32bits project, Mat::create() function should perfectly allocate another matrix in memory when the system has still plenty of free memory (>3.5GB out of 8GB)

Actual behaviour

Mat::create() function crashes and sends a memory allocation error when trying to create a simple matrix in a system with more than 40% of free memory (as said, more than 3.5GB). The issue has been reproduced in 3 different PCs (Win 7 / Win 10) with different RAM sizes (8GB, 12GB, 16GB), when using code compiled for 32bits (as simple as the code provided below, a loop creating and storing matrices). The issue is not present if the code is compiled for 64bits (in this case, the system memory goes up until its maximum [+97%] as expected).

Full error message is as follows:

OpenCV Error: Insufficient memory (Failed to allocate 6220800 bytes) in cv::OutOfMemoryError, file C:\opencv-3.1.0\sources\modules\core\src\alloc.cpp, line 52
OpenCV Error: Assertion failed (u != 0) in cv::Mat::create, file C:\opencv-3.1.0\sources\modules\core\src\matrix.cpp, line 424

Additional description

I've tracked down the error to this portion of the Mat::create() function (lines 423-433 of core/matrix.cpp)

        try
        {
            u = a->allocate(dims, size, _type, 0, step.p, 0, USAGE_DEFAULT);
            CV_Assert(u != 0);
        }
        catch(...)
        {
            if(a != a0)
                u = a0->allocate(dims, size, _type, 0, step.p, 0, USAGE_DEFAULT);
            CV_Assert(u != 0);
        }

The first allocate() throws an exception, that is caught, but a == a0, and so the second allocate is not called, and the CV_Assert fails.

This low-level code and memory management is totally out of my expertise, so this behavior might be expected, but it sounds off to me that when having more than 3.5GB of free memory, it fails to allocate around 6MB (or even less).

Code example to reproduce the issue

#include "opencv2/core.hpp"
#include <iostream>
using namespace std;
using namespace cv;

void main()
{
    Mat src(1920, 1080, CV_8UC3, Scalar(128, 255, 354));  //dummy sizes and contents
    std::vector<cv::Mat> vec;
    for (;;){
        vec.push_back(src.clone());

        imshow("dummy", Mat(3,3,CV_8U));
        waitKey(0);
    }
}

The imshow() and waitKey() lines are there just to control execution (otherwise you may end with an infinite unstoppable loop in 64bits). Issue is reproducible without such lines. Other variants with copyTo() have also failed.

[StevenPuttemans]

@LorenaGdL as far as I understand it, 32bit systems are only capable of addressing 4GB of memory maximally and thus I am pretty sure that a 32 bit process, which you have in this case, cannot assign more then 4 GB of memory.

https://www.quora.com/What-will-happen-if-we-use-more-than-4GB-of-memory-in-a-32-bit-system

[LorenaGdL]

@StevenPuttemans I was aware of this limitation in 32bits systems, but I'm not too sure about the limitation in 32bits processes running in 64bits systems. I'll read the linked web thoroughly, run some tests, and report back. Thank you 😃

[Dikay900]

@StevenPuttemans is right this is an issue of the 32 bit running program since it can only address 32 bit (2^32 byte) of memory. While these are 4GB, the windows system reserves ~500MB for hardware related stuff if i remember correctly which leads to the 3.5GB limitation.
So you kinda have to use 64 bit for this purpose.

[LorenaGdL]

I'm back. You both are right, and a bit wrong at the same time. After some research, I've found that even though 32bit systems can support up to 4GB, each 32bit process is limited by default to 2GB of virtual memory (which makes sense with the tests I've run, as the app crashes when it has consumed around 1.8GB). There is a special flag that can be set (IMAGE_FILE_LARGE_ADDRESS_AWARE) and that allows 32bit processes to consume up to 3GB in 32bit systems, and up to 4GB in 64bit systems.

More interesting limits in this page: https://msdn.microsoft.com/en-us/library/aa366778.aspx#memory_limits

I'll close this. Thank you both, it's good to learn something new every day.

[StevenPuttemans]

Good to know and thank your for the extra information. Always nice to keep in mind!