Mat in opencv leaks off-heap memory and causes OutOfMemoryError #423
Comments
|
If you need to allocate more than that amount of Mat objects, increase either the |
|
@saudet Thanks for your reply. My point is: I didn't keep references to the Mat objects I allocated, so their memory should be deallocated, both the java heap memory and off-heap memory. But the testing shows that the off-heap memory is not correctly released. |
mythly
changed the title
|
I see, does it help if you increase http://bytedeco.org/javacpp/apidocs/org/bytedeco/javacpp/Pointer.html#maxRetries (via the "org.bytedeco.javacpp.maxretries" system property) to something like 1000? |
|
I tried but no help. The gc log shows that the java heap is clean enough after the third full gc. I double checked this using |
|
So even manually calling |
|
Manually calling It is also the work-around solution currently using in our application. |
|
Strange, the deallocator thread just basically calls |
|
I think I found the problem. The deallocator in jni code use static void JavaCPP_org_bytedeco_javacpp_opencv_1core_00024Mat_deallocate(void *p) { delete (::cv::Mat*)p; }and use /** Returns the amount of physical memory currently used by the whole process, or 0 if unknown.
* Also known as "resident set size" (Linux, Mac OS X, etc) or "working set size" (Windows). */
@Name("JavaCPP_physicalBytes") public static synchronized native long physicalBytes();However this link discussed that The solution on linux could be calling while (count++ < maxRetries && ((maxBytes > 0 && DeallocatorReference.totalBytes + r.bytes > maxBytes) || (maxPhysicalBytes > 0 && (lastPhysicalBytes = physicalBytes()) > maxPhysicalBytes))) {
// try to get some more memory back
System.gc();
Thread.sleep(100);
} |
|
That's interesting, thanks for investigating this! |
|
The behavior of those calls are pretty complex. Have you been able to confirm that a call to malloc_trim(0) helps? |
|
Yes, I have tested it on my PC The testing code #include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>
#include <malloc.h>
#include <opencv2/core/core.hpp>
using namespace cv;
const int MAXN = 10000;
void info(int iter)
{
int fd = open("/proc/self/statm", O_RDONLY, 0);
if (fd >= 0) {
char line[256];
char* space;
lseek(fd, 0, SEEK_SET);
if (read(fd, line, sizeof(line)) > 0 && (space = strchr(line, ' ')) != NULL) {
printf("%d: %s\n", iter, line);
}
}
close(fd);
}
int main()
{
Mat* mats[MAXN];
for (int i = 0; i < MAXN; ++i) {
mats[i] = new Mat(200, 100, CV_8UC3);
if (i % 1000 == 0) {
info(i);
}
}
for (int i = 0; i < MAXN; ++i) {
delete mats[i];
if (i % 1000 == 0) {
info(i);
}
}
info(MAXN + 1);
malloc_trim(0);
info(MAXN + 2);
return 0;
}compile and run the output You can see that number of pages keeps 11726 while deleting 10000 Mat objects at one moment. However, this behavior depends on the implementation of glibc and should be tested on more environments if possible. |
|
I wonder what happens if we forget to delete just the last Mat?
|
|
I tested it and it still releases most of the memory, interesting. What would be the equivalent on Mac OS X and Windows? Does Android implement it? |
|
You worry about the hole problem and you want to know "what happens if we forget to delete the Mat with the highest address?" I tested it too and calling
As it claims here, Whether to use this magic trick is on your decision. About other platforms, should try to reproduce this bug first? |
|
Ok, I verified that the same thing happens on Mac (though what is the equivalent of |
|
What's your Mac OS version? I tested (with the code I posted in the very beginning) on OS X Yosemite 10.10.5 and found no OutOfMemoryError. When javacpp reaches the bound of This link shortly describes the memory management of Mac OS and the things are quite different from those on linux. I have no idea whether the equivalent of |
|
Yosemite here as well. I only tried your second "testing code" though. When things get called across threads, maybe it acts differently? |
|
Yes, the second "testing code" shows that the physical memory is not immediately freed after |
|
@mythly In the case of multi-threaded high concurrent request, whether malloc_trim(0) should be locked by std::mutex?At the end of each request need to be called malloc_trim(0) ？ |
|
@youzp |
…mory gets released (issue bytedeco/javacpp-presets#423)
|
Finally got around to adding the call. Let me know if you still have issues with that! Thanks |
|
Fix released in JavaCPP 1.3.3. Thanks for reporting and testing this out! |
|
Hi guys, I am still having this issue... https://stackoverflow.com/q/47112338/2046728 java.lang.OutOfMemoryError: Physical memory usage is too high: physicalBytes = 1G > maxPhysicalBytes = 1G Already downloaded the latest version of JavaCPP (1.3.3) but kept the same behavior :-/ |
|
What's your platform? |
|
Linux (Ubuntu 16.04 64bits) |
|
You're probably just corrupting your memory. For example, you're calling cvClearMemStorage(storage) before accessing that region of memory... |
|
@saudet, so you are saying that I should't call cvClearMemStorage(storage) there? I understand as it is cleaning the storage before calling the loop of cvGetSeqElem(...) in order to place the rectangles in the image - and even if I place cvClearMemStorage(...) after all or before all it still accumulating... what would be the best practice here then? |
|
Just do it like shown here:
https://github.com/bytedeco/javacv#sample-usage
|
|
Hi @saudet , |
I'm using opencv in javacpp-presets and I found the deallocator of Mat is not working properly.
I test this by creating Mat objects (without keeping their references)
The output
One can see that java heap memory is almost not used (30MB) and the mat objects are all collected by gc.
However, the value of
Pointer.physicalBytes()keeps increasing until the limitation ofPointer.maxPhysicalBytes()(930MB), and finally throws OutOfMemoryError.Is this a bug of the deallocator?
package version (I'm using sbt):
"org.bytedeco.javacpp-presets" % "opencv" % "3.2.0-1.3" classifier "" classifier "linux-x86_64".extra java options:
-Xmx500m -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStampsThe text was updated successfully, but these errors were encountered: