Possible Memory Leak

[agtbaskara]

I have suffer from memory leak when running MultiStickSSDwithRealSense_OpenVINO_NCS2.py with a single stick and USB Webcam.

The memory leak occurs on Laptop and Odroid XU4 with rate about 200MB/minutes.

I can run SingleStickSSDwithUSBCamera_OpenVINO_NCS2.py normally without any memory leak, my guess is the problem is at the multiprocessing where the already processed image didn't removed from the memory.

Laptop Specification:
Ubuntu 18.04.1 (x64)
i7-4710HQ (2.5 GHz / 4 Cores)
GTX 850M (2 GB VRAM)
8 GB RAM

ODROID XU4 Specification:
Ubuntu Mate 16.04 (ARM)
Samsung Exynos5422 Cortex™-A15 2Ghz and Cortex™-A7 (2 GHz / 8 Cores)
Mali-T628 MP6
2 GB RAM

Camera:
Laptop Webcam, Logitech C270, PS3 EYE

[PINTO0309]

@agtbaskara

Thank you for giving feedback!
Since I started the process with "Fork Server", it may be causing a problem.
It seems necessary to investigate a little seriously.

[PINTO0309]

@agtbaskara

I tried it.

• My Environment
  USB Camera
  NCS2
  Corei7 + Ubuntu16.04

• Test command

$ python3 MultiStickSSDwithRealSense_OpenVINO_NCS2.py -wd 1280 -ht 720 -mod 1

or

$ python3 MultiStickSSDwithUSBCamera_OpenVINO_NCS2.py -wd 1280 -ht 720 -cn 1

• Results
  320x240 --> Measured for 5 minutes, no memory leak
  640x480 --> Measured for 5 minutes, no memory leak
  1280x720,720p --> Measured for 15 minutes, no memory leak
  

Could you tell me what kind of operation you did?

[blitzvb]

Sorry to be out of topic but are you running an NCS 2 on a odroid XU4 ? I thought it was only compatible with raspberry pi beside intel proc.

[PINTO0309]

@blitzvb
@agtbaskara

Perhaps it works with any CPU that supports the armhf (ARMv7) architecture.

$ sudo dpkg --add-architecture armhf
$ sudo apt-get update
$ sudo apt-get install libusb-1.0-0:armhf libudev1:armhf libstdc++6:armhf

The cause of the memory leak may be caused by the difference in OpenVINO version.

• My OpenVINO
  5.0.1 (R5)

btw, "SingleStickSSDwithUSBCamera_OpenVINO_NCS2.py" does not use OpenVINO.
It uses the OpenCV DNN module.

[onion233]

Hi, @PINTO0309 thanks for your code.
I also notice memory leak on my raspberry pi 3 about 200 kb/s when running MultiStickSSDwithUSBCamera_OpenVINO_NCS2.py :

Raspberry pi 3 details:
OS: Raspbian Stretch
CPU: ARMv7
NCS2

But when I run the same code on my PC, it works without memory leak.

PC details:
OS: Ubuntu 16.04
CPU: x86 64
NCS2

It seems to be related to queue operation when camera frames are put into frameBuffer. I did the following experiments:

1. Running camera process without inference process, the memory leak still occurs.
2. Running camera process without inference process, and comment queue operation, memory leak disappears

[PINTO0309]

@onion233
Thank you very much for your cooperation in the survey.
I will start thinking about how to fix this tonight.

It is very strange that the situation will change depending on the CPU/OS difference.

[PINTO0309]

Memory leak investigation.
RaspberryPi3 + USBCamera + NCS2
MultiStickSSDwithUSBCamera_OpenVINO_NCS2.py

Real-time monitoring command

$ watch -n 1 "cat /proc/meminfo | egrep -e 'Active:|Inactive:|MemFree:'"

$ python3 MultiStickSSDwithUSBCamera_OpenVINO_NCS2.py

5 minutes monitoring
https://youtu.be/9bAVfGUzmrU

1 minutes GIF

I observed it for 5 minutes, but as a result, the memory usage did not exceed 300 MB.
Unnecessary areas appear to be collected in real time.
I have no idea what is different between you and my environment... 😭

[onion233]

@PINTO0309 Thanks for your investigation! Could you provide your python version and opencv-python version?

[PINTO0309]

@onion233
Yes. of course.

Python 3.5.3
OpenCV 4.0.1-openvino

[onion233]

@PINTO0309 @agtbaskara
Hi, guys. I refreshed my raspberry pi and followed the configuration steps again. This time I do not suffer from the memory leak over 30 minutes.
python3 MultiStickSSDwithUSBCamera_OpenVINO_NCS2.py -cn 0 -numncs 1
It's weird that my configuration stays the same but the code behaviors differently. The most likely reason for the leak is the old frame in frameBuffer was not really freed due to "some environment issue"

[PINTO0309]

@onion233 ,@agtbaskara
Thank you for contributing to problem solving!
I care what the essential problem was.