High CPU% after around 10k objects detected (Jetson Nano)

[effort10]

Hi All,

Im experiencing a strange problem using Opendatacam without docker on a jetson nano. ( I am using yolov3-tiny-prn as the weight and i am passing a video feed via a usb camera)

When I start the recording my cpu % is around 25%
After 10k objects have been counted the cpu hits around 95% and the nano starts becoming slightly unresponsive.

The recording carries on but the opendatacam log starts showing the following error (Error with message send by YOLO, not valid JSON)
I am not sure what is happening but it looks like its got something to do with Mongo.

Please have a look at the screenshots that i have provided. Any help will be greatly appreciated.

[effort10]

Or maybe i just need stronger hardware, maybe a Jetson TX2

[tdurand]

Thanks for reporting this.. very possible we are doing something not optimized with mongodb that makes things not performant...

One of the many issue that would need to be investigated to turn OpenDataCam from a prototype to a real reliable product ;-)

[tdurand]

Adding this to the OpenDataCam v4 release backlog to remind to at least do a long run try and see if CPU usage is not exploding

[vsaw]

I've been running some benchmarks on my development machine.

Counter	Tracker	NodeJS CPU [%]	NodeJS Memory [mb]	MongoDB CPU [%]	MongoDB Memory [mb]
1799	5365	9.0	105.1	9.7	94
2078	6180	8.8	105.2	10.5	98.3
2447	7285	8.9	105.9	12.4	103.4
2848	8459	9.4	106.3	11.4	104.2
3178	9436	11.9	106.4	11.1	106.1
3709	11014	10.5	89.3 ▵	12.6	113.7
4027	11967	12.1	104.5	12.8	113.5

Looking at the data it does seem that there is a memory leak in NodeJS and MongoDB, which may also explain the increase in CPU usage. Please note that I do not have a NVIDIA GPU available and therefore was not able to profile Darknet, but only Node and MongoDB.

I suspect it it has the following sources:

1. The entire counter history is kept in RAM.
2. The entire counter history is stored in a single MongoDB document which gets updated on every frame.

Fixing both could potentially decrease the memory leaks and CPU usage.

Footnotes:
▵ Expected Garbage collection happened or something like that.

[vsaw]

While working on #528

I noticed that there is another memory leak coming from the node moving things tacker. A short analysis shows that the itemHistory and map of tracked is growing a lot (up to 200 kb/s)! Will need further investigation, though

[vsaw]

Closed via #528