ZMC high ram and not clearing when events created

[TelepathicWalrus]

Describe Your Environment

• 1.36.12
• PPA
• Ubuntu Focal Fossa 20.04 Server

If the issue concerns a camera
N/A

Describe the bug
If you have a monitor on MOCORD and the monitor detects an event, RAM usage increases and does not decrease when the event ends. I have tried to limit image buffer size to 100 frames to stop ZMC filling RAM but this results in "queue is full" errors in logs. My keyframe interval for my camera is set to 15frames(1 Second) and I'm recording in 4k.

To Reproduce
Steps to reproduce the behavior:

1. Install zoneminder through PPA following read the docs
2. Add camera and set to MOCORD
3. Trigger event by waving hand in front of camera and watch RAM usage increase
4. repeat step 3 and watch RAM increase

Expected behavior
This will cause RAM usage to increase and not clear over time(it does decrease slightly). The RAM usage seems to only increase when the event is longer than the previous one.

Debug Logs
Packet queue error

You have set the max video packets in the queue to 100. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. | zm_packetqueue.cpp | 113

RAM increase with no max buffer size
1 Event RAM increase

2 Events RAM increase

3 Events RAM increase

After a few minutes RAM stays roughly constant

[welcome]

Thanks for opening your first issue here! Just a reminder, this forum is for Bug Reports only. Be sure to follow the issue template!

[tiagogbarbosa]

Same problem here. Cant use Mocord. 30seconds after activation no memory left on device

[chrisyokum]

Same problem here. It was fine on 1.34 but 1.36 is exhausting my system RAM after some time.

[crossan007]

I think I'm seeing this as well; or at least similar.

I have my buffers set to a hard limit (which I've been gradually increasing) currently at 300.

[ionizedatom]

I'm seeing the same issue. I have multiple cameras that should stay around 18 to 22 GB of memory usage but after about 30 minutes of starting Zoneminder, all of my memory is completely filled and spills into swap. This is true if I limit the buffer or set it to 0 for unlimited. This is also true if I give the server 32 GB of ram or 128, Zonminder takes it all regardless.

[Fl1pp3d0ff]

Same issue here - RAM usage to 100%, then something "breaks" and RAM usage drops to almost nothing... No recording while RAM at 100% or just after.
/dev/shm remains at 1%-2% utilized the whole time.
Ten total cams, 1 on monitor, 9 on mocord.
Debian Buster on ProxMox
Installed via APT
40G RAM, 14 vCPUs assigned to the VM
48G of swap dedicated to this VM (on a separate drive, both virtually and physically).
10Gbit link direct to the fileserver where the recordings and databases are stored.
Removing the packet queue limit makes the problem worse.

Zabbix (both of my instances...) report the same issues.

[connortechnology]

Lots of graphs and such but not a single log file. One of you will have to send a debug level 3 log.

[Fl1pp3d0ff]

@connortechnology How large do you want the log?
More than happy to oblige.
Meanwhile, more graphs...

[Fl1pp3d0ff]

@connortechnology 11 hours of level 3 debug logs came to just over 18 GB of loggy goodness. Where would you like me to send it?
It's much too large to upload here...

[Fl1pp3d0ff]

@connortechnology You're going to have to gunzip this.. but... here ya go... https://www.dropbox.com/s/maaqjrht2gsoapg/extra_debug_level_3.log.gz

[herrjon]

I have the same problem. My limited workaround is to set the max buffer size for each stream.

This goes against the claim that Zoneminder will only ever use 1/2 of the available system RAM. It does not - it will use it all up until the server starts killing off zmc tasks.

[connortechnology]

@Fl1pp3d0ff that log shows motion detection is not keeping up. You do not have the cpu to handle that high res camera. You have set it to use 24bit mode, which saves some ram, but at the cost of not being able to use SSE or other optimized instructions. Switch it to 32bit. Also, don't do analysis on every frame, you just can't keep up. Set an analysis fps of 2 or 3 fps.

[Fl1pp3d0ff]

@connortechnology Interesting.. The CPU usage on that VM with 14 vCPUs assigned to it rarely, if ever, runs above 50%...
Also interesting that the one camera (Monitor 10) has the same resolution as monitors 1, 2, 3, and 9, and they're set for 24bit color as well.
Something else interesting... That monitor (10) was the only one I didn't limit analysis to 10FPS on... I just set that and will test, but I don't expect much change.
/dev/shm is still at 2%. Wouldn't caching to memory help on some of this?
Still doesn't change the fact that there seems to be a memory leak or something...and I'm not the only one experiencing it.

[herrjon]

Yeah - changing between 24 / 32 bit did not help my system.

Again - my work around was to change the max buffer size. Per the zoneminder documentation I shouldn't have to do this - and I didn't have to do it before this update.

Zoneminder docs claim that it will never use more than half of the available system RAM. It uses all of it.

My /dev/shm/ remains low through all of this as well.

I'm not a new user. I've been using zoneminder for over a decade. This is the first time this has happened.

[Fl1pp3d0ff]

I've changed the VM slightly... I've set up CPU passthru and assigned 12 threads (Xeon X5690 CPUs, so.. plenty of horsepower to do the job...) of direct host CPU to the VM. I've set all 10 monitors to 32-bit color and I've also moved the analysis down to 5 FPS on all monitors. The memory is still filling then crashing then filling then crashing.
Video has always been 10FPS - I haven't changed that.
I've turned debug logging back on and have gzipped the file after it ran all night.
Maybe you can find something useful and helpful in this new log...
https://www.dropbox.com/s/hrr08r2yg5qnwp1/extra_extra_debug_level_3.log.gz?dl=0

[elvis-epx]

Had the same problem, trying 1.34 right now.

[tommysnello]

i have the same problem wirh zm 1.36.19 on vmware's VM, with only one monitor

[Fl1pp3d0ff]

I've given up on solving this - it seems the dev is more interested in blaming everything else than fixing this issue. I tested it on FreeBSD as well - same issue.
Like I said: I've given up on this - and moved on to a different bit of software.

[jeje42]

Hi,

I had pretty much the same problem with ZM 1.36, here is how I solved it according to my need (v.1.36.17 as of now, did not have time to update to the latest). RAM was filling up with just one 1080p camera, no matter how much RAM was allocated to the VM.
My needs:

• record stream to be able to see plate number, recognize faces, distance being between 10-30 metres from the camera.
• be able to detect incoming cars/moves

As stated in the dummies guide, It is recommended to record 2 streams: https://wiki.zoneminder.com/Dummies_Guide#Motion_Detection
“4K Cameras Mocord on the low res stream, and record on the high res stream. Do NOT use any analysis (zma) on the high res stream. Analysis is where the CPU is eaten up.”

So I ended with the following setup:

• stream 1 for motion detection: low resolution 432p ffmeg/rtsp, Modect, analysis enabled, just 5 i/s to analyse (this might be lower, works fine in my case) , 32 Bits color, Frames + Analysis images saved JPEGs, video writer is Camera Passthrough. ImageBufferCount is set to 30, MaxImageBufferCount to 0 (unlimitted). For detection, zones in the monitor are set as small as possible. In the camera setup (Bosh in my case) the stream contains 25 i/s and since the resolution is low enough this stream does not cause RAM to be filled up.

• stream 2: ffmpeg/rtsp, add the previous monitor as linked monitor so events from the first stream is reported on the second, resolution 1080p. In the Storage tab, set Save JPEGS to Disabled, this is very IMPORTANT and I think it was the cause of the RAM filling up. Set Video Writer to Camera Passthrough. In the camera setup, the stream is set to contain 12.5 i/s which is good enough in my case. Higher i/s or "better" keyframe are limited in my case since I get the streams from a vpn and a different country, but this config with 12.5 i/s and default bosch image optimization leads to good quality videos.

It is only possible to live view the stream1 in the interface, however live view the stream 2 would be useless. I have been testing this for several weeks, RAM usage on the VM (OS+ZM) is stabilized to around 1.40 GB and I use zoneminder-base docker images.
As a result, you can have 2 monitors per camera:

• first low res monitor to be able to quickly check events
• second higher res monitor to see the image with better quality/details for the corresponding events.

What saved me was to disable "Save JPEGs" under Storage tab for the high res stream. As I read in different threads this feature implementation changed a bit between version 1.34 and 1.36, but I am not able to explain in detail :).

I hope these details will be helpful.

[Yetangitu]

#3414 (comment)

While I also see the OOM_KILLER every now and then your situation seems to be radically worse than mine. For comparison, here's the week (average) graphs for my installation (7 cameras, 16 monitors, 7 monitors low-res on modetct, 7 monitors full-res (1920*1080) on nodect, 2 monitors inactive waiting for camera #8) running on a container with 8GB RAM/512MB swap/8 cores (X5675@3.07GHz). Just like you I'm using Debian, Zoneminder version 1.36.12-bullseye1

While I have fewer cameras than your installation (7 instead of 10) I have more monitors (16 instead of 10) running in ⅕ of the memory, 1/96 of the swap and close to ½ of the core count.

The main difference is the use of modect on the low-res streams and nodect on the high-res streams from the same cameras and the fact that I'm using a container instead of a VM. As said I do see the occasional runaway zmc process but as the graphs already show this does not happen all that often. I also have some problems with cameras which just quiet sending data (unrelated to ZM) which I solved with a watchdog script which resets zombie cameras.

Maybe you can try the low-res modect/high-res nodect approach to see whether that solves this issue? You can also check whether using a container instead of a VM makes a difference (unlikely but worth exploring). This installation has been in use for about 1.5 year running ZM 1.34, then 1.36, without problems (other than the mentioned occasional runaway zmc process).

| #3414 (comment)

I do have live-streaming enabled on the high-res monitors since the cameras are also used for monitoring purposes.

[basildane]

I have the same problem. It was working fine on 1.34. Working fine for many years prior. The instant I upgraded to 1.36 I started having serious memory problem. No other configuration changes were made.

[connortechnology]

perhaps you didn't read the release notes. You should change your ImageBuffers to something like 3. Or 5 if live viewing a little stuttery.

[basildane]

if you mean me, my image buffer size = 3 and always has been.

[basildane]

Can you provide a link to the release notes?

[connortechnology]

https://github.com/ZoneMinder/zoneminder/releases/tag/1.36.0

Of course the problem here is that was a long time ago and that line should have been in bold and caps and all that.

Anyways, I've seen a lot of people say that 1.34 worked great and while doing support diagnosing a problem I see that their system has been screaming at them that there is something wrong and they just ignored it.

So it's complicated. 1.36 CAN use a lot more ram than 1.34 because 1.34 used fixed buffers and 1.36's grow as needed (hence the MaxImageBuffer setting). We also buffer when writing to disk, so if disk is slow, then buffers fill up. Same with writing to the db.

@basildane we are going to need a lot more info about your system.

[Martindzi]

Hello, I am a beginner in Linux and Zoneminder, the problem is with a memory leak, I used the 1.36.15 update to 1.36.19 and then started the problem, a ~20 hours after the Zonemider restart the RAM is almost full.
With version 1.36.15 RAM was max 60%
what are the suggestions on what to do?

[MoHf7f4]

Seeing this after upgrading to v1.36.17. Restarting ZM recovers the memory.

[kovaga]

I had the same issue after upgrading to 1.36.

So followed the advice from [jeje42] by creating the linked monitor at lower resolution for analysis.
Also set to passthrough the x264 stream from the cameras instead of reencoding it in x265 and the whole system stabilised.

[MoHf7f4]

What I've found is it's vastly worse (or maybe only) for cameras with zones defined. Trying to reduce my zones to see if it improves.

[MoHf7f4]

I seem to have fixed this on mine.
For the cameras having the issue I had Maximum Image Buffer Size (frames) set to 0.
I set it to 4x my frame interval on the cameras. No longer seeing the aggressive memory increase.
Using v1.36.17

[Martindzi]

Thanks "MoHf7f4" everything seems to be working so far.

[Jsalas424]

Yeah - changing between 24 / 32 bit did not help my system.

Again - my work around was to change the max buffer size. Per the zoneminder documentation I shouldn't have to do this - and I didn't have to do it before this update.

Zoneminder docs claim that it will never use more than half of the available system RAM. It uses all of it.

My /dev/shm/ remains low through all of this as well.

I'm not a new user. I've been using zoneminder for over a decade. This is the first time this has happened.

I run 1 camera in mocord mode, recording a single HD stream. I allocated ZM 12GB RAM and 8 cores of an i7. This is running in a dedicated Ubuntu 22 server VM, it ONLY runs ZM v1.36.33. I got to this thread same as y'all, ZM was using way too much RAM resulting in too much swap usage and acceleration of my disk's death.

I added a "Maximum Image Buffer Size" as my RAM usage immediately dropped:

The logs reflected nothing of interest upon this change.