ZM 1.37.16-1.177 (Fedora 36) with Recording "Always" fails and creates repeated backtraces

[jnikolich]

Describe Your Environment
OS: Fedora 36 (5.17.11-300.fc36.x86_64)
ZoneMinder Version: zoneminder-common-1.37.16-1.177.20220604gitg4be56ec67 from ZMRepo
Browser: n/a, but FireFox 101.0 (Windows 10)
FFMpeg: 5.0.1
Glibc: 2.35-11.fc36.x86_64
Output of rpm -qa | grep -i zoneminder:

zoneminder-common-1.37.16-1.177.20220604gitg4be56ec67.fc36.x86_64
zoneminder-httpd-1.37.16-1.177.20220604gitg4be56ec67.fc36.x86_64
zoneminder-debugsource-1.37.16-1.177.20220604gitg4be56ec67.fc36.x86_64
zoneminder-common-debuginfo-1.37.16-1.177.20220604gitg4be56ec67.fc36.x86_64
zoneminder-1.37.16-1.177.20220604gitg4be56ec67.fc36.x86_64

List of ALL installed packages is available on PasteBin here.

Describe the bug
ZoneMinder 1.37.16-1.177 is installed on a Fedora 36 x86_64 host. One monitor has been configured, and with capture and analysis set to Always but recording None, live-view is possible. Things work as expected.

However as soon as recording is changed to Always, zmc_m1 immediately tosses a SEGV and emits a backtrace. zmc_m1 respawns and again SEGVs, repeating endlessly.

No data gets recorded into /var/data/zm_data/events/1/2022-06-04/34. In fact nothing exists anywhere under /var/data/zm_data/events except for a bunch of subdirectories.

Zoneminder runs as user apache:apache. Perms on /var/data/zm_data/events/1/2022-06-04/34:

$ ls -ald /var/data/zm_data/events/1/2022-06-04/34
drwxr-xr-x 2 apache apache 6 Jun  4 12:15 /var/data/zm_data/events/1/2022-06-04/34

Expected behavior
Recording commences, with no segment violations/backtraces, and with recorded data accumulating in the proper filesystem location.

Debug Logs
Full zmc_m1 debug log (lvl 4) has been preserved, and is available on PasteBin here. Pertinent output at the time of failure looks like:

06/04/22 12:15:34.760585 zmc_m1[36268].DB1-zm_event.cpp/648 [Using storage area at /var/data/zm_data/events/1/2022-06-04/34]
06/04/22 12:15:34.760597 zmc_m1[36268].DB4-zm_swscale.cpp/33 [SWScale object created]
06/04/22 12:15:34.760602 zmc_m1[36260].DB3-zm_monitor.cpp/1888 [signal and capturing and doing motion detection 2]
06/04/22 12:15:34.760607 zmc_m1[36260].DB1-zm_monitor.cpp/1902 [Assigning instead of Detecting]
06/04/22 12:15:34.760609 zmc_m1[36260].DB1-zm_monitor.cpp/1912 [assigning refimage from snap->image]
06/04/22 12:15:34.760614 zmc_m1[36268].DB1-zm_videostore.cpp/113 [Opening video storage stream /var/data/zm_data/events/1/2022-06-04/34/incomplete.mp4 format: mp4]
06/04/22 12:15:34.760682 zmc_m1[36268].ERR-zm_signal.cpp/50 [Got signal 11 (Segmentation fault), crashing]
06/04/22 12:15:34.761306 zmc_m1[36268].DB1-zm_signal.cpp/57 [Signal information: number 11 code 2 errno 0 pid -1465762868 uid 32605 status 0]
06/04/22 12:15:34.761320 zmc_m1[36268].ERR-zm_signal.cpp/80 [Signal address is 0x7f5da8a23bcc, from 0x56317748173e]
06/04/22 12:15:34.761826 zmc_m1[36268].ERR-zm_signal.cpp/101 [Backtrace 0: /usr/bin/zmc(+0x8fabf) [0x563177387abf]]
06/04/22 12:15:34.761913 zmc_m1[36268].ERR-zm_signal.cpp/101 [Backtrace 1: /lib64/libc.so.6(+0x3ea70) [0x7f5da67d0a70]]
06/04/22 12:15:34.761953 zmc_m1[36268].ERR-zm_signal.cpp/101 [Backtrace 2: /usr/bin/zmc(+0x18973e) [0x56317748173e]]
06/04/22 12:15:34.761997 zmc_m1[36268].ERR-zm_signal.cpp/101 [Backtrace 3: /usr/bin/zmc(+0x16c529) [0x563177464529]]
06/04/22 12:15:34.762036 zmc_m1[36268].ERR-zm_signal.cpp/101 [Backtrace 4: /lib64/libstdc++.so.6(+0xdbb73) [0x7f5da6b6cb73]]
06/04/22 12:15:34.762073 zmc_m1[36268].ERR-zm_signal.cpp/101 [Backtrace 5: /lib64/libc.so.6(+0x8ce1d) [0x7f5da681ee1d]]
06/04/22 12:15:34.762112 zmc_m1[36268].ERR-zm_signal.cpp/101 [Backtrace 6: /lib64/libc.so.6(+0x1125e0) [0x7f5da68a45e0]]
06/04/22 12:15:34.762145 zmc_m1[36268].INF-zm_signal.cpp/108 [Backtrace complete, please execute the following command for more information: addr2line -e /usr/bin/zmc 0x563177387abf 0x7f5da67d0a70 0x56317748173e 0x563177464529 0x7f5da6b6cb73 0x7f5da681ee1d 0x7f5da68a45e0]

Executing addr2line as suggested above yields little info:

??:0
??:0
??:0
??:0
??:0
??:0
??:0

[paul-h]

See also https://forums.zoneminder.com/viewtopic.php?p=126842

[connortechnology]

Lots has changed since this. That forum post was resolved by reducing ImageBuffers to a sane number (3). Perhaps that will work here as well. Anyways, please update us on the status of this issue.

[wirerydr]

Re-tested with the OS / apps and ZM updated to the following versions, all-of-which were current as of the time this comment was written:

OS: Fedora 36 (kernel-5.18.16-200.fc36.x86_64
ZoneMinder: zoneminder-1.37.20-1.1.20220810gitg80227d749.fc36.x86_64
FFMpeg: ffmpeg-5.0.1-3.fc36.x86_64
GlibC: glibc-2.35-15.fc36.x86_64

No change - the issue persists. The backtrace looks identical, and the logging info preceding the segv also looks the same. More specifically, there's nothing being mentioned about buffers getting overrun, permissions errors being encountered, etc.

FWIW I never really thought the issue in that forum post was related - they had at least a few monitors working in recording mode, In my case, as soon as I turn on recording for the only defined monitor, the problem occurs.

Memory-wise, the VM has 8GB allocated, with approx. 87% of that free right before I toggle recording on. Once it enable recording, it rapidly decreases to about 80.5% free, before the segv occurs and free memory bounces back up to 87%. I originally had only 4GB memory allocated, but upped it to 8GB in order to conclusively rule out the possibility of systemd-oomd killing off any processes due to memory-constraints.

Please advise on any additional info you might need from me, or next-steps you'd like me to take.

Thanks very much.

[connortechnology]

I can confirm this issue. For now you can record only jpegs. Turn off video writer. I should have a fix soon

[jnikolich]

After some time spent re-testing, I can confirm this is now working. Thanks for the fix - if anything else comes up as I test this fix more thoroughly, I'll raise a new issue.

Thanks again.