malloc() issue #142
Comments
|
Container? Unfortunately since I use that version for a long time and can not reproduce them and since problems with SIGABRT if exists always happens on custom system/devices I would require to run HyperHDR debug version using gdb and reporting the stack during the crash also. Otherwise can't help and the report will be closed due time: too many components or lack of memory can play a role. It's just a realistic approach. |
Yes I run HyperHDR in a docker container using an Ubuntu base image. Can you advise how to install the debug version please and I will update my image to run the debug version. |
|
It needs to be compile https://github.com/awawa-dev/HyperHDR/wiki/Compiling-HyperHDR with one change: instead of release, |
I'm capturing 1920 x 1080p 120hz in nv12 format and everything is fine when there is a signal. When I turn off my sources and try and turn it back on the next day is usually when I find HyperHDR has crashed.
The system is usually around 400-500mb RAM total when running the HyperHDR container. I've built the debug version and installed and built a new docker image with it. Where would the additional debug information be found? |
|
OK, great. Did you check 'quarter of frame' mode in the grabber properties? It's neutral for colors as nv12 delivers only 1/4 of saturation/hue information anyway. |
|
Sorry to be a pain... running the debug version from the command i.e. Any idea what I'm doing wrong? |
|
You are almost there...just type 'run' |
|
BTW if won't work because it can't find its resources run the command directly from build folder (you are in |
|
As I dug deeper that error could be glibc thing, there are a lot of similar reports lately. The cause is not sure, maybe there are multiple reasons. One suggestion is hardware related: RAM overheating which make sense for high speed 1080p capturing & processing. malloc in hyperhdr in used almost purely for creating the image buffers (for the software reason: maybe the memory is too fragmented at some point?). |
|
I left it running through gdb and came back several hours later and found the following in the console: |
|
Thank you. As I suspected it's a Image object that contains captured frame. It's a bit complicated because it uses QExplicitlySharedDataPointer for better performance and its destructor should be called when the frame is disposed (in fact when all copies of that frame are not referenced anymore). As it's done automatically and by book using QT library mechanism can't see why it fails in some condition under stress inside the container. Maybe creating full copy of the frame each time could work as workaround but don't want to do it unless serious reason because it brings higher CPU and RAM usage and it's uncertain if it helps if the problems is in glibc or it's specific for running hyperhdr from inside the container on RPi. |
|
Just as added info, I ended up running HyperHDR natively on the Pi4 (no Docker) for that debug session to eliminate any issues caused by using a container. It always seems to crash after a period of no signal. Do you think this could be caused by the grabber going into sleep mode or something? |
|
But in logs you attached earlier the processing is always happening (probably grabber is sending its no signal board I guess) so attaching the source could only have a hardware effect on the system (the voltage jump/drop when the grabbers start the real capturing). The grabber is directly not related to that exception, but selecting 1080p and very high frame-rate could play a factor because it's a lot of stress for the memory rather than CPU. |
|
I'll try to set my ezcap 320 for full frame 1080p@120@nv capturing for the weekend session. Then I could tell at least if I can reproduce it after few hours of usage. I'm using 64OS Raspberry Pi OS upgraded to Bulls Eye so different QT(5.15) and other key system libraries. |
|
Thank you - let me know if you need any further information or to try any further testing. |
|
If you could try to enable 'quarter of frame' mode in the grabber and to reduce framerate (if not possible then lower resolution) and check if it has any positive effect. I know it is not a fix but want to check if there is any correlation . Do you have signal detection enabled? |
|
Ok, tested it using 1080p 120Hz nv. There are problems related to the environmental memory management: RAM usage increases steadily to over 90% then "garbage collector" kicks in and steadily reduces it to 20%...but sometimes it is not fast enough and out of memory exception happens (I have 2GB model). For now I'm ready to exclude 1080p120 from supported modes for Rpi, similar as I did for 1080p60 MJPEG that is blacklisted for MS2109 under Windows as the whole system can't handle it. 1080p60 works fine without that race for RAM, valgrind can't find serious memory problem in hyperhdr either. |
I'm already using quarter frame mode as I think 1080p at 120hz without quarter mode would be too optimistic 🙂 However my grabber only supports 120hz at 1080p so I can't reduce the resolution further.
Edit: actually before I test at 1080p 60hz, I am going to test again at 120hz but recalibrate the signal detection and see if it crashes. So far with the recalibrated signal detection HyperHDR has dropped the captured framerate down so I will see if this is easier on the memory. |
|
'Save resource' is working only when the signal lost id detected: it simply enables 'software frame-rate decimation' for the grabber since we don't need high frame processing for the grabber 'no-signal' board . That's why you get ~8Hz framerate. But it won't help when then signals is back and 120Hz capturing returns. In fact your system is more stable than mine on 1080p@120. Unfortunately all the Image frame processing is based on automatic object management and we do not control it's lifetime directly: the environmental does it for us. And it works unless it reaches its performance limit. 120Hz could make some sense if the source has similar frame-rate. Most video content are at 23-24Hz, some TV series at 50-60Hz so I guess it leaves only gaming for capturing range over 60Hz. If not, the hyperhdr smoothing configuration handles main aspect of the smooth transition between frames. You can set it well over 100Hz with minimal delay if your LED driver has such high output capability (but you have already done it for the grabber). |
|
I have an idea how to optimize memory management especially for large frames at high framerate (for Rpi's at least because PC/macOS platforms can handle it in a standard way) so will investigate the possible solution. |
|
I want you to install the installer from newest commit: |
I had already deployed this version and so far after about 12 hours (3-4 hours using last night and then no signal throughout the night) it hasn't crashed yet 😀 |
|
After changes malloc invoke number is usually reduced from few thousands per minute to zero (cache provides own buffers after they are created) which increases performance, reduces memory fragmentation and solve that problem by the way (which could be related to lack of resources or their fragmentation). Hope it still works problematic free. |
Bug report, debug log and your config file (FULL LOGS ARE MANDATORY)
Steps to reproduce
Turn off the source and after a period of time HyperHDR crashes due to SIGABRT
What is expected?
No crashes or exceptions in HyperHDR
What is actually happening?
HyperHDR crashes and becomes unresponsive until the container is restarted
System
The text was updated successfully, but these errors were encountered: