Memory leak

[darkcheater]

• I confirm that this is an issue rather than a question.

Bug report

I have to restart hyperion from time to time because eventually, it will eat all my memory.

Steps to reproduce

Start hyperion and use it. I use effects for usual lighting and for ambilight I use the Kodi grabber addon (windows) as source.

What is expected?

RAM usage stays on the same low level.

What is actually happening?

After 4 days hyperion uses more than 2 GB of RAM.
I have attached two pmap outputs.

The first one is directly after the restart of the service (0,3 GB) and the second one is 4 days later (2,1 GB).

fresh.txt
4days.txt

If you need any additional information, please ask.
Thanks!

System

Hyperion Server:

• Build: (HEAD detached at 2.0.12) (GitHub-dc6aa4d/df14958-1637501177)
• Build time: Nov 21 2021 17:25:08
• Git Remote: https://github.com/hyperion-project/hyperion.ng
• Version: 2.0.12
• UI Lang: auto (BrowserLang: de-DE)
• UI Access: expert
• Avail Capt: dispmanx,v4l2,framebuffer,qt
• Config path: /root/.hyperion
• Database: read/write

Hyperion Server OS:

• Distribution: Raspbian GNU/Linux 10 (buster)
• Architecture: arm
• CPU Model: ARMv7 Processor rev 3 (v7l)
• CPU Type: Raspberry Pi 4 Model B Rev 1.1
• CPU Revision: c03111
• CPU Hardware: BCM2711
• Kernel: linux (5.10.63-v7l+ (WS: 32))
• Root/Admin: true
• Qt Version: 5.11.3
• Python Version: 3.7.3
• Browser: Mozilla/5.0 (Linux; Android 8.0.0; SM-G930F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Mobile Safari/537.36

[Paulchen-Panther]

Hyperion is not exactly a small project. To find the memory hog, one would have to know what exactly you did in the 4 days. For example, to exclude the effect engine, you would have to turn off the start effect and the background effect and not use any effects during the 4 days. If the memory still fills up then it is another component.

[ManuVice]

Hi, I have the same problem. I was able to narrow down the error. I have played with some settings and the problem only occurs when i activate the "LED Output". My RPI3b runs slowly out of memory and Hyperion ng process is responsible for it.

My short config:

• RaspberryPi3b
• Karatelight16
• USB Hdmi Video Capture Card

Hyperion System Summary Report (My Hyperion Config), Reported instance: unknown

< ----- System information -------------------- >
Hyperion Server:

• Build: master (GitHub-0027a65/addff6f-1647783741)
• Build time: Mar 21 2022 20:22:17
• Git Remote: https://github.com/hyperion-project/hyperion.ng
• Version: 2.0.13-beta.1+nightly20220321addff6f
• UI Lang: de (BrowserLang: de-DE)
• UI Access: default
• Avail Screen Cap.: dispmanx,framebuffer,qt
• Avail Video Cap.: v4l2
• Avail Services: boblight,cec,effectengine,forwarder,flatbuffer,protobuffer,borderdetection
• Config path: /root/.hyperion
• Database: read/write

Hyperion Server OS:

• Distribution: Raspbian GNU/Linux 11 (bullseye)
• Architecture: arm
• CPU Model: ARMv7 Processor rev 4 (v7l)
• CPU Type: Raspberry Pi 3 Model B Rev 1.2
• CPU Revision: a02082
• CPU Hardware: BCM2835
• Kernel: linux (5.10.103-v7+ (WS: 32))
• Root/Admin: true
• Qt Version: 5.15.2
• Python Version: 3.9.2
• Browser: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36

< ----- Configured Instances ------------------ >
0: First LED Hardware instance Running: true

< ----- This instance's priorities ------------ >
240: VISIBLE! (V4L) Owner: V4L2
priorities_autoselect: true

< ----- This instance components' status ------->
ALL - true
SMOOTHING - true
BLACKBORDER - true
FORWARDER - false
BOBLIGHTSERVER - false
GRABBER - false
V4L - true
LEDDEVICE - true

< ----- This instance's configuration --------- >
{"backgroundEffect":{"color":[255,138,0],"effect":"Warm mood blobs","enable":false,"type":"effect"},"blackborderdetector":{"blurRemoveCnt":1,"borderFrameCnt":50,"enable":true,"maxInconsistentCnt":10,"mode":"letterbox","threshold":10,"unknownFrameCnt":600},"boblightServer":{"enable":false,"port":19333,"priority":128},"color":{"channelAdjustment":[{"backlightColored":false,"backlightThreshold":20,"blue":[13,13,153],"brightness":100,"brightnessCompensation":40,"cyan":[187,250,250],"gammaBlue":1,"gammaGreen":1.2,"gammaRed":1,"green":[150,222,3],"id":"left_right","leds":"0,1,2,3,8,9,10,11","magenta":[244,0,255],"red":[201,78,49],"white":[201,188,188],"yellow":[240,255,0]},{"backlightColored":false,"backlightThreshold":20,"blue":[1,1,161],"brightness":100,"brightnessCompensation":40,"cyan":[194,253,244],"gammaBlue":1,"gammaGreen":1.2,"gammaRed":1,"green":[215,221,2],"id":"up_down","leds":"4,5,6,7,12,13,14,15","magenta":[255,0,220],"red":[194,62,31],"white":[197,165,137],"yellow":[255,216,0]}],"imageToLedMappingType":"multicolor_mean"},"device":{"autoStart":true,"colorOrder":"rgb","delayAfterConnect":1500,"hardwareLedCount":16,"hardwareLedCountList":"16","latchTime":15,"output":"ttyACM0","rate":1000000,"rewriteTime":1000,"type":"karate"},"effects":{"disable":[""],"paths":["$ROOT/custom-effects"]},"flatbufServer":{"enable":false,"port":19400,"timeout":5},"foregroundEffect":{"color":[0,0,255],"duration_ms":3000,"effect":"Rainbow swirl fast","enable":false,"type":"effect"},"forwarder":{"enable":false},"framegrabber":{"available_devices":"Suche Aufnahmegeräte","cropBottom":0,"cropLeft":0,"cropRight":0,"cropTop":0,"device":"auto","device_inputs":"","enable":false,"fps":10,"framerates":"","height":45,"input":0,"pixelDecimation":8,"resolutions":"","width":80},"general":{"configVersion":"2.0.13-beta.1+nightly20220316c75c98e","name":"My Hyperion Config","previousVersion":"2.0.13-beta.1+nightly20220316c75c98e","showOptHelp":true,"watchedVersionBranch":"Stable"},"grabberV4L2":{"available_devices":"USB Video: USB Video","blueSignalThreshold":0,"cecDetection":false,"cropBottom":0,"cropLeft":0,"cropRight":0,"cropTop":0,"device":"/dev/video0","device_inputs":"0","enable":true,"encoding":"MJPEG","flip":"NO_CHANGE","fps":20,"fpsSoftwareDecimation":0,"framerates":"20","greenSignalThreshold":100,"hardware_brightness":-11,"hardware_contrast":148,"hardware_hue":0,"hardware_saturation":180,"height":480,"input":0,"noSignalCounterThreshold":200,"redSignalThreshold":0,"resolutions":"1","sDHOffsetMax":0.46,"sDHOffsetMin":0.4,"sDVOffsetMax":0.9,"sDVOffsetMin":0.1,"signalDetection":false,"sizeDecimation":4,"standard":"NONE","width":720},"instCapture":{"systemEnable":false,"systemGrabberDevice":"NONE","systemPriority":250,"v4lEnable":true,"v4lGrabberDevice":"usbtv","v4lPriority":240},"jsonServer":{"port":19444},"ledConfig":{"classic":{"bottom":4,"edgegap":0,"glength":0,"gpos":0,"hdepth":8,"left":4,"overlap":0,"pblh":0,"pblv":100,"pbrh":100,"pbrv":100,"position":12,"ptlh":0,"ptlv":0,"ptrh":100,"ptrv":0,"reverse":false,"right":4,"top":4,"vdepth":5},"matrix":{"cabling":"snake","ledshoriz":1,"ledsvert":1,"start":"top-left"}},"leds":[{"hmax":0.05,"hmin":0,"vmax":1,"vmin":0.772},{"hmax":0.05,"hmin":0,"vmax":0.772,"vmin":0.514},{"hmax":0.05,"hmin":0,"vmax":0.514,"vmin":0.228},{"hmax":0.05,"hmin":0,"vmax":0.228,"vmin":0},{"hmax":0.25,"hmin":0,"vmax":0.08,"vmin":0},{"hmax":0.5,"hmin":0.25,"vmax":0.08,"vmin":0},{"hmax":0.75,"hmin":0.5,"vmax":0.08,"vmin":0},{"hmax":1,"hmin":0.75,"vmax":0.08,"vmin":0},{"hmax":1,"hmin":0.95,"vmax":0.228,"vmin":0},{"hmax":1,"hmin":0.95,"vmax":0.514,"vmin":0.228},{"hmax":1,"hmin":0.95,"vmax":0.772,"vmin":0.514},{"hmax":1,"hmin":0.95,"vmax":1,"vmin":0.772},{"hmax":1,"hmin":0.75,"vmax":1,"vmin":0.92},{"hmax":0.75,"hmin":0.5,"vmax":1,"vmin":0.92},{"hmax":0.5,"hmin":0.25,"vmax":1,"vmin":0.92},{"hmax":0.25,"hmin":0,"vmax":1,"vmin":0.92}],"logger":{"level":"debug"},"network":{"apiAuth":true,"internetAccessAPI":false,"ipWhitelist":[],"localAdminAuth":true,"localApiAuth":false,"restirctedInternetAccessAPI":false},"protoServer":{"enable":false,"port":19445,"timeout":5},"smoothing":{"decay":1,"dithering":false,"enable":true,"interpolationRate":25,"outputRate":25,"time_ms":200,"type":"linear","updateDelay":0,"updateFrequency":25},"webConfig":{"crtPath":"","document_root":"","keyPassPhrase":"","keyPath":"","port":8090,"sslPort":8092}}

[Lord-Grey]

@ManuVice Could you test, if that happens too, if you have smoothing off and maybe blackborder detection?

Smoothing is off when the LED-Device is off, therefore the issue might not be related to the device itself, but to the part between grabber and device.

On the Karate Code I have ad-hoc not seen any issue…

@darkcheater Would you mind sharing your configuration too. To see, if there are commonalities?

[ManuVice]

Thank you for reply. I have test it in past (February, March) with black bar detection and smoothing off and both off.
Next I have tried was deleting the config db and set only light and grabber new up. Same problem. Last thing I tried was only activate grabber and lights. If I activate lights the problem occurs. Grabber only is fine.

Last version that worked for me was alpha 4 or 5.
I did a new setup in the end of 2021 and updated to the last versions. I didnt test the versions between those.

I have looked at the commits between the last versions but my code understanding is too bad.

My result is that the error has to do with lights enable.
Sorry for my bad english 😅

[Lord-Grey]

@ManuVice Thank you for the derailed summary. I will have another look.
Could you do me a favor and just change the LEDDevice to the debug file one?
Just to see, if it is related to the Serial Karate Device or not.

[ManuVice]

Thank you. I will do it tommorow.

If it helps I had created a forum thread with libleak enabled. Log is posted as attachment.
This was the time I updated my RPi and hyperion.

https://github.com/WuBingzheng/libleak

https://hyperion-project.org/forum/index.php?thread/12423-memory-leak/

[ManuVice]

@Lord-Grey I have set a new clean config with default settings up. Tested a few things. I have found an interesting thing.
If I set up "Capturing Hardware" "Frames per secound" to higher fps memory goes faster up. Hyperion will not free memory. It will eaten all up. If I disable "LED Output" on dashboard memory will get freed. I have added some screenshots.

If I set LED Output to file memory looks fine.
hyperion file output.log

Memory after new clean config

Memory after 2min with led output FILE

Memory after 2min with Karate16

Memory after ~5min with Karate16 and 50fps

Memory after I disable led output in dashboard after ~5min 50fps

Hyperion debug config.txt

[ManuVice]

I have set up a new device (odroid c2) with only hyperion on it. it uses right now 1,4gb of memory 🙈
my system memory + swap are going to burst

[Lord-Grey]

@darkcheater / @ManuVice You are running your grabber with encoding format MJPEG.
Would it be possible to run with a different encoding format and see, if the memory leakage still persists?

Thank you!

Edit: No need for further testing. Seems I identified some issue in context of the LED update…

[ManuVice]

I have an update on this issue.
I have tested Raspbian and Dietpi images based on Debian Stretch, Bullseye and Bookworm bare metal installation, released versions and compiled hyperion ng myself. With all of them I had memory leaks.

I also tried various docker versions.. same issue.
Tested on RaspberryPi 3b, 4b and OdroidC2

But then I discovered https://hub.docker.com/r/bradsjm/hyperion-ng-addon-aarch64
2.0.13, 2.0.14 and 2.0.15 are working fine over serveral weeks.
2.0.15 is running since 2023-02-20 24/7 and memory is now at 14.8MB 👍

[jamieburchell]

I'm seeing the memory usage of Hyperion increase each time I switch to a different effect. It doesn't seem to go back down. Hyperion 2.0.15, Pi 2, OSMC / Kodi 18.

[seirowg]

I can confirm this also. Overnight my LXC container went from 250MB RAM usage to 520MB used. Newest version of Hyperion NG installed on Ubuntu 22.04 LXC container. Hyperion is set to only listen for Android grabber.

[Lord-Grey]

The issue was already open too long...
Luckily we found some time during the holidays to look into the memory issue and found a serious one which is fixed by #1678.
The finding was regarding each image processed and nicely correlates with @ManuVice's observations. Thank you for highlighting the scenario.

@darkcheater, @ManuVice, @jamieburchell, @seirowg and others
I would be really great, if you could test the fix.
In case you are brave you can also test #1679 which build further one the fix (just change the PR number in the instructions below.

Testing the PR on Linux

0. Create a GitHub user, if case you have one continue with the next step

1. Create a fine grained personal access token for your GitHub user.
   Go to https://github.com/settings/tokens?type=beta and do a "Generate new token":
   a) Give the token a name
   b) Select "All Repositories"
   c) Under Permissions/Repository permissions
   Define Read-Only Access to "Actions"
   d) Generate token

2. Copy the new generated token

3. Log on to your Linux system, e.g. ssh into the HyperBian rpi

4. Execute the install_pr script replacing the token with your token copied in 2.
   wget -qO- https://raw.githubusercontent.com/hyperion-project/hyperion.ng/master/bin/scripts/install_pr.sh | bash -s -- -t token 1678

5. run the execution script to start Hyperion under the current user
   ./hyperion_pr1678/1678.sh

6. Test the new build etc.

7. Remove installed version for the given PR

Note:
The script will stop the current hyperion service before execution.
After doing your tests, you should start the service again or just reboot to start the service automatically.
Replace USER with the username, Hyperion runs with.

systemctl start hyperion@USER.service

Note 2:
In case you would like to test with a current configuration, provide the configuration location via the "-c" option.
The configuration will be copied to avoid any side effects on your current configuration,
i.e. any updates to the configuration when testing the PR will also not be reflected in the current setup.

Sample

Current user
wget -qO- https://raw.githubusercontent.com/hyperion-project/hyperion.ng/master/bin/scripts/install_pr.sh | bash -s -- -t token -c ~/.hyperion 1678

[ManuVice]

Hi, I'am sorry but I cannot reproduce the memleak. I have tried some docker containers which had the memleak in past, but now they are working fine.
With the last version of HyperHDR I can reproduce it. I havent changed any hardware, so I dont know why it doesnt leak atm.

I will start some new tests with a fresh native install in the next days

[Lord-Grey]

@ManuVice Thanks for looking into it.
Note: HyperHDR changed lots of its codebase, therefore you cannot compare the outcome of the observations and the test.

[ManuVice]

I've set up my OdroidC2 with a fresh bullseye image, but I could not reproduce the memleak. I've also tested your PR and it worked fine, too.
I'am sorry, but I havent changed any hardware. I dont know why it works now.

Hopefully this PR will solve it for the others with this problem. Thank you for your hard work!!