-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory Leak (TCP_Mem Leak / Internet Slow Down) #164
Comments
For some more info, In addition to using Speed Test Tracker, I also confirmed my internet speed using Speedtest's cli tool and can confirm the internet speed reduction. The internet speed could sometimes stay permanently below 9 Mbits/s! In fact, the only reason I noticed this issue was occurring was that when trying to play Navidrome music using bonob on my Sonos speakers, there would be "buffering"(delay before playing and frequent pausing mid-song). My music server, Navidrome, and Sonos speaker are all on the same Lan, so there should never be buffering. So, the issue affects all TCP connections on the server rather than just ones to the internet. The issue takes several hours to appear, and Ubuntu will never reclaim either the system memory being used or the TCP memory, even after not playing music for several hours/days. You must either stop/restart the bonob container or restart your computer. |
Thank you for the great analysis, that's an interesting issue. How much memory do you give the bonob container? With your listening pattern, are you simply listening to an album/playlist track by track or are you frequently changing tracks manually? Unfortunately I cannot test this right now as my system is in a box whilst I move house. |
@simojenki I use Portainer to managed my docker containers. I don't explicitly set a maximum size. I have about 3000 songs in a queue and rarely manually change tracks. It simply plays for hours until I get bored. |
I don't mean to be adding fuel to the fire here, but I can confirm this. Here is my setup: Then I started doing my rating-my-songs behavior. I left the playlist running, and just started rating the songs that are playing. Once I rated them, I would fast forward to the next song. After doing this for a few minutes, the TCP memory use is now 500MB. At this time, the docker container is using 350MB, and TCP is using 500MB. So that I've said it, I have created a playlist with about 3 hours of music in it. I start this in the morning and then start it again around lunch time. This seems to keep things behaving. One thing I'd love to do is be able to attach a debugger to the containerized app, to see if I can identify where this problem is happening. But I've never debugged a container before, much less an app like this. So I'd need some help doing that. |
Interesting... Why do you say it is TCP memory usage? What command did you run to see that? Are you transcoding your tracks? One thing I have noticed is that sometimes a transcoding (ffmpeg) process in the ND container will be still running well after the track has completed, I think some processes are being left behind, perhaps on skip track? |
No, I'm not transcoding anything. |
What happens if you restart one of either bonob or navidrome, does the memory get released? That may help determine which container is leaking. re: docker-compose.yml, you should be able to just refer to the name of the container, rather than either internel or external IP. Given the wide spread use of docker (and friends) I'd be surprised if the issue is caused by docker/Linux kernel, I'm guessing it's going to be an app issue somewhere. |
I'll do some more testing this evening. Thanks! |
Here's what I can see: It looks like if I restart navidrome, all the memory does not get cleared, only about half. Where as if restart bonob, all the memory gets cleared out... Thoughts? |
I guess for some reason bonob is keeping open a connection to navidrome, perhaps because it is unaware that the sonos device has moved onto the next track. |
I've tidied up the way streams between bonob -> navidrome are handled, could you please try again with the 'master' build/release of bonob and see what behaviour you get? |
This worked! |
I'm running different playlists to two separate speakers. |
I've been at this for a few hours now. I think this is fixed! |
Bear with me here since narrowing down what was happening took two weeks. Anyway, so some background:
I'm using bonob in a docker container on the following system(this system is my media server that's only used to run docker containers):
System:
So, I noticed my internet connection on my Ubuntu media server would become really slow, from 1 Gbs to ~20Mbs. I use an app called Speedtest Tracker to monitor this.
Screenshot from Speedtest Tracker:
I've been noticing the above behavior for weeks. My media server's download/internet speed would drop for the whole system and stay low; only a restart would fix it.
Cause
I figured out that the issue was bonob after killing apps one by one and waiting for the issue to reoccur; I figured out what was happening. There seems to be some TCP memory leak with bonob.
When bonob is first launched, my system looks like this(image from docker stats):
and my TCP Memory looks like this:
After listening to music for a few hours, my stats become this:
And inside my
/var/log/syslog
I'm seeing this message repeated:bonob is using 460MB of TCP Memory! This causes the network speed of my server to drop until a) Killing the bonob container or b) restarting the computer.
Killing bonob's container immediately restores my TCP memory and fixes my speed issue.
The text was updated successfully, but these errors were encountered: