Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: libretime process leaks and lsof high cpu usage #2615

Merged
merged 1 commit into from
Jul 12, 2023

Conversation

conet
Copy link
Contributor

@conet conet commented Jul 6, 2023

Description

The default docker composition had several problems like leaking processes and high lsof cpu usage. The processes were leaking because of this call and no init process to reap the processes once they finished. The lsof thing is a know issue with docker.

Testing Notes

What I did:

Ran libretime with these changes and everything worked as expected

How you can replicate my testing:

Run the docker composition and observe process leak and high cpu usage:

root     3192950  0.0  0.0 720496  9588 ?        Sl   04:03   0:00 /usr/bin/containerd-shim-runc-v2 -namespace moby -id f8742179f10197ea27612dc855a761b6f93fbcdba3e0428f0d2eb31f5587590d -address /run/containerd/containerd.sock
almalin+ 3192970  3.1  0.1 963584 81564 ?        SLsl 04:03   0:05  \_ libretime-liquidsoap --verbose /app/radio.liq
almalin+ 3197149  0.0  0.0      0     0 ?        Z    04:04   0:00      \_ [timeout] <defunct>
almalin+ 3197199  0.0  0.0      0     0 ?        Z    04:04   0:00      \_ [timeout] <defunct>
...
root       26497  0.0  0.0 720752  6620 ?        Sl   Jun28   1:51 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 262d9318917de816be525b437b1f1634ab4d240333053075a375f6293eb28da7 -address /run/containerd/containerd.sock
almalin+   26570  0.0  0.0 574872 44016 ?        Ssl  Jun28   1:49  \_ /usr/local/bin/python /usr/local/bin/libretime-playout
almalin+ 3139332 18.8  0.0   3836   808 ?        S    03:55   2:07      \_ lsof -- /app/scheduler/15.mp3
almalin+ 3196381 46.0  0.0   3836   148 ?        R    04:04   1:10          \_ lsof -- /app/scheduler/15.mp3

@conet conet changed the title Fix libretime process leaks and lsof high cpu usage fix: libretime process leaks and lsof high cpu usage Jul 6, 2023
@paddatrapper
Copy link
Contributor

Thanks for the PR.

  • lsof is only invoked from the playout service, so the nofile ulimit is only required for that service.
  • libretime-playout-notify is only called from the liquidsoap service, so init should only be needed for that service.

@conet
Copy link
Contributor Author

conet commented Jul 12, 2023

That is correct but I see not reason to not have those settings on the other services as well just to keep things future proof, anyway it's up to you, we already solved our problem and just wanted to contribute to the community to avoid others going through the same hurdles.

@paddatrapper paddatrapper merged commit bd6822f into libretime:main Jul 12, 2023
11 checks passed
@jooola
Copy link
Contributor

jooola commented Jul 14, 2023

Hey @conet, could you please explain what you mean with "future-proof" ? Is that a recommendation from docker ?

@conet
Copy link
Contributor Author

conet commented Jul 15, 2023

By future proof I meant that if libretime changes in the future to call subprocesses or lsof from other places having an init process in the container and proper open file limits is a good thing without any drawbacks (that I know of).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

When using libretime docker compose there are process leaks and high cpu usage from lsof
3 participants