Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

earlyoom service: Cannot configure second part of -m/-s parameters #83504

Closed
nh2 opened this issue Mar 27, 2020 · 5 comments · Fixed by #163663
Closed

earlyoom service: Cannot configure second part of -m/-s parameters #83504

nh2 opened this issue Mar 27, 2020 · 5 comments · Fixed by #163663
Labels
2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md 6.topic: nixos

Comments

@nh2
Copy link
Contributor

nh2 commented Mar 27, 2020

Issue description

The earlyoom module requires type = int:

freeMemThreshold = mkOption {
type = types.int;
default = 10;
description = ''
Minimum of availabe memory (in percent).
If the free memory falls below this threshold and the analog is true for
<option>services.earlyoom.freeSwapThreshold</option>
the killing begins.
'';
};

but in fact it can be given docs

  -m PERCENT[,KILL_PERCENT] set available memory minimum to PERCENT of total
                            (default 10 %).
                            earlyoom sends SIGTERM once below PERCENT, then
                            SIGKILL once below KILL_PERCENT (default PERCENT/2).
  -s PERCENT[,KILL_PERCENT] set free swap minimum to PERCENT of total (default
                            10 %).
                            Note: both memory and swap must be below minimum for
                            earlyoom to act.

so e.g. -m 10,5 to configure KILL_PERCENT.

That means the KILL_PERCENT setting is currently not configurable in NixOS.

It is very important to configure it, because as written:

both memory and swap must be below minimum for earlyoom to act.

That means earlyoom will only start terminating processes after a large amount of swap is full, which can take forever to happen, thus defeating a major point of earlyoom. (You can work around that by completely disabling swap, but then you can't suspend-to-disk either.)

Even if you set freeSwapThreshold = 100, that will translate to to -s 100,50 by default, so that's 50% of swap filled!

This is why we need to allow the user to configure KILL_PERCENT, so that they can set it to an impactful value like -s 100,100.

We will have to change away from type = int, probably to accept both int and string for backwards compatibility.

Steps to reproduce

Set up earlyoom with swap, run out of memory, see your system freeze.

Technical details

19.09 (same for current master).


CC @wedens @ljli as past contributors.

@ljli
Copy link
Contributor

ljli commented Mar 27, 2020

It seems they changed the behavior of the flag: rfjakob/earlyoom@bcd1ad4.
Previously it only accepted one value and always SIGKILL'd, now it still uses the limit in case one value is passed but tries SIGTERM instead of SIGKILL.
Personally I would like freeMemThreshold and freeSwapThreshold to be used in both tuple components to restore the original behavior.
The new flag behavior could be covered with new options freeMemThreshold{SigKill,SigTerm} respective freeSwapThreshold{SigKill,SigTerm} that would be mutually exclusive with freeMemThreshold respective freeSwapThreshold.
On the other hand the changed behavior was introduced some time ago in e5508a4, so theoretical someone could rely on the new behavior.

@nh2
Copy link
Contributor Author

nh2 commented Mar 28, 2020

@ljli I also liked the old behaviour much better, but I think it would be extra confusing to deviate from upstream's defaults.

Would you be up to implementing

The new flag behavior could be covered with new options freeMemThreshold{SigKill,SigTerm} respective freeSwapThreshold{SigKill,SigTerm} that would be mutually exclusive with freeMemThreshold respective freeSwapThreshold.

? And then we would add extra documentation to highlight

both memory and swap must be below minimum for earlyoom to act.

and point out what you have to set to get the old behaviour back.

@stale
Copy link

stale bot commented Sep 24, 2020

Hello, I'm a bot and I thank you in the name of the community for opening this issue.

To help our human contributors focus on the most-relevant reports, I check up on old issues to see if they're still relevant. This issue has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

The community would appreciate your effort in checking if the issue is still valid. If it isn't, please close it.

If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me". If you'd like it to get more attention, you can ask for help by searching for maintainers and people that previously touched related code and @ mention them in a comment. You can use Git blame or GitHub's web interface on the relevant files to find them.

Lastly, you can always ask for help at our Discourse Forum or at #nixos' IRC channel.

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Sep 24, 2020
@nh2
Copy link
Contributor Author

nh2 commented Sep 26, 2020

Still relevant

@stale stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Sep 26, 2020
@stale
Copy link

stale bot commented Mar 26, 2021

I marked this as stale due to inactivity. → More info

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Mar 26, 2021
ncfavier added a commit to ncfavier/nixpkgs that referenced this issue Mar 22, 2022
peterhoeg pushed a commit that referenced this issue Mar 24, 2022
* nixos/earlyoom: bring the module up to date

Removes deprecated option `ignoreOOMScoreAdjust`, introduces `killHook`
as a replacement for `notificationsCommand`, and adds an `extraArgs`
option for things not covered by the module.

* nixos/earlyoom: add nixos test

* nixos/earlyoom: add reportInterval

Allows setting the interval for logging a memory report. Defaults to
3600 following upstream
     (https://github.com/rfjakob/earlyoom/blob/master/earlyoom.default#L5)
to avoid flooding logs.

* nixos/earlyoom: add free{Mem,Swap}KillThreshold

Fixes #83504
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md 6.topic: nixos
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants