RFC: Notifications/actions before systemd-oomd kills

[anitazha]

We've had some discussions offline about customers who are interested in a pre-kill notifications and also pre-kill hooks to run certain things before systemd-oomd kills a process. Starting this issue to track the discussions in one place.

For pre-kill hooks, we discussed the possibility of a generic design where we define a socket for data processing and it goes away when the unit is OOM killed. Apparently this is similar to systemd-coredump. We also discussed using different signals to tell the service that an OOM kill is about to happen, but this is less flexible.

For pre-kill notifications I think we had discussed using dbus to detect the state changes. But I'm not sure what kind of hook up the desktop folks want. Or what kind of settings we need to configure when the notification would send. If possible we should try to tie this in with the pre-kill hooks above and get one generic solution.

@michel-slm @benzea @msekletar

[benzea]

I assume we are talking about a system wide warning (rather than a cgroup specific one)? And, I guess oomd needs to fire it because user applications do not have access to the configuration to estimate whether kills are imminent?

Right now we have a GMemoryMonitor already in glib. So, I guess we would just want to switch that over to an enhanced API (right now there is a small daemon just firing a signal). But, all that has is a low-memory-warning signal, so as long as we somehow can receive a signal in applications, we are good. Also, I doubt there is any real user out there ...

[Werkov]

I've also seen related requests -- to tell the application memory is running low and it should reduce its memory consumption (think of custom allocators in java or browsers). The natural answer would be -- just watch PSI in your cgroup and act based on that.

In a more generic sense, I considered issuing a DBus signal from oomd based on available info -- I learn now that's effectively what GMemoryMonitor already does and it's sad that it's apparently unused -- maybe moving the same functionality into into oomd would make it more accessible/popular.

To the systemd-coredump-like idea -- does it mean that with imminent oomd kill you'd start yet another consumer?

[ruckc]

Just dropping this as examples of why notifications prior to killing a cgroup...

Applications just disappear. If I don't restart daily, it randomly just kills a process, some with drastic effects (killing gnome-shell, Nautilus, gnome-terminal).

Apr 13 10:51:39 omnixps systemd-oomd[3562]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-google\x2dchrome-11396.scope due to memory used (16109408256) / total (16474247168) and swap used (1982627840) / total (2147479552) being more than 90.00%
Apr 13 13:09:57 omnixps systemd-oomd[3363]: Killed /user.slice/user-1000.slice/user@1000.service/session.slice/org.gnome.Shell@wayland.service due to memory used (16195358720) / total (16474247168) and swap used (2053111808) / total (2147479552) being more than 90.00%
Apr 14 13:21:21 omnixps systemd-oomd[3248]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-google\x2dchrome-41482.scope due to memory used (16056369152) / total (16474247168) and swap used (1945407488) / total (2147479552) being more than 90.00%
Apr 18 09:58:14 omnixps systemd-oomd[3457]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-org.gnome.Nautilus-0.scope due to memory used (16223395840) / total (16474243072) and swap used (2027683840) / total (2147479552) being more than 90.00%
Apr 18 09:58:15 omnixps systemd-oomd[3457]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-org.gnome.Nautilus-0.scope due to memory used (16091160576) / total (16474243072) and swap used (2145886208) / total (2147479552) being more than 90.00%
Apr 18 09:58:15 omnixps systemd-oomd[3457]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-org.gnome.Nautilus-0.scope due to memory used (16159051776) / total (16474243072) and swap used (2073485312) / total (2147479552) being more than 90.00%
Apr 18 09:58:15 omnixps systemd-oomd[3457]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-org.gnome.Nautilus-0.scope due to memory used (16212561920) / total (16474243072) and swap used (2008236032) / total (2147479552) being more than 90.00%
Apr 18 09:58:15 omnixps systemd-oomd[3457]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-org.gnome.Nautilus-0.scope due to memory used (15865094144) / total (16474243072) and swap used (1974681600) / total (2147479552) being more than 90.00%
Apr 20 16:00:06 omnixps systemd-oomd[3457]: Killed /user.slice/user-1000.slice/user@1000.service/session.slice/org.gnome.Shell@wayland.service due to memory used (16120066048) / total (16474243072) and swap used (1999306752) / total (2147479552) being more than 90.00%
Apr 20 16:04:18 omnixps systemd-oomd[3457]: Killed /user.slice/user-1000.slice/user@1000.service/session.slice/org.gnome.Shell@wayland.service due to memory used (16117116928) / total (16474243072) and swap used (2039713792) / total (2147479552) being more than 90.00%
Apr 26 08:07:19 omnixps systemd-oomd[3457]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-code-2704032.scope due to memory used (16189804544) / total (16474243072) and swap used (1975377920) / total (2147479552) being more than 90.00%
May 03 12:17:16 omnixps systemd-oomd[3253]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-jetbrains\x2dtoolbox-10574.scope due to memory used (15012593664) / total (16474185728) and swap used (1944969216) / total (2147479552) being more than 90.00%
May 04 22:31:52 omnixps systemd-oomd[3253]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-google\x2dchrome-1560985.scope due to memory used (15096467456) / total (16474185728) and swap used (2009534464) / total (2147479552) being more than 90.00%
May 05 12:34:52 omnixps systemd-oomd[3354]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-google\x2dchrome-47763.scope due to memory used (16104947712) / total (16474189824) and swap used (1948254208) / total (2147479552) being more than 90.00%
May 09 10:26:05 omnixps systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/session.slice/org.gnome.Shell@wayland.service due to memory used (16139558912) / total (16474193920) and swap used (2147348480) / total (2147479552) being more than 90.00%
May 10 19:40:25 omnixps systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-google\x2dchrome-950608.scope due to memory used (16102866944) / total (16474193920) and swap used (2071912448) / total (2147479552) being more than 90.00%
May 10 19:49:30 omnixps systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-16c14424-f9a9-4c04-8ff2-0755377a7539.scope due to memory used (16022802432) / total (16474193920) and swap used (1948741632) / total (2147479552) being more than 90.00%
May 11 09:00:04 omnixps systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-71b574b8-17e5-408b-b91f-431a458d55bb.scope due to memory used (15930691584) / total (16474193920) and swap used (2041720832) / total (2147479552) being more than 90.00%
May 11 10:20:57 omnixps systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-google\x2dchrome-2201754.scope due to memory used (15575441408) / total (16474193920) and swap used (1939279872) / total (2147479552) being more than 90.00%
May 11 12:05:21 omnixps systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-24f78600-4612-4805-9b29-a96ca4f432cb.scope due to memory used (15433670656) / total (16474193920) and swap used (1950101504) / total (2147479552) being more than 90.00%
May 11 12:05:51 omnixps systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-google\x2dchrome-2814868.scope due to memory used (15329484800) / total (16474193920) and swap used (1946755072) / total (2147479552) being more than 90.00%
May 11 12:06:29 omnixps systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-netbeans-2803190.scope due to memory used (15826632704) / total (16474193920) and swap used (2145792000) / total (2147479552) being more than 90.00%
May 11 14:33:29 omnixps systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-google\x2dchrome-2909372.scope due to memory used (15613919232) / total (16474193920) and swap used (1933402112) / total (2147479552) being more than 90.00%
May 11 14:56:16 omnixps systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/session.slice/org.gnome.Shell@wayland.service due to memory used (16104726528) / total (16474193920) and swap used (1968328704) / total (2147479552) being more than 90.00%
May 12 08:25:14 omnixps systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-jetbrains\x2dpycharm-3483077.scope due to memory used (15616532480) / total (16474193920) and swap used (1937936384) / total (2147479552) being more than 90.00%
May 12 11:15:23 omnixps systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-google\x2dchrome-3361444.scope due to memory used (15812100096) / total (16474193920) and swap used (1955221504) / total (2147479552) being more than 90.00%
May 12 17:58:07 omnixps systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-google\x2dchrome-722172.scope due to memory used (16039403520) / total (16474193920) and swap used (1988845568) / total (2147479552) being more than 90.00%
May 13 11:30:57 omnixps systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-google\x2dchrome-1218538.scope due to memory used (16068583424) / total (16474193920) and swap used (2087342080) / total (2147479552) being more than 90.00%
May 13 11:30:59 omnixps systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/session.slice/org.gnome.Shell@wayland.service due to memory used (16112533504) / total (16474193920) and swap used (1946112000) / total (2147479552) being more than 90.00%
May 18 15:47:46 omnixps systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-921c3683-819b-4f55-ba40-5bcd85681825.scope due to memory used (16058290176) / total (16474193920) and swap used (2002198528) / total (2147479552) being more than 90.00%
May 20 09:03:27 omnixps systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-google\x2dchrome-1534842.scope due to memory used (15276462080) / total (16474193920) and swap used (1937162240) / total (2147479552) being more than 90.00%
Jun 03 12:15:36 omnixps systemd-oomd[3298]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-jetbrains\x2dwebstorm-541749.scope due to memory used (15719739392) / total (16474193920) and swap used (1967783936) / total (2147479552) being more than 90.00%
Jun 08 00:21:43 omnixps systemd-oomd[3768]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-google\x2dchrome-16072.scope due to memory used (15445958656) / total (16474218496) and swap used (1942192128) / total (2147479552) being more than 90.00%
Jun 08 07:14:28 omnixps systemd-oomd[3768]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-teams-2369963.scope due to memory used (15633932288) / total (16474218496) and swap used (1964466176) / total (2147479552) being more than 90.00%
Jun 08 07:16:38 omnixps systemd-oomd[3768]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-jetbrains\x2dgoland-2978322.scope due to memory used (15238217728) / total (16474218496) and swap used (1939480576) / total (2147479552) being more than 90.00%
Jun 13 10:14:46 omnixps systemd-oomd[7467]: Killed /user.slice/user-1000.slice/user@1000.service/session.slice/org.gnome.Shell@wayland.service due to memory used (15284772864) / total (16474218496) and swap used (1937956864) / total (2147479552) being more than 90.00%

[hadess]

In a more generic sense, I considered issuing a DBus signal from oomd based on available info -- I learn now that's effectively what GMemoryMonitor already does and it's sad that it's apparently unused -- maybe moving the same functionality into into oomd would make it more accessible/popular.

We already have a GLib API that uses low-memory-monitor's D-Bus API, and portal support for Flatpak. The API is also implemented on Windows through Windows native calls.

The D-Bus API is simple, and extensible:
https://gitlab.freedesktop.org/hadess/low-memory-monitor/-/blob/master/src/org.freedesktop.LowMemoryMonitor.xml

The simplest would be for systemd-oomd to implement that D-Bus interface, while owning the org.freedesktop.LowMemoryMonitor name, and we could obsolete low-memory-monitor itself.

Changing the D-Bus API to use another name or another path would require changing GIO, xdg-desktop-portal and python-dbusmock, at least.

[John-Nagle]

I see this is still open.
Just a data point. I'm getting Rust compiles killed somewhat randomly, on a 32GB system that is not showing any memory pressure.

Dec 18 19:34:56 user-desktop systemd-oomd[840]: Killed /user.slice/user-1001.slice/user@1001.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-dbb4a75e-031f-480d-924b-5de5af7579d2.scope due to memory pressure for /user.slice/user-1001.slice/user@1001.service being 51.07% > 50.00% for > 20s with reclaim activity
Dec 18 19:34:56 user-desktop systemd[1837]: vte-spawn-dbb4a75e-031f-480d-924b-5de5af7579d2.scope: systemd-oomd killed 12 process(es) in this unit.
Dec 18 19:34:57 user-desktop systemd[1837]: vte-spawn-dbb4a75e-031f-480d-924b-5de5af7579d2.scope: Consumed 5h 17min 34.893s CPU time.

Here's the system performance graph running at the time.

Machine is x86_64. No virtualization. 32GB RAM, 2GB swap.
Only about 6GB of memory is in use, according to the performance graph. This suggests that the out-of-memory daemon triggered when it should not have. Maybe this is a false alarm bug rather than a policy problem. Comments?

rustc 1.74.0 (79e9716c9 2023-11-13)
Ubuntu 22.04.3 LTS

[Werkov]

I see this is still open.

Not sure if this doesn't classify as a separate (bug) report.

Comments?

I assume the resource graphs cover the moment of the unxpected kill somewhere after the CPU peaks. (And that those >5h CPU time includes multiple compiles out of the captured window.)

Are you sure there is no memory limit configured in any of the /user.slice/user-1001.slice/user@1001.service units?
What kind of HW is the swap partition on? Can anything else use it in concurrently with troublesome builds?

[John-Nagle]

I assume the resource graphs cover the moment of the unexpected kill somewhere after the CPU peaks.

Yes, that's the compute blip in the middle of the graph.

Are you sure there is no memory limit configured in any of the /user.slice/user-1001.slice/user@1001.service units?

I dunno. Ubuntu puts a huge amount of stuff in those directories, none of which I ever touched. This is a desktop system, and it's not acting as a server at all.

/sys/fs/cgroup/user.slice/user-1001.slice$ ls -lt
total 0
-rw-r--r-- 1 root root 0 Dec 12 15:52 cgroup.subtree_control
-r--r--r-- 1 root root 0 Nov 24 23:33 cgroup.controllers
-rw-r--r-- 1 root root 0 Nov 24 23:33 cgroup.freeze
--w------- 1 root root 0 Nov 24 23:33 cgroup.kill
-rw-r--r-- 1 root root 0 Nov 24 23:33 cgroup.max.depth
-rw-r--r-- 1 root root 0 Nov 24 23:33 cgroup.max.descendants
-rw-r--r-- 1 root root 0 Nov 24 23:33 cgroup.pressure
-rw-r--r-- 1 root root 0 Nov 24 23:33 cgroup.procs
-r--r--r-- 1 root root 0 Nov 24 23:33 cgroup.stat
-rw-r--r-- 1 root root 0 Nov 24 23:33 cgroup.threads
-rw-r--r-- 1 root root 0 Nov 24 23:33 cgroup.type
-rw-r--r-- 1 root root 0 Nov 24 23:33 cpu.pressure
-rw-r--r-- 1 root root 0 Nov 24 23:33 io.pressure
-r--r--r-- 1 root root 0 Nov 24 23:33 memory.current
-r--r--r-- 1 root root 0 Nov 24 23:33 memory.events
-r--r--r-- 1 root root 0 Nov 24 23:33 memory.events.local
-r--r--r-- 1 root root 0 Nov 24 23:33 memory.numa_stat
-r--r--r-- 1 root root 0 Nov 24 23:33 memory.peak
-rw-r--r-- 1 root root 0 Nov 24 23:33 memory.pressure
--w------- 1 root root 0 Nov 24 23:33 memory.reclaim
-r--r--r-- 1 root root 0 Nov 24 23:33 memory.stat
-r--r--r-- 1 root root 0 Nov 24 23:33 memory.swap.current
-r--r--r-- 1 root root 0 Nov 24 23:33 memory.swap.events
-rw-r--r-- 1 root root 0 Nov 24 23:33 memory.swap.high
-r--r--r-- 1 root root 0 Nov 24 23:33 memory.zswap.current
-rw-r--r-- 1 root root 0 Nov 24 23:33 memory.zswap.max
-r--r--r-- 1 root root 0 Nov 24 23:33 pids.current
-r--r--r-- 1 root root 0 Nov 24 23:33 pids.events
-r--r--r-- 1 root root 0 Nov 24 23:33 pids.peak
drwxr-xr-x 2 root root 0 Nov 24 23:33 session-6.scope
drwxr-xr-x 2 root root 0 Nov 24 23:32 session-c2.scope
drwxr-xr-x 2 root root 0 Nov 24 23:32 session-c1.scope
drwxr-xr-x 6 john john 0 Nov 24 23:32 user@1001.service
-r--r--r-- 1 root root 0 Nov 24 23:32 cgroup.events
-r--r--r-- 1 root root 0 Nov 24 23:32 cpu.stat
-rw-r--r-- 1 root root 0 Nov 24 23:32 memory.high
-rw-r--r-- 1 root root 0 Nov 24 23:32 memory.low
-rw-r--r-- 1 root root 0 Nov 24 23:32 memory.max
-rw-r--r-- 1 root root 0 Nov 24 23:32 memory.min
-rw-r--r-- 1 root root 0 Nov 24 23:32 memory.oom.group
-rw-r--r-- 1 root root 0 Nov 24 23:32 memory.swap.max
-rw-r--r-- 1 root root 0 Nov 24 23:32 pids.max

john@user-desktop:/sys/fs/cgroup/user.slice/user-1001.slice$

What kind of HW is the swap partition on?

3TB rotating hard drive.

Can anything else use it in concurrently with troublesome builds?

It's the main hard drive. There's an SSD, but the system swap isn't on it.