OOM killer on RPiOS Trixie under certain conditions: MGLRU tweak implicated

[andrum993]

Describe the bug

I'm getting spurious OOM killer events on Raspberry Pi OS Trixie 64-bit after loading ZFS using DKMS. By 'spurious' I mean that there is plenty of free RAM.

Steps to reproduce the behaviour

0. Pi 5 8GB RAM with 64-bit Raspberry Pi OS Trixie

sudo apt update && sudo apt full-upgrade -y
sudo apt install --no-install-recommends dkms
sudo apt install zfsutils-linux samba

2. Create or import a zpool.
3. Share out a directory on the pool over samba.
4. dmesg -w
5. Write lots of data to the share.
6. Observe OOM killer event in dmesg output. Sometimes the samba bit isn't needed - it just OOMs on its own.

Device (s)

Raspberry Pi 5

System

Trixie fully up to date. I've now wiped the machine and loaded RPiOS Bookworm on it, which is working correctly with ZFS.

Logs

[  302.364958] CPU: 3 UID: 0 PID: 72 Comm: kswapd3 Tainted: P           O       6.12.47+rpt-rpi-2712 #1  Debian 1:6.12.47-1+rpt1
[  302.364968] Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE
[  302.364970] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT)
[  302.364973] Call trace:
[  302.364977]  dump_backtrace.part.0+0xe0/0x100
[  302.365002]  show_stack+0x20/0x40
[  302.365005]  dump_stack_lvl+0x60/0x80
[  302.365017]  dump_stack+0x18/0x28
[  302.365020]  dump_header+0x48/0x178
[  302.365025]  oom_kill_process+0x29c/0x320
[  302.365033]  out_of_memory+0xf4/0x5b0
[  302.365037]  balance_pgdat+0x554/0x958
[  302.365045]  kswapd+0x248/0x500
[  302.365051]  kthread+0x118/0x128
[  302.365057]  ret_from_fork+0x10/0x20
[  302.365063] Mem-Info:
[  302.365083] active_anon:5097 inactive_anon:0 isolated_anon:0
                active_file:8249 inactive_file:5019 isolated_file:0
                unevictable:0 dirty:0 writeback:0
                slab_reclaimable:1564 slab_unreclaimable:5738
                mapped:3925 shmem:1177 pagetables:237
                sec_pagetables:0 bounce:0
                kernel_misc_reclaimable:0
                free:229191 free_pcp:6721 free_cma:3062
[  302.365093] Node 0 active_anon:10256kB inactive_anon:0kB active_file:16576kB inactive_file:9744kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:7984kB dirty:0kB writeback:0kB shmem:2336kB writeback_tmp:0kB kernel_stack:1936kB pagetables:608kB sec_pagetables:0kB all_unreclaimable? no
[  302.365099] Node 1 active_anon:6912kB inactive_anon:0kB active_file:16560kB inactive_file:9888kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:8016kB dirty:0kB writeback:0kB shmem:2176kB writeback_tmp:0kB kernel_stack:448kB pagetables:448kB sec_pagetables:0kB all_unreclaimable? no
[  302.365104] Node 2 active_anon:7536kB inactive_anon:0kB active_file:16672kB inactive_file:10240kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:7920kB dirty:0kB writeback:0kB shmem:2368kB writeback_tmp:0kB kernel_stack:496kB pagetables:480kB sec_pagetables:0kB all_unreclaimable? no
[  302.365109] Node 3 active_anon:14896kB inactive_anon:0kB active_file:16448kB inactive_file:10208kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:7776kB dirty:0kB writeback:0kB shmem:2352kB writeback_tmp:0kB kernel_stack:144kB pagetables:528kB sec_pagetables:0kB all_unreclaimable? no
[  302.365115] Node 4 active_anon:15632kB inactive_anon:0kB active_file:16384kB inactive_file:10048kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:7728kB dirty:0kB writeback:0kB shmem:2560kB writeback_tmp:0kB kernel_stack:368kB pagetables:432kB sec_pagetables:0kB all_unreclaimable? no
[  302.365122] Node 5 active_anon:7744kB inactive_anon:0kB active_file:16368kB inactive_file:9904kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:7776kB dirty:0kB writeback:0kB shmem:2320kB writeback_tmp:0kB kernel_stack:160kB pagetables:416kB sec_pagetables:0kB all_unreclaimable? no
[  302.365126] Node 6 active_anon:7920kB inactive_anon:0kB active_file:16416kB inactive_file:10128kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:7744kB dirty:0kB writeback:0kB shmem:2400kB writeback_tmp:0kB kernel_stack:560kB pagetables:432kB sec_pagetables:0kB all_unreclaimable? no
[  302.365131] Node 7 active_anon:10656kB inactive_anon:0kB active_file:16560kB inactive_file:10144kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:7856kB dirty:0kB writeback:0kB shmem:2320kB writeback_tmp:0kB kernel_stack:144kB pagetables:448kB sec_pagetables:0kB all_unreclaimable? no
[  302.365136] Node 0 DMA free:84416kB boost:32768kB min:34784kB low:35792kB high:36800kB reserved_highatomic:0KB active_anon:10256kB inactive_anon:0kB active_file:16576kB inactive_file:9744kB unevictable:0kB writepending:0kB present:1044480kB managed:1016096kB mlocked:0kB bounce:0kB free_pcp:3120kB local_pcp:928kB free_cma:48992kB
[  302.365145] lowmem_reserve[]: 0 0 0 0
[  302.365150] Node 1 DMA free:35824kB boost:32768kB min:34832kB low:35872kB high:36912kB reserved_highatomic:0KB active_anon:6912kB inactive_anon:0kB active_file:16560kB inactive_file:9888kB unevictable:0kB writepending:0kB present:1048576kB managed:1044464kB mlocked:0kB bounce:0kB free_pcp:2672kB local_pcp:1040kB free_cma:0kB
[  302.365158] lowmem_reserve[]: 0 0 0 0
[  302.365164] Node 2 DMA free:35808kB boost:32768kB min:34832kB low:35872kB high:36912kB reserved_highatomic:0KB active_anon:7536kB inactive_anon:0kB active_file:16672kB inactive_file:10240kB unevictable:0kB writepending:0kB present:1048576kB managed:1044464kB mlocked:0kB bounce:0kB free_pcp:2800kB local_pcp:864kB free_cma:0kB
[  302.365169] lowmem_reserve[]: 0 0 0 0
[  302.365176] Node 3 DMA free:35232kB boost:32768kB min:34704kB low:35680kB high:36656kB reserved_highatomic:0KB active_anon:14896kB inactive_anon:0kB active_file:16448kB inactive_file:10208kB unevictable:0kB writepending:0kB present:1048576kB managed:978928kB mlocked:0kB bounce:0kB free_pcp:25184kB local_pcp:928kB free_cma:0kB
[  302.365182] lowmem_reserve[]: 0 0 0 0
[  302.365188] Node 4 Normal free:771184kB boost:0kB min:2064kB low:3104kB high:4144kB reserved_highatomic:0KB active_anon:15632kB inactive_anon:0kB active_file:16384kB inactive_file:10048kB unevictable:0kB writepending:0kB present:1048576kB managed:1044464kB mlocked:0kB bounce:0kB free_pcp:8272kB local_pcp:224kB free_cma:0kB
[  302.365193] lowmem_reserve[]: 0 0 0 0
[  302.365198] Node 5 Normal free:900560kB boost:0kB min:2064kB low:3104kB high:4144kB reserved_highatomic:0KB active_anon:7744kB inactive_anon:0kB active_file:16368kB inactive_file:9904kB unevictable:0kB writepending:0kB present:1048576kB managed:1044464kB mlocked:0kB bounce:0kB free_pcp:26080kB local_pcp:624kB free_cma:0kB
[  302.365203] lowmem_reserve[]: 0 0 0 0
[  302.365209] Node 6 Normal free:908752kB boost:0kB min:2064kB low:3104kB high:4144kB reserved_highatomic:0KB active_anon:7920kB inactive_anon:0kB active_file:16416kB inactive_file:10128kB unevictable:0kB writepending:0kB present:1048576kB managed:1044464kB mlocked:0kB bounce:0kB free_pcp:20096kB local_pcp:384kB free_cma:0kB
[  302.365214] lowmem_reserve[]: 0 0 0 0
[  302.365219] Node 7 Normal free:895280kB boost:0kB min:2048kB low:3072kB high:4096kB reserved_highatomic:0KB active_anon:10656kB inactive_anon:0kB active_file:16560kB inactive_file:10144kB unevictable:0kB writepending:0kB present:1048576kB managed:1039184kB mlocked:0kB bounce:0kB free_pcp:19280kB local_pcp:736kB free_cma:0kB
[  302.365224] lowmem_reserve[]: 0 0 0 0
[  302.365230] Node 0 DMA: 98*16kB (E) 61*32kB (UMEC) 50*64kB (UMEC) 25*128kB (UMC) 23*256kB (UMC) 12*512kB (UC) 5*1024kB (UMC) 0*2048kB 2*4096kB (MC) 2*8192kB (MC) 0*16384kB 1*32768kB (C) = 84416kB
[  302.365254] Node 1 DMA: 31*16kB (UME) 43*32kB (ME) 57*64kB (UME) 56*128kB (UME) 41*256kB (UME) 16*512kB (UME) 2*1024kB (UE) 1*2048kB (M) 0*4096kB 0*8192kB 0*16384kB 0*32768kB = 35472kB
[  302.365277] Node 2 DMA: 51*16kB (UE) 47*32kB (UME) 41*64kB (UME) 34*128kB (UME) 21*256kB (UE) 18*512kB (UE) 3*1024kB (UME) 0*2048kB 0*4096kB 1*8192kB (M) 0*16384kB 0*32768kB = 35152kB
[  302.365306] Node 3 DMA: 28*16kB (UME) 7*32kB (UE) 14*64kB (UME) 21*128kB (UME) 10*256kB (UME) 5*512kB (ME) 1*1024kB (U) 2*2048kB (UM) 1*4096kB (M) 0*8192kB 1*16384kB (U) 0*32768kB = 34976kB
[  302.365331] Node 4 Normal: 113*16kB (UME) 105*32kB (UME) 63*64kB (UME) 27*128kB (UM) 13*256kB (UME) 11*512kB (UME) 0*1024kB 2*2048kB (ME) 2*4096kB (UM) 2*8192kB (UE) 2*16384kB (UE) 21*32768kB (M) = 771184kB
[  302.365354] Node 5 Normal: 16*16kB (UME) 15*32kB (UE) 12*64kB (UE) 6*128kB (UE) 6*256kB (UME) 2*512kB (UE) 1*1024kB (E) 3*2048kB (E) 3*4096kB (UME) 1*8192kB (U) 3*16384kB (UME) 25*32768kB (M) = 900832kB
[  302.365379] Node 6 Normal: 6*16kB (UE) 9*32kB (E) 14*64kB (UME) 7*128kB (UE) 7*256kB (UE) 3*512kB (UME) 2*1024kB (ME) 2*2048kB (ME) 1*4096kB (U) 1*8192kB (E) 2*16384kB (ME) 26*32768kB (M) = 908672kB
[  302.365402] Node 7 Normal: 5*16kB (UME) 13*32kB (UME) 13*64kB (UE) 7*128kB (UME) 7*256kB (ME) 5*512kB (UME) 2*1024kB (UM) 3*2048kB (UME) 1*4096kB (U) 3*8192kB (UME) 2*16384kB (UE) 25*32768kB (M) = 895408kB
[  302.365429] 14433 total pagecache pages
[  302.365432] 0 pages in swap cache
[  302.365433] Free swap  = 2097136kB
[  302.365435] Total swap = 2097136kB
[  302.365437] 524032 pages RAM
[  302.365439] 0 pages HighMem/MovableOnly
[  302.365440] 7999 pages reserved
[  302.365441] 4096 pages cma reserved
[  302.365442] Tasks state (memory values in pages):
[  302.365444] [  pid  ]   uid  tgid total_vm      rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
[  302.366701] [    318]     0   318     1686      522       96      394        32    98304        0          -250 systemd-journal
[  302.366733] [    389]   991   389     5785      489       96      393         0   114688        0             0 systemd-timesyn
[  302.366742] [    393]     0   393     2248      623      192      431         0   114688        0         -1000 systemd-udevd
[  302.366760] [    765]   101   765      406      196       32      164         0    98304        0             0 avahi-daemon
[  302.366767] [    766]     0   766      822      330       32      298         0    98304        0             0 bluetoothd
[  302.366771] [    767]     0   767      447      134        0      134         0    98304        0             0 cron
[  302.366776] [    769]   990   769      573      265       64      201         0   114688        0          -900 dbus-daemon
[  302.366782] [    772]   988   772    19244      464       64      400         0   180224        0             0 polkitd
[  302.366786] [    775]     0   775     1207      522       96      426         0    98304        0             0 systemd-logind
[  302.366791] [    776]     0   776    15295      357       96      261         0   163840        0             0 zed
[  302.366795] [    791]   101   791      397       98       32       66         0    98304        0             0 avahi-daemon
[  302.366799] [   1061]     0  1061    21413     1185      224      961         0   196608        0             0 NetworkManager
[  302.366806] [   1062]     0  1062     1104      397       64      333         0   114688        0             0 wpa_supplicant
[  302.366810] [   1072]     0  1072    20335      819      224      595         0   196608        0             0 ModemManager
[  302.366817] [   1190]     0  1190      655      265       64      201         0    98304        0             0 login
[  302.366821] [   1195]     0  1195      466      131        0      131         0   114688        0             0 agetty
[  302.366824] [   1301]  1000  1301     1467      747      160      587         0   114688        0           100 systemd
[  302.366829] [   1303]  1000  1303     1589      296      163      133         0   114688        0           100 (sd-pam)
[  302.366833] [   1321]  1000  1321      458      201       32      169         0    98304        0           200 mpris-proxy
[  302.366838] [   1322]  1000  1322      560      325      128      197         0    98304        0             0 bash
[  302.366841] [   1323]  1000  1323      543      235       32      203         0   114688        0           200 dbus-daemon
[  302.366846] [   1481]     0  1481      720      425       64      361         0   114688        0         -1000 sshd
[  302.366851] [   1493]     0  1493     1438      763      160      603         0   114688        0             0 sshd-session
[  302.366857] [   1501]  1000  1501     1445      543      179      364         0   114688        0             0 sshd-session
[  302.366865] [   1502]  1000  1502      567      324      128      196         0    98304        0             0 bash
[  302.366890] [   2410]     0  2410     7061     1800      384     1192       224   163840        0             0 smbd
[  302.366895] [   2413]     0  2413     6676      613      382      231         0   163840        0             0 smbd-notifyd
[  302.366900] [   2414]     0  2414     6678      581      382      199         0   163840        0             0 smbd-cleanupd
[  302.366905] [   2420]  1000  2420      361      102        0      102         0    98304        0             0 dmesg
[  302.366908] [   2422]  1000  2422    11468     1605      547      891       167   196608        0             0 smbd[192.168.1.
[  302.366915] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0-7,global_oom,task_memcg=/,task=dbus-daemon,pid=1323,uid=1000
[  302.366952] Out of memory: Killed process 1323 (dbus-daemon) total-vm:8688kB, anon-rss:512kB, file-rss:3248kB, shmem-rss:0kB, UID:1000 pgtables:112kB oom_score_adj:200

Additional context

This was first found on a system that was happily chugging away, until I loaded ZFS on it. I verified the abnormal behaviour on a completely clean install of Raspberry Pi OS Trixie. I also tested with zram swap disabled, which made no difference.

Note that I've also successfully used ZFS on Debian Trixie on x86-64, so this is Raspberry Pi OS Trixie-specific.

[popcornmix]

While you have enough system memory available, there are other pools that can trigger OOM like CMA.

Can you try editing the vc4-kms-v3d line in config.txt to:

dtoverlay=vc4-kms-v3d,cma-128

(you could try 256 if this doesn't help).

[XECDesign]

May also be related to the Multi-Gen LRU thrashing prevention commit?

If running echo 0 | sudo tee /sys/kernel/mm/lru_gen/min_ttl_ms makes the problem go away, that would be good to know too.

[andrum993]

May also be related to the Multi-Gen LRU thrashing prevention commit?

If running echo 0 | sudo tee /sys/kernel/mm/lru_gen/min_ttl_ms makes the problem go away, that would be good to know too.

Thanks - that seems to have cured it. I will remove that setting from all my Pis, since the description of what it does doesn't exactly fill me with confidence: assuming that tunable works in the way documented on the page you have linked to, it seems like a rather daft way to fix janks. It sounds like a sliding scale with high likelihood of janks at one end, and high likelihood of OOMs at the other. The range should really have janks at one end, and completely fine but no janks at the other. Certainly not OOMs.

I will leave the bug open since it occurs on Raspberry Pi OS Trixie out of the box, once ZFS is added.

[danielschlenzig]

I also have that issue on a PI5 with 4 GB of RAM.

For testing I set the dtoverlay parameter to cma-256 and so far it seems stable. cma-128 did not work and the oom-killer started shortly after the SMB copy process started.

(On a side note, I also updated to the latest ZFS from trixie-backports as I previously thought it was ARC/dnode cache related)

[danielschlenzig]

For testing I set the dtoverlay parameter to cma-256 and so far it seems stable. cma-128 did not work and the oom-killer started shortly after the SMB copy process started.

Well, that wasn't it. Actually, setting /sys/kernel/mm/lru_gen/min_ttl_ms to 0 also fixed the issue for my system.

[popcornmix]

Does setting it to a lower than default value avoid the issue? If it's currently 1000, then try 100.
If there are still issues then 50 and then 10 would be useful data points.

[andrum993]

Does setting it to a lower than default value avoid the issue? If it's currently 1000, then try 100. If there are still issues then 50 and then 10 would be useful data points.

I will not be doing this, since I have no need for it.

[andrum993]

See also https://forums.raspberrypi.com/viewtopic.php?t=392785 for a user who has the same problem, fixed in the same way, when using aptitude and git on their RPiOS Trixie-based Pi. I've amended the bug title accordingly.

[kmhofmann]

Encountered a very similar problem: spurious OOM kills on Trixie while plenty of RAM remains free. Mine are unrelated to aptitude or git or ZFS but instead relate to gocryptfs filling kernel and own caches during file operations.

And I can confirm that setting echo 0 | sudo tee /sys/kernel/mm/lru_gen/min_ttl_ms seems to fix the issue.

[davy7125]

The same thing happens here with a Java application using 2.3 GB of RAM, opened at startup. It is killed by OOM with Trixie, and this bug did not appear with Bookworm. Using one of the following solutions resolves the issue:

• dtoverlay=vc4-kms-v3d,cma-256 in file /boot/firmware/config.txt (",cma-256" appended to a line)
• echo 0 | sudo tee /sys/kernel/mm/lru_gen/min_ttl_ms

[kmhofmann]

Does setting it to a lower than default value avoid the issue? If it's currently 1000, then try 100. If there are still issues then 50 and then 10 would be useful data points.

IMO, this is useless - any value but 0 (disabled) is likely to lead to problems eventually.

Either way, I can explicitly confirm that a value of 100 (as per latest update) does NOT fix the issues for me.

[m1nkm4n]

I am having the oom problem related to ZFS usage. Trixie installation was an upgrade from bookworm. Neither solution, adding cma-256 or setting min_ttl_ms to zero helped for long. They both lengthened the time between oom's, but oom's still happened.
I did a fresh install of bookworm on a separate sd card and it is stable.

[andrum993]

Does setting it to a lower than default value avoid the issue? If it's currently 1000, then try 100. If there are still issues then 50 and then 10 would be useful data points.

IMO, this is useless - any value but 0 (disabled) is likely to lead to problems eventually.

That's my feeling too. Anything which makes the system less reliable or stable is a bad thing in my book. Attempting to fix desktop stuttering by deliberately making the system less reliable and stable makes no sense to me.

[kmhofmann]

Indeed. Desktop stutter also seems like a very minor issue to have, compared to random and vital processes being stopped by the OOM killer while >70% of physical RAM is reported free (with swap unused), which I would classify as a very critical bug.