Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

4.19: ext4_inode_cache slab memory usage, doesn't get reclaimed #2829

Open
cinch opened this issue Jan 26, 2019 · 21 comments

Comments

Projects
None yet
8 participants
@cinch
Copy link

commented Jan 26, 2019

i tried out the next branch of https://github.com/raspberrypi/firmware/ with "sudo BRANCH=next rpi-update". this gave me kernel version 4.19.16. i had no issues with it until i noticed that the memory usage was going up and up.
after executing "sudo slabtop", ext4_inode_cache was using ~170 MB. normally this would get reclaimed when the rest of the memory is getting full, however this was not the case here. even when the memory was fully used up by userspace programs, and the kernel was agressively swapping, the ext4_inode_cache would not go down. increasing "vm.vfs_cache_pressure" and "echo 3 > /proc/sys/vm/drop_caches" did not help. i've since went back to the older 4.14 kernel and the issue doesn't appear there.

  • Which model of Raspberry Pi? e.g. Pi3B+, PiZeroW
    Pi3B+
  • Which OS and version (cat /etc/rpi-issue)?
    Raspbian Stretch with desktop

Additional context
I'm not using a microSD card: the system is on a USB attached SSD drive. Let me know if you need more info to reproduce this issue.

@DrJosh9000

This comment has been minimized.

Copy link

commented Mar 1, 2019

No idea if it's the same bug, but this seems similar to an issue I have been wrangling recently with the 4.19.16 and 4.19.20 Linux kernels on some non-Raspberry Pis.

In my case it turned out that the kernel was leaking memory associated with cgroups - and over time it grew to a large size because we had some faulty systemd services restarting over and over (creating and removing a cgroup each time, but the kernel leaked).
Here's an LKML thread discussing backporting the patches: https://lkml.org/lkml/2018/11/2/160
And a bug in Ubuntu: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1792349

@Pmant

This comment has been minimized.

Copy link

commented Mar 6, 2019

I think I'm experiencing the same issue on my Pi 3.

root@raspberrypi:~# slabtop -s c
 Active / Total Objects (% used)    : 1718215 / 1738017 (98.9%)
 Active / Total Slabs (% used)      : 66822 / 66822 (100.0%)
 Active / Total Caches (% used)     : 88 / 102 (86.3%)
 Active / Total Size (% used)       : 599262.30K / 603247.54K (99.3%)
 Minimum / Average / Maximum Object : 0.02K / 0.35K / 8.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
638336 638336 100%    0.71K  44561       22    712976K ext4_inode_cache
361062 356987  98%    0.30K  13887       26    111096K radix_tree_node
607744 607744 100%    0.03K   4748      128     18992K ext4_extent_status
 10017   9933  99%    0.38K    477       21      3816K inode_cache
 22530  14787  65%    0.13K    751       30      3004K dentry
 16170  16170 100%    0.09K    385       42      1540K kernfs_node_cache

System info:

root@raspberrypi:~# cat /sys/firmware/devicetree/base/model
Raspberry Pi 3 Model B Rev 1.2root

root@raspberrypi:~# uname -a
Linux raspberrypi 4.19.25-v7+ #1205 SMP Mon Feb 25 18:19:20 GMT 2019 armv7l GNU/Linux

root@raspberrypi:~# cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 8 (jessie)"
NAME="Raspbian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
@DrJosh9000

This comment has been minimized.

Copy link

commented Mar 8, 2019

What does cat /proc/cgroups say?

@Pmant

This comment has been minimized.

Copy link

commented Mar 8, 2019

I'm sorry, I had to switch back to 4.14 since the Pi was almost unusable

@JamesH65

This comment has been minimized.

Copy link
Contributor

commented Mar 8, 2019

@Pmant What specifically are you doing that caused the issue on 4.19.16

Note we have some of the cgroup patches referenced above, in our latest 4.19.27 tree, so it might be worth trying that.

@Pmant

This comment has been minimized.

Copy link

commented Mar 8, 2019

I upgraded to 4.19.25 (not 4.19.16) via rpi-update, thats all. After a reboot it took 5-10 minutes (with running node processes) until the whole ram was used and the Pi got unresponsive.

@JamesH65

This comment has been minimized.

Copy link
Contributor

commented Mar 8, 2019

What do you mean by running node process? I got 4.19.27 running here with no memory problems that I can see, so trying to understand specifically what you are doing that is soaking up the RAM.

@Pmant

This comment has been minimized.

Copy link

commented Mar 8, 2019

Just node.js processes with some memory/disk io, nothing heavy. As you can see ext4_inode_cache is eating up all the memory. There were no processes in top/htop that had more than a couple of Megabytes ram usage. Then I checked with slabtop which finally led me here.

@Pmant

This comment has been minimized.

Copy link

commented Mar 8, 2019

here is what it looks like on 4.14.98 if that helps

 Active / Total Objects (% used)    : 174004 / 179501 (96.9%)
 Active / Total Slabs (% used)      : 4401 / 4401 (100.0%)
 Active / Total Caches (% used)     : 72 / 86 (83.7%)
 Active / Total Size (% used)       : 30628.44K / 31375.33K (97.6%)
 Minimum / Average / Maximum Object : 0.02K / 0.17K / 8.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
 10475  10475 100%    0.69K    457       23      7312K ext4_inode_cache
 24240  24222  99%    0.13K    808       30      3232K dentry
  7392   7238  97%    0.36K    336       22      2688K inode_cache
   440    433  98%    3.75K     55        8      1760K task_struct
  4810   4810 100%    0.30K    185       26      1480K radix_tree_node
 14448  14448 100%    0.09K    344       42      1376K kernfs_node_cache

@kapitainsky

This comment has been minimized.

Copy link

commented Mar 12, 2019

I have similar issue with 4.19.27

The easiest way to reproduce it for me is to run e4defrag on ext4 eternal disk (but it also happens when doing tar or even du on folders with many entries).

I use external disk to make my Pi backups with rclone - so backup folder contains 1m+ files (about 10 full backups)

Now when I start some operations on this folder (e4dfrag, tar or du) I can see ext4_inode_cache grwoing - when I CTRL-C it does not truncate. When I start something else it grows again. If I let it go it grows until my Pi crashes. All of these worked perfectly well with 4.14.98

example of slabtop -s c just before Pi becomes unresponsive

Active / Total Objects (% used)    : 3491438 / 3497484 (99.8%)
Active / Total Slabs (% used)      : 127158 / 127158 (100.0%)
Active / Total Caches (% used)     : 88 / 102 (86.3%)
Active / Total Size (% used)       : 851559.98K / 852126.67K (99.9%)
Minimum / Average / Maximum Object : 0.02K / 0.24K / 8.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
1542420 1542420 100%    0.13K  51414       30    205656K dentry
810321 810321 100%    0.71K  60309       22    764944K ext4_inode_cache
600640 600640 100%    0.06K   9385       64     37540K kmalloc-64
335240 335240 100%    0.02K   1972      170      7888K scsi_data_buffer
 82688  82688 100%    0.03K    646      128      2584K ext4_extent_status
 16968  16968 100%    0.09K    404       42      1616K kernfs_node_cache
 15012  15012 100%    0.11K    417       36      1668K ext4_groupinfo_4k
 11622  11622 100%    0.30K    447       26      3576K radix_tree_node
  9984   9984 100%    0.03K     78      128       312K anon_vma_chain

@kapitainsky

This comment has been minimized.

Copy link

commented Mar 12, 2019

and also in 4.19.27 case "echo 3 | sudo tee /proc/sys/vm/drop_caches "does not truncate ext4_inode_cache which is the case in 4.14.98
It always grows until RPi is dead - the pace depends on number of ext4 disk operations - fair amount is required to eat all memory.

@kapitainsky

This comment has been minimized.

Copy link

commented Mar 21, 2019

4.19.29 no change. I can kill my raspberry pi in reproducible way by simply copying big folder few times (copy -R source A; copy -R source B; copy -R source C) With 10GB of source code files I can do it after three copy commands . After few more tests I can see that none of caches are released not just ext4_inode_cache - but this one is the largest in size so fills all memory first.

When not using heavy file operations system seems to be stable as caches are within manageable size. They grow forever though as well so the end comes just much later.

Having swap file does not help as caches are not swappable

@JamesH65

This comment has been minimized.

Copy link
Contributor

commented Mar 21, 2019

This is looking more and more like an upstream bug that has been reported, and we are currently awaiting a fix - there is some argument, apparently, over the proposed fix.

@P33M

This comment has been minimized.

Copy link
Contributor

commented Mar 21, 2019

The "big" folder full of files doesn't even need files with any sensible contents - 0 byte files created by "touch" are sufficient to trigger the leak.

@kapitainsky

This comment has been minimized.

Copy link

commented Mar 21, 2019

in case of ext4_inode_cache any filesystem access which requires inode information not already in cache adds new info to cache which only grows until there is no memory. From user perspective nothing to do at the moment but wait for upstream fix. 4.14 is fully usable,

P33M pushed a commit to P33M/linux that referenced this issue Mar 26, 2019

P33M
defconfigs: disable memory and IO cgroups
Due to an upstream bug, memory is leaked in the inode cache when cgroups
are enabled. Disable as this is causing crashes.

See: raspberrypi#2829

pelwell added a commit that referenced this issue Mar 26, 2019

defconfigs: disable memory and IO cgroups (#2908)
Due to an upstream bug, memory is leaked in the inode cache when cgroups
are enabled. Disable as this is causing crashes.

See: #2829

popcornmix added a commit to raspberrypi/firmware that referenced this issue Mar 26, 2019

kernel: config: disable memory and IO cgroups
See: raspberrypi/linux#2829

kernel: overlays: Add max17040 support to i2c-sensor
See: raspberrypi/linux#2906

kernel: configs: Re-enable CONFIG_NETFILTER_XT_MATCH_SOCKET
See: raspberrypi/linux#2905

kernel: overlays: Fix multiple-instantiation of sc16is7xx*
See: https://www.raspberrypi.org/forums/viewtopic.php?f=107&t=235650

kernel: overlays: sdio: Added 4-bit support on GPIOs 34-39
See: raspberrypi/linux#2903

kernel: bcm2835-sdhost: Allow for sg entries that cross pages

kernel: bcm2835-mmc: Fix DMA channel leak
See: raspberrypi/linux#2816

popcornmix added a commit to Hexxeh/rpi-firmware that referenced this issue Mar 26, 2019

kernel: config: disable memory and IO cgroups
See: raspberrypi/linux#2829

kernel: overlays: Add max17040 support to i2c-sensor
See: raspberrypi/linux#2906

kernel: configs: Re-enable CONFIG_NETFILTER_XT_MATCH_SOCKET
See: raspberrypi/linux#2905

kernel: overlays: Fix multiple-instantiation of sc16is7xx*
See: https://www.raspberrypi.org/forums/viewtopic.php?f=107&t=235650

kernel: overlays: sdio: Added 4-bit support on GPIOs 34-39
See: raspberrypi/linux#2903

kernel: bcm2835-sdhost: Allow for sg entries that cross pages

kernel: bcm2835-mmc: Fix DMA channel leak
See: raspberrypi/linux#2816
@popcornmix

This comment has been minimized.

Copy link
Collaborator

commented Mar 26, 2019

Anyone affected by this issue, please run rpi-update and report back if issue is still present.

pelwell added a commit that referenced this issue Mar 26, 2019

defconfigs: disable memory and IO cgroups (#2908)
Due to an upstream bug, memory is leaked in the inode cache when cgroups
are enabled. Disable as this is causing crashes.

See: #2829

pelwell added a commit that referenced this issue Mar 26, 2019

defconfigs: disable memory and IO cgroups (#2908)
Due to an upstream bug, memory is leaked in the inode cache when cgroups
are enabled. Disable as this is causing crashes.

See: #2829

popcornmix added a commit that referenced this issue Apr 2, 2019

defconfigs: disable memory and IO cgroups (#2908)
Due to an upstream bug, memory is leaked in the inode cache when cgroups
are enabled. Disable as this is causing crashes.

See: #2829

popcornmix added a commit that referenced this issue Apr 2, 2019

defconfigs: disable memory and IO cgroups (#2908)
Due to an upstream bug, memory is leaked in the inode cache when cgroups
are enabled. Disable as this is causing crashes.

See: #2829

popcornmix added a commit that referenced this issue Apr 2, 2019

defconfigs: disable memory and IO cgroups (#2908)
Due to an upstream bug, memory is leaked in the inode cache when cgroups
are enabled. Disable as this is causing crashes.

See: #2829

artynet added a commit to artynet/rpi-linux that referenced this issue Apr 3, 2019

defconfigs: disable memory and IO cgroups (raspberrypi#2908)
Due to an upstream bug, memory is leaked in the inode cache when cgroups
are enabled. Disable as this is causing crashes.

See: raspberrypi#2829

popcornmix added a commit that referenced this issue Apr 8, 2019

defconfigs: disable memory and IO cgroups (#2908)
Due to an upstream bug, memory is leaked in the inode cache when cgroups
are enabled. Disable as this is causing crashes.

See: #2829

popcornmix added a commit that referenced this issue Apr 8, 2019

defconfigs: disable memory and IO cgroups (#2908)
Due to an upstream bug, memory is leaked in the inode cache when cgroups
are enabled. Disable as this is causing crashes.

See: #2829

Gadgetoid added a commit to Gadgetoid/linux that referenced this issue Apr 10, 2019

defconfigs: disable memory and IO cgroups (raspberrypi#2908)
Due to an upstream bug, memory is leaked in the inode cache when cgroups
are enabled. Disable as this is causing crashes.

See: raspberrypi#2829

popcornmix added a commit that referenced this issue Apr 18, 2019

defconfigs: disable memory and IO cgroups (#2908)
Due to an upstream bug, memory is leaked in the inode cache when cgroups
are enabled. Disable as this is causing crashes.

See: #2829

popcornmix added a commit that referenced this issue Apr 18, 2019

defconfigs: disable memory and IO cgroups (#2908)
Due to an upstream bug, memory is leaked in the inode cache when cgroups
are enabled. Disable as this is causing crashes.

See: #2829
@rkkoszewski

This comment has been minimized.

Copy link

commented Apr 19, 2019

in case of ext4_inode_cache any filesystem access which requires inode information not already in cache adds new info to cache which only grows until there is no memory. From user perspective nothing to do at the moment but wait for upstream fix. 4.14 is fully usable,

Is there any upstream bugreport that can be followed to see the progress on this issue? I cannot find anything, or is the leak only happening on rpi?

@popcornmix

This comment has been minimized.

Copy link
Collaborator

commented Apr 19, 2019

From earlier in this issue:

Here's an LKML thread discussing backporting the patches: https://lkml.org/lkml/2018/11/2/160
And a bug in Ubuntu: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1792349

popcornmix added a commit that referenced this issue Apr 23, 2019

defconfigs: disable memory and IO cgroups (#2908)
Due to an upstream bug, memory is leaked in the inode cache when cgroups
are enabled. Disable as this is causing crashes.

See: #2829
@P33M

This comment has been minimized.

Copy link
Contributor

commented Apr 23, 2019

My script still crashes on downstream versions 4.19.36 through to 5.1.0-rc6, taking the top of each of rpi-4.19.y, rpi-5.0.y, rpi-5.1.y branches. I suppose the next step is to replicate it on an upstream config and start making noise in bug trackers.

@P33M

This comment has been minimized.

Copy link
Contributor

commented Apr 23, 2019

For reference, this script on an ext4 filesystem (with cgroups enabled) will crash a Pi in about 2 minutes:

#!/bin/bash

mkdir trivial
echo trivial/{1..50000} | xargs touch

COUNT=1
while true; do
        cp -R trivial trivial$COUNT
        echo -n "."
        let COUNT=COUNT+1
done

Note that you need a filesystem with enough free inodes to leak enough memory to crash.

@P33M

This comment has been minimized.

Copy link
Contributor

commented Apr 24, 2019

I've been unable to get an upstream kernel to crash in the same fashion. Reverting this commit cd6ce4d causes the issue to disappear. If we have the commit applied but pass cgroup_enable=memory on the commandline, the issue disappears.

@P33M P33M referenced this issue Apr 24, 2019

Merged

cgroup mayhem #2941

popcornmix added a commit that referenced this issue Apr 30, 2019

defconfigs: disable memory and IO cgroups (#2908)
Due to an upstream bug, memory is leaked in the inode cache when cgroups
are enabled. Disable as this is causing crashes.

See: #2829

popcornmix added a commit that referenced this issue Apr 30, 2019

defconfigs: disable memory and IO cgroups (#2908)
Due to an upstream bug, memory is leaked in the inode cache when cgroups
are enabled. Disable as this is causing crashes.

See: #2829

popcornmix added a commit that referenced this issue May 7, 2019

defconfigs: disable memory and IO cgroups (#2908)
Due to an upstream bug, memory is leaked in the inode cache when cgroups
are enabled. Disable as this is causing crashes.

See: #2829

popcornmix added a commit that referenced this issue May 13, 2019

defconfigs: disable memory and IO cgroups (#2908)
Due to an upstream bug, memory is leaked in the inode cache when cgroups
are enabled. Disable as this is causing crashes.

See: #2829

TiejunChina added a commit that referenced this issue Jun 19, 2019

defconfigs: disable memory and IO cgroups (#2908)
Due to an upstream bug, memory is leaked in the inode cache when cgroups
are enabled. Disable as this is causing crashes.

See: #2829
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.