arc_reclaim takes lots of cpu and memory disappears #292
Comments
|
Oh, I also get this in the logs: Jun 24 12:08:26 xback1 kernel: [ 1910.782053] kworker/2:1: page allocation failure. order:0, mode:0x20 |
|
There's also a trace from the 1st crash, which appears to be from within zfs: Jun 24 02:18:22 xback1 kernel: [147666.084757] arc_reclaim: page allocation failure. order:0, mode:0x4020 |
|
Thought it might have been https://github.com/behlendorf/zfs/issues/251. Tried setting zfs_arc_max to 3GB, 3221225472 for the module options, (on a 4GB machine), but all the memory has gone still: [root@xback1 ~]# cat /proc/meminfo |
|
The memory isn't lost, it looks like it's being consumed by the ARC. The ARC will be by far the biggest vmalloc consumer on your system and you can see this is where the bulk of the memory is: VmallocUsed: 3200212 kB What's odd in that for some reason the ARC can't release that memory. The only time that should happen is if it's all dirty, thus dropping it would result in data loss. Something must be preventing it from being safely flushed to disk. In your second comment I do see memory allocation failures from the SCSI driver so perhaps that's preventing the I/O. This may be a bit of a catch 22 due to getting in to a low memory condition. You may be able to prevent this is the short term by further reducing your zfs_arc_max to say 2GB. Longer term this memory handling will be improved. |
|
Thanks - I've reverted to 2.6.35 and it seems to be working much better there, though the memory usage gets close to maximum at times. |
The Linux 3.1 kernel has introduced the concept of per-filesystem shrinkers which are directly assoicated with a super block. Prior to this change there was one shared global shrinker. The zfs code relied on being able to call the global shrinker when the arc_meta_limit was exceeded. This would cause the VFS to drop references on a fraction of the dentries in the dcache. The ARC could then safely reclaim the memory used by these entries and honor the arc_meta_limit. Unfortunately, when per-filesystem shrinkers were added the old interfaces were made unavailable. This change adds support to use the new per-filesystem shrinker interface so we can continue to honor the arc_meta_limit. The major benefit of the new interface is that we can now target only the zfs filesystem for dentry and inode pruning. Thus we can minimize any impact on the caching of other filesystems. In the context of making this change several other important issues related to managing the ARC were addressed, they include: * The dnlc_reduce_cache() function which was called by the ARC to drop dentries for the Posix layer was replaced with a generic zfs_prune_t callback. The ZPL layer now registers a callback to drop these dentries removing a layering violation which dates back to the Solaris code. This callback can also be used by other ARC consumers such as Lustre. arc_add_prune_callback() arc_remove_prune_callback() * The arc_reduce_dnlc_percent module option has been changed to arc_meta_prune for clarity. The dnlc functions are specific to Solaris's VFS and have already been largely eliminated already. The replacement tunable now represents the number of bytes the prune callback will request when invoked. * Less aggressively invoke the prune callback. We used to call this whenever we exceeded the arc_meta_limit however that's not strictly correct since it results in over zeleous reclaim of dentries and inodes. It is now only called once the arc_meta_limit is exceeded and every effort has been made to evict other data from the ARC cache. * More promptly manage exceeding the arc_meta_limit. When reading meta data in to the cache if a buffer was unable to be recycled notify the arc_reclaim thread to invoke the required prune. * Added arcstat_prune kstat which is incremented when the ARC is forced to request that a consumer prune its cache. Remember this will only occur when the ARC has no other choice. If it can evict buffers safely without invoking the prune callback it will. * This change is also expected to resolve the unexpect collapses of the ARC cache. This would occur because when exceeded just the arc_meta_limit reclaim presure would be excerted on the arc_c value via arc_shrink(). This effectively shrunk the entire cache when really we just needed to reclaim meta data. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#466 Closes openzfs#292
Adding test case to fetch specific snapshot information using target Signed-off-by: mayank <mayank.patel@mayadata.io>
This includes some incidental breaking changes to the CLI. Specifically, subcommand names and arg long names are all kebab-case, and a few argument names are simplified.
arc_reclaim was taking most of the CPU. The system had no free memory, but all the user processes were swapped out. The machine stopped responding during shutdown. It seems to have happened again (running rsync). I caught it early and killed the rsync, but lots of memory is "missing" from top (memory order sorting):
Mem: 4024644k total, 3839860k used, 184784k free, 5392k buffers
Swap: 4194300k total, 26844k used, 4167456k free, 17356k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2148 root 20 0 200m 1668 1048 S 0.0 0.0 0:31.63 snmpd
3047 root 20 0 15268 1292 812 R 1.6 0.0 0:00.02 top
2551 root 20 0 106m 1024 672 S 0.0 0.0 0:00.07 bash
arc_reclaim has stopped using lots of CPU after killing rsync.
[root@xback1 logs]# cat /proc/meminfo
MemTotal: 4024644 kB
MemFree: 173504 kB
Buffers: 13532 kB
Cached: 20672 kB
SwapCached: 8668 kB
Active: 7128 kB
Inactive: 38660 kB
Active(anon): 1172 kB
Inactive(anon): 10408 kB
Active(file): 5956 kB
Inactive(file): 28252 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 4194300 kB
SwapFree: 4167768 kB
Dirty: 4 kB
Writeback: 0 kB
AnonPages: 5212 kB
Mapped: 4064 kB
Shmem: 0 kB
Slab: 403116 kB
SReclaimable: 7940 kB
SUnreclaim: 395176 kB
KernelStack: 2848 kB
PageTables: 4424 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 6206620 kB
Committed_AS: 160528 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 3334504 kB
VmallocChunk: 34355911836 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 13888 kB
DirectMap2M: 2050048 kB
DirectMap1G: 2097152 kB
This is with 2.6.38.8 running rc4. Is there any debugging I should try? Should I run the latest git version?
The text was updated successfully, but these errors were encountered: