Almost empty L2ARC on raidz2 (bleeding out after reboot)

[clefru]

System information

Type	Version/Name
Distribution Name	NixOS
Distribution Version	20.09.git.896270d629e
Linux Kernel	5.9.6
Architecture	x86_64
ZFS Version	2.0.0-rc5
SPL Version	2.0.0-rc5

Describe the problem you're observing

My L2ARC fill ratio stays at 2% and therefore makes the cache for less useful than it could be.

The graphs below show that after reboot (around 14:00), my L2ARC cache warms up and fills up completely. Then over the next 24h, it bleeds out to only around 3-4GB and remains there. I presume that the cache is expiring items, as the box is pretty idle. However this behavior is wrong, as the L2ARC space isn't competing with anything (in contrast to ARC space which is competing with system memory). In short, I never want L2ARC items to expire without pressure. Am I missing a tunable for that?

Note that my zpool is a raidz2 and currently a leg is missing:

NAME                                                 SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
z                                                   18.2T  3.90T  14.3T        -         -     1%    21%  1.00x  DEGRADED  -
  raidz2                                            18.2T  3.90T  14.3T        -         -     1%  21.4%      -  DEGRADED
    ata-WDC_WD40EFRX-68N32N0_WD-WCC7K7YXKH3T-part2      -      -      -        -         -      -      -      -  ONLINE  
    ata-WDC_WD40EFRX-68N32N0_WD-WCC7K4UH8AJL-part2      -      -      -        -         -      -      -      -  ONLINE  
    ata-WDC_WD40EFRX-68N32N0_WD-WCC7K4UH89J1-part2      -      -      -        -         -      -      -      -  ONLINE  
    ata-HGST_HDN724040ALE640_PK2331PAGXDJZT-part2       -      -      -        -         -      -      -      -  ONLINE  
    /tmp/fake2.raw                                      -      -      -        -         -      -      -      -  OFFLINE 
cache                                                   -      -      -        -         -      -      -      -  -
  nvme-SAMSUNG_MZVPV128HDGM-00000_S1XVNYAG910189     119G  1.99G   117G        -         -     0%  1.67%      -  ONLINE  
  nvme-SAMSUNG_MZVPV128HDGM-00000_S1XVNYAG910178     119G  2.45G   117G        -         -     0%  2.05%      -  ONLINE  

Here are some graphs from my Prometheus node exporter:

Describe how to reproduce the problem

Create an raidz2 array, attach 2 cache devices, reboot, and monitor your L2ARC size over the next hours without pressure.

Include any warning/errors/backtraces from the system logs

ARC summary: arc_summary.txt

I would appreciate any hints.

[IvanVolosyuk]

I wonder what the 'zpool iostat -v' looks like. I think there are 2 ways to expire the cache unless there is a bug: stream of new l2arc writes and deletions of files/snapshots/datasets. iostat for cache device may give a clue what is going on. I wonder if there a small set of files which is constantly overwritten with new data which makes l2arc rotate and get filled with deleted blocks.

[clefru]

TL:DR; A misconfigured qemu VM created the guest's memory as file on a ZFS partition and generated tons of useless IO. The L2ARC filled up with short-lived blocks, and in the end all L2ARC blocks were dead. Can we new L2ARC writes target dead block space over pressuring blocks that are alive?

@IvanVolosyuk: Thanks for your poke towards iostat. My assertion that this is an system without pressure turned out to be wrong. I finally found that I had a VM instance running that was generating ZFS IO even when the VM was idle. It turned out that the backing file for shared memory access between the host and the guest landed not in /dev/shm but on my ZFS root fs. Shared memory is required for the virtiofsd access that I use between guest/host. Of course, the memory of a VM should not actually be a ZFS backed file. /dev/shm (=tmpfs) is the right location for that. That my config ever worked is surprising.

So Ivan, your speculation that a small set of files (=1 in this case) get constantly overwritten is exactly what happened here, probably filling up the L2ARC with blocks that soon after get deleted.

What's surprising though is, that L2ARC writes wouldn't overwrite dead L2ARC blocks. Instead they seem to evict blocks that are alive and eventually the L2ARC ends up with mostly dead blocks. Can we make dead block space to be the target of L2ARC writes?

[behlendorf]

@clefru thanks for following up. Glad you got to the root cause. The L2ARC is implemented as an on-disk ring buffer which is why eventually all of the block were overwritten. This is at the heart of the design and wouldn't be easy to change.