ZFS still writes to HDDs vdev while dataset is configure to store only in special vdev for Docker

[lealoureiro]

I'm facing a strange behaviour when using Docker together with ZFS.

I have one pool with 2x10TB HDDs as mirror vdev and 2x500GB SSDs as mirror for special vdev (to store metadata and datasets with small config files etc).

I decided to put my Docker Root (containers/images) in ZFS dataset using ZFS driver:

Docker deamon.json:

{  
  "data-root": "/docker-root",  
  "storage-driver": "zfs"  
}

I've also set the recordsize of /docker-root dataset to 32K and special_small_blocks to 128K, to make sure all files store in dataset get allocated and store in special vdev with fast SSDs. (I understood if you set the recordsize equal or smaller than special_small_blocks, all files go to special vdev)

But the problem is that I still see quite a lot writes to HDDs vdevs when I'm starting, stopping and update container's image. As well frequent rights while containers are running, which is not the case when I stop all containers. I also made sure that all containers mounted volumes are mounted in config dataset that is configured the same to store all files special vdev.

I also notice when HDDs vdev are scrubing, which can take hours, managing the containers becomes super slow.

The atime property is also off for datasets in the pool.

Why this is still happening even though I'm forcing all docker related files to be store in special vdev? Is this still expected or I'm missing something?

Layout of the pool:

config:

	NAME                                               STATE     READ WRITE CKSUM
	storage-pool-01                                    ONLINE       0     0     0
	  mirror-0                                         ONLINE       0     0     0
	    ata-ST10000NE0008-2JM101_XXXXXXXX              ONLINE       0     0     0
	    ata-ST20000NM007D-3DJ103_XXXXXXXX              ONLINE       0     0     0
	special	
	  mirror-1                                         ONLINE       0     0     0
	    ata-Samsung_SSD_870_EVO_500GB_XXXXXXXXXXXXXXX  ONLINE       0     0     0
	    ata-Samsung_SSD_870_EVO_500GB_XXXXXXXXXXXXXXX  ONLINE       0     0     0

Dataset Config:

zfs get recordsize,special_small_blocks storage-pool-01/docker-root
NAME                         PROPERTY              VALUE                 SOURCE
storage-pool-01/docker-root  recordsize            32K                   local
storage-pool-01/docker-root  special_small_blocks  128K                  local

System Information:

Type	Version/Name
Distribution Name	Ubuntu
Distribution Version	24.04.2 LTS
Kernel Version	6.8.0-58-generic
Architecture	x86_64
OpenZFS Version (kernel)	zfs-kmod-2.2.2-0ubuntu9.1
OpenZFS Version (userspace)	zfs-2.2.2-0ubuntu9.2

[amotin]

Interesting question. I can only guess few things. First, recordsize reduction might be ineffective once the data is already written, since it affects only files written beyond that size after that. Second, if the workload includes any sync writes, then in absence of SLOG they will go to Embedded SLOG on normal vdev. This are might benefit from some attention.

[amotin]

Just to be sure, I mean recordsize for the dataset should be set before the data is written, not special vdev added.

I'd think sync=disabled should be enough, but you might experiment.

[lealoureiro]

Yeap, I've set the recordsize and special_small_blocks before start installing docker etc.

I will experiment with SLOG for writes.

[IvanVolosyuk]

@amotin I'm pretty sure there are some data which will still be written to HDDs, like transaction groups? I don't think it is possible to completely idle HDDs when using only special small blocks.

[amotin]

All metadata go to special vdev. If one was added during the pool creation and if special_small_blocks is set early enough it is possible to make ZFS not write to normal vdevs at all:

NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
optane      256G   783M   255G        -         -     0%     0%  1.00x    ONLINE  -
  nvd2p1    256G      0   254G        -         -     0%  0.00%      -    ONLINE
special        -      -      -        -         -      -      -      -         -
  md0         2G   783M  1.11G        -         -     7%  40.8%      -    ONLINE

Though ZFS will still update uberblocks there, so they won't be able to go to sleep, unfortunately.

[lealoureiro]

I tried to add log vdev with both sync=standard and sync=disabled and still the same behaviour.

I've also tried to just generate a random file in docker-root dataset with the following command:
dd if=/dev/zero of=./test bs=512K count=8912 and I still small activity in hdd vdevs but not as impactful as pulling new docker images etc.

Any debug tooling available that allow me to get more insights what zfs is trying to store in hdd vdevs?

I guess at end of day I will just create another pool with just SSDs for docker containers.

[pojlFDlxCOvZ4Kg8y1l4]

My system is also showing this behavior, with a very similar setup.

But maybe more activity attracts someone that fixes this unexpected behaviour. Would be cool!

I migrated from a seperate mirrored hard drive pool (2x12TB) and mirrored NVME pool (2x1.6TB) to a single pool, hard drives with the SSDs as a special vdev.
Expecting to end up in a similar situation as before, silent docker operation with bulk storage that doesn't get touched a lot.
But with one pool instead of two pools. And of course the blistering speed from a special metadata device on a HDD pool.
This expectation did not end up to be fulfilled fully.
The hard drive operations are now way faster, but the system sounds like there's a pair of tiny machineguns firing constantly.
Thinking it was a fluke, since finding anything related to this on the Google is quite hard.
I have reinstalled the OS, I have started from scratch with the pool, verified I set the special_small_blocks and recordsize to the proper values before writing data. Sadly, the constant IO on the hard drives does not go away.
With and without snapshots, even many (3000+) did not give noticeable change to the noise and IO.
Fiddling with sync=disabled, sync=disabled and back to sync=standard changed it a bit, but not that much.

The system:
My system is running IXSystem's TrueNAS 25.04, running 30-ish docker containers.
IXSystem's TrueNAS does not use the ZFS storage driver for Docker, but rather overlay2.
Data root for docker is mounted at /mnt/.ix-apps/docker, dataset zp0/ix-apps/docker
Both hard drives and NVME's are used datacenter gear.

ZFS version:

zfs-2.3.0-1
zfs-kmod-2.3.0-1

Pool layout:

  pool: zp0
 state: ONLINE
  scan: scrub repaired 0B in 11:28:53 with 0 errors on Sat May  3 23:58:07 2025
config:

        NAME                                      STATE     READ WRITE CKSUM
        zp0                                       ONLINE       0     0     0
          mirror-0                                ONLINE       0     0     0
            88c88557-dd36-4d68-8d0b-4a52bb57b90d  ONLINE       0     0     0
            098ac9de-28cb-4a61-81b1-57dfb51a121d  ONLINE       0     0     0
        special
          mirror-1                                ONLINE       0     0     0
            5aa3490e-72f7-4c1f-9382-8977a26d708b  ONLINE       0     0     0
            b6f408d4-1c02-4941-9d78-2f56ec2bbe18  ONLINE       0     0     0

errors: No known data errors

Atime, xattr and record sizes for ix-apps (automatically created by TrueNAS) and data/docker (manually created), here lives most of the data used by Docker containers:

/sbin/zfs get recordsize,special_small_blocks,atime,xattr zp0/ix-apps/docker zp0/data/docker
NAME                PROPERTY              VALUE                 SOURCE
zp0/data/docker     recordsize            128K                  local
zp0/data/docker     special_small_blocks  512K                  inherited from zp0
zp0/data/docker     atime                 off                   inherited from zp0
zp0/data/docker     xattr                 on                    local
zp0/ix-apps/docker  recordsize            128K                  inherited from zp0
zp0/ix-apps/docker  special_small_blocks  512K                  inherited from zp0
zp0/ix-apps/docker  atime                 off                   local
zp0/ix-apps/docker  xattr                 on                    local

Zpool list output

/sbin/zpool list -v zp0
NAME                                       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zp0                                       12.4T  6.89T  5.47T        -         -     0%    55%  1.00x    ONLINE  /mnt
  mirror-0                                10.9T  6.38T  4.52T        -         -     0%  58.5%      -    ONLINE
    88c88557-dd36-4d68-8d0b-4a52bb57b90d  10.9T      -      -        -         -      -      -      -    ONLINE
    098ac9de-28cb-4a61-81b1-57dfb51a121d  10.9T      -      -        -         -      -      -      -    ONLINE
special                                       -      -      -        -         -      -      -      -         -
  mirror-1                                1.45T   515G   973G        -         -    21%  34.6%      -    ONLINE
    5aa3490e-72f7-4c1f-9382-8977a26d708b  1.45T      -      -        -         -      -      -      -    ONLINE
    b6f408d4-1c02-4941-9d78-2f56ec2bbe18  1.45T      -      -        -         -      -      -      -    ONLINE
    

The space used on the special vdev is within expectations of space usage, as well as for the standard hard drive mirror.
I am also thinking if I would revert back to the previous situation, hard drives detached from the SSD pool instead of a special mirror. Mainly due noise concerns, I didn't notice a performance drop.
Is this information useful? I hope it is, even it it were just a little.