ZFS Native Encryption is slow on Intel Atom

[oberien]

System information

Type	Version/Name
Distribution Name	debian
Distribution Version	buster / bullseye (testing)
Linux Kernel	4.19.0-13-amd64 / 5.10.0-1-amd64
Architecture	x86_64
ZFS Version	0.8.6-1~bpo10+1 / 2.0.1-2
SPL Version	??? / 2.0.1-2

Describe the problem you're observing

ZFS native encryption is very slow compared to non-encrypted datasets and ZFS on LUKS. It shouldn't be a CPU throughput problem, as both cryptsetup and Openssl reach very high speeds during benchmarks. I've performed most tests with debian stable (buster) and ZFS 0.8.6-1. In the end I updated to debian testing (bullseye) to confirm with ZFS 2.0.1-2 (last test), which has the same problem.

Describe how to reproduce the problem

Setup:

• 4×8TB hdds, raidz2, 120GB SSD cache
  • on zfs-0.8.6 L2ARC is dropped with export/import
  • on zfs-2.0.1 L2ARC isn't used for sequential reads (which I tested)
• no compression, no dedup
• between read and write test zpool export and import (drop caches)
• zfs uses aesni (perf top shows kernel aes_aesni_encrypt at 60% when running dd on encrypted datasets)
• CPU: Intel(R) Atom(TM) CPU C3538 @ 2.10GHz
• performance tested with
  • write: dd if=/dev/zero of=/foo/foo bs=1M count=20k
  • drop caches between write / read test: zpool export tank && zpool import tank (-l)
  • read: dd if=/foo/foo of=/dev/null bs=1M

no encryption

# zfs create -o mountpoint=/foo tank/foo
write: 21474836480 bytes (21 GB, 20 GiB) copied, 84.4587 s, 254 MB/s
read:  21474836480 bytes (21 GB, 20 GiB) copied, 73.5051 s, 292 MB/s

zpool in luks (aes-xts-plain64 512b key)

# dd if=/dev/urandom of=key bs=4096 count=1
# cryptsetup luksFormat -d key /dev/sd...
# zpool create -O mountpoint=none tank raidz2 /dev/mapper/hdd* cache /dev/mapper/cache
# zfs create -o mountpoint=/foo tank/foo
write:  21474836480 bytes (21 GB, 20 GiB) copied, 86.9323 s, 247 MB/s
read: 21474836480 bytes (21 GB, 20 GiB) copied, 64.4583 s, 333 MB/s

zfs native encryption

dataset creation commands:

# zpool create -O mountpoint=none tank raidz2 <hdds...> cache <ssd>
# zfs create -o encryption=aes-... -o mountpoint=/foo -o keyformat=passphrase -o pbkdf2iters=10000000 tank/foo

aes-256-gcm:

write: 21474836480 bytes (21 GB, 20 GiB) copied, 222.886 s, 96.3 MB/s
read:  21474836480 bytes (21 GB, 20 GiB) copied, 230.363 s, 93.2 MB/s

aes-128-gcm:

write: 21474836480 bytes (21 GB, 20 GiB) copied, 218.149 s, 98.4 MB/s
read:  21474836480 bytes (21 GB, 20 GiB) copied, 226.480 s, 94.8 MB/s

aes-128-ccm:

write: 21474836480 bytes (21 GB, 20 GiB) copied, 226.67 s, 94.7 MB/s
read:  21474836480 bytes (21 GB, 20 GiB) copied, 309.31 s, 69.4 MB/s

aes-256-ccm:

write: 21474836480 bytes (21 GB, 20 GiB) copied, 241.022 s, 89.1 MB/s
read:  21474836480 bytes (21 GB, 20 GiB) copied, 268.546 s, 80.0 MB/s

aes-256-gcm on ZFS 2.0.1-2 (zfs-2.0.1-2, zfs-kmod-2.0.1-2) after updating from debian buster to bullseye:

write: 21474836480 bytes (21 GB, 20 GiB) copied, 226.848 s, 94.7 MB/s
read:  21474836480 bytes (21 GB, 20 GiB) copied, 615.195 s, 34.9 MB/s

cryptsetup benchmark

# cryptsetup benchmark
      Algorithm |       Key |      Encryption |      Decryption
        aes-cbc        128b       437.3 MiB/s       941.4 MiB/s
    serpent-cbc        128b        42.1 MiB/s        96.4 MiB/s
    twofish-cbc        128b        87.0 MiB/s       100.3 MiB/s
        aes-cbc        256b       329.3 MiB/s       768.9 MiB/s
    serpent-cbc        256b        42.1 MiB/s        96.0 MiB/s
    twofish-cbc        256b        86.9 MiB/s       100.0 MiB/s
        aes-xts        256b       673.9 MiB/s       683.5 MiB/s
    serpent-xts        256b        92.0 MiB/s        91.6 MiB/s
    twofish-xts        256b        95.1 MiB/s        95.3 MiB/s
        aes-xts        512b       580.1 MiB/s       576.6 MiB/s
    serpent-xts        512b        92.3 MiB/s        91.7 MiB/s
    twofish-xts        512b        95.2 MiB/s        95.5 MiB/s

openssl speed (aesni)

# openssl speed -elapsed -evp aes-128-gcm
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-128-gcm     166390.27k   426037.03k   705026.65k   841382.23k   872259.58k   873578.50k

# openssl speed -elapsed -multi 4 -evp aes-128-gcm
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
evp            663211.95k  1701381.89k  2814948.35k  3357190.14k  3522813.95k  3538889.39k

openssl speed (software)

# OPENSSL_ia32cap="~0x200000200000000" openssl speed -elapsed -evp aes-128-gcm
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-128-gcm      49673.40k    65665.94k   71541.16k    73072.98k    73708.89k    73777.15k

# OPENSSL_ia32cap="~0x200000200000000" openssl speed -elapsed --multi 4 -evp aes-128-gcm
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
evp            199244.69k   262287.83k   285695.32k   291686.74k   294166.53k   294655.32k

[AttilaFueloep]

The current SIMD accelerated AES-GCM encryption requires a CPU with AVX support. Unfortunately your Atom CPU, while having the AES-NI and PCLMULQDQ extensions, does not have the required AVX extension but just SSE. See #10846. I'm working on SSE4.2 support albeit with slow progress due to limited time. With SSE4.2 support in place you can expect encryption performance comparable to LUKS and OpenSSL.

[oberien]

Thanks for the quick response. I'll close this issue then as a duplicate of #10846.