When native encryption is enabled, sync writes are slow.

[Maciej-Poleski]

System information

Type	Version/Name
Distribution Name	Gentoo Linux
Distribution Version	amd64 (stable) 17.1/no-multilib
Linux Kernel	4.19.72-gentoo
Architecture	x86_64
ZFS Version	8.2
SPL Version	N/A

Describe the problem you're observing

When native encryption is enabled, sync writes are slow.

Describe how to reproduce the problem

Prepare test environment in the following way (assuming that tmpfs is mounted to /tmp):

dd if=/dev/zero of=/tmp/disk bs=1G count=4
dd if=/dev/zero of=/tmp/log bs=1G count=4

For no-encyption tests create the pool in the following way:

zpool create -m /tmp/storage -O compression=off -O logbias=latency test /tmp/disk log /tmp/log

For encryption tests create the pool in the following way:

zpool create -m /tmp/storage -O compression=off -O encryption=aes-256-gcm -O keyformat=raw -O keylocation=file:///tmp/test.key -O logbias=latency test /tmp/disk log /tmp/log

Perform async tests in the following way:

dd if=/dev/zero of=/tmp/storage/zero bs=10M count=350

Perform sync tests in the following way:

dd if=/dev/zero of=/tmp/storage/zero bs=10M count=350 oflag=sync

Results of the tests on my machine are summarized in the table below:

x	async	sync
no encryption	1.7 GB/s	1.2 GB/s
encryption	819 MB/s	117 MB/s

My machine is a Xeon 4208 (8x 2.1GHz, AES-NI, AVX2, AVX512f, AVX512dq, AVX512cd, AVX512bw, AVX512vl).

It happens that async encryption performance above is roughly the same as async encryption/no-encryption performance of zpool build on actual HDDs (see #9375) thus I don't observe encryption slowdown on async write operations. The encryption/sync slowdown reported above is somewhat concerning.

I understand that native encryption is a new feature and not all performance optimizations are in place yet (are we using aes-ni?). I will let you triage this issue as a bug/feature request depending on your preferences.

Include any warning/errors/backtraces from the system logs

None

[v0112358]

I have the same issue. I have tested write and read speed on 8 NVMe disk:

Write: ~ 1.1GB/s, ~100k IOPS
Read: ~ 30MB/s, ~ 2k IOPS.

Zpool config:
compression=off
recordsize=16k
atime=off
primarycache=metadata

ZoL 0.8.2

[PaulZ-98]

While it's not conclusive, I determined that logbias=throughput would result in fewer encryption calls needing to occur:

In my test
dd if=/dev/zero of=/tmp/storage/zero bs=10M count=35 oflag=sync

There were 6306 calls to zio_encrypt with logbias=latency.
There were 4145 calls to zio_encrypt with logbias=throughput.

This is because with logbias=throughput, zfs will attempt to encrypt and write the blocks to the main pool instead of once to the slog, and once to the main pool.

This may not account for all the performance penalty you are seeing but it is likely part of it.

[ovizii]

I had a pool with GELI encryption(under freenas) which I have removed the GELI encryption from and moved the pool into a Debian Buster envrironment (same underpowered server though)

I now wanted to move to native encryption. As it is not possible to "zfs send" from an unencrypted pool and "zfs receive" into an encrypted pool I had to basically do a "mv pool/unenc/* pool/enc/" and the progress was extremely slow. It felt like 10x slower than when copying files on a pool which was GELI encrypted.

[stale]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.