OpenZFS on FreeBSD needs to support BIO_ORDERED

[peterjeremy]

System information

Type	Version/Name
Distribution Name	FreeBSD
Distribution Version	13.0-stable
Kernel Version	13.0-STABLE 0fc6eebbf763
Architecture	amd64
OpenZFS Version	zfs-2.1.2-FreeBSD_gaf88d47f1, zfs-kmod-2.1.2-FreeBSD_gaf88d47f1

Describe the problem you're observing

OpenZFS doesn't include the BIO_ORDERED flag on bio operations it generates. This allows the FreeBSD I/O subsystem to arbitrarily reorder I/O operations, which breaks ZFS expectations regarding which data is on stable storage.

Describe how to reproduce the problem

1. OpenZFS code search.
2. With a geom_gate(4) patched to detect BIO_ORDERED operations, do a zpool create, zfs recv and zpool export to a geom_gate device and note the absence of any BIO_ORDERED operations.

Include any warning/errors/backtraces from the system logs

N/A

[amotin]

For some time ZFS on FreeBSD indeed marked all BIO_FLUSH operations with BIO_ORDERED flag. But from one side it badly affected NVMe performance, since NVMe with its multiple queues makes its implementation problematic, while from another ZFS always waits for depending writes to complete before sending BIO_FLUSH requests, so explicit request ordering is not required. So I've removed that. Do you know any specific scenario where any expectations are broken?

[peterjeremy]

There was some discussion regarding BIO reordering in freebsd-fs@FreeBSD.org and some messages around https://lists.freebsd.org/archives/freebsd-fs/2022-February/000696.html noted that OpenZFS didn't appear to use BIO_ORDERED. If OpenZFS internally serialises IO requests then this bug is probably obsolete.

Seems not applicable.

For context:

https://lists.freebsd.org/archives/freebsd-fs/2022-February/000730.html

Unrelated to the above but for completeness: OpenZFS avoids the need
for BIO_ORDERED by not issuing additional I/Os until previous I/Os have
been retired when ordering is important. (It does rely on BIO_FLUSH).

https://lists.freebsd.org/archives/freebsd-fs/2022-February/000731.html

To be clear: OpenZFS won't schedule new I/Os until the BIO_FLUSH it sends
down w/o the BIO_ORDERED flag completes, right? The parenthetical confuses
me on how to parse it: BIO_FLUSH is needed and ZFS depends on it completing
with all blocks flushed to stable media, or ZFS depends on BIO_FLUSH being
strongly ordered relative to other commands. I think you mean the former,
but want to make sure.

ZIL writes do ensure ordering by flush completion as needed, not by relying on the order issued. See various comments about FLUSH in module/zfs/zil.c for details.