either always use fsync on btrfs or use chattr +C on heap and log files

[spacejam]

as mentioned here, BTRFS uses functionality that is incompatible w/ sync_file_range when running in its default mode, but CoW can be disabled using chattr +C <file> on these systems, OR we can dynamically detect BTRFS and always use a full fsync instead of sync_file_range.

[cberner]

I did a bit more research into sync_file_range because I wanted to know why lmdb still uses fsyncdata. As far as I can tell, sync_file_range is only safe to use on drives that have power loss protection (i.e. the write-cache is durable). From the sync_file_range manpage:

This system call does not flush disk
write caches and thus does not provide any data integrity on
systems with volatile disk write caches

This probably explains why the performance gain is so enormous: the data isn't actually being written out to disk, just flushed to the disk's write-cache.

Unfortunately, it looks like power loss protection is generally only available on enterprise grade drives: https://image-us.samsung.com/SamsungUS/b2b/resource/2016/07/08/b2b_resource_WHP-SSD-POWERLOSSPROTECTION-r1-JUL16J.pdf

So I think you also need to check for power loss protection support, when using sync_file_range.

[spacejam]

@cberner that's a good point, thank you for digging in!

As time goes on, the incentive for not using O_DIRECT becomes smaller and smaller, and I'm starting to think that maybe it's time to use that instead of fsync/sync_file_range.

[tv42]

I definitely would not write new code using O_DIRECT, it's.. let's say "not welcome". https://yarchive.net/comp/linux/o_direct.html

These days, the good answer would probably be io_uring.
Last I looked, RocksDB was getting nice performance improvements from it.

[tv42]

And apparently zfs also does not guarantee much for sync_file_range: "For example, it appears that with zfs, the sync_file_range call does not actually flush data" https://www.evanjones.ca/durability-filesystem.html

RocksDB disables sync_file_range on ZFS: https://github.com/facebook/rocksdb/blob/74a652a45ff63350640245ba7994bebc748bb0d4/env/io_posix.cc#L160-L166

[spacejam]

O_DIRECT has well-known pros and cons, and especially in combination with io_uring seems to be the clear winner for point-read situations like what tends to dominate b-tree like storage engines.

[ecton]

From my investigation, sync_file_range is not enough for durable writes on any filesystem. The man page has this caveat:

This system call does not flush disk write caches and thus does not provide any data integrity on systems with volatile disk write caches.

Without flushing the volatile cache, there is a window of time between when sync_file_range has returned success and when the write is actually persisted to disk.