Enhance documentation support for interleavings of options.

[TimHe95]

Description of the new feature

I am doing some raw disk tests using fio. I found some behaviour is not clear when specifying --filename=/dev/device_name and --direct=1.

• (1) default ioengine is sync, which use read(2)/write(2), but for raw disks, there is no filesystem. So how fio read/write raw disks? The same problem for libaio, posix_aio engine.

• (2) what will fio do for the raw device when specifying sync=sync. It is claimed that fio use O_SYNC when testing files. But what for raw disks. The same problem for fsync=1 option.
  Further, if using libaio or posix_aio engine, what will fio do if specifying sync=sync and fsync=1 options. (In case one may want to make sure the write is right on the storage medium (e.g., TLC NAND for SSD) rather than disk cache.)

• (3) the result report has an "in_queue" time which is documented as "Total time spent in the disk queue". What is the unit of this data (ns/us/ms).

Thank you very much for the patience!

[sitsofe]

Hi @TimHe95,

(Hmm I'm not sure about this enhancement request but I'll take a look at this and use what happens to guide how I respond to other similar situations in the future)

I'm guessing you're referring to Linux since you mentioned libaio so the I'll use that as the context of my answers but if not again we can follow up on the mailing list.

(Pedantry: raw device implies no buffering. Linux doesn't have "raw devices" any more so I will interpret all of your uses of the term to be block device so we can progress)

(1) default ioengine is sync, which use read(2)/write(2), but for raw [sic] disks, there is no filesystem. So how fio read/write raw [sic] disks? The same problem for libaio, posix_aio engine.

Well this doesn't sound like an fio question because as you said sync ioengine just makes read()/write() calls... But I ask you, why can't the read(2)/write(2) operating system syscalls work on block devices? If so, for those engines issuing I/O like that (e.g. sync/posix_aio) there's no extra work. For libaio I'm less clear what happens but I think the so long as the filesystem or block device implements vectored I/O it looks like it will work. I'm basing this on what I see in https://elixir.bootlin.com/linux/v5.10.13/source/fs/aio.c#L1797 and then tracing through its calls (like https://elixir.bootlin.com/linux/v5.10.13/source/fs/aio.c#L1544 , https://elixir.bootlin.com/linux/v5.10.13/source/lib/iov_iter.c#L1814 , https://elixir.bootlin.com/linux/v5.10.13/source/lib/iov_iter.c#L1736 ). This looks a lot like a Linux kernel question though...

(2) what will fio do for the raw device when specifying sync=sync. It is claimed that fio use O_SYNC when testing files. But what for raw disks. The same problem for fsync=1 option.

Well, we can see that setting sync=sync on https://github.com/axboe/fio/blob/fio-3.25/options.c#L3733-L3755 causes fio to set store a value of O_SYNC in sync_io. Searching for sync_io we can see it gets used in a number of places but in this case filesetup.c is the most interesting one where we can see it is stored in flags and flags gets passed to file_lookup_open() which eventually calls open(3) which I would guess eventually calls open(2). So another OS question! Why do you think block devices would refuse the O_SYNC flag? If they chose to implement it why would it not be possible?

The same problem for fsync=1 option.

Well again, what makes you think block devices can't support the fsync(2) syscall? Doing a quick search of lxr for fsync gets us to blkdev_fsync(). Yes the VFS layer implements vfs_fsync_range but if you think about it, wouldn't that layer have to pass it down to a block device if the disk is local so wouldn't that layer have to know how to deal with it too?

if using libaio or posix_aio engine, what will fio do if specifying sync=sync and fsync=1 options

fio will pass it on to the C library/operating system. I've covered O_SYNC above. I can't remember if Linux's AIO interface actually implements an asynchronous fsync() but it's documented as having it so the libaio ioengine tries to queue up such a command (see https://github.com/axboe/fio/blob/fio-3.25/engines/libaio.c#L130 )... In fact, now I look further you can see fio itself documents that libaio doesn't support it!. So if it's not implemented the libaio ioengine will, (after draining the queue) call do_io_u_sync() which is actually the same call the sync ioengine will take for issuing the fsync().

(3) the result report has an "in_queue" time which is documented as "Total time spent in the disk queue". What is the unit of this data (ns/us/ms).

Whatever the Linux kernel reports the disk stats in. Would you like to submit a patch to document this in the manual and HOWTO?

OK having gotten this far it's clear this isn't really the form an enhancement request should take so I'll change my approach going forward. However, it is very reasonable to ask "how does OS A implement feature X?" to the OS developers and from fio's perspective we can sometimes can point people in the right direction when those questions are sent to the fio mailing list so if you have more follow up clarifications or queries we can continue over there (obviously no guarantees but it's a better place). Thanks!

[sitsofe]

(Setting to needreporterinfo pending a reply from the reporter)

[sitsofe]

Closing due to lack of reply from reporter. If this issue is still happening with the latest fio (see https://github.com/axboe/fio/releases to find out which version that is) please reopen. Thanks!