This script tests options related to copy operations.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>

This script collects copy performance data over a set of block sizes.
Also included is a python script to create a CSV from JSON copy output.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>

Add an example that writes the first half of a file with a known
pattern, copies the first half to the second half of the file, and then
reads the entire file verifying the expected pattern.

Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>

Support copy operations via copy_file_range() in the psync ioengine.
Also add an option to emulate copy commands with read and write
operations.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>

Add the dest_offset_delta option that sets the copy destination to be a
fixed offset from the beginning of the source range for each copy
command. This is intended for use with sequential copy operations.

For example, dest_offset_delta=128M will copy blocks 128M forward in the
LBA range.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>

Make sure that we only issue copy operations when rw=[rand]copy was
specified.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>

The source ranges for copy operation uses the existing offset generation
algorithm.  The data buffer will contain the destination offset.

For generating the destination offset added a new function. Each
successful copy operation will copy one block of data.

Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>

The copy operation requires a new fio option, dest_offset. This
specifies the starting destination offset for the copy operation.
The existing fio offset option will be the starting source offset.

Also update the 'rw' option to accept 'copy' and 'randcopy'.

Data to be copied from source range can be random or sequential
according to rw=copy or randcopy option. The destination location
for the copy operation will be sequential.

This patch only adds parsing for these options but does not implement
the actual copy workload.

Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>

Update the code that prints live eta and final summary statistics to
accommodate copy operations. In the eta per-job status map, use 'q' and
'Q' for running copy jobs since 'C' is already taken for TD_CREATED.

To maintain backward compatibility no copy data is included in the terse
output.

Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>

Allow the bssplit option to set transfer sizes for copy operations.

When we have automated bssplit testing we should clean this up to reduce
duplicate code.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>

Update parsing for blocksize, alignment, percent random, and rate
thresholds to support copy operations. These options allow users to
specify different values for different data directions and thus need
updating to support copy operations.

Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>

Add a new copy data direction and uppdate associated symbols.

Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>

The same tls issue that occurred with AppVeyor msys2 builds has appeared
in GitHub Actions msys2 builds. Disable thread local storage for GitHub
Actions msys2 builds as well.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>

Since we are no long using AppVeyor for Windows tests, we can use the
installers built by GitHub Actions for Windows releases.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>

Stop running Windows tests using AppVeyor since it only runs tests
serially. GitHub Actions Windows tests have been running for a few weeks
and seem to be working ok.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>

Signed-off-by: Jens Axboe <axboe@kernel.dk>

* 'master' of https://github.com/bvanassche/fio:
  zbd: Report the zone capacity
  zbd: Make an error message more detailed

The zone capacity is important information. Hence report the zone
capacity if it is identical for all zones and if ZBD debugging is enabled.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>

This could probably be cleaner in the error handling, but jump to
the normal error handling case for ddir_sync() as well.

Fixes: axboe#1577
Signed-off-by: Jens Axboe <axboe@kernel.dk>

Knowing which part of a zone report is invalid is useful when debugging
firmware of zoned devices. Hence report the number of zones that have
been parsed successfully when reporting a report zones error.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>

* 'master' of https://github.com/huajingyun01/fio:
  Add LoongArch64 support

Signed-off-by: Jingyun Hua <huajingyun@loongson.cn>

Fix the dependency install problem for 32-bit builds by explicitly
installing libc6:i386 and libgcc-s1:i386 as mentioned in the link below.

actions/runner-images#4589 (comment)

Deleting microsoft-prod.list and adding --allow-downgrades were not
needed for our use case.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>

This reverts commit 83b2d4b.

There now seems to be a solution to the 32-bit GitHub Actions Ubuntu
breakage.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>

Describe in the HOWTO and man page the windowsaio ioengine
no_completion_thread option.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>

Move the description for experimental_verify next to the other
verify-related options.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>

Move the description for rate_cycle from the I/O latency section to the
I/O rate section.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>

*** CID 455020:  Integer handling issues  (BAD_SHIFT)
/engines/io_uring.c: 1201 in fio_ioring_cmd_open_file()

In expression "1 << data->lba_shift", left shifting by more than 31 bits
has undefined behavior.  The shift amount, "data->lba_shift", is 4294967295

Fixes: 345fa8f ("engines/io_uring_cmd: add extended LBA support")

Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Link: https://lore.kernel.org/r/20230516121717.28508-2-ankit.kumar@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>

The GitHub Actions image has had a longstanding problem with the
dependencies required for 32-bit builds. The current work around no
longer fixes the issue. Stop testing 32-bit builds to avoid false
positive failures with our CI testing.

We can re-enable 32-bit builds when this issue is fixed upstream.

Details: actions/runner-images#4589

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>

The io_uring_cmd ioengine assumes that logical block size is always
power of 2. But with namespace formats where metadata is transferred at
the end of logical block i.e. an extended logical block this is not
true.

This patch calculates the correct extended logical block size and uses
division operation for start and number of logical block calculation.
The existing calculation for power of 2 logical block size will remain
same.

This also add checks to verify that block sizes are multiple of LBA
size.

This current implementation however doesn't support protection info and
metadata transferred as separate buffer. Return error for those.

Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
[Vincent: edited commit message]

The NVM command set specification 1.0c supports 64 LBA formats.
The 0-based nlbaf field specifies the number of LBA formats.
The flbas field is used to calculate the current LBA format, in which
bit 0-3 indicates lsb and bit 5-6 indicates msb of the format index.

Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>

Update HOWTO.rst and fio.1 to provide more details about the descriptive
statistics interval-based bw and iops measurements.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>

The test job that measures rates is actually t0011.

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>

We can never reach this branch.

Signed-off-by: Jens Axboe <axboe@kernel.dk>

'work' will be overwritten shortly, no point in clearing it to NULL.

Signed-off-by: Jens Axboe <axboe@kernel.dk>