fio testing
Fio has a large, complex code base that supports myriad options for IO generation. Like virtually all software projects, fio benefits from automated testing. This blog post is an overview of testing fio (as of fio 3.30) and covers automated testing of fio features and fio's continuous integration testing infrastructure. At the end is an assessment of fio's test infrastructure and discussion of areas where it can be improved. To be clear, this blog post discusses tests for fio's features (not how to use fio to test storage devices).
Let us begin with a tour of components fio has for automated testing. These components include test jobs, compiled test programs, unit tests, and test scripts.
The first general class of fio's test components are test jobs. Fio has
fourteen sets of test jobs in the subdirectory t/jobs.
root@ubuntu:~/fio-dev/fio-canonical# ls -l t/jobs
total 64
-rw-r--r-- 1 root root 106 Jan 28 20:23 t0001-52c58027.fio
-rw-r--r-- 1 root root 272 Jan 28 20:23 t0002-13af05ae-post.fio
-rw-r--r-- 1 root root 258 Jan 28 20:23 t0002-13af05ae-pre.fio
-rw-r--r-- 1 root root 323 Jan 28 20:23 t0003-0ae2c6e1-post.fio
-rw-r--r-- 1 root root 243 Jan 28 20:23 t0003-0ae2c6e1-pre.fio
-rw-r--r-- 1 root root 364 Jan 28 20:23 t0004-8a99fdf6.fio
-rw-r--r-- 1 root root 198 Jan 28 20:23 t0005-f7078f7b.fio
-rw-r--r-- 1 root root 307 Jan 28 20:23 t0006-82af2a7c.fio
-rw-r--r-- 1 root root 167 Jan 28 20:23 t0007-37cf9e3c.fio
-rw-r--r-- 1 root root 191 Jan 28 20:23 t0008-ae2fafc8.fio
-rw-r--r-- 1 root root 608 Jan 28 20:23 t0009-f8b0bd10.fio
-rw-r--r-- 1 root root 146 Jan 28 20:23 t0010-b7aae4ba.fio
-rw-r--r-- 1 root root 293 Jan 28 20:23 t0011-5d2788d5.fio
-rw-r--r-- 1 root root 369 Jan 28 20:23 t0012.fio
-rw-r--r-- 1 root root 262 Jan 28 20:23 t0013.fio
-rw-r--r-- 1 root root 589 Jan 28 20:23 t0014.fio
Most of the test jobs include a commit hash in the name. For jobs referencing a
commit, the hash in question refers to the patch that fixed a bug exposed by
the job file. Thus, commits before the hash in question will exhibit buggy
behavior when the job in question is run. Comments are included in the job
files describing the correct and buggy behavior. For instance,
t0004-8a99fdf6.fio contains the following:
# Expected result: fio runs to completion
# Buggy result: fio segfaults
[global]
ioengine=libaio
direct=1
filename=foo
iodepth=128
size=10M
loops=1
group_reporting=1
readwrite=write
do_verify=0
verify=md5
numjobs=1
thread
verify_dump=1
[small_writes]
offset=0G
blocksize=512
verify_interval=1M
[large_writes]
stonewall
offset=1G
blocksize=1M
verify_interval=512
Prior to commit 8a99fdf6, the job in question would cause a segmentation
fault. This bug was fixed in 8a99fdf6 and from that commit onwards fio should
complete the job without error. Many of these test jobs are related to
verification workloads. Some of these test jobs include a preconditioning step.
The final few jobs in this set do not reference a specific commit.
Fio has a set of test programs exercising various portions of the code base.
These are located in the t/ subdirectory and are built by the Makefile
along with the main fio executable. These test programs include:
| Program | Description |
|---|---|
axmap |
This tests fio's axmap data structure which is used to keep track of which blocks fio has accessed when (by default) norandommap is disabled. |
fio-genzipf |
This exercises fio's probability distribution generating code and can be used to generate random values following uniform, zipf, pareto, and normal distributions. |
gen-rand |
This tests fio's random number generator. |
ieee754 |
This tests fio's library functions packing IEEE754 floating point numbers into 64-bit values and unpacking them. |
lfsr-test |
This tests fio's linear feedback shift register functionality which is used by random_generator=lfsr to have fio access all blocks of a device or file without using a random map. |
stest |
This tests fio's shared memory allocator. In actual use fio has multiple processes or threads using its shared memory allocator whereas this test program is single-threaded. |
Fio's codebase is integrated with the CUnit unit testing framework.
Relevant code is located in the unittests subdirectory. If CUnit support is
detected by the configure script, unit tests will be automatically built. Unit
tests exist for only seven fio components and are mostly for fio's string
manipulation functions.
Also in fio's repository is fuzzing for fio's option parsing functions. By
default a test program is built that can take a fio command line and offer it
to fio's option parsing routines to check for problems. By defining appropriate
environment variables alternative fuzzing strategies can be used. To build this
test program the symbol FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION must be
defined. One way to accomplish this is to run fio's configure script with the
--extra-cflags=-DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION option. When the
executable is built it can be used with fio command lines such as the
following:
root@ubuntu:~/fio-dev/fio-canonical# t/fuzz/fuzz_parseini examples/aio-read.fio
root@ubuntu:~/fio-dev/fio-canonical# echo $?
0
root@ubuntu:~/fio-dev/fio-canonical# t/fuzz/fuzz_parseini examples/aio-read.png
fio: option <▒PNG> outside of [] job section
root@ubuntu:~/fio-dev/fio-canonical# t/fuzz/fuzz_parseini kflfhd
root@ubuntu:~/fio-dev/fio-canonical# echo $?
2
Fuzzing is also incorporated into fio's continuous integration tests which will be discussed later.
Fio's source repository also contains a set of bash and Python test scripts.
Notable are the bash scripts in the t/zbd subdirectory. As the directory name
suggests, these scripts test fio's zoned block device support.
test-zbd-support runs up to 58 tests against a device, testing various
aspects of fio's support for zoned storage. Tests ensure that error conditions
are correctly handled, exercise many of the options related to fio's support
for zoned storage devices, and carry out a variety of read and write operations
using the libaio and sg ioengines. run-tests-against-nullb is a wrapper
for test-zbd-support. It creates a zoned null block device and uses this
device as a test target for test-zbd-support. This script has 13 sections
covering various configurations of zoned null block devices. These sections
have different configurations for the number of conventional and sequential
zones, various combinations of zone capacity and zone size, and devices with
and without limitations on the maximum number of open zones.
Fio's repository also contains a set of Python test scripts. These include:
| Script | Description |
|---|---|
jsonplus2csv_test.py |
With output-format=json+ detailed latency data is available. This script tests the jsonplus2csv helper script that converts fio json+ output to CSV format. |
latency_percentiles.py |
This runs twenty test jobs testing various aspects of fio's latency reporting features. |
readonly.py |
This tests fio's --readonly command line option. |
sgunmap-perf.py |
This carries out a performance test of the sg ioengine's unmap commands. |
sgunmap-test.py |
This carries out functional tests of the sg ioengine's unmap commands. |
steadystate_tests.py |
This tests fio's steady state detection feature. |
strided.py |
This tests fio's zonemode=strided option. |
As the file extension implies, this is a Python script, but it deserves special
mention as it does not contain any actual test cases. Instead it is a wrapper
script for the other test tools cataloged here. It runs the test jobs in
t/jobs, fio's compiled test programs, the first two sections of
run-tests-against-nullb, and fio's other Python test scripts. This script
supports Linux, Windows, and macOS and is able to automatically skip test cases
on platforms where they cannot be run. This script forms the basis of fio's
continuous integration testing which will be described next.
How are all of these test components actually put into regular use? Fio relies
on GitHub Actions for Linux and macOS
continuous integration testing and
AppVeyor for Windows continuous
integration testing. A set of tests is run with every new push to the main
branch of the fio repository on GitHub and for updated pull requests. Most of
the setup for continuous integration testing is available in the repository's
ci subdirectory.
Within the .github/workflows subdirectory of the fio repository are
configuration files for GitHub Actions. ci.yml specifies four fio builds:
64-bit gcc, 64-bit clang, 32-bit gcc, and macOS. Inside the repository's ci
subdirectory are a set of shell scripts that set up the environments and carry
out the tests. actions-install.sh installs the required packages for building
fio, actions-build.sh actually carries out the build, actions-smoke-test.sh
carries out a quick, basic test of fio functionality using the Makefile's
test target, and actions-fulltest.sh executes the run-fio-tests.py script
and also does a test build of the HTML documentation.
On AppVeyor, fio is built on three Windows platforms: cygwin 64-bit, cygwin
32-bit, and msys2 64-bit. At ci/appveyor-install.sh is a shell script for
creating the Windows build environments. This is a useful reference for
developers building fio on Windows. Also useful is that installers are created
as part of the build process and these can be downloaded as artifacts from
AppVeyor. So the latest fio Windows builds are easily available. However,
artifacts are available only for one month. So installer builds for tagged
versions will no longer be available one month after a version is tagged. After
the executable and installer are built, AppVeyor runs run-fio-tests.py and
results are available in the test log. Test artifacts are also uploaded and can
be examined.
At .github/workflows/cifuzz.yml are configuration settings for a Github
Actions workflow that carries out fuzz testing for fio's option parsing
routines. It builds the fuzzer and then feeds it the files in the fio
repository's examples subdirectory. Fio then attempts to parse the job files
as well as the image files contained therein. Any resulting crashes or memory
leaks can be detected.
Fio actually has a substantial set of test components. However, the test components have not been systematically developed and cover only a subset of fio's features. There is no systematic test coverage of fio's myriad features. For instance, some basic options have no explicit test coverage. Does fio actually issue sequential IO when directed to? Are generated offsets actually random when fio is directed to issue random IO? Some of this functionality may be indirectly assessed by existing test jobs but the fact remains that no explicit testing exists for these and other basic features. Other areas that merit test coverage include client/server mode and bandwidth/IOPS logging. Adding tests for basic and more advanced fio features would guard against regressions and improve reliability.
As for existing test components, one area crying out for attention is the Python test scripts. Many of them share a common pattern where test objects are created for fio jobs. These test objects have methods to actually run the tests and evaluate the output for success or failure. Such scripts would benefit from having their duplicated elements abstracted into a testing library that could be imported into each of the separate test scripts. Finally it may be worthwhile to consider whether these test cases could be integrated into a formal test framework such as Gauge or whether the current homegrown framework is sufficient.
Most fio code has not been written in a way conducive to unit testing because functions in the fio source code typically depend on many different data structures that would have to be laboriously mocked up for unit testing. However, augmenting the existing unit tests where possible would also be a worthwhile contribution.
As for continuous integration testing, one improvement that could be made is to save the AppVeyor installer builds for tagged versions. AppVeyor artifacts expire after one month. So installers for tagged releases are available from AppVeyor for only a short period. If installers built on AppVeyor were, for instance, deposited on GitHub as releases, Windows users would have easier access to tagged versions.
The (free) continuous integration testing infrastructure also is unable to run
some tests because of platform limitations. Thus, the t/zbd zoned block
device scripts are not publicly run regularly. Not running these tests
regularly is a lost opportunity.
Overall, fio has test coverage for many use cases. However, like almost all projects, its test coverage can be improved. Given how widely used fio is in the storage industry, improving test coverage would be of great benefit.