Tests Disk fio

Tests: Disk (fio)

In short

YABS measures random read/write disk performance with fio, running four block sizes (4k, 64k, 512k, 1m) against a test file on the partition you launched it from. Results print as speed and IOPS for Read, Write, and Total. If fio can't run at all, YABS falls back to simple dd sequential tests so you still get some number.

For everyone

The disk section looks like this:

fio Disk Speed Tests (Mixed R/W 50/50) (Partition /dev/vda2):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 404.90 MB/s (101.2k) | 402.40 MB/s (6.3k)
Write      | 406.57 MB/s (101.6k) | 407.52 MB/s (6.4k)
Total      | 811.47 MB/s (202.8k) | 809.93 MB/s (12.7k)
           |                      |
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 381.08 MB/s (743)    | 389.50 MB/s (380)
Write      | 393.54 MB/s (768)    | 408.20 MB/s (398)
Total      | 774.62 MB/s (1.5k)   | 797.70 MB/s (778)

Three things to know before reading it:

• Read and Write are nearly identical on purpose. The test issues a 50/50 random read/write mix and reports both directions over the same run, so they should be similar. What the numbers mean — and when an asymmetry is a red flag — is covered in Interpreting Results.
• "Total" is read + write summed, the aggregate throughput of the device under a mixed load.
• Only the current partition is tested — the one containing the directory you ran YABS from.

Which number to look at depends on the workload you're benchmarking for:

Block size	Represents	Metric to watch
4k	Small random I/O: databases, OS files, mail spools, web app trees	IOPS (throughput numbers stay modest even on fast disks)
64k	Middle ground: many filesystem and virtual-disk operations	both
512k / 1m	Large transfers: backups, media, sequential-ish streams	MB/s (IOPS naturally drops as each op carries more data)

A healthy SSD shows a smooth pattern: high 4k IOPS that fall off as block size grows while throughput climbs toward the device's ceiling. Wild anomalies — 4k IOPS in the hundreds of thousands paired with HDD-class 1m throughput, or numbers that collapse between runs — usually point to a cache absorbing the small I/O rather than honest media speed. The full field guide to these fingerprints lives in Interpreting Results.

How this test was designed (and why it looks this way)

The disk test you see today was negotiated in public. A brief history:

• Oct 2019, day one: the disk test was plain dd sequential writes — immediately criticised on LowEndTalk as cache-skewed and unrealistic. ioping was added for read tests the next day, then removed in Jan 2020 ("just as affected by cache as dd").
• Dec 2019: Daniel15 opened issue #1, "Replace dd with fio?", pointing at the BinaryLane fio how-to and ServerScope as prior art. The blocker was distribution: fio needs root to install from packages, so a prebuilt static binary was the only way to keep the "no dependencies" promise — see Precompiled Binaries for that saga.
• Jan 2020 (LowEndSpirit thread): the community hashed out the parameters in the open. The initial draft borrowed BinaryLane's settings (--randrepeat=1 --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75) and had both Sequential and Random columns. Through discussion the shape settled: sequential tests dropped as "essentially useless as they aren't testing real-world conditions", the file went 1G→2G, the per-test duration 60s→30s, read-only/write-only runs dropped in favour of a single mixed test, and rwmixread went 75→50 because "it's better to read and write the same amounts of data". Daniel15 countered that real workloads are read-heavy; Mason's verdict was that "creating a 'one-size-fits-all' test is HARD" — and 50/50 stuck. The (Mixed R/W 50/50) text in the header was a community suggestion (beagle) so readers know the caveat.
• Feb 2020 (v2020-02-04): fio replaced dd for real; dd survives only as the fallback path described below.
• Nov 2023: the (Partition /dev/...) label was added to the header so multi-disk hosts know which filesystem was measured.
• Jul 2026: the KiB/KB display correction (#120, see "Units" below).

Under the hood: the exact commands

Setup pass

Before measuring, YABS lays out the test file with a quick throwaway run:

fio --name=setup --ioengine=libaio --rw=read --bs=64k --iodepth=64 \
    --numjobs=2 --size=$FIO_SIZE --runtime=1 --gtod_reduce=1 \
    --filename=$DISK_PATH/test.fio --direct=1 --minimal

This creates/extends test.fio so subsequent runs measure I/O, not file allocation.

Measurement runs

Once per block size, for BS in 4k 64k 512k 1m:

timeout 35 fio --name=rand_rw_$BS --ioengine=libaio --rw=randrw --rwmixread=50 \
    --bs=$BS --iodepth=64 --numjobs=2 --size=$FIO_SIZE --runtime=30 \
    --gtod_reduce=1 --direct=1 --filename=$DISK_PATH/test.fio \
    --group_reporting --minimal

Every parameter, and why

Parameter	Value	Why
--ioengine	libaio	Linux native async I/O — the engine that supports true queue depth with O_DIRECT. (On systems without io_setup, e.g. QEMU user-mode emulation, libaio fails; that is a test-env caveat, not a real host one.)
--rw / --rwmixread	randrw, 50	Random mixed read/write, exactly half reads. Chosen in Jan 2020 community discussion: sequential tests were scrapped as "essentially useless ... not testing real-world conditions", and 50/50 over 75/25 because "better to read and write the same amounts of data".
--bs	4k, 64k, 512k, 1m	From tiny random I/O (databases, OS) to large streaming-ish transfers (media, backups). Set suggested by LES community members.
--iodepth	64	Outstanding requests per job.
--numjobs	2	Two worker jobs → up to 128 I/Os in flight combined — enough to expose queue depth without an absurd load.
--size	2G (x86/x64), 512M (aarch64/arm/riscv64)	Test file size. Bumped 1G→2G during the Jan 2020 design discussion; halved on ARM/RISC-V for slower/lower-memory targets.
--runtime	30	Up to 30 s per block size — no --time_based, so fio stops early if it completes size of I/O first (fast NVMe finishes early).
timeout 35	external	Hard kill at 35 s if fio hangs.
--direct=1	on	O_DIRECT: bypasses the guest page cache. It does not bypass host/hypervisor caches, RAID controller caches, or ZFS ARC — one reason numbers can look impossibly good.
--gtod_reduce=1	on	Cheaper timestamps inside fio; slightly less overhead per I/O. (Also means latency percentiles aren't meaningful and aren't reported.)
--group_reporting	on	One aggregate result across both jobs.
--minimal	on	Terse semicolon-separated output for parsing.

How output is parsed

--minimal produces fio "terse v3" output — a single line of ~121 ;-separated fields covering every metric fio collects (throughput, IOPS, latency stats, percentiles, per-job breakdowns). YABS reads only four, with awk -F';':

Field	Meaning
$7	read bandwidth, KiB/s
$8	read IOPS
$48	write bandwidth, KiB/s
$49	write IOPS

Totals = read + write for both metrics. (That the bandwidth fields are KiB/s and not something else was verified empirically: bw = iops × 4 KiB exactly at --bs=4k.) Everything else fio measured — including the latency distribution — is simply discarded; it's in the raw output if you run fio by hand, but YABS never sees it.

Each iteration of the loop looks like:

DISK_TEST=$(timeout 35 "$FIO_CMD" ... --minimal | grep rand_rw_"$BS")
# then: $7 -> read bw, $8 -> read iops, $48 -> write bw, $49 -> write iops

If a run produces no matching line (binary killed, crashed, engine unsupported), the arrays end up empty and YABS falls through to the dd fallback.

Units and the KiB/KB story

fio's terse bandwidth fields are always KiB/s regardless of --kb_base. YABS's format_speed converts to decimal units for display:

• ≥ 976563 KiB/s (≈ 1 GB/s decimal) → x.xx GB/s
• ≥ 977 KiB/s → x.xx MB/s
• otherwise → x.xx KB/s (KiB/s × 1.024)

format_iops prints raw IOPS until 1000, then x.yk.

History: until July 2026 (#120, commit 3125fed), YABS treated the KiB/s values as if they were already decimal KB/s, so displayed speeds were ~2.4% low. The JSON schema still labels these fields "speed_units": "KBps" for compatibility — consumers should read them as KiB/s. See JSON Output.

The arithmetic of one block-size run

Why the numbers look the way they do:

• iodepth=64 × numjobs=2 → up to 128 I/Os in flight at once — deep enough to expose a fast NVMe's parallelism, and deep enough that a slow disk spends the whole 30 s saturated.
• --runtime=30 without --time_based means fio stops either at 30 s or when size worth of I/O completes — on a fast device with the 2 GiB file, the 1m run finishes early. So a "30 s" cap is just a cap.
• timeout 35 wraps the whole thing — the wall-clock kill if fio itself hangs.
• Bandwidth and IOPS are two views of the same thing: bw ≈ iops × bs. At 4k, 100k IOPS is ~400 MiB/s; the same ~800 MB/s Total at 1m blocks corresponds to only ~800 combined IOPS — which is why the table prints both and why 4k is quoted in IOPS while 1m is quoted in MB/s.

The test file ($YABS_PATH/disk/test.fio) is created by the setup pass, written and read through the whole run, and deleted with the temp directory at the end — it never outlives the run.

Pre-flight checks

Before fio runs, YABS checks:

1. Free space. df -k . on the current directory: < 2097152 KiB (2 GiB) on x86/x64 or < 524288 KiB (512 MiB) on ARM/RISC-V → prints Less than 2GB of space available. Skipping disk test... (or 512MB variant).

2. ZFS inflation. If /sys/module/zfs/parameters/spa_asize_inflation exists, YABS multiplies its value (24 by default) by the 2 GB test size and warns when the current dataset has less than spa_asize_inflation × 2 GB free:

   Warning! You are running YABS on a ZFS Filesystem and your disk space is too low for the fio test. Your test results will be inaccurate. You need at least <N> GB free in order to complete this test accurately.

   Why: ZFS accounts inflight writes at spa_asize_inflation× their logical size, so a pool that's nearly full forces extreme write throttling and fio results "crumble" — reported up to 10× slow (issue #13, first seen on webdock.io hosts). The check finds the longest df -Th mountpoint that is a prefix of $PWD, reads its Avail column, and converts T/G/M/K/B suffixes to GB.

flowchart TD
    A[Disk test?] -->|"-f / -d flag"| Z1[skipped]
    A --> B{"df -k . free ≥ test size?"}
    B -->|no| Z2["print 'Less than 2GB/512MB...' and skip"]
    B -->|yes| C{"ZFS loaded AND free < spa_asize_inflation × 2G?"}
    C -->|yes| C1[print ZFS low-space warning, continue anyway]
    C -->|no| D
    C1 --> D{fio available?}
    D -->|"local fio, unless -b"| F[use local fio]
    D -->|else| E[download fio_$ARCH from GitHub release]
    E -->|download empty/failed| E1["'Fio binary download failed' -> dd fallback"]
    F --> G[setup pass + 4 block-size runs]
    E -->|ok| G
    G -->|no output / killed| G1["'fio disk speed tests failed' -> dd fallback"]
    G -->|ok| H[print fio table + JSON]
    E1 --> I[dd sequential test x3]
    G1 --> I
    I --> J[print dd table]

Loading

Edge cases in the ZFS and free-space checks

• The free-space probe is df -k . on $PWD — so it's the filesystem you'll write test.fio into that matters, not / or any other mount.
• The ZFS check only runs when the spa_asize_inflation sysfs parameter exists — i.e. the ZFS module is loaded. A non-ZFS host never enters that block at all.
• The ZFS math: mul_spa = spa_asize_inflation × 2 (GB) — the fio file is 2 GiB, so free space below inflation×2 GiB means inflight-write accounting can throttle the test. Default inflation of 24 → the warning needs ≥48 GB free. The check then finds the longest df -Th mountpoint that prefixes $PWD (most specific mount wins), reads Avail from the zfs-typed line, and converts suffixes (T/G/M/K/B) to integer GB.
• If the mount can't be identified (Note: No relevant filesystem path detected...) or the Avail field can't be parsed (Warning: Could not parse free space format...), the script says so and proceeds — the guard warns, never blocks.

The dd fallback

If the fio binary can't be downloaded (Fio binary download failed. Running dd test as fallback....) or fio runs but produces nothing (fio disk speed tests failed. Run manually to determine cause.\nRunning dd test as fallback...), YABS runs the original 2019-era dd test instead:

• Write ×3: dd if=/dev/zero of=$DISK_PATH/$DATE.test bs=64k count=16k oflag=direct — a 1 GiB direct-I/O sequential write.
• Read ×3: dd if=$DISK_PATH/$DATE.test of=/dev/null bs=8k — buffered sequential read, with no direct flag, so the page cache inflates it.

Printed as a dd Sequential Disk Speed Tests table (Test 1/2/3/Avg). The code comment is candid: "dd sequential speed tests are not indiciative or real-world results, however, some form of disk speed measure is better than nothing." Never compare dd numbers to fio numbers — different workload (sequential vs random) and different cache exposure (the read side is essentially cached). Two subtleties worth knowing if you do stare at a dd table:

• The write numbers are the more honest half — oflag=direct bypasses the page cache for the 1 GiB write.
• The read numbers often look suspiciously fast: the 1 GiB file was just written, so on any host with ≥ 1 GiB free RAM it may be served straight out of the page cache at RAM speed. A read:write ratio far above 1 is a tell.

What this test does NOT measure

• Latency — no avg/percentile latency anywhere (--minimal + --gtod_reduce=1 means it's not even collected meaningfully). Run fio manually for p99 numbers.
• Sequential throughput — randrw at 512k/1m approximates it, but there is no dedicated sequential pass (deliberately removed in Jan 2020 as unrepresentative).
• Other disks/partitions — only the partition holding $PWD is exercised.
• Write durability / fsync behaviour, filesystem vs raw device differences, and steady-state behaviour beyond ~35 s per block size.

Testing a different disk or partition

cd into a directory on the target filesystem and re-run with the other tests disabled:

cd /mnt/other-disk
curl -sL yabs.sh | bash -s -- -ig

df -P . determines which partition is named in the header, and test.fio is created under ./<timestamp>/disk/ — i.e. on whatever filesystem $PWD lives on.

Going beyond YABS: running fio yourself

YABS is deliberately a fixed snapshot; for anything deeper, run fio directly. The binary YABS downloaded is gone after the run (the temp dir is deleted), so install fio from your distro or grab fio_$ARCH from the repo's releases. Ideas:

• Latency and percentiles (what YABS never reports): drop --minimal --gtod_reduce, add --time_based --runtime=60 and read the lat percentiles section of the human output — p99 matters more than the mean for interactive workloads.
• Steady-state / cache unmasking: increase --size well beyond RAM and run longer (--runtime=300 --time_based); if numbers sink as the run progresses, a cache was fronting the disk.
• Sequential: --rw=read or --rw=write with --bs=1m.
• Different mix: --rwmixread=70 approximates read-heavier real workloads.
• Another device: point --filename at a path on it (a file, not the raw device — raw-device testing is a different sport and not what YABS does).

Keep --ioengine=libaio --direct=1 --iodepth=64 if you want results comparable to YABS output.

See also

• Interpreting Results — what good/bad disk numbers look like, cache fingerprints
• Troubleshooting and FAQ — "fio shows the same read and write", ZFS warning, download failures
• System Information — how the disk size total is computed (different code path)
• Precompiled Binaries — where fio_$ARCH comes from
• Benchmarks Are Snapshots — why one run isn't the whole story