Help me get reasonable bandwidth numbers

[yuzhu]

I noticed the benchmark only has latency numbers, so I used fio to write and read some large files. Writes, in particular, are much slower than the nic maximum. Any reason for this? any tuning tips?
Env: m5.2xlarge and s3 bucket
fio command
fio --name=initial_write --ioengine=libaio --iodepth=32 --rw=write --bs=4k --size=1g --nrfiles=5 --filename=testfile.0:testfile.1:testfile.2:testfile.3:testfile.4 --group_reporting

Jobs: 1 (f=1): [W(1)][-.-%][eta 00m:00s]
initial_write: (groupid=0, jobs=1): err= 0: pid=4284: Wed Oct 29 23:46:06 2025
write: IOPS=10.8k, BW=42.3MiB/s (44.3MB/s)(4096MiB/96894msec); 0 zone resets
slat (nsec): min=1949, max=11928k, avg=2859.98, stdev=12840.81
clat (usec): min=2, max=12173, avg=118.07, stdev=77.41
lat (usec): min=7, max=12178, avg=120.93, stdev=78.72
clat percentiles (usec):
| 1.00th=[ 103], 5.00th=[ 104], 10.00th=[ 105], 20.00th=[ 105],
| 30.00th=[ 106], 40.00th=[ 108], 50.00th=[ 109], 60.00th=[ 110],
| 70.00th=[ 110], 80.00th=[ 114], 90.00th=[ 155], 95.00th=[ 180],
| 99.00th=[ 221], 99.50th=[ 243], 99.90th=[ 433], 99.95th=[ 668],
| 99.99th=[ 1106]
bw ( KiB/s): min= 8, max=1163744, per=100.00%, avg=699050.00, stdev=533332.85, samples=12
iops : min= 2, max=290936, avg=174762.50, stdev=133333.21, samples=12
lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec) : 250=99.57%, 500=0.37%, 750=0.03%, 1000=0.02%
lat (msec) : 2=0.01%, 10=0.01%, 20=0.01%
cpu : usr=1.03%, sys=3.96%, ctx=50722, majf=0, minf=12
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1048575,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
WRITE: bw=42.3MiB/s (44.3MB/s), 42.3MiB/s-42.3MiB/s (44.3MB/s-44.3MB/s), io=4096MiB (4295MB), run=96894-96894msec

[Barre]

What does monitoring network traffic looks like when running the benchmarks? ZeroFS uses more bandwidth than the raw data written due to compaction and writing the WAL.

[yuzhu]

yes, I expect some overhead, but not this much. Have you done any bandwidth testing? I am just wondering if I have benchmarked incorrectly.

[Barre]

Bandwidth usage is going to vary wildly based on your workload. fio is not really representative of real-world use-cases. It'd help to see what you are trying to do with ZeroFS and work from there.