sab incomplete/complete folder speed test

[thezoggy]

When we do the speed test we open file, then loop write line of data + flush, repeat until 0.5 sec then close file. and use counter on how big the data written would be..

        while total_time < maxtime:
            start = time.time()
            os.write(fp_testfile, dump_data)
            os.fsync(fp_testfile)
            total_time += time.time() - start
            total_written += _DUMP_DATA_SIZE

I've always wondered why this speed test was not ever as close to other drive benchmarks.. and why any io wait just tanks its performance... I finally got around to adding a newer m2 drive in my box and saw the speed test go down when it technically is a faster drive with faster specs and nego at a faster speed.

lspci -vv for both drives so I could fact check what their cap/speed is:

/downloads -- 1TB Samsung 980 Pro (PCIe 4.0 x4 nvme m.2 drive)

01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO (prog-if 02 [NVM Express])
		LnkCap:	Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
		LnkSta:	Speed 16GT/s, Width x4

				[N:0:6:1]    disk    Samsung SSD 980 PRO 1TB__1                 /dev/nvme0n1  1.00TB

/download-970 -- 1TB Samsung 970 EVO Plus (PCIe 3.0 x4 nvme m.2 drive)

04:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 (prog-if 02 [NVM Express])
		LnkCap:	Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
		LnkSta:	Speed 8GT/s, Width x4

				[N:1:4:1]    disk    Samsung SSD 970 EVO Plus 1TB__1            /dev/nvme1n1  1.00TB

---

Using same sabnzbd container, mounting both m2 drives into docker and running diskspeed to each with nothing else going on.

loading up sabnzbd docker from linuxserver.io, dropping sab git repo in a folder so I could access diskspeed from sab util.
logged into that docker to run various tests. box has linux kernel 6.1.79

:/downloads/sabnzbd/sabnzbd/utils# python3 -V
Python 3.11.9

/download-970 -- 1TB Samsung 970 EVO Plus (PCIe 3.0 x4 nvme m.2 drive)

manuf specs of sequential read/writes are rated at 3,500 MB/s read and 3,300 MB/s

# python diskspeed.py /download-970/
Let's go
Disk writing speed: 760.70 Mbytes per second
Done

/downloads -- 1TB Samsung 980 Pro (PCIe 4.0 x4 nvme m.2 drive)

manuf specs of sequential read/writes are rated at 7,000 MB/s read and 5,000 MB/s

# python diskspeed.py /downloads/
Let's go
Disk writing speed: 664.70 Mbytes per second
Done

Can see that newer m2 that has higher specs is getting slower speed per this test.

Commenting out the fsync line,

      # os.fsync(fp_testfile)

re-running test, can see they both get roughly 2,550 MB/s

# python diskspeed.py /download-970/
Let's go
Disk writing speed: 2546.80 Mbytes per second
Done

# python diskspeed.py /downloads/
Let's go
Disk writing speed: 2561.80 Mbytes per second
Done

which then were closer aligned to just doing dd:

:/download-970# dd if=/dev/zero of=/download-970/deleteme bs=1M count=2048 oflag=direct
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 0.948926 s, 2.3 GB/s

/downloads# dd if=/dev/zero of=/downloads/deleteme bs=1M count=2048 oflag=direct
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 0.756706 s, 2.8 GB/s

Debated if how we are doing the speed test is optimal and if we were purposely flushing to disk to not skew things with buffers or what. So I redid it using with open, and did the normal recommendation of flush + fsync.. and showing file size of each run:

    total_time = 0.0
    try:
        with open(filename, "wb") as f:
            while total_time < maxtime:
                start = time.time()
                f.write(dump_data)
                f.flush()
                os.fsync(f.fileno())
                total_time += time.time() - start
                total_written += _DUMP_DATA_SIZE
        print("File size:", os.path.getsize(filename), "bytes")
        os.remove(filename)
    except OSError:
        # Could not write, so ... report 0.0
        logging.debug("Failed to measure disk speed on %s", dirname)
        return 0.0

can see that even with flushing buffers this newer method yields 2x the speed (but still shows the older drive is still slightly faster):

# python diskspeed2.py /download-970/
Let's go
File size: 629145600 bytes
File size: 629145600 bytes
Disk writing speed: 1141.50 Mbytes per second
Done

# python diskspeed2.py /downloads/
Let's go
File size: 629145600 bytes
File size: 629145600 bytes
Disk writing speed: 1089.70 Mbytes per second
Done

and if I comment out doing the flush+fsync on this method:

# python diskspeed2.py /download-970/
Let's go
File size: 1363148800 bytes
File size: 1363148800 bytes
Disk writing speed: 2457.00 Mbytes per second
Done

# python diskspeed2.py /downloads/
Let's go
File size: 1363148800 bytes
File size: 1363148800 bytes
Disk writing speed: 2475.90 Mbytes per second
Done

Both drives use btrfs and have discard=async, but still ran trim on both drives just in case. No change.
Running the test longer (0.5 -> 1.0 / 3.0 / 5.0 / 10.0), just results in slightly slower speed as things prob normalize a bit.

So got me curious, when we save data from article cache to the drive, it does not look like we flush/sync to disk after writing:
https://github.com/sabnzbd/sabnzbd/blob/develop/sabnzbd/filesystem.py#L1128

So then why do we do it for the diskspeed test?

Doing the extra step causes io strain, which will cause suboptimal setups to rear their head.. which I see all the time with people using horrible paths that go through virtual overlay/filesystems or slow external drives / remote mounts.

[mnightingale]

Without the syncs it’s just testing how quick it can write to caches, system or SSD dram.
If I remove the fsync line and point it at ZFS it claims impossible speeds.
But in normal use you want all the caching.

Perhaps there would be benefits if the file was opened with a combination of O_DIRECT and O_SYNC, to make it more like your dd tests.

O_DIRECT = direct memory access, try to avoid caches.
O_SYNC = synchronise writes, return once written to disk - disk could be lying though.

Opening with O_SYNC would remove the need for the fsync calls and losing time in python code.

Could also try tightening the loop and using better time measurements.
I didn't bother using perf_counter_ns()

For some ideas: https://gist.github.com/mnightingale/8d2b2ce5181783e2ee29e640a8daa2da
WD SN850X 1TB claims around 3GB/s which is the same as dd, but still not particularly close to what it should do sequentially.

[thezoggy]

comparing the current diskspeed.py (nothing commented out) and your version "diskspeed-mnight.py".

i see even slightly less speeds:

/downloads -- 1TB Samsung 980 Pro (PCIe 4.0 x4 nvme m.2 drive)

:/downloads/sabnzbd/sabnzbd/utils# python diskspeed.py /downloads/
Let's go
Disk writing speed: 665.80 Mbytes per second
Done

:/downloads/sabnzbd/sabnzbd/utils# python diskspeed-mnight.py /downloads/
Let's go
Disk writing speed: 635.50 Mbytes per second
Done

/download-970 -- 1TB Samsung 970 EVO Plus (PCIe 3.0 x4 nvme m.2 drive)

:/downloads/sabnzbd/sabnzbd/utils# python diskspeed.py /download-970/
Let's go
Disk writing speed: 793.30 Mbytes per second
Done

:/downloads/sabnzbd/sabnzbd/utils# python diskspeed-mnight.py /download-970/
Let's go
Disk writing speed: 777.40 Mbytes per second
Done

mounted volume of /dev/shm (ramdrive) to the docker to test just how fast the speed test would report.
so looks like ~2500 is just limit i can do on this box, and to note your version wouldnt run with it:

# python diskspeed.py /ramdrive/
Let's go
Disk writing speed: 2543.60 Mbytes per second
Done

# python diskspeed-mnight.py /ramdrive/
Let's go
No measurement possible. Check that directory is writable.
Done

box has DDR4-3600, slight above default system says 2666 MT/s
https://www.gskill.com/product/165/184/1562833535/F4-3600C18D-32GVK

So guessing memory is my bottle neck there on getting full speeds out of my nvme drives from that pov...

Some point I need to run test to drives directly outside of docker and see how much performance overhead the container is doing...

testing directly to disk outside of docker, normal mount and using dd

1TB Samsung 970 EVO Plus (PCIe 3.0 x4 nvme m.2 drive)

writing 2G.. zeros quick:

# dd if=/dev/zero of=/mnt/disks/S59ANMFNA03845Z/deleteme bs=1M count=2048 oflag=direct
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 0.948204 s, 2.3 GB/s

without direct

# dd if=/dev/zero of=/mnt/disks/S59ANMFNA03845Z/deleteme bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 1.32663 s, 1.6 GB/s

without direct, but trying to make sure stuff is flush to disk

# dd if=/dev/zero of=/mnt/disks/S59ANMFNA03845Z/deleteme bs=1M count=2048 conv=fdatasync
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 1.8669 s, 1.2 GB/s

we dont really do a new random value each time we write a new line (so above 'zero' figure it closer to our actual test).
but lets use /dev/urandom and see how much different things are.. can see it tanks performance:

# dd if=/dev/urandom of=/mnt/disks/S59ANMFNA03845Z/deleteme bs=1M count=2048 oflag=direct
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 6.01464 s, 357 MB/s

# dd if=/dev/urandom of=/mnt/disks/S59ANMFNA03845Z/deleteme bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 6.42399 s, 334 MB/s

# dd if=/dev/urandom of=/mnt/disks/S59ANMFNA03845Z/deleteme bs=1M count=2048 conv=fdatasync
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 6.81496 s, 315 MB/s

goal is to find out if our speed test is really valid / whats the cli method to sanity check. since people join discord saying their manuf specs say one thing but dont realize where bottle necks are.. then run benchmark tools and see wildly different results than what we show.. or of course suboptimal paths or overhead due to how the mounts are done/filesystem used and so on.

[mnightingale]

Perhaps compare to a different filesystem or playing with BTRFS options; I’ve never used it but try disabling COW (and checksums) for a directory? chattr +C /dir/file

I choose to use XFS for my scratch/downloads disk but may give BTRFS a go sometime for the checksums.

dd direct reports 4.4GB/s, diskspeed.py 2.4GB/s, with my changes around 4.8GB/s

Benchmarking with fio --name=fiotest --filename=/mnt/downloads/deleteme --size=16Gb --rw=write --bs=1M --direct=1 --ioengine=libaio --iodepth=8 --group_reporting --runtime=10 gives 6.4GB/s which meets the drive spec.
There is also a --numjobs=8 option which may be of interest, but SAB only performs one assemble at a time anyway.
Likewise, I don't know what iodepth would be most representative.

[thezoggy]

so logging into my lsio-sab docker, installing fio. and doing test to each nvme. both drives are btrfs... which ideally i want because of docker containers and such on it.

the 980 one is my current cache drive, while the 970 used to be my current one and now i just have it mounted as an unused disk so i could just do some direct tests on. (not sure why discard isnt being set on that one, but i run trim manually after doing some tests to just make sure it gets cleared)

# mount | grep nvme
/dev/nvme0n1p1 on /mnt/cache type btrfs (rw,noatime,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/)
/dev/nvme1n1p1 on /mnt/disks/S59ANMFNA03845Z type btrfs (rw,relatime,ssd,discard,space_cache=v2,subvolid=5,subvol=/)

updated needed to run trim as for whatever reason even though discard=async is enabled performance still was hit until i did. did not affect sab's writespeed test.. still 650-700 MB/s.

/downloads -- 1TB Samsung 980 Pro (PCIe 4.0 x4 nvme m.2 drive)

# fio --name=fiotest --filename=/downloads/deleteme --size=16Gb --rw=write --bs=1M --direct=1 --ioengine=libaio --iodepth=8 --group_reporting --runtime=10
fiotest: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
fio-3.36
Starting 1 process
fiotest: Laying out IO file (1 file / 16384MiB)
Jobs: 1 (f=1): [W(1)][-.-%][w=4724MiB/s][w=4724 IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=2004: Sat May 11 00:36:03 2024
  write: IOPS=4710, BW=4711MiB/s (4940MB/s)(16.0GiB/3478msec); 0 zone resets
    slat (usec): min=64, max=398, avg=90.64, stdev=25.99
    clat (usec): min=924, max=9970, avg=1606.45, stdev=415.28
     lat (usec): min=1129, max=10089, avg=1697.09, stdev=413.44
    clat percentiles (usec):
     |  1.00th=[ 1483],  5.00th=[ 1516], 10.00th=[ 1532], 20.00th=[ 1549],
     | 30.00th=[ 1549], 40.00th=[ 1565], 50.00th=[ 1582], 60.00th=[ 1582],
     | 70.00th=[ 1582], 80.00th=[ 1598], 90.00th=[ 1598], 95.00th=[ 1598],
     | 99.00th=[ 1876], 99.50th=[ 5800], 99.90th=[ 5800], 99.95th=[ 5932],
     | 99.99th=[ 9896]
   bw (  MiB/s): min= 4652, max= 4744, per=99.96%, avg=4708.67, stdev=32.56, samples=6
   iops        : min= 4652, max= 4744, avg=4708.67, stdev=32.56, samples=6
  lat (usec)   : 1000=0.02%
  lat (msec)   : 2=99.07%, 4=0.03%, 10=0.88%
  cpu          : usr=3.05%, sys=40.52%, ctx=18222, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=4711MiB/s (4940MB/s), 4711MiB/s-4711MiB/s (4940MB/s-4940MB/s), io=16.0GiB (17.2GB), run=3478-3478msec

/download-970 -- 1TB Samsung 970 EVO Plus (PCIe 3.0 x4 nvme m.2 drive)

# fio --name=fiotest --filename=/download-970/deleteme --size=16Gb --rw=write --bs=1M --direct=1 --ioengine=libaio --iodepth=8 --group_reporting --runtime=10
fiotest: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
fio-3.36
Starting 1 process
fiotest: Laying out IO file (1 file / 16384MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=3349MiB/s][w=3349 IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=315: Thu May  9 00:25:13 2024
  write: IOPS=3339, BW=3340MiB/s (3502MB/s)(16.0GiB/4906msec); 0 zone resets
    slat (usec): min=61, max=278, avg=107.69, stdev=22.71
    clat (usec): min=1439, max=7846, avg=2286.40, stdev=205.18
     lat (usec): min=1597, max=8052, avg=2394.08, stdev=203.98
    clat percentiles (usec):
     |  1.00th=[ 2180],  5.00th=[ 2212], 10.00th=[ 2212], 20.00th=[ 2245],
     | 30.00th=[ 2245], 40.00th=[ 2245], 50.00th=[ 2245], 60.00th=[ 2245],
     | 70.00th=[ 2245], 80.00th=[ 2278], 90.00th=[ 2311], 95.00th=[ 2442],
     | 99.00th=[ 3326], 99.50th=[ 3425], 99.90th=[ 4015], 99.95th=[ 4359],
     | 99.99th=[ 7767]
   bw (  MiB/s): min= 3304, max= 3356, per=99.99%, avg=3339.33, stdev=17.94, samples=9
   iops        : min= 3304, max= 3356, avg=3339.33, stdev=17.94, samples=9
  lat (msec)   : 2=0.05%, 4=99.85%, 10=0.10%
  cpu          : usr=2.02%, sys=34.70%, ctx=17486, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=3340MiB/s (3502MB/s), 3340MiB/s-3340MiB/s (3502MB/s-3502MB/s), io=16.0GiB (17.2GB), run=4906-4906msec

and now testing drives outside of docker...

getting almost drive spec now,

/mnt/cache -- 1TB Samsung 980 Pro (PCIe 4.0 x4 nvme m.2 drive)

# fio --name=fiotest --filename=/mnt/cache/deleteme --size=16Gb --rw=write --bs=1M --direct=1 --ioengine=libaio --iodepth=8 --group_reporting --runtime=10
fiotest: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
fio-3.36
Starting 1 process
fiotest: Laying out IO file (1 file / 16384MiB)
Jobs: 1 (f=1): [W(1)][-.-%][w=4731MiB/s][w=4731 IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=25017: Sat May 11 00:32:57 2024
  write: IOPS=4532, BW=4532MiB/s (4752MB/s)(16.0GiB/3615msec); 0 zone resets
    slat (usec): min=63, max=445, avg=91.48, stdev=26.53
    clat (usec): min=651, max=8743, avg=1672.14, stdev=564.69
     lat (usec): min=810, max=8824, avg=1763.62, stdev=566.64
    clat percentiles (usec):
     |  1.00th=[ 1483],  5.00th=[ 1516], 10.00th=[ 1532], 20.00th=[ 1549],
     | 30.00th=[ 1549], 40.00th=[ 1565], 50.00th=[ 1582], 60.00th=[ 1582],
     | 70.00th=[ 1598], 80.00th=[ 1598], 90.00th=[ 1598], 95.00th=[ 1614],
     | 99.00th=[ 4228], 99.50th=[ 5800], 99.90th=[ 7570], 99.95th=[ 7898],
     | 99.99th=[ 8717]
   bw (  MiB/s): min= 3416, max= 4740, per=99.98%, avg=4531.14, stdev=492.11, samples=7
   iops        : min= 3416, max= 4740, avg=4531.14, stdev=492.11, samples=7
  lat (usec)   : 750=0.01%, 1000=0.05%
  lat (msec)   : 2=95.84%, 4=2.60%, 10=1.49%
  cpu          : usr=2.85%, sys=39.46%, ctx=18043, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=4532MiB/s (4752MB/s), 4532MiB/s-4532MiB/s (4752MB/s-4752MB/s), io=16.0GiB (17.2GB), run=3615-3615msec

that 970 drive outside of docker, via direct disk mount. pretty much the same as when i did it within docker. hitting write spec of manufc.

/mnt/disks/S59ANMFNA03845Z/ -- 1TB Samsung 970 EVO Plus (PCIe 3.0 x4 nvme m.2 drive)

# fio --name=fiotest --filename=/mnt/disks/S59ANMFNA03845Z/deleteme --size=16Gb --rw=write --bs=1M --direct=1 --ioengine=libaio --iodepth=8 --group_reporting --runtime=10
fiotest: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
fio-3.36
Starting 1 process
fiotest: Laying out IO file (1 file / 16384MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=3324MiB/s][w=3324 IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=28601: Sat May 11 00:34:29 2024
  write: IOPS=3337, BW=3338MiB/s (3500MB/s)(16.0GiB/4909msec); 0 zone resets
    slat (usec): min=60, max=304, avg=108.17, stdev=19.38
    clat (usec): min=2051, max=8045, avg=2287.49, stdev=204.62
     lat (usec): min=2258, max=8143, avg=2395.66, stdev=204.20
    clat percentiles (usec):
     |  1.00th=[ 2180],  5.00th=[ 2212], 10.00th=[ 2212], 20.00th=[ 2245],
     | 30.00th=[ 2245], 40.00th=[ 2245], 50.00th=[ 2245], 60.00th=[ 2245],
     | 70.00th=[ 2245], 80.00th=[ 2278], 90.00th=[ 2311], 95.00th=[ 2442],
     | 99.00th=[ 3261], 99.50th=[ 3392], 99.90th=[ 4113], 99.95th=[ 4752],
     | 99.99th=[ 7832]
   bw (  MiB/s): min= 3298, max= 3354, per=100.00%, avg=3338.22, stdev=21.43, samples=9
   iops        : min= 3298, max= 3354, avg=3338.22, stdev=21.43, samples=9
  lat (msec)   : 4=99.90%, 10=0.10%
  cpu          : usr=2.42%, sys=34.35%, ctx=17531, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=3338MiB/s (3500MB/s), 3338MiB/s-3338MiB/s (3500MB/s-3500MB/s), io=16.0GiB (17.2GB), run=4909-4909msec

but can see with fio speeds are great.

[thezoggy]

trying out ioping, can see just writing 4Kb over and over can see idea of what avg latency is.. 9ms vs older nvme 6.65ms

# ioping -D -Y -WWW -s4k /downloads/
4 KiB >>> /downloads/ (  931.5 GiB): request=1 time=42.5 us (warmup)
4 KiB >>> /downloads/ (  931.5 GiB): request=2 time=8.61 ms
4 KiB >>> /downloads/ (  931.5 GiB): request=3 time=8.54 ms
4 KiB >>> /downloads/ (  931.5 GiB): request=4 time=8.57 ms
4 KiB >>> /downloads/ (  931.5 GiB): request=5 time=9.31 ms
4 KiB >>> /downloads/ (  931.5 GiB): request=6 time=9.23 ms
4 KiB >>> /downloads/ (  931.5 GiB): request=7 time=9.22 ms
4 KiB >>> /downloads/ (  931.5 GiB): request=8 time=9.10 ms
4 KiB >>> /downloads/ (  931.5 GiB): request=9 time=9.19 ms
4 KiB >>> /downloads/ (  931.5 GiB): request=10 time=9.29 ms
^C
--- /downloads/ (  931.5 GiB) ioping statistics ---
9 requests completed in 81.0 ms, 36 KiB written, 111 iops, 444.2 KiB/s
generated 10 requests in 9.83 s, 40 KiB, 1 iops, 4.07 KiB/s
min/avg/max/mdev = 8.54 ms / 9.01 ms / 9.31 ms / 313.1 us

# ioping -D -Y -WWW -s4k /download-970/
4 KiB >>> /download-970/ (  931.5 GiB): request=1 time=43.5 us (warmup)
4 KiB >>> /download-970/ (  931.5 GiB): request=2 time=6.65 ms
4 KiB >>> /download-970/ (  931.5 GiB): request=3 time=6.66 ms
4 KiB >>> /download-970/ (  931.5 GiB): request=4 time=6.60 ms
4 KiB >>> /download-970/ (  931.5 GiB): request=5 time=6.84 ms
4 KiB >>> /download-970/ (  931.5 GiB): request=6 time=6.66 ms
4 KiB >>> /download-970/ (  931.5 GiB): request=7 time=6.67 ms
4 KiB >>> /download-970/ (  931.5 GiB): request=8 time=6.61 ms
4 KiB >>> /download-970/ (  931.5 GiB): request=9 time=6.64 ms
4 KiB >>> /download-970/ (  931.5 GiB): request=10 time=6.59 ms
4 KiB >>> /download-970/ (  931.5 GiB): request=11 time=6.61 ms
4 KiB >>> /download-970/ (  931.5 GiB): request=12 time=6.64 ms
4 KiB >>> /download-970/ (  931.5 GiB): request=13 time=6.58 ms
^C
--- /download-970/ (  931.5 GiB) ioping statistics ---
12 requests completed in 79.8 ms, 48 KiB written, 150 iops, 601.9 KiB/s
generated 13 requests in 12.9 s, 52 KiB, 1 iops, 4.03 KiB/s
min/avg/max/mdev = 6.58 ms / 6.65 ms / 6.84 ms / 64.5 us

https://www.anandtech.com/show/16087/the-samsung-980-pro-pcie-4-ssd-review/6

The 980 PRO's sequential read performance doesn't saturate until around QD16: rather late in the test compared to most drives, but that's because high-end PCIe Gen3 drives have been hitting the host bandwidth limit at moderate queue depths. The 1TB 980 PRO does show decent performance scaling through the lower queue depths, taking it past the PCIe Gen3 limits by QD8. This is a clear improvement over the Phison E16-based Seagate FireCuda 520, which doesn't start gaining speed until after QD4.

will have to play with queue depth to see if it makes much of a difference.. anyways just noting some info as i come across stuff.