Getting very inconsistent results between Vmware VM 2025 Datacenter Servers and 2019 Datacenter Servers

[Jerkysan]

I have been testing performance of my new vmware cluster with hosts running 2xAMD EPYC 9575F, 2x Dual Port 32GB FC, 1.5TB of RAM, and running NVME of FC to a AFF900 2x controller 2x shelves fully populated with NVME
VM - Both Fresh test machines spun up with 2019 and 2025 then updated.. They are running 2xVMWara paravirtual SCSI with thick formatted drives for this test. The reason for 2x is because the "test drive" has it's own controller.(the data below is paravirtual but I also tested the nvme controllers with similar results).

diskspd -c2G -b4K -d60 -o64 -t4 -r -L -w100 F:\testfile.dat

Running the above command on a 2025 results in nearly 1 second(1000ms) of latency being generated and it shows the drive being literally maxed at near 100% utilization. (I noticed it had high utilization even when updating windows). Also the MiB/s is extremely low at 12ish. I checked the hosts and it's queues(esxtop disk and vm) along with the Netapp and it's Queues/load and nothing is standing out at all. Same host, same nvme over FC datastore, and basically same everything except the OS. Vmware tools all up to date and latest updates on both. I performed NO TUNING on either and yes I know there is tuning I can do on the machines themselves but I'm attempting to get as close to stock results as possible. I put different controller cards in like nvme, lsi sas, and pvscsi.. They all result in the same issue with 2025. The machine isn't bound to any domain so it isn't a GPO or any
Literally just installed, vmware tools, updated, and then test for both the 2019 and 2025. As stated same host, same datastores, and same basic hardware configuration with only differing in the OS department.

I also thought "maybe this is the new hosts or something" so I moved it all back to the old hosts we still have running and same results albit a bit slower on the 2019 because it is FC-scsi instead of NVME of FC(also didn't test the nvme controller with 2019 there).
2025

Write IO
thread |       bytes     |     I/Os     |    MiB/s   |  I/O per s |  AvgLat  | LatStdDev |  file

     0 |       184086528 |        44943 |       2.93 |     749.05 |   91.485 |   924.938 | F:\testfile.dat (2GiB)
     1 |       189886464 |        46359 |       3.02 |     772.65 |   88.704 |   910.875 | F:\testfile.dat (2GiB)
     2 |       194134016 |        47396 |       3.09 |     789.93 |   86.758 |   900.958 | F:\testfile.dat (2GiB)
     3 |       197582848 |        48238 |       3.14 |     803.96 |   85.258 |   893.158 | F:\testfile.dat (2GiB)

total:         765689856 |       186936 |      12.17 |    3115.59 |   87.990 |   907.250
 

Total latency distribution:
  %-ile |  Read (ms) | Write (ms) | Total (ms)

    min |        N/A |      0.002 |      0.002
   25th |        N/A |     36.873 |     36.873
   50th |        N/A |     48.215 |     48.215
   75th |        N/A |     57.664 |     57.664
   90th |        N/A |     73.541 |     73.541
   95th |        N/A |    114.973 |    114.973
   99th |        N/A |    335.611 |    335.611
3-nines |        N/A |  24534.166 |  24534.166
4-nines |        N/A |  24538.762 |  24538.762
5-nines |        N/A |  24539.043 |  24539.043
6-nines |        N/A |  24539.059 |  24539.059
7-nines |        N/A |  24539.059 |  24539.059
8-nines |        N/A |  24539.059 |  24539.059
9-nines |        N/A |  24539.059 |  24539.059
    max |        N/A |  24539.059 |  24539.059

2019

Write IO
thread |       bytes     |     I/Os     |    MiB/s   |  I/O per s |  AvgLat  | LatStdDev |  file

     0 |     30943436800 |      7554550 |     491.70 |  125876.44 |    0.008 |     0.014 | F:\testfile.dat (2GiB)
     1 |     30847483904 |      7531124 |     490.18 |  125486.11 |    0.008 |     0.013 | F:\testfile.dat (2GiB)
     2 |     31022792704 |      7573924 |     492.97 |  126199.25 |    0.008 |     0.013 | F:\testfile.dat (2GiB)
     3 |     31056547840 |      7582165 |     493.50 |  126336.57 |    0.008 |     0.013 | F:\testfile.dat (2GiB)

total:      123870261248 |     30241763 |    1968.35 |  503898.36 |    0.008 |     0.013
 

Total latency distribution:
  %-ile |  Read (ms) | Write (ms) | Total (ms)

    min |        N/A |      0.002 |      0.002
   25th |        N/A |      0.004 |      0.004
   50th |        N/A |      0.006 |      0.006
   75th |        N/A |      0.009 |      0.009
   90th |        N/A |      0.014 |      0.014
   95th |        N/A |      0.018 |      0.018
   99th |        N/A |      0.036 |      0.036
3-nines |        N/A |      0.127 |      0.127
4-nines |        N/A |      0.504 |      0.504
5-nines |        N/A |      1.100 |      1.100
6-nines |        N/A |      3.019 |      3.019
7-nines |        N/A |      6.529 |      6.529
8-nines |        N/A |      9.798 |      9.798
9-nines |        N/A |      9.798 |      9.798
    max |        N/A |      9.798 |      9.798

Trying with Crystal Disk I get expected results(though crystal disk does q32 and 16 threads and changing the settings yields the same results) so I'm assuming it might be an issue with diskspd..

2019

2025

[dl2n]

Trivia: CrystalDiskMark is actually using DISKSPD for its results. If you look in its installation directory, I believe its 2.0.18 (old but OK).

Are any warning/info messages shown when you launch your runs? What are you looking at when you say "nearly 1 second(1000ms) of latency being generated"?

Utilization is somewhat odd as a statistic. Its actually just average queue depth (x100), which itself simply states the aggregate latency accumulated by IOs over the sample period (say, a second). If one IO takes 1s, that's 1 aqd. If 1000 IOs take 1ms/ea, that's also 1 aqd. Most modern devices (esp. SSDs) can process in parallel. So "100% utilization" is just saying that there's at least one thing executing at any given time, which is actually on the low side of being busy for modern devices.

Task Manager may be clipping it at 100%, I'd have to check, but regardless its a detail. There is absolutely no notion of 100% being a true max, though.

[Jerkysan]

Trivia: CrystalDiskMark is actually using DISKSPD for its results. If you look in its installation directory, I believe its 2.0.18 (old but OK).

Are any warning/info messages shown when you launch your runs? What are you looking at when you say "nearly 1 second(1000ms) of latency being generated"?

Utilization is somewhat odd as a statistic. Its actually just average queue depth (x100), which itself simply states the aggregate latency accumulated by IOs over the sample period (say, a second). If one IO takes 1s, that's 1 aqd. If 1000 IOs take 1ms/ea, that's also 1 aqd. Most modern devices (esp. SSDs) can process in parallel. So "100% utilization" is just saying that there's at least one thing executing at any given time, which is actually on the low side of being busy for modern devices.

Task Manager may be clipping it at 100%, I'd have to check, but regardless its a detail. There is absolutely no notion of 100% being a true max, though.

Are any warning/info messages shown when you launch your runs?
There are no errors/warnings/info besides the standard readout at the end of the test that shows all the settings and what the configuration of the machine was at the time.

What are you looking at when you say "nearly 1 second(1000ms) of latency being generated"?
As I showed in the readouts diskspd shows 907.250 LatStdDev. Taskmanager shows similar latency on the disk while the job as running as seen in the picture I also posted of 979ms.

Task Manager may be clipping it at 100%, I'd have to check, but regardless its a detail. There is absolutely no notion of 100% being a true max, though.

It never hits 100% but around 90% utilization.

[dl2n]

Right, there are some things that now jump out.

One, LatStdDev is the standard deviation of the latencies. Its not the latency, its a measure of spread. This is huge.

I'd suggest looking at the Physical Disk performance counters (perfmon). These will be clearer, and I may take a look at what TM is presenting. What you're seeing seems like a data problem.

Your initial base cmd:

diskspd -c2G -b4K -d60 -o64 -t4 -r -L -w100 F:\testfile.dat

This is doing cached IO through the filesystem. Its not what CDM is doing with diskspd, that would be -Sh / -Suw (these are equivalent): unbuffered write-through IO.

It also re-creates the file each time. This tears down the cache and exposes however the lower stack may or may not be also caching, the filesystem potentially reallocating to the same or a totally different range of the F: device, etc. This is a usual source of strange results. I'd suggest creating the file once, writing it through, and then reusing the file repeatedly for measured results.

Regardless of those two points, what's strange about your results even there is that it looks like the slow one managed to issue 64 IOs below (0.091485 s * 749.05 IO/s = 68.5 IOs, which is ~= -o64). This isn't normal for cached IO in a single thread. Its usually synchronous.

The second case is synchronous. One tell is the 8us latency (0.008 ms). 125876 IO/s * 0.000008 s =1.007 IO outstanding on average, i.e. the file was fully cached and it was just reading out of the cache immediately and returning. Maybe there were a few faults, maybe your lower storage subsystem was also caching the range of the logical device and was able to return that fast so the cache would get re-primed quickly. This is another way -c may be confounding things.

This doesn't explain 91ms cached IOs. I'd like to see the PD counters show the actual average queue depth at that layer. Is it 64? Is it a lot lower? If Windows Defender is active, can we exempt F:? Is that a difference between the two?

Regardless you should switch to unbuffered writethrough, I don't think cached is relevant to what you're after.