Sprandom w/ FIO Q&A

[steven-sprouse-sndk]

Creating this area for Sprandom related questions.

[goodmancode]

Is there more info on setting an optimal spr_num_regions value? Should there be a minimum # regions per TB or should it primarily scale to the memory available in a test system? What are the benefits/tradeoffs of using more regions? Does preconditioning take longer with more regions?

[steven-sprouse-sndk]

TL;DR SSD's aggregate die or blocks in to larger "erase units". I good rule of thumb is spr_num_regions should be in the range of .5 - 2x the number of "erase units" in the drive. The number of regions should not have any effect on the preconditioning time.

Longer answer:
If you don't know the number of erase units in the drive I think setting spr_num_regions to a value of 1000 is a good number for most users. We set it lower to 200 or 400 during prototyping of the fio code, but that was just to have fewer data points to review.

The main benefit of increasing spr_num_regions is it reduces the amount of host DRAM fio needs to reserve for a datastructure to track where to issue invalidating writes. This structure's size is proportional to the amount of invalid space in the most invalid region. So increasing the number of regions reduces this space. The parameter is exposed to the user mainly for this purpose.

The other effect of increasing spr_num_regions is it better approximates the ideal curve that describes how OP is distributed within the drive. Sprandom samples this curve with discrete regions, so more regions improves the approximation. In practice, after a short time of running random write, the discrete "step" curve will smooth out and approach the ideal curve fairly quickly.

You also don't want to make spr_num_regions too high as each region will add some processing overhead.

[steven-sprouse-sndk]

Q:What is the definition of OP?
Is it 1-User capacity/physical capacity(LDPC excluded) or is it (physical capacity-user capacity)/user capacity-1?
The OP for newer drives potentially include quite a bit of spares or bad block reserves.
There are also XOR areas in which users are inaccessible in the total physical capacity.
If you could help with an example with the definition, that would be very helpful.

[steven-sprouse-sndk]

In the context of FIO Sprandom, we define Overprovisioning (OP) is as (physical_capacity - logical_capacity)/logical_capacity.
logical_capacity is the drive's exported capacity seen by the host.
physical_capacity is the portion of the SSD that is used to store host data. So for example, capacity reserved for XOR would not count as OP. Spare blocks may count towards OP if they are spare but actively used, but won't count if they are spare but are not used to store user data until activated.

[steven-sprouse-sndk]

Q: How accurate does the spr_op setting need to be to get good results?

[steven-sprouse-sndk]

The OP needs to be fairly accurate to get good results.

But one should err on the side of underestimating the OP.

We recommend a 2 step preconditioning process before running your random tests.
Step 1 is to run fio/sprandom followed by Step 2 which is a short, conventional random write.

If you monitor the write bandwidth during Step 1 and 2, you ideally want to see a high write throughput during all of Step 1, and then a short period of high write BW in Step 2 before it drops down to the steady state.

If you see a drop in write bandwidth in Step 1, it means your spr_op setting is too high, and you should repeat with a slightly lower value.
Then try to set the spr_op value so that the performance drop in Step 2 occurs as early as possible, but does not occur in Step 1.

[steven-sprouse-sndk]

@DataRexOn , I answered your question in a new thread below. If possible, can you delete your question above to keep this thread clean?

[steven-sprouse-sndk]

Q: We had a 30TB drive and used the following FIO script to run the test, it took about 60hrs to finish 1 drive write, is that right? Based on your presentation, the precondition should be done within several hours. What could be the issue? Do we have to use the exact OP setting? if OP is is a couple of % more or less, does this still work?

FIO: 3.41
CentOS 9 stream
kernel: 5.14.0-533.el9.x86_64

Here is the FIO script:

[global]
ioengine=libaio
rw=randwrite
bs=4096
blockalign=4096
direct=1
norandommap=1
iodepth=64
[preconditioning]
sprandom=1
spr_op=0.15
spr_num_regions=400

[steven-sprouse-sndk]

@DataRexOn There are a couple possibilities:

1. Most likely the drive's indirection unit (IU), which is the resolution of the drive's logical to physical translation table, is larger than 4KiB, and most likely 16KiB. So when you use a bs=4096 the drive is doing read-modify-writes which will be slower performance than the larger write.
2. Another possibility is that some drives do not optimize for small sequential writes and can have control path bottlenecks.

The solution for both of these cases is to the bs and ba settings to be 16k. You can also experiement with increasing these settings to 32k or 64k to see if you get even further performance improvements without impact on the final test measurements.

[DataRexOn]

Will give a try. Thank you.

[DataRexOn]

We tried the SPRANDOM FIO with different OP settings, we still feel that the pattern written to the medium is not fully randomized. I wonder whether it only works for SanDisk's OP partition architecture. Do you know any other companies are able to be successful with the SPRANDOM FIO?

[steven-sprouse-sndk]

@DataRexOn Can you be more specific about what behavior you observe to indicate that the medium is not fully randomized? I'd say at least 3 other vendors and a few customers have reported that Sprandom yielded preconditioning behavior that was good enough to adopt in to their test methodologies.

Can you also clarify what you mean by "SanDisk's OP partition architecture"? Sprandom should work with any single namespace format on a drive that uses a "greedy" garbage collection strategy.

That said, we have observed cases where there are other behaviors some drives have where certain data structures or features require a "warm up" period. This warm up period is provided by the long conventional precon process, but may not be warmed up properly during the Sprandom precon.

Sprandom solve's the problem of distributing OP within the drive with a particular pattern. But it won't be able to ensure that other drive or vendor unique behaviors are also initialized.

[jamesdoung]

Sometimes we don't know the exactly OP rate(--spr_op), if we set a approximated --spr_op value(for example we know a drive OP rate is 19.5%, we intentionally set --spr_op value is 1.20), so the test result still meaningful, right?
As you said, the --spr_op value was set too high, we would see performance drop during setp1. But for step2 result, we still can treat it as steady status performance, correct?
Thanks!

[anrahman4]

When setting up a fio jobfile to use sprandom preconditioning, what is the iodepth and numjobs that should be used?

Why is it recommended to do a short random write workload after the sprandom preconditioning, shouldn't the drop to steady state numbers occur during the sprandom preconditioning and not following it after? How long should this short random write be before you measure steady state?

Should the block size and rw values be set appropriately based off of the workload you are trying to precondition for? For example, if I want to do a 1M sequential workload, should I pass the bs=1M, with rw=write? Or if I were to do a 16k random workload, I would use bs=16k with rw=randwrite?

[steven-sprouse-sndk]

iodepth can be reasonable for your drive. Usually something in the QD32-QD128 range.
But numjobs must be 1.

The short random write workload after sprandom is there because sprandom is not exact in consuming all the OP in the drive. It is actually better to get close to the ideal validity distribution with sprandom and then let the short post ssprandom random write finish consuming the remaining OP. Also, some vendors will have small drive to drive variation in OP which sprandom cannot compensate for.

In general is good practice to use a bs that is the same as the test you are going to run with some caveats:

• Don't use a bs smaller than the drive's L2P table "indirection unit" size.
• The bs you use will depend a lot on the drive's internal design. You'll need to get feedback from your drive vendor. If you want to test random write you should always use rw-randwrite whether it is a large or small bs.
• Some drive's have cpu bottlenecks and cannot do a high IO rate with small bs. E.g. if sprandom can write at the sequential write speed, some drives that can give high sequential write with 128K commands cannot meet the same performance with 4K sprandom writes. The fix is to increase the bs to the smallest size that gets you high write bandwidth during sprandom.

[vincentkfu]

@steven-sprouse-sndk @tomas-winkler-sndk Currently SPRandom is constrained to work only with power of 2 block sizes.

sprandom_init() contains:

	if (!is_power_of_2(align_bs)) {
		log_err("fio: sprandom: bs [%"PRIu64"] should be power of 2",
			align_bs);
		return -EINVAL;
	}

The only location where I found this requirement was used was in calculating the device physical size.

static uint64_t sprandom_physical_size(double over_provisioning, uint64_t logical_sz,
				       uint64_t align_bs)
{
	uint64_t size;

	size = logical_sz + ceil((double)logical_sz * over_provisioning);
	return (size + (align_bs - 1)) & ~(align_bs - 1);
}

I can think of no reason why the general SPRandom approach should not work with block sizes that are not a power of 2. Do you agree?

We have users interested in running SPRandom on devices formatted in extended LBA DIF mode with 520-byte or 4160-byte sectors.

[steven-sprouse-sndk]

@vincentkfu , @tomas-winkler-sndk I agree, there is no reason to constrain the block sizes to a power of 2. The block size should be sized such that it spans and aligns to a in integer multiple of the drive's L2P indirection unit. But this is something that the user needs ensure.

[danshoudasiwusong]

I encountered the following issues while testing with SPRandom. Could you please help me with them? Thank you!

1：When filling the disk using the SPRandom algorithm, is it normal for the disk speed not to drop at all during the process?
2：After preprocessing with the SPRandom algorithm, how can I intuitively determine whether a disk has reached steady state?

fio result：WRITE: bw=465MiB/s (488MB/s), 465MiB/s-465MiB/s (488MB/s-488MB/s), io=3827GiB (4109GB), run=8425899-8425899msec.

FIO: 3.42
system：win10
capaticy：3.84TB
Firmware：V1027A0
Model： YSS25G1YD3T8EPNNA

[global]
direct=1
ioengine=windowsaio
bs=16k
iodepth=64
numjobs=1
randrepeat=1
randseed=12345
group_reporting
filename=\.\PHYSICALDRIVE2
size=100%
loops=1

[sprandom]
rw=randwrite
sprandom=1
spr_op=0.07
spr_num_regions=1000

[steven-sprouse-sndk]

@danshoudasiwusong Please see this response above: #1990 (reply in thread)

In the 2-step precon method we recommend we do not want the performance to drop during the sprandom phase. Performance should drop in the 2nd step where we run a true 100% random write. Increase the spr_op value so that the performance drop occurs soon after the 2nd step starts, but not so much that the drop occurs in the first step.

With sprandom preconditioning the drive should be in steady state as soon as the performance drops. But you'll have to monitor the bandwidth profile to see that it is stable. You may need to run a baseline conventional 100% random write test to determine that the sprandom method and conventional get the same results.