Request for a stricter limit on worst-case space-amplification

[mdcallag]

This is a feature request.

Expected behavior

With respect to worst-case space-amplification:

• Leveled compaction in theory has an implied limit of 1 + 1/plf where plf is the per-level fanout. But see There are too many write stalls because the write slowdown stage is frequently skipped #9423 to understand why in practice the limit is larger and the results below show that.
• Universal doesn't have a limit. There is max_size_amplification_percent but that isn't a limit, it just determines when major compaction is triggered. Universal can also benefit from level0_file_num_compaction_trigger but that is also not a strict limit in practice. ScyllaDB has an optional space-amp goal (SAG) for size-tiered.
• Integrated BlobDB has blob_garbage_collection_force_threshold and blob_garbage_collection_age_cutoff. While these have a strong impact on worst-case space-amp it isn't clear yet that they provide a strict limit.

Actual behavior

Full details were provided to the team (sorry, bad OSS behavior) and I used 2 workloads with db_bench

• in-memory - use fillseq to load 40M KV pairs, run readwhilewriting for 30 minutes, then run overwrite with no writer rate limit for 1 hour
• IO-bound - same as in-memory, except fillseq loads 800M KV pairs

Then I compared the size at the end of readrandom vs the max size of database directory during overwrite and I will list the ratio of max / initial size here for the in-memory workload.

• 2.0 for leveled
• 5.6 for universal
• 3.5 for blob with a naive config
• 23.2 for blob with a better config. This is too large and I need to revisit the problem to try and explain it.

The max/initial size ratio for the IO-bound workload using buffered IO and then O_DIRECT

• 1.2 and 1.4 for leveled
• 3.8 and failed for universal - by failed I mean the overwrite test failed when the 3T database directory became full. At that point the space-amp was ~6X
• 1.4 and 1.3 for blob with the naive config
• 1.7 and 1.7 for blob with the better config