Use Env::IO_MID for the L0->L0 && L0->L1 compaction

[lhsoft]

Fixes #9918

Test Plan:

./db_bench --benchmarks "fillrandom,stats" -db ./db --compression_type None  --level0_file_num_compaction_trigger 4  --write_buffer_size 1048576 -benchmark_write_rate_limit 1048576 --max_bytes_for_level_base 10485760 --rate_limiter_bytes_per_sec 10485760 --level0_slowdown_writes_trigger 8 --max_bytes_for_level_multiplier 5 -num 3000000 --max_background_jobs 10

[ajkr]

It turns out not everyone has the same intuition on this so we need some data to show it is beneficial. For example, if you run db_bench -benchmarks=fillrandom with a fixed write rate and rate limiter bytes/sec, does this PR increase the duration until stall happens or reduce the amount of stalling?

[lhsoft]

@ajkr
I used following cmd and got the result that this PR has reduced 40% stalling time.

./db_bench --benchmarks "fillrandom,stats" -db ./db --compression_type None  --level0_file_num_compaction_trigger 4  --write_buffer_size 1048576 -benchmark_write_rate_limit 1048576 --max_bytes_for_level_base 10485760 --rate_limiter_bytes_per_sec 10485760 --level0_slowdown_writes_trigger 8 --max_bytes_for_level_multiplier 5 -num 3000000 --max_background_jobs 10

result with PR

** DB Stats **
Uptime(secs): 343.5 total, 340.4 interval
Cumulative writes: 3000K writes, 3000K keys, 3000K commit groups, 1.0 writes per commit group, ingest: 0.37 GB, 1.09 MB/s
Cumulative WAL: 3000K writes, 0 syncs, 3000000.00 writes per sync, written: 0.37 GB, 1.09 MB/s
Cumulative stall: 00:00:2.901 H:M:S, 0.8 percent
Interval writes: 2973K writes, 2973K keys, 2973K commit groups, 1.0 writes per commit group, ingest: 371.49 MB, 1.09 MB/s
Interval WAL: 2973K writes, 0 syncs, 2973592.00 writes per sync, written: 0.36 GB, 1.09 MB/s
Interval stall: 00:00:2.901 H:M:S, 0.9 percent

result without PR

** DB Stats **
Uptime(secs): 346.0 total, 342.9 interval
Cumulative writes: 3000K writes, 3000K keys, 3000K commit groups, 1.0 writes per commit group, ingest: 0.37 GB, 1.08 MB/s
Cumulative WAL: 3000K writes, 0 syncs, 3000000.00 writes per sync, written: 0.37 GB, 1.08 MB/s
Cumulative stall: 00:00:7.559 H:M:S, 2.2 percent
Interval writes: 2973K writes, 2973K keys, 2973K commit groups, 1.0 writes per commit group, ingest: 371.52 MB, 1.08 MB/s
Interval WAL: 2973K writes, 0 syncs, 2973786.00 writes per sync, written: 0.36 GB, 1.08 MB/s
Interval stall: 00:00:7.559 H:M:S, 2.2 percent

[lhsoft]

@ajkr Is there any more comments for this PR?:)

[ajkr]

I don't have more comments right now. We (RocksDB team) just need to find time to discuss/evaluate it more internally since this would be a change to behavior that's been there forever and affects many workloads.