MyRocks: Bulk load requires an insane number of file descriptors

[RickPizzi]

Hi,

still trying to load (using myloader) a dump from a large database (830+ tables), we see that MyRocks is eating an insane amount of file descriptors. We were unable to complete the load with 25,000 open files limit! Now we are retrying with 10x that value to see if it arrives to completion, however, this just doesn't seem right. Maybe a file descriptors leak?

Here's the relevant config.

Thanks,

Rick

rocksdb
skip-innodb
default-storage-engine=rocksdb
default-tmp-storage-engine=MyISAM
collation-server=latin1_bin
binlog_format = ROW

rocksdb_max_open_files=-1
rocksdb_base_background_compactions=1
rocksdb_max_total_wal_size=4G
rocksdb_block_size=16384
rocksdb_block_cache_size=16G
rocksdb_table_cache_numshardbits=6
rocksdb_skip_unique_check=1
rocksdb_commit_in_the_middle=1
rocksdb_write_disable_wal=1
rocksdb_max_background_flushes=40
rocksdb_max_background_compactions=40
rocksdb_default_cf_options=write_buffer_size=128m;level0_file_num_compaction_trigger=4;level0_slowdown_writes_trigger=256;level0_stop_writes_trigger=256;max_write_buffer_number=16;memtable=vector:1024

[yoshinorim]

Please try the followings.

• Add "target_file_size_base=32m;max_bytes_for_level_base=512m;" into rocksdb_default_cf_options parameter. Default sst file size is 4MB, which is generally too small. Increasing to 32MB will reduce the number of sst files by 1/8. See https://github.com/facebook/mysql-5.6/wiki/my.cnf-tuning for details.
• Add rocksdb_override_cf_options=system={memtable=skip_list:16} parameter in my.cnf. vector memtable is not efficient for reads. Note that we're working on more efficient bulk loading. Once it's ready, you won't need to set vector memtable at all. Will update wiki once it's done.

[mdcallag]

Using his my.cnf and building from HEAD today. Not sure bug reporter used the same source

1. target_file_size_base and max_bytes_for_level_base aren't printed in SHOW GLOBAL VARIABLE. Wile file task for that.

2. target_file_size_base default is 64MB, L1 size default is 256M.
   see https://github.com/facebook/rocksdb/blob/master/util/options.cc#L54

2016/10/21-11:22:44.089431 7f26046ab940 Options.target_file_size_base: 67108864
2016/10/21-11:22:44.089432 7f26046ab940 Options.target_file_size_multiplier: 1
2016/10/21-11:22:44.089432 7f26046ab940 Options.max_bytes_for_level_base: 268435456

[mdcallag]

Defaults changed in facebook/rocksdb@2feafa3

[yoshinorim]

@RickPizzi Could you paste "SHOW ENGINE ROCKSDB STATUS" output?

[RickPizzi]

Please note that I have used two different configurations for bulk load and for normal use.
For bulk load I used the config above, which I have taken from your wiki page. The bulk load succeeded (even if I had very long stalls, as I described in a separate issue). Then after the myloader process completed I have shut down and have restarted with your recommended parameters:

rocksdb_max_open_files=-1
rocksdb_base_background_compactions=1
rocksdb_max_background_compactions=8
rocksdb_max_total_wal_size=4G
rocksdb_max_background_flushes=4
rocksdb_block_size=16384
rocksdb_block_cache_size=16G
rocksdb_table_cache_numshardbits=6
rocksdb_default_cf_options=write_buffer_size=128m;target_file_size_base=32m;max_bytes_for_level_base=512m;level0_file_num_compaction_trigger=4;level0_slowdown_writes_trigger=10;level0_stop_writes_trigger=15;max_write_buffer_number=4;compression_per_level=kNoCompression:kNoCompression:kNoCompression:kZlibCompression:kZlibCompression:kZlibCompression;bottommost_compression=kZlibCompression;compression_opts=-14:1:0;block_based_table_factory={cache_index_and_filter_blocks=1;filter_policy=bloomfilter:10:false;whole_key_filtering=1};level_compaction_dynamic_level_bytes=true;optimize_filters_for_hits=true

Two notes here:

• with 64 GB of system RAM I had to reduce block cache size from 32GB to 16GB to avoid OOM killer kicking in
• with the recommended options for normal use (the ones I am running right now and that you can see above) the bulk load crashed, see separate issue

I have just checked, the 2.3 TB InnoDB dataset size is occupying 2.0 TB in RocksDB, and the server currently has 34,160 file descriptors open. I expected some more space saving here. By the way, some of the InnoDB tables are compressed in InnoDB but I removed compression when importing in RocksDB, while some other have blobs with compressed content (typically zlib) and these were imported unchanged.

Since I am going to retry the bulk load, if you want me to try a different bulk load setting just please tell me what to try.

Output of SHOW ENGINE (with above settings in effect) follows. Thank you!

mysql> show engine rocksdb status\G
*************************** 1. row ***************************
Type: DBSTATS
Name: rocksdb
Status:
** DB Stats **
Uptime(secs): 36754.5 total, 36120.0 interval
Cumulative writes: 5 writes, 1366 keys, 5 commit groups, 0.8 writes per commit group, ingest: 0.00 GB, 0.00 MB/s
Cumulative WAL: 5 writes, 1 syncs, 2.50 writes per sync, written: 0.00 GB, 0.00 MB/s
Cumulative stall: 00:12:43.342 H:M:S, 2.1 percent
Interval writes: 3 writes, 1366 keys, 3 commit groups, 0.8 writes per commit group, ingest: 0.14 MB, 0.00 MB/s
Interval WAL: 3 writes, 0 syncs, 3.00 writes per sync, written: 0.00 MB, 0.00 MB/s
Interval stall: 00:12:43.342 H:M:S, 2.1 percent
** Level 0 read latency histogram (micros):
Count: 5 Average: 138.8000 StdDev: 270.12
Min: 1 Median: 3.5000 Max: 679

Percentiles: P50: 3.50 P75: 8.75 P99: 679.00 P99.9: 679.00 P99.99: 679.00

[ 0, 1 ) 2 40.000% 40.000% ########
[ 3, 4 ) 1 20.000% 60.000% ####
[ 8, 9 ) 1 20.000% 80.000% ####
[ 600, 700 ) 1 20.000% 100.000% ####

** Level 1 read latency histogram (micros):
Count: 12326 Average: 4.7115 StdDev: 1.46
Min: 0 Median: 4.0540 Max: 52

Percentiles: P50: 4.05 P75: 4.62 P99: 7.73 P99.9: 23.51 P99.99: 38.84

[ 0, 1 ) 11 0.089% 0.089%
[ 1, 2 ) 3 0.024% 0.114%
[ 2, 3 ) 95 0.771% 0.884%
[ 3, 4 ) 5761 46.739% 47.623% #########
[ 4, 5 ) 5430 44.053% 91.676% #########
[ 5, 6 ) 653 5.298% 96.974% #
[ 6, 7 ) 197 1.598% 98.572%
[ 7, 8 ) 72 0.584% 99.156%
[ 8, 9 ) 27 0.219% 99.375%
[ 9, 10 ) 9 0.073% 99.448%
[ 10, 12 ) 8 0.065% 99.513%
[ 12, 14 ) 4 0.032% 99.546%
[ 14, 16 ) 2 0.016% 99.562%
[ 18, 20 ) 22 0.178% 99.740%
[ 20, 25 ) 28 0.227% 99.968%
[ 25, 30 ) 2 0.016% 99.984%
[ 35, 40 ) 1 0.008% 99.992%
[ 50, 60 ) 1 0.008% 100.000%

** Level 2 read latency histogram (micros):
Count: 73049 Average: 53.1012 StdDev: 870.21
Min: 0 Median: 3.9673 Max: 66686

Percentiles: P50: 3.97 P75: 5.76 P99: 97.66 P99.9: 13722.82 P99.99: 35868.87

[ 0, 1 ) 588 0.805% 0.805%
[ 1, 2 ) 1564 2.141% 2.946%
[ 2, 3 ) 8656 11.850% 14.796% ##
[ 3, 4 ) 26585 36.393% 51.189% #######
[ 4, 5 ) 14331 19.618% 70.807% ####
[ 5, 6 ) 4017 5.499% 76.306% #
[ 6, 7 ) 1537 2.104% 78.410%
[ 7, 8 ) 997 1.365% 79.775%
[ 8, 9 ) 777 1.064% 80.839%
[ 9, 10 ) 577 0.790% 81.629%
[ 10, 12 ) 790 1.081% 82.710%
[ 12, 14 ) 499 0.683% 83.393%
[ 14, 16 ) 297 0.407% 83.800%
[ 16, 18 ) 195 0.267% 84.067%
[ 18, 20 ) 221 0.303% 84.369%
[ 20, 25 ) 332 0.454% 84.824%
[ 25, 30 ) 1213 1.661% 86.484%
[ 30, 35 ) 2701 3.698% 90.182% #
[ 35, 40 ) 2076 2.842% 93.024% #
[ 40, 45 ) 1644 2.251% 95.274%
[ 45, 50 ) 1132 1.550% 96.824%
[ 50, 60 ) 1020 1.396% 98.220%
[ 60, 70 ) 261 0.357% 98.578%
[ 70, 80 ) 101 0.138% 98.716%
[ 80, 90 ) 72 0.099% 98.814%
[ 90, 100 ) 177 0.242% 99.057%
[ 100, 120 ) 231 0.316% 99.373%
[ 120, 140 ) 60 0.082% 99.455%
[ 140, 160 ) 36 0.049% 99.504%
[ 160, 180 ) 29 0.040% 99.544%
[ 180, 200 ) 12 0.016% 99.561%
[ 200, 250 ) 18 0.025% 99.585%
[ 250, 300 ) 7 0.010% 99.595%
[ 300, 350 ) 4 0.005% 99.600%
[ 350, 400 ) 1 0.001% 99.602%
[ 400, 450 ) 5 0.007% 99.608%
[ 450, 500 ) 5 0.007% 99.615%
[ 500, 600 ) 2 0.003% 99.618%
[ 600, 700 ) 3 0.004% 99.622%
[ 700, 800 ) 3 0.004% 99.626%
[ 800, 900 ) 2 0.003% 99.629%
[ 900, 1000 ) 1 0.001% 99.630%
[ 1000, 1200 ) 1 0.001% 99.632%
[ 1200, 1400 ) 2 0.003% 99.634%
[ 1400, 1600 ) 3 0.004% 99.639%
[ 1600, 1800 ) 2 0.003% 99.641%
[ 1800, 2000 ) 4 0.005% 99.647%
[ 2000, 2500 ) 5 0.007% 99.654%
[ 2500, 3000 ) 8 0.011% 99.665%
[ 3000, 3500 ) 4 0.005% 99.670%
[ 3500, 4000 ) 9 0.012% 99.682%
[ 4000, 4500 ) 5 0.007% 99.689%
[ 4500, 5000 ) 11 0.015% 99.704%
[ 5000, 6000 ) 12 0.016% 99.721%
[ 6000, 7000 ) 21 0.029% 99.749%
[ 7000, 8000 ) 17 0.023% 99.773%
[ 8000, 9000 ) 30 0.041% 99.814%
[ 9000, 10000 ) 18 0.025% 99.838%
[ 10000, 12000 ) 26 0.036% 99.874%
[ 12000, 14000 ) 22 0.030% 99.904%
[ 14000, 16000 ) 11 0.015% 99.919%
[ 16000, 18000 ) 15 0.021% 99.940%
[ 18000, 20000 ) 13 0.018% 99.958%
[ 20000, 25000 ) 15 0.021% 99.978%
[ 25000, 30000 ) 6 0.008% 99.986%
[ 30000, 35000 ) 2 0.003% 99.989%
[ 35000, 40000 ) 4 0.005% 99.995%
[ 40000, 45000 ) 2 0.003% 99.997%
[ 50000, 60000 ) 1 0.001% 99.999%
[ 60000, 70000 ) 1 0.001% 100.000%

** Level 3 read latency histogram (micros):
Count: 177611 Average: 110.1864 StdDev: 1599.17
Min: 0 Median: 3.6821 Max: 132639

Percentiles: P50: 3.68 P75: 4.87 P99: 126.74 P99.9: 21697.54 P99.99: 66238.90

[ 0, 1 ) 1224 0.689% 0.689%
[ 1, 2 ) 7701 4.336% 5.025% #
[ 2, 3 ) 38944 21.927% 26.952% ####
[ 3, 4 ) 60014 33.790% 60.741% #######
[ 4, 5 ) 29246 16.466% 77.207% ###
[ 5, 6 ) 5355 3.015% 80.223% #
[ 6, 7 ) 2170 1.222% 81.444%
[ 7, 8 ) 1544 0.869% 82.314%
[ 8, 9 ) 1099 0.619% 82.932%
[ 9, 10 ) 780 0.439% 83.372%
[ 10, 12 ) 1216 0.685% 84.056%
[ 12, 14 ) 690 0.388% 84.445%
[ 14, 16 ) 386 0.217% 84.662%
[ 16, 18 ) 245 0.138% 84.800%
[ 18, 20 ) 296 0.167% 84.967%
[ 20, 25 ) 542 0.305% 85.272%
[ 25, 30 ) 5201 2.928% 88.200% #
[ 30, 35 ) 8419 4.740% 92.940% #
[ 35, 40 ) 3832 2.158% 95.098%
[ 40, 45 ) 2418 1.361% 96.459%
[ 45, 50 ) 1630 0.918% 97.377%
[ 50, 60 ) 1434 0.807% 98.184%
[ 60, 70 ) 456 0.257% 98.441%
[ 70, 80 ) 146 0.082% 98.523%
[ 80, 90 ) 81 0.046% 98.569%
[ 90, 100 ) 303 0.171% 98.739%
[ 100, 120 ) 414 0.233% 98.972%
[ 120, 140 ) 145 0.082% 99.054%
[ 140, 160 ) 108 0.061% 99.115%
[ 160, 180 ) 63 0.035% 99.150%
[ 180, 200 ) 27 0.015% 99.166%
[ 200, 250 ) 35 0.020% 99.185%
[ 250, 300 ) 12 0.007% 99.192%
[ 300, 350 ) 9 0.005% 99.197%
[ 350, 400 ) 10 0.006% 99.203%
[ 400, 450 ) 7 0.004% 99.207%
[ 450, 500 ) 8 0.005% 99.211%
[ 500, 600 ) 11 0.006% 99.217%
[ 600, 700 ) 8 0.005% 99.222%
[ 700, 800 ) 7 0.004% 99.226%
[ 800, 900 ) 4 0.002% 99.228%
[ 900, 1000 ) 9 0.005% 99.233%
[ 1000, 1200 ) 10 0.006% 99.239%
[ 1200, 1400 ) 6 0.003% 99.242%
[ 1400, 1600 ) 6 0.003% 99.246%
[ 1600, 1800 ) 11 0.006% 99.252%
[ 1800, 2000 ) 5 0.003% 99.255%
[ 2000, 2500 ) 26 0.015% 99.269%
[ 2500, 3000 ) 17 0.010% 99.279%
[ 3000, 3500 ) 23 0.013% 99.292%
[ 3500, 4000 ) 29 0.016% 99.308%
[ 4000, 4500 ) 30 0.017% 99.325%
[ 4500, 5000 ) 29 0.016% 99.341%
[ 5000, 6000 ) 77 0.043% 99.385%
[ 6000, 7000 ) 78 0.044% 99.429%
[ 7000, 8000 ) 145 0.082% 99.510%
[ 8000, 9000 ) 221 0.124% 99.635%
[ 9000, 10000 ) 70 0.039% 99.674%
[ 10000, 12000 ) 125 0.070% 99.744%
[ 12000, 14000 ) 61 0.034% 99.779%
[ 14000, 16000 ) 52 0.029% 99.808%
[ 16000, 18000 ) 111 0.062% 99.871%
[ 18000, 20000 ) 31 0.017% 99.888%
[ 20000, 25000 ) 63 0.035% 99.923%
[ 25000, 30000 ) 33 0.019% 99.942%
[ 30000, 35000 ) 32 0.018% 99.960%
[ 35000, 40000 ) 14 0.008% 99.968%
[ 40000, 45000 ) 12 0.007% 99.975%
[ 45000, 50000 ) 9 0.005% 99.980%
[ 50000, 60000 ) 12 0.007% 99.986%
[ 60000, 70000 ) 10 0.006% 99.992%
[ 70000, 80000 ) 7 0.004% 99.996%
[ 80000, 90000 ) 4 0.002% 99.998%
[ 100000, 120000 ) 2 0.001% 99.999%
[ 120000, 140000 ) 1 0.001% 100.000%

** Level 4 read latency histogram (micros):
Count: 2927839 Average: 206.4811 StdDev: 865.60
Min: 0 Median: 6.2307 Max: 466494

Percentiles: P50: 6.23 P75: 33.68 P99: 5398.18 P99.9: 33839.59 P99.99: 77159.11

[ 0, 1 ) 11180 0.382% 0.382%
[ 1, 2 ) 73702 2.517% 2.899% #
[ 2, 3 ) 405739 13.858% 16.757% ###
[ 3, 4 ) 472723 16.146% 32.903% ###
[ 4, 5 ) 300210 10.254% 43.157% ##
[ 5, 6 ) 173479 5.925% 49.082% #
[ 6, 7 ) 116567 3.981% 53.063% #
[ 7, 8 ) 88555 3.025% 56.088% #
[ 8, 9 ) 67781 2.315% 58.403%
[ 9, 10 ) 50615 1.729% 60.131%
[ 10, 12 ) 65942 2.252% 62.384%
[ 12, 14 ) 38269 1.307% 63.691%
[ 14, 16 ) 21545 0.736% 64.427%
[ 16, 18 ) 11430 0.390% 64.817%
[ 18, 20 ) 7724 0.264% 65.081%
[ 20, 25 ) 14508 0.496% 65.576%
[ 25, 30 ) 106935 3.652% 69.229% #
[ 30, 35 ) 229547 7.840% 77.069% ##
[ 35, 40 ) 195102 6.664% 83.733% #
[ 40, 45 ) 141535 4.834% 88.567% #
[ 45, 50 ) 94782 3.237% 91.804% #
[ 50, 60 ) 87980 3.005% 94.809% #
[ 60, 70 ) 28163 0.962% 95.771%
[ 70, 80 ) 9899 0.338% 96.109%
[ 80, 90 ) 6498 0.222% 96.331%
[ 90, 100 ) 19342 0.661% 96.991%
[ 100, 120 ) 23905 0.816% 97.808%
[ 120, 140 ) 7140 0.244% 98.052%
[ 140, 160 ) 4785 0.163% 98.215%
[ 160, 180 ) 2973 0.102% 98.317%
[ 180, 200 ) 1664 0.057% 98.374%
[ 200, 250 ) 2748 0.094% 98.467%
[ 250, 300 ) 1269 0.043% 98.511%
[ 300, 350 ) 714 0.024% 98.535%
[ 350, 400 ) 583 0.020% 98.555%
[ 400, 450 ) 530 0.018% 98.573%
[ 450, 500 ) 648 0.022% 98.595%
[ 500, 600 ) 1134 0.039% 98.634%
[ 600, 700 ) 508 0.017% 98.651%
[ 700, 800 ) 445 0.015% 98.667%
[ 800, 900 ) 380 0.013% 98.680%
[ 900, 1000 ) 312 0.011% 98.690%
[ 1000, 1200 ) 600 0.020% 98.711%
[ 1200, 1400 ) 503 0.017% 98.728%
[ 1400, 1600 ) 447 0.015% 98.743%
[ 1600, 1800 ) 420 0.014% 98.757%
[ 1800, 2000 ) 440 0.015% 98.773%
[ 2000, 2500 ) 992 0.034% 98.806%
[ 2500, 3000 ) 948 0.032% 98.839%
[ 3000, 3500 ) 902 0.031% 98.870%
[ 3500, 4000 ) 927 0.032% 98.901%
[ 4000, 4500 ) 980 0.033% 98.935%
[ 4500, 5000 ) 1040 0.036% 98.970%
[ 5000, 6000 ) 2189 0.075% 99.045%
[ 6000, 7000 ) 2398 0.082% 99.127%
[ 7000, 8000 ) 2528 0.086% 99.213%
[ 8000, 9000 ) 3224 0.110% 99.323%
[ 9000, 10000 ) 1827 0.062% 99.386%
[ 10000, 12000 ) 3241 0.111% 99.496%
[ 12000, 14000 ) 2302 0.079% 99.575%
[ 14000, 16000 ) 1871 0.064% 99.639%
[ 16000, 18000 ) 1709 0.058% 99.697%
[ 18000, 20000 ) 1147 0.039% 99.737%
[ 20000, 25000 ) 2445 0.084% 99.820%
[ 25000, 30000 ) 1445 0.049% 99.869%
[ 30000, 35000 ) 1167 0.040% 99.909%
[ 35000, 40000 ) 726 0.025% 99.934%
[ 40000, 45000 ) 532 0.018% 99.952%
[ 45000, 50000 ) 368 0.013% 99.965%
[ 50000, 60000 ) 410 0.014% 99.979%
[ 60000, 70000 ) 233 0.008% 99.987%
[ 70000, 80000 ) 133 0.005% 99.991%
[ 80000, 90000 ) 85 0.003% 99.994%
[ 90000, 100000 ) 28 0.001% 99.995%
[ 100000, 120000 ) 40 0.001% 99.997%
[ 120000, 140000 ) 15 0.001% 99.997%
[ 140000, 160000 ) 7 0.000% 99.997%
[ 160000, 180000 ) 7 0.000% 99.998%
[ 180000, 200000 ) 1 0.000% 99.998%
[ 200000, 250000 ) 11 0.000% 99.998%
[ 250000, 300000 ) 24 0.001% 99.999%
[ 300000, 350000 ) 18 0.001% 99.999%
[ 350000, 400000 ) 16 0.001% 100.000%
[ 400000, 450000 ) 2 0.000% 100.000%
[ 450000, 500000 ) 1 0.000% 100.000%

** Level 5 read latency histogram (micros):
Count: 1874930 Average: 524.1551 StdDev: 2002.65
Min: 0 Median: 5.7513 Max: 420122

Percentiles: P50: 5.75 P75: 33.94 P99: 15313.78 P99.9: 46624.96 P99.99: 91418.96

[ 0, 1 ) 70777 3.775% 3.775% #
[ 1, 2 ) 93693 4.997% 8.772% #
[ 2, 3 ) 232467 12.399% 21.171% ##
[ 3, 4 ) 274506 14.641% 35.812% ###
[ 4, 5 ) 187411 9.996% 45.807% ##
[ 5, 6 ) 104637 5.581% 51.388% #
[ 6, 7 ) 68514 3.654% 55.042% #
[ 7, 8 ) 52045 2.776% 57.818% #
[ 8, 9 ) 40419 2.156% 59.974%
[ 9, 10 ) 30419 1.622% 61.596%
[ 10, 12 ) 40133 2.141% 63.737%
[ 12, 14 ) 23745 1.266% 65.003%
[ 14, 16 ) 13349 0.712% 65.715%
[ 16, 18 ) 7306 0.390% 66.105%
[ 18, 20 ) 4638 0.247% 66.352%
[ 20, 25 ) 9010 0.481% 66.833%
[ 25, 30 ) 54632 2.914% 69.747% #
[ 30, 35 ) 125126 6.674% 76.420% #
[ 35, 40 ) 110140 5.874% 82.295% #
[ 40, 45 ) 82516 4.401% 86.696% #
[ 45, 50 ) 56481 3.012% 89.708% #
[ 50, 60 ) 54490 2.906% 92.614% #
[ 60, 70 ) 18090 0.965% 93.579%
[ 70, 80 ) 6565 0.350% 93.929%
[ 80, 90 ) 3633 0.194% 94.123%
[ 90, 100 ) 10698 0.571% 94.694%
[ 100, 120 ) 13678 0.730% 95.423%
[ 120, 140 ) 4077 0.217% 95.641%
[ 140, 160 ) 2566 0.137% 95.777%
[ 160, 180 ) 1674 0.089% 95.867%
[ 180, 200 ) 1048 0.056% 95.923%
[ 200, 250 ) 1671 0.089% 96.012%
[ 250, 300 ) 833 0.044% 96.056%
[ 300, 350 ) 477 0.025% 96.082%
[ 350, 400 ) 433 0.023% 96.105%
[ 400, 450 ) 453 0.024% 96.129%
[ 450, 500 ) 586 0.031% 96.160%
[ 500, 600 ) 925 0.049% 96.210%
[ 600, 700 ) 440 0.023% 96.233%
[ 700, 800 ) 384 0.020% 96.253%
[ 800, 900 ) 314 0.017% 96.270%
[ 900, 1000 ) 300 0.016% 96.286%
[ 1000, 1200 ) 480 0.026% 96.312%
[ 1200, 1400 ) 416 0.022% 96.334%
[ 1400, 1600 ) 367 0.020% 96.354%
[ 1600, 1800 ) 358 0.019% 96.373%
[ 1800, 2000 ) 358 0.019% 96.392%
[ 2000, 2500 ) 958 0.051% 96.443%
[ 2500, 3000 ) 1038 0.055% 96.498%
[ 3000, 3500 ) 1136 0.061% 96.559%
[ 3500, 4000 ) 1243 0.066% 96.625%
[ 4000, 4500 ) 1398 0.075% 96.700%
[ 4500, 5000 ) 1494 0.080% 96.779%
[ 5000, 6000 ) 3195 0.170% 96.950%
[ 6000, 7000 ) 3615 0.193% 97.143%
[ 7000, 8000 ) 11123 0.593% 97.736%
[ 8000, 9000 ) 7618 0.406% 98.142%
[ 9000, 10000 ) 3517 0.188% 98.330%
[ 10000, 12000 ) 5917 0.316% 98.645%
[ 12000, 14000 ) 3943 0.210% 98.856%
[ 14000, 16000 ) 4122 0.220% 99.075%
[ 16000, 18000 ) 3625 0.193% 99.269%
[ 18000, 20000 ) 1760 0.094% 99.363%
[ 20000, 25000 ) 4170 0.222% 99.585%
[ 25000, 30000 ) 2091 0.112% 99.697%
[ 30000, 35000 ) 1842 0.098% 99.795%
[ 35000, 40000 ) 838 0.045% 99.840%
[ 40000, 45000 ) 942 0.050% 99.890%
[ 45000, 50000 ) 591 0.032% 99.921%
[ 50000, 60000 ) 620 0.033% 99.954%
[ 60000, 70000 ) 356 0.019% 99.973%
[ 70000, 80000 ) 198 0.011% 99.984%
[ 80000, 90000 ) 105 0.006% 99.989%
[ 90000, 100000 ) 67 0.004% 99.993%
[ 100000, 120000 ) 48 0.003% 99.996%
[ 120000, 140000 ) 19 0.001% 99.997%
[ 140000, 160000 ) 5 0.000% 99.997%
[ 160000, 180000 ) 4 0.000% 99.997%
[ 180000, 200000 ) 8 0.000% 99.998%
[ 200000, 250000 ) 4 0.000% 99.998%
[ 250000, 300000 ) 13 0.001% 99.998%
[ 300000, 350000 ) 14 0.001% 99.999%
[ 350000, 400000 ) 11 0.001% 100.000%
[ 400000, 450000 ) 4 0.000% 100.000%

*************************** 2. row ***************************
Type: CF_COMPACTION
Name: system
Status:
** Compaction Stats [system] **

Level Files Size(MB) Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop

L0 1/0 0.30 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0
L1 1/0 0.64 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0
L6 1/0 0.44 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0
Sum 3/0 1.38 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0
Int 0/0 0.00 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0
Uptime(secs): 36754.5 total, 36754.5 interval
AddFile(GB): cumulative 0.000, interval 0.000
Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count

*************************** 3. row ***************************
Type: CF_COMPACTION
Name: default
Status:
** Compaction Stats [default] **

Level Files Size(MB) Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop

L0 1/0 64.21 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0
L1 1/0 6.32 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0
L2 202/0 500.56 1.0 0.0 0.0 0.0 0.0 0.0 0.2 0.0 0.0 0.0 0 0 0.000 0 0
L3 310/0 5105.58 1.0 2.2 2.2 0.0 0.4 0.4 0.0 0.2 74.8 12.7 30 36 0.825 5659K 0
L4 1451/0 51136.58 1.0 3.2 1.8 1.4 1.1 -0.3 18.5 0.6 39.0 13.5 84 32 2.637 29M 0
L5 9915/0 511585.00 1.0 213.4 135.0 78.4 25.5 -52.9 83.2 0.2 69.8 8.4 3130 1241 2.522 24M 0
L6 22214/0 1427155.20 0.0 0.0 0.0 0.0 0.0 0.0 1393.7 0.0 0.0 0.0 0 0 0.000 0 0
Sum 34094/0 1995553.45 0.0 218.8 139.0 79.8 27.0 -52.8 1495.6 29011020983.0 69.1 8.5 3244 1309 2.478 59M 0
Int 0/0 0.00 0.0 16.5 8.5 8.0 2.2 -5.7 234.3 2408273249.0 77.7 10.6 217 129 1.685 13M 0
Uptime(secs): 36754.5 total, 36754.5 interval
AddFile(GB): cumulative 0.000, interval 0.000
Cumulative compaction: 27.02 GB write, 0.75 MB/s write, 218.81 GB read, 6.10 MB/s read, 3243.8 seconds
Interval compaction: 2.24 GB write, 0.06 MB/s write, 16.50 GB read, 0.46 MB/s read, 217.3 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 24216 stop for pending_compaction_bytes, 724 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 3653 total count

*************************** 4. row ***************************
Type: Memory_Stats
Name: rocksdb
Status:
MemTable Total: 173312
MemTable Unflushed: 173312
Table Readers Total: 0
Cache Total: 235264945
Default Cache Capacity: 0
4 rows in set (0.01 sec)

[yoshinorim]

default column family shows that you have 34094 sst files and total size is 1995553MB.

Sum 34094/0 1995553.45 0.0 218.8 139.0 79.8 27.0 -52.8 1495.6 29011020983.0 69.1 8.5 3244 1309 2.478 59M 0

Average sst file size is 58MB. Since you have 2TB instance size, having tens of thousands of open sst files is not uncommon. Just allocate large enough file descriptors (/etc/security/limits.conf etc). If you don't like too many open files, consider increasing target_file_size_base (my previous 32MB suggestion was not appropriate. Now default is 64MB).

[mdcallag]

--- for OOM

For the OOM issue do you know what RSS was for mysqld when it was killed?

What malloc do you use for mysqld - tcmalloc, jemalloc or glibc? In the past I have had lousy results with glibc malloc. Almost all of my tests use jemalloc with mysqld. One result from my past testing showed that RSS was twice larger with glibc malloc -> http://smalldatum.blogspot.com/2015/10/myrocks-versus-allocators-glibc.html

I run days long tests using linkbench as the workload and with rocksdb_block_cache_size=35G the mysqld RSS was about 41G.

The other potential issue for using too much memory is using a large number of column families because each CF gets its own memtable, or write_buffer_size * write_buffer_number bytes of RAM. But I don't think you are doing that.

--- for compression

Are you comparing space used immediately after the load? If the tables will then be subject to random updates, InnoDB usually grows much faster than RocksDB as it suffers more from fragmentation.

If RocksDB is loaded in key order, then there is an optimization for leveled compaction and with that the memtable is flushed to write L0 files and then the L0 files are pushed down the tree, so compaction doesn't occur beyond L0. If L0 files are not to be compressed then uncompressed files are pushed all the way down the tree. Looking at the table schema you pasted for deeplink, the table has a PK and 2 secondary indexes. If loads are done into that table and the load is in PK order, I am not sure whether this optimization occurs.

Looking at the compaction IO stats that you pasted the interesting data is:

• Write(GB) - this is column 8 and the amount of data written into a level by compaction
• Moved(GB) - this is column 10 and the amounted of data moved into a level by the file pushdown optimization. And this optimization means that data might not be compressed as expected.

From the "Sum" line Write(GB)=27.0 and Moved(GB)=1495.6
So you might not be getting the compression on L3, L4, L5, L6 that you need.

If you are using a special my.cnf during the bulk load then you can configure compression for L0, L1, L2. Change kNoCompression to one of kZlibCompression or kSnappyCompression in the my.cnf used during the bulk load.

Level    Files   Size(MB) Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      1/0      64.21   0.2      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
  L1      1/0       6.32   0.1      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
  L2    202/0     500.56   1.0      0.0     0.0      0.0       0.0      0.0       0.2   0.0      0.0      0.0         0         0    0.000       0      0
  L3    310/0    5105.58   1.0      2.2     2.2      0.0       0.4      0.4       0.0   0.2     74.8     12.7        30        36    0.825   5659K      0
  L4   1451/0   51136.58   1.0      3.2     1.8      1.4       1.1     -0.3      18.5   0.6     39.0     13.5        84        32    2.637     29M      0
  L5   9915/0   511585.00   1.0    213.4   135.0     78.4      25.5    -52.9      83.2   0.2     69.8      8.4      3130      1241    2.522     24M      0
  L6  22214/0   1427155.20   0.0      0.0     0.0      0.0       0.0      0.0    1393.7   0.0      0.0      0.0         0         0    0.000       0      0
 Sum  34094/0   1995553.45   0.0    218.8   139.0     79.8      27.0    -52.8    1495.6 29011020983.0     69.1      8.5      3244      1309    2.478     59M      0

[mdcallag]

Maybe I missed this but the my.cnf you pasted at the start that was used for the bulk load doesn't enable compression.

Yoshinori, can you suggest a my.cnf?

[mdcallag]

One more thing. I have a test that doesn't get compression as requested. Not sure whether you hit this - #350

[RickPizzi]

OK, I guess (since I just started playing with MyRocks) that my config for bulk load was not appropriate for the operation, as pointed out by @mdcallag . In fact, I didn't look at all the options carefully and just took the suggested config from the wiki and started loading, to see how it performed.

Can you guys recommend a my.cnf config for bulk load that allows for bulk load with maximum compression? As the idea here is to switch from InnoDB to MyRocks for this stats cluster where we have almost INSERTs only, so looks like a good candidate for the LSM storage, also because the master here gets quite a lot of write traffic and we expect it to grow in the next months even more.

Also, for file descriptors, thanks @yoshinorim for confirming that this is normal behaviour for MyRocks with large datasets; sure I can live with that, but seemed odd at first :-) I will use your suggestion about target_file_size_base.

Re. the OOM, here is what the mysqld had at the time it kicked in:

Oct 17 13:50:36 dbbackup01 kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Oct 17 13:50:36 dbbackup01 kernel: [ 8035]   496  8035 23029196 16114547   1       0             0 mysqld

Thanks
Rick

[RickPizzi]

UPDATE - I have given it another try, after following Mark's and Yoshinori's suggestions above, and results are MUCH better!

• no crash (yay)
• dataset is 600G vs 2.3T from InnoDB!!!!! Now we're talking :-)
• number of file descriptors dropped from 36,000 to 12,500 - still a high number but I now understand this is normal behaviour for MyRocks

Relevant config plus rocksdb status below; if there are other tweaks you want to suggest to further improve compression please let me know as I will repeat it once again with fresh data anyways....

Thanks!

rocksdb
skip-innodb
default-storage-engine=rocksdb
default-tmp-storage-engine=MyISAM
collation-server=latin1_bin
binlog_format                  = ROW

# config for bulk load

rocksdb_max_open_files=-1
rocksdb_base_background_compactions=1
rocksdb_max_total_wal_size=4G
rocksdb_block_size=16384
rocksdb_block_cache_size=16G
rocksdb_table_cache_numshardbits=6
rocksdb_default_cf_options=write_buffer_size=128m;target_file_size_base=64m;max_bytes_for_level_base=512m;level0_file_num_compaction_trigger=4;level0_slowdown_writes_trigger=256;level0_stop_writes_trigger=256;max_write_buffer_number=16;compression_per_level=kNoCompression:kNoCompression:kNoCompression:kZlibCompression:kZlibCompression:kZlibCompression;bottommost_compression=kZlibCompression;compression_opts=-14:1:0;block_based_table_factory={cache_index_and_filter_blocks=1;filter_policy=bloomfilter:10:false;whole_key_filtering=1};level_compaction_dynamic_level_bytes=true;optimize_filters_for_hits=true
rocksdb_override_cf_options=system={memtable=skip_list:16}

rocksdb_skip_unique_check=1
rocksdb_commit_in_the_middle=1
rocksdb_write_disable_wal=1
rocksdb_max_background_flushes=40
rocksdb_max_background_compactions=40

mysql> show engine rocksdb status\G
*************************** 1. row ***************************
  Type: DBSTATS
  Name: rocksdb
Status: 
** DB Stats **
Uptime(secs): 86190.7 total, 26388.5 interval
Cumulative writes: 12M writes, 12G keys, 8048K commit groups, 1.6 writes per commit group, ingest: 0.06 GB, 0.00 MB/s
Cumulative WAL: 64K writes, 3589 syncs, 18.04 writes per sync, written: 0.06 GB, 0.00 MB/s
Cumulative stall: 01:15:24.946 H:M:S, 5.2 percent
Interval writes: 7075 writes, 7053K keys, 7074 commit groups, 1.0 writes per commit group, ingest: 0.01 MB, 0.00 MB/s
Interval WAL: 29 writes, 0 syncs, 29.00 writes per sync, written: 0.00 MB, 0.00 MB/s
Interval stall: 00:00:0.000 H:M:S, 0.0 percent
** Level 0 read latency histogram (micros):
Count: 104416857 Average: 6.7008  StdDev: 39.46
Min: 0  Median: 4.6197  Max: 865750
Percentiles: P50: 4.62 P75: 5.70 P99: 37.18 P99.9: 85.38 P99.99: 274.43
------------------------------------------------------
[       0,       1 )    24297   0.023%   0.023% 
[       1,       2 )    36266   0.035%   0.058% 
[       2,       3 )  4869581   4.664%   4.722% #
[       3,       4 ) 30420897  29.134%  33.856% ######
[       4,       5 ) 27204207  26.053%  59.909% #####
[       5,       6 ) 22629443  21.672%  81.581% ####
[       6,       7 )  6907881   6.616%  88.197% #
[       7,       8 )  2539814   2.432%  90.629% 
[       8,       9 )  1392574   1.334%  91.963% 
[       9,      10 )   938726   0.899%  92.862% 
[      10,      12 )  1320469   1.265%  94.127% 
[      12,      14 )   941449   0.902%  95.028% 
[      14,      16 )   763703   0.731%  95.760% 
[      16,      18 )   612257   0.586%  96.346% 
[      18,      20 )   551145   0.528%  96.874% 
[      20,      25 )   992366   0.950%  97.824% 
[      25,      30 )   598436   0.573%  98.397% 
[      30,      35 )   471943   0.452%  98.849% 
[      35,      40 )   361423   0.346%  99.196% 
[      40,      45 )   257291   0.246%  99.442% 
[      45,      50 )   162299   0.155%  99.597% 
[      50,      60 )   160562   0.154%  99.751% 
[      60,      70 )    89476   0.086%  99.837% 
[      70,      80 )    49929   0.048%  99.885% 
[      80,      90 )    29744   0.028%  99.913% 
[      90,     100 )    19927   0.019%  99.932% 
[     100,     120 )    22597   0.022%  99.954% 
[     120,     140 )    10921   0.010%  99.964% 
[     140,     160 )     8547   0.008%  99.973% 
[     160,     180 )     5480   0.005%  99.978% 
[     180,     200 )     3967   0.004%  99.982% 
[     200,     250 )     6655   0.006%  99.988% 
[     250,     300 )     4386   0.004%  99.992% 
[     300,     350 )     2462   0.002%  99.995% 
[     350,     400 )      648   0.001%  99.995% 
[     400,     450 )      401   0.000%  99.996% 
[     450,     500 )      379   0.000%  99.996% 
[     500,     600 )     1120   0.001%  99.997% 
[     600,     700 )      678   0.001%  99.998% 
[     700,     800 )      189   0.000%  99.998% 
[     800,     900 )      102   0.000%  99.998% 
[     900,    1000 )       66   0.000%  99.998% 
[    1000,    1200 )      101   0.000%  99.998% 
[    1200,    1400 )       90   0.000%  99.998% 
[    1400,    1600 )       96   0.000%  99.998% 
[    1600,    1800 )      103   0.000%  99.998% 
[    1800,    2000 )       72   0.000%  99.998% 
[    2000,    2500 )      139   0.000%  99.999% 
[    2500,    3000 )       99   0.000%  99.999% 
[    3000,    3500 )       61   0.000%  99.999% 
[    3500,    4000 )       33   0.000%  99.999% 
[    4000,    4500 )       49   0.000%  99.999% 
[    4500,    5000 )       32   0.000%  99.999% 
[    5000,    6000 )       76   0.000%  99.999% 
[    6000,    7000 )       68   0.000%  99.999% 
[    7000,    8000 )       56   0.000%  99.999% 
[    8000,    9000 )       37   0.000%  99.999% 
[    9000,   10000 )       45   0.000%  99.999% 
[   10000,   12000 )      810   0.001% 100.000% 
[   12000,   14000 )       41   0.000% 100.000% 
[   14000,   16000 )       48   0.000% 100.000% 
[   16000,   18000 )       35   0.000% 100.000% 
[   18000,   20000 )       22   0.000% 100.000% 
[   20000,   25000 )       21   0.000% 100.000% 
[   25000,   30000 )       11   0.000% 100.000% 
[   30000,   35000 )        4   0.000% 100.000% 
[   35000,   40000 )        2   0.000% 100.000% 
[   45000,   50000 )        1   0.000% 100.000% 
[  500000,  600000 )        1   0.000% 100.000% 
[  800000,  900000 )        1   0.000% 100.000% 

** Level 2 read latency histogram (micros):
Count: 13494331 Average: 4.5493  StdDev: 15.64
Min: 0  Median: 3.7432  Max: 18764
Percentiles: P50: 3.74 P75: 4.42 P99: 7.20 P99.9: 23.30 P99.99: 211.67
------------------------------------------------------
[       0,       1 )     9540   0.071%   0.071% 
[       1,       2 )     5365   0.040%   0.110% 
[       2,       3 )  1273368   9.436%   9.547% ##
[       3,       4 )  7345381  54.433%  63.980% ###########
[       4,       5 )  3575516  26.496%  90.476% #####
[       5,       6 )   883953   6.551%  97.027% #
[       6,       7 )   254371   1.885%  98.912% 
[       7,       8 )    58122   0.431%  99.343% 
[       8,       9 )    20728   0.154%  99.496% 
[       9,      10 )    12145   0.090%  99.586% 
[      10,      12 )    10385   0.077%  99.663% 
[      12,      14 )     2400   0.018%  99.681% 
[      14,      16 )     1168   0.009%  99.690% 
[      16,      18 )     1451   0.011%  99.700% 
[      18,      20 )    16027   0.119%  99.819% 
[      20,      25 )    16522   0.122%  99.942% 
[      25,      30 )      581   0.004%  99.946% 
[      30,      35 )      740   0.005%  99.951% 
[      35,      40 )      988   0.007%  99.959% 
[      40,      45 )      489   0.004%  99.962% 
[      45,      50 )      495   0.004%  99.966% 
[      50,      60 )     1250   0.009%  99.975% 
[      60,      70 )      562   0.004%  99.979% 
[      70,      80 )      269   0.002%  99.981% 
[      80,      90 )      194   0.001%  99.983% 
[      90,     100 )      141   0.001%  99.984% 
[     100,     120 )      170   0.001%  99.985% 
[     120,     140 )       92   0.001%  99.986% 
[     140,     160 )      139   0.001%  99.987% 
[     160,     180 )      234   0.002%  99.989% 
[     180,     200 )      186   0.001%  99.990% 
[     200,     250 )       41   0.000%  99.990% 
[     250,     300 )       46   0.000%  99.991% 
[     300,     350 )       62   0.000%  99.991% 
[     350,     400 )       75   0.001%  99.992% 
[     400,     450 )       47   0.000%  99.992% 
[     450,     500 )       68   0.001%  99.992% 
[     500,     600 )      104   0.001%  99.993% 
[     600,     700 )      133   0.001%  99.994% 
[     700,     800 )      245   0.002%  99.996% 
[     800,     900 )      285   0.002%  99.998% 
[     900,    1000 )      142   0.001%  99.999% 
[    1000,    1200 )       66   0.000% 100.000% 
[    1200,    1400 )       12   0.000% 100.000% 
[    1600,    1800 )        5   0.000% 100.000% 
[    2000,    2500 )        4   0.000% 100.000% 
[    2500,    3000 )        2   0.000% 100.000% 
[    3000,    3500 )        3   0.000% 100.000% 
[    4500,    5000 )        1   0.000% 100.000% 
[    9000,   10000 )        1   0.000% 100.000% 
[   10000,   12000 )       14   0.000% 100.000% 
[   12000,   14000 )        2   0.000% 100.000% 
[   18000,   20000 )        1   0.000% 100.000% 

** Level 3 read latency histogram (micros):
Count: 133350251 Average: 10.6061  StdDev: 15.30
Min: 0  Median: 4.8979  Max: 511818
Percentiles: P50: 4.90 P75: 5.93 P99: 44.66 P99.9: 111.49 P99.99: 11479.48
------------------------------------------------------
[       0,       1 )    98046   0.074%   0.074% 
[       1,       2 )   111628   0.084%   0.157% 
[       2,       3 )  3879352   2.909%   3.066% #
[       3,       4 ) 31122052  23.339%  26.405% #####
[       4,       5 ) 35040565  26.277%  52.682% #####
[       5,       6 ) 32030144  24.020%  76.702% #####
[       6,       7 ) 11311812   8.483%  85.184% ##
[       7,       8 )  4115819   3.086%  88.271% #
[       8,       9 )  2262492   1.697%  89.968% 
[       9,      10 )  1518978   1.139%  91.107% 
[      10,      12 )  2056473   1.542%  92.649% 
[      12,      14 )  1411880   1.059%  93.708% 
[      14,      16 )  1153385   0.865%  94.572% 
[      16,      18 )   919071   0.689%  95.262% 
[      18,      20 )   801295   0.601%  95.863% 
[      20,      25 )  1451182   1.088%  96.951% 
[      25,      30 )   948067   0.711%  97.662% 
[      30,      35 )   759633   0.570%  98.231% 
[      35,      40 )   589514   0.442%  98.674% 
[      40,      45 )   467214   0.350%  99.024% 
[      45,      50 )   363158   0.272%  99.296% 
[      50,      60 )   395807   0.297%  99.593% 
[      60,      70 )   198247   0.149%  99.742% 
[      70,      80 )   101416   0.076%  99.818% 
[      80,      90 )    52424   0.039%  99.857% 
[      90,     100 )    31900   0.024%  99.881% 
[     100,     120 )    44127   0.033%  99.914% 
[     120,     140 )    27152   0.020%  99.934% 
[     140,     160 )    16998   0.013%  99.947% 
[     160,     180 )     9979   0.007%  99.955% 
[     180,     200 )     7446   0.006%  99.960% 
[     200,     250 )     8608   0.006%  99.967% 
[     250,     300 )     3417   0.003%  99.969% 
[     300,     350 )     2085   0.002%  99.971% 
[     350,     400 )     1506   0.001%  99.972% 
[     400,     450 )     1076   0.001%  99.973% 
[     450,     500 )      956   0.001%  99.973% 
[     500,     600 )     1554   0.001%  99.975% 
[     600,     700 )      952   0.001%  99.975% 
[     700,     800 )      863   0.001%  99.976% 
[     800,     900 )      862   0.001%  99.977% 
[     900,    1000 )      637   0.000%  99.977% 
[    1000,    1200 )      656   0.000%  99.978% 
[    1200,    1400 )      488   0.000%  99.978% 
[    1400,    1600 )      445   0.000%  99.978% 
[    1600,    1800 )      406   0.000%  99.979% 
[    1800,    2000 )      355   0.000%  99.979% 
[    2000,    2500 )      874   0.001%  99.980% 
[    2500,    3000 )      713   0.001%  99.980% 
[    3000,    3500 )      734   0.001%  99.981% 
[    3500,    4000 )      704   0.001%  99.981% 
[    4000,    4500 )      693   0.001%  99.982% 
[    4500,    5000 )      687   0.001%  99.982% 
[    5000,    6000 )     1393   0.001%  99.983% 
[    6000,    7000 )     1464   0.001%  99.984% 
[    7000,    8000 )     1582   0.001%  99.986% 
[    8000,    9000 )     1420   0.001%  99.987% 
[    9000,   10000 )     1230   0.001%  99.988% 
[   10000,   12000 )     4461   0.003%  99.991% 
[   12000,   14000 )     1872   0.001%  99.992% 
[   14000,   16000 )     1732   0.001%  99.994% 
[   16000,   18000 )     1299   0.001%  99.995% 
[   18000,   20000 )      916   0.001%  99.995% 
[   20000,   25000 )     1832   0.001%  99.997% 
[   25000,   30000 )     1131   0.001%  99.997% 
[   30000,   35000 )      841   0.001%  99.998% 
[   35000,   40000 )      672   0.001%  99.999% 
[   40000,   45000 )      524   0.000%  99.999% 
[   45000,   50000 )      366   0.000%  99.999% 
[   50000,   60000 )      402   0.000% 100.000% 
[   60000,   70000 )      197   0.000% 100.000% 
[   70000,   80000 )      106   0.000% 100.000% 
[   80000,   90000 )       66   0.000% 100.000% 
[   90000,  100000 )       55   0.000% 100.000% 
[  100000,  120000 )       60   0.000% 100.000% 
[  120000,  140000 )       28   0.000% 100.000% 
[  140000,  160000 )       15   0.000% 100.000% 
[  160000,  180000 )       12   0.000% 100.000% 
[  180000,  200000 )        4   0.000% 100.000% 
[  200000,  250000 )       18   0.000% 100.000% 
[  250000,  300000 )        7   0.000% 100.000% 
[  300000,  350000 )        8   0.000% 100.000% 
[  350000,  400000 )        7   0.000% 100.000% 
[  400000,  450000 )        3   0.000% 100.000% 
[  500000,  600000 )        1   0.000% 100.000% 

** Level 4 read latency histogram (micros):
Count: 152152466 Average: 11.1141  StdDev: 17.94
Min: 0  Median: 4.7517  Max: 692723
Percentiles: P50: 4.75 P75: 5.73 P99: 34.50 P99.9: 106.86 P99.99: 15872.31
------------------------------------------------------
[       0,       1 )  3646414   2.397%   2.397% 
[       1,       2 ) 14284348   9.388%  11.785% ##
[       2,       3 )  6659419   4.377%  16.162% #
[       3,       4 ) 25050992  16.464%  32.626% ###
[       4,       5 ) 35166135  23.112%  55.738% #####
[       5,       6 ) 40281624  26.475%  82.213% #####
[       6,       7 ) 14772300   9.709%  91.922% ##
[       7,       8 )  4124132   2.711%  94.632% #
[       8,       9 )  1750377   1.150%  95.783% 
[       9,      10 )   994883   0.654%  96.437% 
[      10,      12 )  1065220   0.700%  97.137% 
[      12,      14 )   566208   0.372%  97.509% 
[      14,      16 )   432826   0.284%  97.793% 
[      16,      18 )   356537   0.234%  98.028% 
[      18,      20 )   300740   0.198%  98.225% 
[      20,      25 )   532260   0.350%  98.575% 
[      25,      30 )   359540   0.236%  98.811% 
[      30,      35 )   318794   0.210%  99.021% 
[      35,      40 )   241199   0.159%  99.179% 
[      40,      45 )   228181   0.150%  99.329% 
[      45,      50 )   263606   0.173%  99.503% 
[      50,      60 )   326442   0.215%  99.717% 
[      60,      70 )   153699   0.101%  99.818% 
[      70,      80 )    67330   0.044%  99.862% 
[      80,      90 )    28317   0.019%  99.881% 
[      90,     100 )    17793   0.012%  99.893% 
[     100,     120 )    32051   0.021%  99.914% 
[     120,     140 )    23780   0.016%  99.929% 
[     140,     160 )    15029   0.010%  99.939% 
[     160,     180 )     8084   0.005%  99.945% 
[     180,     200 )     6376   0.004%  99.949% 
[     200,     250 )     8427   0.006%  99.954% 
[     250,     300 )     3879   0.003%  99.957% 
[     300,     350 )     2349   0.002%  99.958% 
[     350,     400 )     1716   0.001%  99.960% 
[     400,     450 )     1360   0.001%  99.961% 
[     450,     500 )     1228   0.001%  99.961% 
[     500,     600 )     2197   0.001%  99.963% 
[     600,     700 )     1562   0.001%  99.964% 
[     700,     800 )     1373   0.001%  99.965% 
[     800,     900 )     1124   0.001%  99.965% 
[     900,    1000 )      845   0.001%  99.966% 
[    1000,    1200 )     1048   0.001%  99.967% 
[    1200,    1400 )      790   0.001%  99.967% 
[    1400,    1600 )      676   0.000%  99.968% 
[    1600,    1800 )      631   0.000%  99.968% 
[    1800,    2000 )      576   0.000%  99.968% 
[    2000,    2500 )     1443   0.001%  99.969% 
[    2500,    3000 )     1303   0.001%  99.970% 
[    3000,    3500 )     1189   0.001%  99.971% 
[    3500,    4000 )     1169   0.001%  99.972% 
[    4000,    4500 )     1184   0.001%  99.973% 
[    4500,    5000 )     1154   0.001%  99.973% 
[    5000,    6000 )     2428   0.002%  99.975% 
[    6000,    7000 )     2397   0.002%  99.976% 
[    7000,    8000 )     2593   0.002%  99.978% 
[    8000,    9000 )     2314   0.002%  99.980% 
[    9000,   10000 )     1971   0.001%  99.981% 
[   10000,   12000 )     7535   0.005%  99.986% 
[   12000,   14000 )     3159   0.002%  99.988% 
[   14000,   16000 )     3199   0.002%  99.990% 
[   16000,   18000 )     2144   0.001%  99.992% 
[   18000,   20000 )     1622   0.001%  99.993% 
[   20000,   25000 )     3140   0.002%  99.995% 
[   25000,   30000 )     2103   0.001%  99.996% 
[   30000,   35000 )     1387   0.001%  99.997% 
[   35000,   40000 )     1127   0.001%  99.998% 
[   40000,   45000 )      953   0.001%  99.998% 
[   45000,   50000 )      664   0.000%  99.999% 
[   50000,   60000 )      729   0.000%  99.999% 
[   60000,   70000 )      373   0.000%  99.999% 
[   70000,   80000 )      243   0.000% 100.000% 
[   80000,   90000 )      131   0.000% 100.000% 
[   90000,  100000 )       93   0.000% 100.000% 
[  100000,  120000 )      101   0.000% 100.000% 
[  120000,  140000 )       35   0.000% 100.000% 
[  140000,  160000 )       36   0.000% 100.000% 
[  160000,  180000 )       24   0.000% 100.000% 
[  180000,  200000 )       11   0.000% 100.000% 
[  200000,  250000 )       40   0.000% 100.000% 
[  250000,  300000 )       26   0.000% 100.000% 
[  300000,  350000 )       13   0.000% 100.000% 
[  350000,  400000 )        9   0.000% 100.000% 
[  400000,  450000 )        5   0.000% 100.000% 
[  450000,  500000 )        1   0.000% 100.000% 
[  600000,  700000 )        1   0.000% 100.000% 

** Level 5 read latency histogram (micros):
Count: 248585749 Average: 6.6961  StdDev: 15.42
Min: 0  Median: 2.3200  Max: 996117
Percentiles: P50: 2.32 P75: 3.11 P99: 7.97 P99.9: 59.21 P99.99: 11340.88
------------------------------------------------------
[       0,       1 ) 14925201   6.004%   6.004% #
[       1,       2 ) 82178009  33.058%  39.062% #######
[       2,       3 ) 84970064  34.181%  73.244% #######
[       3,       4 ) 38615855  15.534%  88.778% ###
[       4,       5 ) 14643921   5.891%  94.669% #
[       5,       6 )  7306599   2.939%  97.608% #
[       6,       7 )  2696597   1.085%  98.693% 
[       7,       8 )   784232   0.315%  99.008% 
[       8,       9 )   297778   0.120%  99.128% 
[       9,      10 )   143224   0.058%  99.186% 
[      10,      12 )   118349   0.048%  99.233% 
[      12,      14 )    48991   0.020%  99.253% 
[      14,      16 )    36076   0.015%  99.268% 
[      16,      18 )    53366   0.021%  99.289% 
[      18,      20 )    57609   0.023%  99.312% 
[      20,      25 )    78510   0.032%  99.344% 
[      25,      30 )   152876   0.061%  99.405% 
[      30,      35 )   292149   0.118%  99.523% 
[      35,      40 )   149507   0.060%  99.583% 
[      40,      45 )   192195   0.077%  99.660% 
[      45,      50 )   323624   0.130%  99.790% 
[      50,      60 )   295693   0.119%  99.909% 
[      60,      70 )    64214   0.026%  99.935% 
[      70,      80 )    16715   0.007%  99.942% 
[      80,      90 )     7972   0.003%  99.945% 
[      90,     100 )     7279   0.003%  99.948% 
[     100,     120 )    14548   0.006%  99.954% 
[     120,     140 )     9442   0.004%  99.958% 
[     140,     160 )     9508   0.004%  99.962% 
[     160,     180 )     5393   0.002%  99.964% 
[     180,     200 )     3679   0.001%  99.965% 
[     200,     250 )     5236   0.002%  99.967% 
[     250,     300 )     2568   0.001%  99.968% 
[     300,     350 )     1870   0.001%  99.969% 
[     350,     400 )     1699   0.001%  99.970% 
[     400,     450 )     1594   0.001%  99.970% 
[     450,     500 )     1375   0.001%  99.971% 
[     500,     600 )     1988   0.001%  99.972% 
[     600,     700 )     1388   0.001%  99.972% 
[     700,     800 )     1209   0.000%  99.973% 
[     800,     900 )     1172   0.000%  99.973% 
[     900,    1000 )      973   0.000%  99.974% 
[    1000,    1200 )     1737   0.001%  99.974% 
[    1200,    1400 )     1431   0.001%  99.975% 
[    1400,    1600 )     1184   0.000%  99.975% 
[    1600,    1800 )     1052   0.000%  99.976% 
[    1800,    2000 )      909   0.000%  99.976% 
[    2000,    2500 )     2280   0.001%  99.977% 
[    2500,    3000 )     2230   0.001%  99.978% 
[    3000,    3500 )     2199   0.001%  99.979% 
[    3500,    4000 )     2135   0.001%  99.980% 
[    4000,    4500 )     2166   0.001%  99.981% 
[    4500,    5000 )     2048   0.001%  99.981% 
[    5000,    6000 )     3863   0.002%  99.983% 
[    6000,    7000 )     3703   0.001%  99.984% 
[    7000,    8000 )     3488   0.001%  99.986% 
[    8000,    9000 )     3034   0.001%  99.987% 
[    9000,   10000 )     2724   0.001%  99.988% 
[   10000,   12000 )     6653   0.003%  99.991% 
[   12000,   14000 )     3840   0.002%  99.992% 
[   14000,   16000 )     3093   0.001%  99.994% 
[   16000,   18000 )     2349   0.001%  99.995% 
[   18000,   20000 )     2007   0.001%  99.995% 
[   20000,   25000 )     3782   0.002%  99.997% 
[   25000,   30000 )     2351   0.001%  99.998% 
[   30000,   35000 )     1634   0.001%  99.999% 
[   35000,   40000 )     1018   0.000%  99.999% 
[   40000,   45000 )      677   0.000%  99.999% 
[   45000,   50000 )      488   0.000%  99.999% 
[   50000,   60000 )      558   0.000% 100.000% 
[   60000,   70000 )      291   0.000% 100.000% 
[   70000,   80000 )      153   0.000% 100.000% 
[   80000,   90000 )       93   0.000% 100.000% 
[   90000,  100000 )       66   0.000% 100.000% 
[  100000,  120000 )       63   0.000% 100.000% 
[  120000,  140000 )       46   0.000% 100.000% 
[  140000,  160000 )       37   0.000% 100.000% 
[  160000,  180000 )       21   0.000% 100.000% 
[  180000,  200000 )       16   0.000% 100.000% 
[  200000,  250000 )       27   0.000% 100.000% 
[  250000,  300000 )       16   0.000% 100.000% 
[  300000,  350000 )       18   0.000% 100.000% 
[  350000,  400000 )       10   0.000% 100.000% 
[  400000,  450000 )        3   0.000% 100.000% 
[  450000,  500000 )        1   0.000% 100.000% 
[  500000,  600000 )        5   0.000% 100.000% 
[  900000, 1000000 )        3   0.000% 100.000% 

** Level 6 read latency histogram (micros):
Count: 181771036 Average: 8.8980  StdDev: 21.23
Min: 0  Median: 1.8603  Max: 733266
Percentiles: P50: 1.86 P75: 2.60 P99: 36.41 P99.9: 81.71 P99.99: 17708.62
------------------------------------------------------
[       0,       1 ) 22830049  12.560%  12.560% ###
[       1,       2 ) 79102213  43.518%  56.077% #########
[       2,       3 ) 57032242  31.376%  87.453% ######
[       3,       4 ) 16082981   8.848%  96.301% ##
[       4,       5 )  2934438   1.614%  97.915% 
[       5,       6 )   796736   0.438%  98.354% 
[       6,       7 )   305698   0.168%  98.522% 
[       7,       8 )   109916   0.060%  98.582% 
[       8,       9 )    44399   0.024%  98.607% 
[       9,      10 )    20127   0.011%  98.618% 
[      10,      12 )    16704   0.009%  98.627% 
[      12,      14 )     7004   0.004%  98.631% 
[      14,      16 )     6561   0.004%  98.635% 
[      16,      18 )    25159   0.014%  98.648% 
[      18,      20 )    24050   0.013%  98.662% 
[      20,      25 )    37855   0.021%  98.682% 
[      25,      30 )   130937   0.072%  98.754% 
[      30,      35 )   361562   0.199%  98.953% 
[      35,      40 )   300088   0.165%  99.118% 
[      40,      45 )   229821   0.126%  99.245% 
[      45,      50 )   396181   0.218%  99.463% 
[      50,      60 )   602310   0.331%  99.794% 
[      60,      70 )   154006   0.085%  99.879% 
[      70,      80 )    35195   0.019%  99.898% 
[      80,      90 )    17728   0.010%  99.908% 
[      90,     100 )    19869   0.011%  99.919% 
[     100,     120 )    22613   0.012%  99.931% 
[     120,     140 )    14729   0.008%  99.940% 
[     140,     160 )    11114   0.006%  99.946% 
[     160,     180 )     6493   0.004%  99.949% 
[     180,     200 )     4978   0.003%  99.952% 
[     200,     250 )     5931   0.003%  99.955% 
[     250,     300 )     2128   0.001%  99.956% 
[     300,     350 )     1308   0.001%  99.957% 
[     350,     400 )     1274   0.001%  99.958% 
[     400,     450 )      948   0.001%  99.958% 
[     450,     500 )      676   0.000%  99.959% 
[     500,     600 )      908   0.000%  99.959% 
[     600,     700 )      753   0.000%  99.960% 
[     700,     800 )      712   0.000%  99.960% 
[     800,     900 )      667   0.000%  99.960% 
[     900,    1000 )      683   0.000%  99.961% 
[    1000,    1200 )     1143   0.001%  99.961% 
[    1200,    1400 )     1151   0.001%  99.962% 
[    1400,    1600 )     1129   0.001%  99.963% 
[    1600,    1800 )     1170   0.001%  99.963% 
[    1800,    2000 )     1031   0.001%  99.964% 
[    2000,    2500 )     2539   0.001%  99.965% 
[    2500,    3000 )     2448   0.001%  99.967% 
[    3000,    3500 )     2214   0.001%  99.968% 
[    3500,    4000 )     2134   0.001%  99.969% 
[    4000,    4500 )     2127   0.001%  99.970% 
[    4500,    5000 )     2039   0.001%  99.971% 
[    5000,    6000 )     3819   0.002%  99.973% 
[    6000,    7000 )     3803   0.002%  99.975% 
[    7000,    8000 )     3503   0.002%  99.977% 
[    8000,    9000 )     3019   0.002%  99.979% 
[    9000,   10000 )     2759   0.002%  99.981% 
[   10000,   12000 )     6976   0.004%  99.984% 
[   12000,   14000 )     4105   0.002%  99.987% 
[   14000,   16000 )     3560   0.002%  99.989% 
[   16000,   18000 )     2863   0.002%  99.990% 
[   18000,   20000 )     2455   0.001%  99.992% 
[   20000,   25000 )     4734   0.003%  99.994% 
[   25000,   30000 )     3191   0.002%  99.996% 
[   30000,   35000 )     2044   0.001%  99.997% 
[   35000,   40000 )     1389   0.001%  99.998% 
[   40000,   45000 )      976   0.001%  99.998% 
[   45000,   50000 )      650   0.000%  99.999% 
[   50000,   60000 )      793   0.000%  99.999% 
[   60000,   70000 )      440   0.000%  99.999% 
[   70000,   80000 )      232   0.000% 100.000% 
[   80000,   90000 )      160   0.000% 100.000% 
[   90000,  100000 )      117   0.000% 100.000% 
  L2      0/0       0.00   0.0    209.5   209.5      0.0     209.1    209.1       0.0   1.0     74.2     74.1      2891       515    5.613   3541M      0
  L3      9/0     517.74   0.9   2917.3  2840.0     77.3    2914.5   2837.2      94.4   1.0    107.3    107.2     27838      3464    8.036     18G      0
  L4    381/0    5453.39   1.0   2593.2  1449.5   1143.7    2449.8   1306.1     776.9   1.7     62.0     58.6     42824     12303    3.481     32G      0
  L5   2995/0   54969.72   1.0   2902.3  1935.5    966.8    1617.6    650.8      77.3   0.8     18.6     10.4    159836     21735    7.354     51G      0
  L6   8943/0   549770.10   0.0   1472.5   669.3    803.2    1336.8    533.6       3.5   2.0      8.1      7.3    186393      5148   36.207     86G      0
 Sum  12331/0   610929.98   0.0  10094.8  7103.8   2991.0   10691.4   7700.4     952.1   4.9     23.8     25.2    434140     64681    6.712    192G      0
 Int      0/0       0.00   0.0      4.6     2.2      2.4       4.4      2.1       0.3   8.9     11.0     10.6       426        29   14.698    495M      0
Uptime(secs): 86190.7 total, 86190.7 interval
AddFile(GB): cumulative 0.000, interval 0.000
Cumulative compaction: 10691.44 GB write, 127.02 MB/s write, 10094.83 GB read, 119.93 MB/s read, 434139.6 seconds
Interval compaction: 4.43 GB write, 0.05 MB/s write, 4.60 GB read, 0.05 MB/s read, 426.2 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 2428 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count

*************************** 4. row ***************************
  Type: Memory_Stats
  Name: rocksdb
Status: 
MemTable Total: 2227538856
MemTable Unflushed: 146426592
Table Readers Total: 0
Cache Total: 6440651900
Default Cache Capacity: 0
4 rows in set (0.50 sec)

[gunnarku]

Please re-open if you see any other issues.