Prevents Table Cache to open same files more times #6707

koldat · 2020-04-15T08:09:03Z

In highly concurrent requests table cache opens same file more times which lowers purpose of max_open_files. Fixes (#6699)

koldat · 2020-04-15T08:09:50Z

@pdillinger @siying please take a look and comment

siying · 2020-04-15T17:01:09Z

util/mutexlock.h

+  }
+
+ private:
+  std::vector<T> locks_;


Can we put those locks in different cache lines to prevent false sharing? We do something like this in other places:

rocksdb/monitoring/statistics.h

Line 83 in fdf882d

struct ALIGN_AS(CACHE_LINE_SIZE) StatisticsData {

Please review if I got that correctly

OK I forgot destructor. Sorry...

facebook-github-bot

@siying has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

siying

Is there a way to test this in unit test?

siying · 2020-04-15T17:49:36Z

db/table_cache.cc

@@ -62,14 +62,17 @@ void AppendVarint64(IterKey* key, uint64_t v) {

 }  // namespace

+#define LOAD_CONCURENCY 128


Can it be a const instead of a macro? The memory is dynamically allocated anyway.

It can. I was waiting for feedback if you want it in options (I would not put it there). And I saw other macros as well. But yes can change it.

koldat · 2020-04-15T18:01:11Z

I have unit test in Java. Unfortunately I did not get check points framework. But if you require I will try to learn it.

siying · 2020-04-15T18:04:02Z

Test might be a little bit hard to write though. If you run Linux, can you try to run "make crash_test" and see whether it can pass"? (it takes several hours).

koldat · 2020-04-15T18:21:55Z

I was testing in debugger to stop threads at correct place and it works as expected. Java multithread test I attached to issue also works as expected after fix. I have deployed build on our stress test and will start test you have mentioned. I will keep you posted.

koldat · 2020-04-16T09:57:37Z

I have run the make crash_test. It was running many hours as you have said. Basically it looks to me it was running in some cycle some tool. So it was repeating some command. All of them looks like this (hopefully it is ok):

Running db_stress with pid=4710: ./db_stress --acquire_snapshot_one_in=10000 --avoid_unnecessary_blocking_io=1 --block_size=16384 --bloom_bits=6.08920299324703 --bottommost_compression_type=disable --cache_index_and_filter_blocks=0 --cache_size=1048576 --checkpoint_one_in=1000000 --checksum_type=kxxHash --clear_column_family_one_in=0 --compact_files_one_in=1000000 --compact_range_one_in=1000000 --compaction_ttl=0 --compression_max_dict_bytes=16384 --compression_type=bzip2 --compression_zstd_max_train_bytes=0 --continuous_verification_interval=0 --db=/tmp/rocksdb_crashtest_blackbox9sb1kq8p --db_write_buffer_size=134217728 --delpercent=5 --delrangepercent=0 --destroy_db_initially=0 --enable_pipelined_write=1 --expected_values_path=/tmp/tmpimwgqhkc --flush_one_in=1000000 --format_version=5 --get_current_wal_file_one_in=0 --get_live_files_one_in=1000000 --get_sorted_wal_files_one_in=0 --index_block_restart_interval=11 --index_type=0 --key_len_percent_dist=1,30,69 --level_compaction_dynamic_level_bytes=True --long_running_snapshots=0 --max_background_compactions=20 --max_bytes_for_level_base=10485760 --max_key=100000000 --max_key_len=3 --max_manifest_file_size=1073741824 --max_write_batch_group_size_bytes=1048576 --max_write_buffer_number=3 --memtablerep=skip_list --mmap_read=1 --nooverwritepercent=1 --open_files=500000 --ops_per_thread=100000000 --partition_filters=0 --pause_background_one_in=1000000 --periodic_compaction_seconds=0 --prefixpercent=5 --progress_reports=0 --read_fault_one_in=0 --readpercent=45 --recycle_log_file_num=1 --reopen=20 --set_options_one_in=10000 --snapshot_hold_ops=100000 --subcompactions=3 --sync=0 --target_file_size_base=2097152 --target_file_size_multiplier=2 --test_batches_snapshots=1 --use_direct_io_for_flush_and_compaction=0 --use_direct_reads=0 --use_full_merge_v1=1 --use_merge=0 --use_multiget=0 --verify_checksum=1 --verify_checksum_one_in=1000000 --verify_db_one_in=100000 --write_buffer_size=4194304 --write_dbid_to_manifest=1 --writepercent=35


2020/04/16-06:24:02  Initializing db_stress
RocksDB version           : 6.9
Format version            : 5
TransactionDB             : false
BlobDB                    : false
Read only mode            : false
Atomic flush              : false
Column families           : 10
Number of threads         : 32
Ops per thread            : 100000000
Time to live(sec)         : unused
Read percentage           : 45%
Prefix percentage         : 5%
Write percentage          : 35%
Delete percentage         : 5%
Delete range percentage   : 0%
No overwrite percentage   : 1%
Iterate percentage        : 10%
DB-write-buffer-size      : 134217728
Write-buffer-size         : 4194304
Iterations                : 10
Max key                   : 100000000
Ratio #ops/#keys          : 32.000000
Num times DB reopens      : 20
Batches/snapshots         : 1
Do update in place        : 0
Num keys per lock         : 4
Compression               : BZip2
Bottommost Compression    : DisableOption
Checksum type             : kxxHash
Bloom bits / key          : 6.089203
Max subcompactions        : 3
Use MultiGet              : false
Memtablerep               : skip_list
Test kill odd             : 0
Periodic Compaction Secs  : 0
Compaction TTL            : 0
Background Purge          : 1
Write DB ID to manifest   : 1
Max Write Batch Group Size: 1048576
Use dynamic level         : 1
Read fault one in         : 0
------------------------------------------------
DB path: [/tmp/rocksdb_crashtest_blackbox9sb1kq8p]
Choosing random keys with no overwrite
No lock creation because test_batches_snapshots set
2020/04/16-06:24:06  Initializing worker threads
Crash-recovery verification passed :)
2020/04/16-06:24:06  Starting database operations
KILLED 4710

koldat · 2020-04-16T10:02:49Z

Our stress test does not show any deviation in response times and NO_FILES_OPENS looks beautiful.

facebook-github-bot

@siying has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

…nt requests. Fixes (facebook#6699)

facebook-github-bot · 2020-04-18T11:33:03Z

@koldat has updated the pull request. Re-import the pull request

koldat · 2020-04-18T11:37:50Z

Fixed reviews and squashed / rebased. Please let me know if you want to do further modifications.

facebook-github-bot

@siying has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2020-04-21T20:38:33Z

This pull request has been merged in 6ee66cf.

koldat · 2020-04-22T14:23:40Z

If anyone interested, I have posted LRU concurrent performance analysis here: https://groups.google.com/forum/#!topic/rocksdb/1_uo3Pr6DiE

facebook-github-bot added the CLA Signed label Apr 15, 2020

siying reviewed Apr 15, 2020

View reviewed changes

facebook-github-bot reviewed Apr 15, 2020

View reviewed changes

siying reviewed Apr 15, 2020

View reviewed changes

facebook-github-bot reviewed Apr 16, 2020

View reviewed changes

Prevents Table Cache to open same files more times in highly concurre…

c315442

…nt requests. Fixes (facebook#6699)

koldat force-pushed the table_cache_loader branch from a011dc9 to c315442 Compare April 18, 2020 11:32

facebook-github-bot reviewed Apr 20, 2020

View reviewed changes

siying approved these changes Apr 20, 2020

View reviewed changes

facebook-github-bot closed this in 6ee66cf Apr 21, 2020

facebook-github-bot added the Merged label Apr 21, 2020

koldat deleted the table_cache_loader branch April 22, 2020 14:23

pdillinger mentioned this pull request Apr 22, 2020

TableCache in highly concurrent query with limited max_open_files #6699

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prevents Table Cache to open same files more times #6707

Prevents Table Cache to open same files more times #6707

koldat commented Apr 15, 2020

koldat commented Apr 15, 2020

siying Apr 15, 2020

koldat Apr 15, 2020 •

edited

Loading

koldat Apr 15, 2020

facebook-github-bot left a comment

siying left a comment

siying Apr 15, 2020

koldat Apr 15, 2020

koldat Apr 18, 2020

koldat commented Apr 15, 2020

siying commented Apr 15, 2020

koldat commented Apr 15, 2020

koldat commented Apr 16, 2020

koldat commented Apr 16, 2020

facebook-github-bot left a comment

facebook-github-bot commented Apr 18, 2020

koldat commented Apr 18, 2020

facebook-github-bot left a comment

facebook-github-bot commented Apr 21, 2020

koldat commented Apr 22, 2020

		@@ -62,14 +62,17 @@ void AppendVarint64(IterKey* key, uint64_t v) {

		} // namespace

		#define LOAD_CONCURENCY 128

Prevents Table Cache to open same files more times #6707

Prevents Table Cache to open same files more times #6707

Conversation

koldat commented Apr 15, 2020

koldat commented Apr 15, 2020

siying Apr 15, 2020

Choose a reason for hiding this comment

koldat Apr 15, 2020 • edited Loading

Choose a reason for hiding this comment

koldat Apr 15, 2020

Choose a reason for hiding this comment

facebook-github-bot left a comment

Choose a reason for hiding this comment

siying left a comment

Choose a reason for hiding this comment

siying Apr 15, 2020

Choose a reason for hiding this comment

koldat Apr 15, 2020

Choose a reason for hiding this comment

koldat Apr 18, 2020

Choose a reason for hiding this comment

koldat commented Apr 15, 2020

siying commented Apr 15, 2020

koldat commented Apr 15, 2020

koldat commented Apr 16, 2020

koldat commented Apr 16, 2020

facebook-github-bot left a comment

Choose a reason for hiding this comment

facebook-github-bot commented Apr 18, 2020

koldat commented Apr 18, 2020

facebook-github-bot left a comment

Choose a reason for hiding this comment

facebook-github-bot commented Apr 21, 2020

koldat commented Apr 22, 2020

koldat Apr 15, 2020 •

edited

Loading