Lock free MultiGet #4754
Lock free MultiGet #4754
Conversation
There was a problem hiding this comment.
@anand1976 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
db/db_impl.cc
Outdated
| mgcfd->cfd = cfd; | ||
| struct MultiGetColumnFamilyData mgcfd; | ||
| mgcfd.cfd = cfd; | ||
| mgcfd.super_version = nullptr; |
There was a problem hiding this comment.
The most typical way of avoiding this new is multiget_cf_data.emplace(...).
There was a problem hiding this comment.
Good point. I made the change
|
@anand1976 has updated the pull request. Re-import the pull request |
There was a problem hiding this comment.
@anand1976 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
|
@ajkr should give it another pass, but it looks good to me. |
db/db_impl.cc
Outdated
| }; | ||
| std::unordered_map<uint32_t, MultiGetColumnFamilyData*> multiget_cf_data; | ||
| // fill up and allocate outside of mutex | ||
| std::unordered_map<uint32_t, MultiGetColumnFamilyData> multiget_cf_data(column_family.size()); |
db/db_impl.cc
Outdated
| auto mgcfd = new MultiGetColumnFamilyData(); | ||
| mgcfd->cfd = cfd; | ||
| multiget_cf_data.insert({cfd->GetID(), mgcfd}); | ||
| multiget_cf_data.emplace(cfd->GetID(), MultiGetColumnFamilyData(cfd, nullptr)); |
There was a problem hiding this comment.
I belive this should be multiget_cf_data.emplace(cfd->GetID(), cfd, nullptr)
There was a problem hiding this comment.
It doesn't work. It has to be something like multiget_cf_data.emplace(std::piecewise_construct, std::forward_as_tuple(cfd->GetID()), std::forward_as_typle(cfd, nullptr)), but the current version looks simpler and not much overhead anyway.
|
@anand1976 has updated the pull request. Re-import the pull request |
Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:
Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:
Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:
Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:
Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:
Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:
|
@anand1976 has updated the pull request. Re-import the pull request |
There was a problem hiding this comment.
@anand1976 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
Avoid locking the DB mutex in order to reference SuperVersions. Instead, we get the thread local cached SuperVersion for each column family in the list. It depends on finding a sequence number that overlaps with all the open memtables. We start with the latest published sequence number, and if any of the memtables is sealed before we can get all the SuperVersions, the process is repeated. After a few times, give up and lock the DB mutex.
Tests:
TEST_TMPDIR=/dev/shm ./db_bench -use_existing_db=true -benchmarks=readrandom -write_buffer_size=4194304 -target_file_size_base=4194304 -max_bytes_for_level_base=16777216 -num=5000000 -reads=1000000 -threads=32 -compression_type=none -cache_size=1048576000 -batch_size=1 -bloom_bits=1
readrandom : 0.167 micros/op 5983920 ops/sec; 426.2 MB/s (1000000 of 1000000 found)
Multireadrandom with batch size 1:
multireadrandom : 0.176 micros/op 5684033 ops/sec; (1000000 of 1000000 found)