add multibatch write into memtable #131
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
yiwu-arbug
reviewed
Dec 18, 2019
db/db_test_util.cc
Outdated
| @@ -550,6 +550,13 @@ Options DBTestBase::GetOptions( | |||
| options.enable_pipelined_write = true; | |||
| break; | |||
| } | |||
| case kMultiThreadWrite: { | |||
| if (options.allow_concurrent_memtable_write) { | |||
There was a problem hiding this comment.
should this be enable_multi_thread_write = true regardless of allow_concurrent_memtable_write value?
There was a problem hiding this comment.
I can not understand. allow_concurrent_memtable_write is necessary so that we can insert into skiplist concurrently.
There was a problem hiding this comment.
But DBImpl::SanitizeOptions is setting allow_concurrent_memtable_write=true when enable_multi_thread_write=true is set.
| @@ -114,6 +117,7 @@ class WriteThread { | |||
| // Information kept for every waiting writer. | |||
| struct Writer { | |||
| WriteBatch* batch; | |||
There was a problem hiding this comment.
Is it easy to remove batch in favor of batches?
| while (memtable_write_group.running.load(std::memory_order_acquire) > 0) { | ||
| std::function<void()> work; | ||
| if (write_thread_.write_queue_.PopFront(work)) { | ||
| work(); |
There was a problem hiding this comment.
here the leader works on the writer queue. how does followers help with the work?
There was a problem hiding this comment.
Followers would do this work in AwaitState.
yiwu-arbug
reviewed
Dec 19, 2019
| @@ -137,6 +141,43 @@ uint8_t WriteThread::AwaitState(Writer* w, uint8_t goal_mask, | |||
| // 1/sampling_base. | |||
| const int sampling_base = 256; | |||
|
|
|||
| if (enable_multi_thread_write_) { | |||
| #ifdef NDEBUG | |||
There was a problem hiding this comment.
what's the reason debug build and release build logic are different?
There was a problem hiding this comment.
Because unit test need to block and wait SYNC_POINT to wake up. But in product environment, we need it not block thread so that it could do more jobs.
There was a problem hiding this comment.
Would be great if release build and debug build can use the same logic. Otherwise rocksdb unit tests (maybe also tikv unit tests and schrodinger tests?) is not testing the real logic.
yiwu-arbug
reviewed
Dec 19, 2019
Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
Little-Wallace
force-pushed
the
multibatch
branch
from
2670855
to
a6637d7
Compare
Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
|
/run-all-tests |
yiwu-arbug
approved these changes
Dec 25, 2019
|
/run-all-tests |
tabokie
pushed a commit
to tabokie/rocksdb
that referenced
this pull request
May 12, 2022
Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> Support write multiple WriteBatch into RocksDB together. These WriteBatch will be assigned sequence number in order and pushed into queue. If a thread is waiting for some state, it could steal some job from work queue. Signed-off-by: tabokie <xy.tao@outlook.com>
tabokie
added a commit
that referenced
this pull request
May 12, 2022
* add multibatch write into memtable (#131) Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> Support write multiple WriteBatch into RocksDB together. These WriteBatch will be assigned sequence number in order and pushed into queue. If a thread is waiting for some state, it could steal some job from work queue. Signed-off-by: tabokie <xy.tao@outlook.com> * Improve Multi Batch Write (#154) * perform like normal pipelined write Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> Signed-off-by: tabokie <xy.tao@outlook.com> * pass enable_multi_thread_write to BuildDBOptions (#170) Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> Signed-off-by: tabokie <xy.tao@outlook.com> * Fix life time of `memtable_write_group` (#171) * fix life time of memtable_write_group Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> Signed-off-by: tabokie <xy.tao@outlook.com> * Commit pipeline when write a WriteBatch for linearizability (#267) * support commit pipeline Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> Signed-off-by: tabokie <xy.tao@outlook.com> * format Signed-off-by: tabokie <xy.tao@outlook.com> * remove useless code Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: Wallace <bupt2013211450@gmail.com>
tabokie
added a commit
that referenced
this pull request
May 25, 2022
* add multibatch write into memtable (#131) Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> Support write multiple WriteBatch into RocksDB together. These WriteBatch will be assigned sequence number in order and pushed into queue. If a thread is waiting for some state, it could steal some job from work queue. Signed-off-by: tabokie <xy.tao@outlook.com> * Improve Multi Batch Write (#154) * perform like normal pipelined write Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> Signed-off-by: tabokie <xy.tao@outlook.com> * pass enable_multi_thread_write to BuildDBOptions (#170) Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> Signed-off-by: tabokie <xy.tao@outlook.com> * Fix life time of `memtable_write_group` (#171) * fix life time of memtable_write_group Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> Signed-off-by: tabokie <xy.tao@outlook.com> * Commit pipeline when write a WriteBatch for linearizability (#267) * support commit pipeline Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> Signed-off-by: tabokie <xy.tao@outlook.com> * format Signed-off-by: tabokie <xy.tao@outlook.com> * remove useless code Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: Wallace <bupt2013211450@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Little-Wallace bupt2013211450@gmail.com
Support write multiple WriteBatch into RocksDB together. These WriteBatch will be assigned sequence number in order and pushed into queue. If a thread is waiting for some state, it could steal some job from work queue.