Add an option to trigger flush when the number of range deletions reach a threshold #11358

vrdhn · 2023-04-07T17:15:37Z

Add a mutable column family option memtable_max_range_deletions. When non-zero, RocksDB will try to flush the current memtable after it has at least memtable_max_range_deletions range deletions. Java API is added and crash test is updated accordingly to randomly enable this option.

Test plan:

New unit test: DBRangeDelTest.MemtableMaxRangeDeletions
Ran crash test python3 ./tools/db_crashtest.py whitebox --simple --memtable_max_range_deletions=20 and saw logs showing flushed memtables usually with 20 range deletions.

include/rocksdb/options.h

yao-xiao-github

Thanks for making the change and it's very helpful for our use case (#11407)!

db/memtable.cc

options/cf_options.h

yao-xiao-github · 2023-04-26T20:12:36Z

db/memtable.cc

@@ -170,6 +172,11 @@ size_t MemTable::ApproximateMemoryUsage() {
 }

 bool MemTable::ShouldFlushNow() {


It is possible to log the reason for flush? Maybe consider adding the reason to FlushReason .

Okay, let me see if i can identify where to generate the flush reason.

vrdhn · 2023-04-28T17:52:21Z

hi @yao-xiao-github ,@cbi42 I did the requested changes. I couldn't figure out where to set the FlushReason.
Any hints will be hightly appreciated.
Thanks.

yao-xiao-github · 2023-05-05T23:03:56Z

hi @yao-xiao-github ,@cbi42 I did the requested changes. I couldn't figure out where to set the FlushReason.
Any hints will be hightly appreciated.
Thanks.

Thanks for making the changes. I also tried to expose the FlushReason but didn't find a good way. All the memtable related flushes seem to be put under the same reason. @cbi42 Could you provide some hints? Or is it possible to put info log here?

cbi42 · 2023-05-08T18:44:53Z

hi @yao-xiao-github ,@cbi42 I did the requested changes. I couldn't figure out where to set the FlushReason.
Any hints will be hightly appreciated.
Thanks.

Thanks for making the changes. I also tried to expose the FlushReason but didn't find a good way. All the memtable related flushes seem to be put under the same reason. @cbi42 Could you provide some hints? Or is it possible to put info log here?

Hi - I also don't see an easy way to add FlushReason for this change, especially because ShouldFlushNow() was exclusively used to check if a memtable was full. Maybe we can just log the number of range tombstones here for now:

rocksdb/db/flush_job.cc

Line 883 in a5909f8

<< GetFlushReasonString(flush_reason_);

Also could you add a unit test to check that flush is triggered correctly based on the new option?

saintstack

Don't have perms to +1 this PR but LGTM.

Patched a 8.1.1 checkout then ran against a dataset that was stuck in recovery, OOM'ing every time because too many range deletes (Yao's case). Ran patched binary with all defaults and still OOM. Then I set memtable_max_range_deletions=1000, recompiled and retried. Lots of flushes but was able to open the database.

saintstack · 2023-05-11T05:12:16Z

include/rocksdb/options.h

@@ -331,6 +331,11 @@ struct ColumnFamilyOptions : public AdvancedColumnFamilyOptions {
  // Default: nullptr
  std::shared_ptr<SstPartitionerFactory> sst_partitioner_factory = nullptr;

+  // Automatic flush after range deletions count in memtable hits this limit.
+  // helps with workloads having lot of range deletes.


Suggestion: If making another edit, you might say more than 'helps'; it is a guard against accumulating too many range deletes in memtable such that you get into a state where you cannot flush w/o OOM'ing.

options/options.cc

yao-xiao-github · 2023-06-06T22:47:42Z

@cbi42 Could you please take another look? Thanks!

cbi42 · 2023-06-08T07:21:22Z

@cbi42 Could you please take another look? Thanks!

Hi - there are some requested changes like testing and logging that are not done. I have made these changes locally but realize this PR is not open for maintainer to edit yet. @vrdhn - can you update this PR to allow maintainer to edit? Thanks.

bremac · 2023-06-08T22:55:39Z

@cbi42 Could you please take another look? Thanks!

Hi - there are some requested changes like testing and logging that are not done. I have made these changes locally but realize this PR is not open for maintainer to edit yet. @vrdhn - can you update this PR to allow maintainer to edit? Thanks.

Hi @cbi42. Apparently GitHub forbids opening PRs for a maintainer to edit on repos associated with an organization. At @vrdhn's request I did make you a contributor on the repo. Let me know if that works for you. Otherwise, we can create a new PR from a repo associated with a personal account instead.

facebook-github-bot · 2023-06-09T01:28:34Z

@cbi42 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2023-06-21T18:11:36Z

@vrdhn has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2023-06-21T19:32:51Z

@cbi42 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2023-07-05T21:24:24Z

@vrdhn has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2023-07-05T21:25:13Z

@cbi42 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

ajkr · 2023-07-21T00:53:02Z

db/memtable.h

+      if (memtable_max_range_deletions_ > 0 &&
+          memtable_max_range_deletions_reached_ == false &&
+          val >= (uint64_t)memtable_max_range_deletions_) {
+        memtable_max_range_deletions_reached_ = true;


This feels like a data race. What about comparing num_range_deletes_ against memtable_max_range_deletions_ in ShouldFlushNow()?

Related (similar scenario where the flag started with one value and could only ever flip once to the other value): #4801

Updated to check num_range_deletes_ against memtable_max_range_deletions_ in ShouldFlushNow().

The idea is to force flush after few range-delete operations. Experimentation shows 32 or 64 giving good performance. memtable_max_range_deletions is available as a ColumnFamilyOption. The default is 0, which disables. it.

facebook-github-bot · 2023-07-31T22:16:09Z

@vrdhn has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2023-07-31T22:25:37Z

@cbi42 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2023-08-01T00:19:47Z

@vrdhn has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2023-08-01T00:21:13Z

@cbi42 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

ajkr

Looks great!

facebook-github-bot · 2023-08-03T00:03:19Z

@vrdhn has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2023-08-03T00:09:33Z

@cbi42 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2023-08-03T03:02:41Z

@cbi42 merged this pull request in 87a21d0.

…ch a threshold (#11358) Summary: Add a mutable column family option `memtable_max_range_deletions`. When non-zero, RocksDB will try to flush the current memtable after it has at least `memtable_max_range_deletions` range deletions. Java API is added and crash test is updated accordingly to randomly enable this option. Pull Request resolved: facebook/rocksdb#11358 Test Plan: * New unit test: `DBRangeDelTest.MemtableMaxRangeDeletions` * Ran crash test `python3 ./tools/db_crashtest.py whitebox --simple --memtable_max_range_deletions=20` and saw logs showing flushed memtables usually with 20 range deletions. Reviewed By: ajkr Differential Revision: D46582680 Pulled By: cbi42 fbshipit-source-id: f23d6fa8d8264ecf0a18d55c113ba03f5e2504da

facebook-github-bot added the CLA Signed label Apr 7, 2023

vrdhn force-pushed the vrdhn-upstream-range-del-read branch 2 times, most recently from 2287f58 to 857b8eb Compare April 7, 2023 17:18

vrdhn changed the title ~~Improve performance the mix of range-delete and read workloads.~~ Improve performance of mix of range-delete and read workloads. Apr 8, 2023

cbi42 reviewed Apr 25, 2023

View reviewed changes

include/rocksdb/options.h Outdated Show resolved Hide resolved

cbi42 mentioned this pull request Apr 25, 2023

OOM while flushing memtable with range deletes #11407

Open

yao-xiao-github reviewed Apr 26, 2023

View reviewed changes

vrdhn force-pushed the vrdhn-upstream-range-del-read branch from 857b8eb to b44ef52 Compare April 28, 2023 17:49

vrdhn force-pushed the vrdhn-upstream-range-del-read branch from b44ef52 to e7949f0 Compare May 2, 2023 01:56

saintstack reviewed May 11, 2023

View reviewed changes

vrdhn force-pushed the vrdhn-upstream-range-del-read branch 2 times, most recently from 2222b8e to a1e333c Compare May 24, 2023 17:05

cbi42 self-assigned this Jun 8, 2023

cbi42 force-pushed the vrdhn-upstream-range-del-read branch 2 times, most recently from 0a99266 to 956cb69 Compare June 9, 2023 01:25

cbi42 requested a review from ajkr June 9, 2023 01:30

ajkr reviewed Jul 21, 2023

View reviewed changes

cbi42 changed the title ~~Improve performance of mix of range-delete and read workloads.~~ Add option to trigger flush when the number of range deletions reach a threshold Jul 26, 2023

cbi42 changed the title ~~Add option to trigger flush when the number of range deletions reach a threshold~~ Add an option to trigger flush when the number of range deletions reach a threshold Jul 27, 2023

vrdhn and others added 14 commits July 31, 2023 14:33

Improve performance the mix of range-delete and read workloads.

e2352bc

The idea is to force flush after few range-delete operations. Experimentation shows 32 or 64 giving good performance. memtable_max_range_deletions is available as a ColumnFamilyOption. The default is 0, which disables. it.

Update option comment.

fe6b11e

Count range tombstones whether concurrent update or not.

f0fef7a

Make option mutable.

e828729

Log num_range_deletes.

91de341

Fix option logging indent.

bddc35c

Add TODO for correct flush_reson.

64b7ee2

Correct num_range_deletes counting.

36b83bf

Add unit test.

9c03ad0

Add change log.

d0628f9

Add cast.

88ed110

Remove flag memtable_max_range_deletions_reached_.

e59a4d2

Update num_range_delets_ correctly.

4135a99

Enable option in crash test.

bdfe899

cbi42 force-pushed the vrdhn-upstream-range-del-read branch from 4c6d625 to bdfe899 Compare July 31, 2023 22:16

Fix Java API comment.

8663e79

ajkr approved these changes Aug 2, 2023

View reviewed changes

Merge branch 'main' into vrdhn-upstream-range-del-read

2629a8c

facebook-github-bot closed this in 87a21d0 Aug 3, 2023

facebook-github-bot added the Merged label Aug 3, 2023

igorcanadi mentioned this pull request Jan 17, 2024

[SYS-6913] Upgrade RocksDB-Cloud to 8.9.1 rockset/rocksdb-cloud#315

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add an option to trigger flush when the number of range deletions reach a threshold #11358

Add an option to trigger flush when the number of range deletions reach a threshold #11358

vrdhn commented Apr 7, 2023 •

edited by cbi42

yao-xiao-github left a comment

yao-xiao-github Apr 26, 2023

vrdhn Apr 27, 2023

vrdhn commented Apr 28, 2023

yao-xiao-github commented May 5, 2023

cbi42 commented May 8, 2023

saintstack left a comment

saintstack May 11, 2023

yao-xiao-github commented Jun 6, 2023

cbi42 commented Jun 8, 2023

bremac commented Jun 8, 2023

facebook-github-bot commented Jun 9, 2023

facebook-github-bot commented Jun 21, 2023

facebook-github-bot commented Jun 21, 2023

facebook-github-bot commented Jul 5, 2023

facebook-github-bot commented Jul 5, 2023

ajkr Jul 21, 2023

ajkr Jul 21, 2023

cbi42 Jul 31, 2023

facebook-github-bot commented Jul 31, 2023

facebook-github-bot commented Jul 31, 2023

facebook-github-bot commented Aug 1, 2023

facebook-github-bot commented Aug 1, 2023

ajkr left a comment

facebook-github-bot commented Aug 3, 2023

facebook-github-bot commented Aug 3, 2023

facebook-github-bot commented Aug 3, 2023

		@@ -170,6 +172,11 @@ size_t MemTable::ApproximateMemoryUsage() {
		}

		bool MemTable::ShouldFlushNow() {

Add an option to trigger flush when the number of range deletions reach a threshold #11358

Add an option to trigger flush when the number of range deletions reach a threshold #11358

Conversation

vrdhn commented Apr 7, 2023 • edited by cbi42

yao-xiao-github left a comment

Choose a reason for hiding this comment

yao-xiao-github Apr 26, 2023

Choose a reason for hiding this comment

vrdhn Apr 27, 2023

Choose a reason for hiding this comment

vrdhn commented Apr 28, 2023

yao-xiao-github commented May 5, 2023

cbi42 commented May 8, 2023

saintstack left a comment

Choose a reason for hiding this comment

saintstack May 11, 2023

Choose a reason for hiding this comment

yao-xiao-github commented Jun 6, 2023

cbi42 commented Jun 8, 2023

bremac commented Jun 8, 2023

facebook-github-bot commented Jun 9, 2023

facebook-github-bot commented Jun 21, 2023

facebook-github-bot commented Jun 21, 2023

facebook-github-bot commented Jul 5, 2023

facebook-github-bot commented Jul 5, 2023

ajkr Jul 21, 2023

Choose a reason for hiding this comment

ajkr Jul 21, 2023

Choose a reason for hiding this comment

cbi42 Jul 31, 2023

Choose a reason for hiding this comment

facebook-github-bot commented Jul 31, 2023

facebook-github-bot commented Jul 31, 2023

facebook-github-bot commented Aug 1, 2023

facebook-github-bot commented Aug 1, 2023

ajkr left a comment

Choose a reason for hiding this comment

facebook-github-bot commented Aug 3, 2023

facebook-github-bot commented Aug 3, 2023

facebook-github-bot commented Aug 3, 2023

vrdhn commented Apr 7, 2023 •

edited by cbi42