Optimization for avoiding bloom filter during compaction is gone #14091
Comments
|
Fixed by 38b226f. |
|
@raphaelsc - any idea why this issue is still open? |
|
#13908 is the one fixing it, I reckon? |
That's my fault. Closing it with 38b226f... |
|
@raphaelsc , @avikivity - do we plan to backport this? |
It's fixing a regression, so my understanding is yes, we need to backport this. Users that are using, default TTL / heavy deletion, can be really affected by this. |
mykaul
added
backport/5.2
|
@raphaelsc , @avikivity - please backport then. |
|
@raphaelsc please prepare backports. |
|
raphaelsc
added a commit
to raphaelsc/scylla
that referenced
this issue
Oct 19, 2023
Commit 8c4b5e4 introduced an optimization which only calculates max purgeable timestamp when a tombstone satisfy the grace period. Commit 'repair: Get rid of the gc_grace_seconds' inverted the order, probably under the assumption that getting grace period can be more expensive than calculating max purgeable, as repair-mode GC will look up into history data in order to calculate gc_before. This caused a significant regression on tombstone heavy compactions, where most of tombstones are still newer than grace period. A compaction which used to take 5s, now takes 35s. 7x slower. The reason is simple, now calculation of max purgeable happens for every single tombstone (once for each key), even the ones that cannot be GC'ed yet. And each calculation has to iterate through (i.e. check the bloom filter of) every single sstable that doesn't participate in compaction. Flame graph makes it very clear that bloom filter is a heavy path without the optimization: 45.64% 45.64% sstable_compact sstable_compaction_test_g [.] utils::filter::bloom_filter::is_present With its resurrection, the problem is gone. This scenario can easily happen, e.g. after a deletion burst, and tombstones becoming only GC'able after they reach upper tiers in the LSM tree. Before this patch, a compaction can be estimated to have this # of filter checks: (# of keys containing *any* tombstone) * (# of uncompacting sstable runs[1]) [1] It's # of *runs*, as each key tend to overlap with only one fragment of each run. After this patch, the estimation becomes: (# of keys containing a GC'able tombstone) * (# of uncompacting runs). With repair mode for tombstone GC, the assumption, that retrieval of gc_before is more expensive than calculating max purgeable, is kept. We can revisit it later. But the default mode, which is the "timeout" (i.e. gc_grace_seconds) one, we still benefit from the optimization of deferring the calculation until needed. Cherry picked from commit 38b226f Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com> Fixes scylladb#14091. Closes scylladb#13908 Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
raphaelsc
added a commit
to raphaelsc/scylla
that referenced
this issue
Oct 19, 2023
Commit 8c4b5e4 introduced an optimization which only calculates max purgeable timestamp when a tombstone satisfy the grace period. Commit 'repair: Get rid of the gc_grace_seconds' inverted the order, probably under the assumption that getting grace period can be more expensive than calculating max purgeable, as repair-mode GC will look up into history data in order to calculate gc_before. This caused a significant regression on tombstone heavy compactions, where most of tombstones are still newer than grace period. A compaction which used to take 5s, now takes 35s. 7x slower. The reason is simple, now calculation of max purgeable happens for every single tombstone (once for each key), even the ones that cannot be GC'ed yet. And each calculation has to iterate through (i.e. check the bloom filter of) every single sstable that doesn't participate in compaction. Flame graph makes it very clear that bloom filter is a heavy path without the optimization: 45.64% 45.64% sstable_compact sstable_compaction_test_g [.] utils::filter::bloom_filter::is_present With its resurrection, the problem is gone. This scenario can easily happen, e.g. after a deletion burst, and tombstones becoming only GC'able after they reach upper tiers in the LSM tree. Before this patch, a compaction can be estimated to have this # of filter checks: (# of keys containing *any* tombstone) * (# of uncompacting sstable runs[1]) [1] It's # of *runs*, as each key tend to overlap with only one fragment of each run. After this patch, the estimation becomes: (# of keys containing a GC'able tombstone) * (# of uncompacting runs). With repair mode for tombstone GC, the assumption, that retrieval of gc_before is more expensive than calculating max purgeable, is kept. We can revisit it later. But the default mode, which is the "timeout" (i.e. gc_grace_seconds) one, we still benefit from the optimization of deferring the calculation until needed. Cherry picked from commit 38b226f Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com> Fixes scylladb#14091. Closes scylladb#13908 Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
denesb
pushed a commit
that referenced
this issue
Oct 20, 2023
Commit 8c4b5e4 introduced an optimization which only calculates max purgeable timestamp when a tombstone satisfy the grace period. Commit 'repair: Get rid of the gc_grace_seconds' inverted the order, probably under the assumption that getting grace period can be more expensive than calculating max purgeable, as repair-mode GC will look up into history data in order to calculate gc_before. This caused a significant regression on tombstone heavy compactions, where most of tombstones are still newer than grace period. A compaction which used to take 5s, now takes 35s. 7x slower. The reason is simple, now calculation of max purgeable happens for every single tombstone (once for each key), even the ones that cannot be GC'ed yet. And each calculation has to iterate through (i.e. check the bloom filter of) every single sstable that doesn't participate in compaction. Flame graph makes it very clear that bloom filter is a heavy path without the optimization: 45.64% 45.64% sstable_compact sstable_compaction_test_g [.] utils::filter::bloom_filter::is_present With its resurrection, the problem is gone. This scenario can easily happen, e.g. after a deletion burst, and tombstones becoming only GC'able after they reach upper tiers in the LSM tree. Before this patch, a compaction can be estimated to have this # of filter checks: (# of keys containing *any* tombstone) * (# of uncompacting sstable runs[1]) [1] It's # of *runs*, as each key tend to overlap with only one fragment of each run. After this patch, the estimation becomes: (# of keys containing a GC'able tombstone) * (# of uncompacting runs). With repair mode for tombstone GC, the assumption, that retrieval of gc_before is more expensive than calculating max purgeable, is kept. We can revisit it later. But the default mode, which is the "timeout" (i.e. gc_grace_seconds) one, we still benefit from the optimization of deferring the calculation until needed. Cherry picked from commit 38b226f Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com> Fixes #14091. Closes #13908 Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com> Closes #15744
denesb
pushed a commit
that referenced
this issue
Oct 20, 2023
Commit 8c4b5e4 introduced an optimization which only calculates max purgeable timestamp when a tombstone satisfy the grace period. Commit 'repair: Get rid of the gc_grace_seconds' inverted the order, probably under the assumption that getting grace period can be more expensive than calculating max purgeable, as repair-mode GC will look up into history data in order to calculate gc_before. This caused a significant regression on tombstone heavy compactions, where most of tombstones are still newer than grace period. A compaction which used to take 5s, now takes 35s. 7x slower. The reason is simple, now calculation of max purgeable happens for every single tombstone (once for each key), even the ones that cannot be GC'ed yet. And each calculation has to iterate through (i.e. check the bloom filter of) every single sstable that doesn't participate in compaction. Flame graph makes it very clear that bloom filter is a heavy path without the optimization: 45.64% 45.64% sstable_compact sstable_compaction_test_g [.] utils::filter::bloom_filter::is_present With its resurrection, the problem is gone. This scenario can easily happen, e.g. after a deletion burst, and tombstones becoming only GC'able after they reach upper tiers in the LSM tree. Before this patch, a compaction can be estimated to have this # of filter checks: (# of keys containing *any* tombstone) * (# of uncompacting sstable runs[1]) [1] It's # of *runs*, as each key tend to overlap with only one fragment of each run. After this patch, the estimation becomes: (# of keys containing a GC'able tombstone) * (# of uncompacting runs). With repair mode for tombstone GC, the assumption, that retrieval of gc_before is more expensive than calculating max purgeable, is kept. We can revisit it later. But the default mode, which is the "timeout" (i.e. gc_grace_seconds) one, we still benefit from the optimization of deferring the calculation until needed. Cherry picked from commit 38b226f Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com> Fixes #14091. Closes #13908 Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com> Closes #15745
|
Backports queued, removing backport candidate labels. |
denesb
removed
Backport candidate
backport/5.2
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Commit 8c4b5e4 introduced an optimization which only calculates max purgeable timestamp when a tombstone satisfy the grace period.
Commit 'repair: Get rid of the gc_grace_seconds' removed it.
Compaction is much slower as a result when facing tombstone heavy workloads.
The text was updated successfully, but these errors were encountered: