New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tombstone gc with 'immediate' mode deletes all of records when flush to sstable #13572
Comments
The immediate mode is similar to timeout mode with gc_grace_seconds zero. Thus, the gc_before returned should be the query_time instead of gc_clock::time_point::max in immediate mode. Setting gc_before to gc_clock::time_point::max, a row could be dropped by compaction even if the ttl is not expired yet. The following procedure reproduces the issue: - Start 2 nodes - Insert data ``` CREATE KEYSPACE ks2a WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 2 }; CREATE TABLE ks2a.tb (pk int, ck int, c0 text, c1 text, c2 text, PRIMARY KEY(pk, ck)) WITH tombstone_gc = {'mode': 'immediate'}; INSERT into ks2a.tb (pk,ck, c0, c1, c2) values (10 ,1, 'x', 'y', 'z') USING TTL 1000000; INSERT into ks2a.tb (pk,ck, c0, c1, c2) values (20 ,1, 'x', 'y', 'z') USING TTL 1000000; INSERT into ks2a.tb (pk,ck, c0, c1, c2) values (30 ,1, 'x', 'y', 'z') USING TTL 1000000; ``` - Run nodetool flush and nodetool compact - Compaction drops all data ``` ~128 total partitions merged to 0. ``` Fixes scylladb#13572
@glenn-kim Thanks for the report. A fix is posted here #13800. |
The immediate mode is similar to timeout mode with gc_grace_seconds zero. Thus, the gc_before returned should be the query_time instead of gc_clock::time_point::max in immediate mode. Setting gc_before to gc_clock::time_point::max, a row could be dropped by compaction even if the ttl is not expired yet. The following procedure reproduces the issue: - Start 2 nodes - Insert data ``` CREATE KEYSPACE ks2a WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 2 }; CREATE TABLE ks2a.tb (pk int, ck int, c0 text, c1 text, c2 text, PRIMARY KEY(pk, ck)) WITH tombstone_gc = {'mode': 'immediate'}; INSERT into ks2a.tb (pk,ck, c0, c1, c2) values (10 ,1, 'x', 'y', 'z') USING TTL 1000000; INSERT into ks2a.tb (pk,ck, c0, c1, c2) values (20 ,1, 'x', 'y', 'z') USING TTL 1000000; INSERT into ks2a.tb (pk,ck, c0, c1, c2) values (30 ,1, 'x', 'y', 'z') USING TTL 1000000; ``` - Run nodetool flush and nodetool compact - Compaction drops all data ``` ~128 total partitions merged to 0. ``` Fixes scylladb#13572
The immediate mode is similar to timeout mode with gc_grace_seconds zero. Thus, the gc_before returned should be the query_time instead of gc_clock::time_point::max in immediate mode. Setting gc_before to gc_clock::time_point::max, a row could be dropped by compaction even if the ttl is not expired yet. The following procedure reproduces the issue: - Start 2 nodes - Insert data ``` CREATE KEYSPACE ks2a WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 2 }; CREATE TABLE ks2a.tb (pk int, ck int, c0 text, c1 text, c2 text, PRIMARY KEY(pk, ck)) WITH tombstone_gc = {'mode': 'immediate'}; INSERT into ks2a.tb (pk,ck, c0, c1, c2) values (10 ,1, 'x', 'y', 'z') USING TTL 1000000; INSERT into ks2a.tb (pk,ck, c0, c1, c2) values (20 ,1, 'x', 'y', 'z') USING TTL 1000000; INSERT into ks2a.tb (pk,ck, c0, c1, c2) values (30 ,1, 'x', 'y', 'z') USING TTL 1000000; ``` - Run nodetool flush and nodetool compact - Compaction drops all data ``` ~128 total partitions merged to 0. ``` Fixes scylladb#13572
The immediate mode is similar to timeout mode with gc_grace_seconds zero. Thus, the gc_before returned should be the query_time instead of gc_clock::time_point::max in immediate mode. Setting gc_before to gc_clock::time_point::max, a row could be dropped by compaction even if the ttl is not expired yet. The following procedure reproduces the issue: - Start 2 nodes - Insert data ``` CREATE KEYSPACE ks2a WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 2 }; CREATE TABLE ks2a.tb (pk int, ck int, c0 text, c1 text, c2 text, PRIMARY KEY(pk, ck)) WITH tombstone_gc = {'mode': 'immediate'}; INSERT into ks2a.tb (pk,ck, c0, c1, c2) values (10 ,1, 'x', 'y', 'z') USING TTL 1000000; INSERT into ks2a.tb (pk,ck, c0, c1, c2) values (20 ,1, 'x', 'y', 'z') USING TTL 1000000; INSERT into ks2a.tb (pk,ck, c0, c1, c2) values (30 ,1, 'x', 'y', 'z') USING TTL 1000000; ``` - Run nodetool flush and nodetool compact - Compaction drops all data ``` ~128 total partitions merged to 0. ``` Fixes scylladb#13572 Closes scylladb#13800
While searching for when this feature was introduced (to determine backport candidate versions), I found that this feature is either not documented at all or it is very hard to find. I tried looking in our |
The immediate mode is similar to timeout mode with gc_grace_seconds zero. Thus, the gc_before returned should be the query_time instead of gc_clock::time_point::max in immediate mode. Setting gc_before to gc_clock::time_point::max, a row could be dropped by compaction even if the ttl is not expired yet. The following procedure reproduces the issue: - Start 2 nodes - Insert data ``` CREATE KEYSPACE ks2a WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 2 }; CREATE TABLE ks2a.tb (pk int, ck int, c0 text, c1 text, c2 text, PRIMARY KEY(pk, ck)) WITH tombstone_gc = {'mode': 'immediate'}; INSERT into ks2a.tb (pk,ck, c0, c1, c2) values (10 ,1, 'x', 'y', 'z') USING TTL 1000000; INSERT into ks2a.tb (pk,ck, c0, c1, c2) values (20 ,1, 'x', 'y', 'z') USING TTL 1000000; INSERT into ks2a.tb (pk,ck, c0, c1, c2) values (30 ,1, 'x', 'y', 'z') USING TTL 1000000; ``` - Run nodetool flush and nodetool compact - Compaction drops all data ``` ~128 total partitions merged to 0. ``` Fixes #13572 Closes #13800 (cherry picked from commit 7fcc403)
The immediate mode is similar to timeout mode with gc_grace_seconds zero. Thus, the gc_before returned should be the query_time instead of gc_clock::time_point::max in immediate mode. Setting gc_before to gc_clock::time_point::max, a row could be dropped by compaction even if the ttl is not expired yet. The following procedure reproduces the issue: - Start 2 nodes - Insert data ``` CREATE KEYSPACE ks2a WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 2 }; CREATE TABLE ks2a.tb (pk int, ck int, c0 text, c1 text, c2 text, PRIMARY KEY(pk, ck)) WITH tombstone_gc = {'mode': 'immediate'}; INSERT into ks2a.tb (pk,ck, c0, c1, c2) values (10 ,1, 'x', 'y', 'z') USING TTL 1000000; INSERT into ks2a.tb (pk,ck, c0, c1, c2) values (20 ,1, 'x', 'y', 'z') USING TTL 1000000; INSERT into ks2a.tb (pk,ck, c0, c1, c2) values (30 ,1, 'x', 'y', 'z') USING TTL 1000000; ``` - Run nodetool flush and nodetool compact - Compaction drops all data ``` ~128 total partitions merged to 0. ``` Fixes #13572 Closes #13800 (cherry picked from commit 7fcc403)
Backported to 5.1, 5.2 and 2022.1. |
https://www.scylladb.com/2022/06/30/preventing-data-resurrection-with-repair-based-tombstone-garbage-collection/ (not a replacement for proper documentation of course) |
@mykaul Tombstones GC options is documented here https://docs.scylladb.com/stable/cql/ddl.html#tombstones-gc-options |
I worked with the doc team long time ago for this option. The link is here: https://docs.scylladb.com/stable/cql/ddl.html#tombstones-gc-options |
So is our docs search engine not that great? Perhaps we can improve by asking Google to index it or something? |
What search term did you use? |
Indeed - now I can find it (tried 'tombstone immediate' and got multiple reasons right away!). Strange. |
I can find it too now. Although it helps that I now know which page to look for. The first result is often talking about |
@annastuchlik please open an issue for the doc serach problems |
Sure, I can open an issue, but I'll need to understand better the problem you were facing. |
For "tombstone gc", the right page is the 5th. |
Tables with
TOMBSTONE_GC = {'mode': 'immediate'}
will delete all records with TTLs, regardless of whether they are expired or not.During the flush from memtable to sstable (compacting sstables originated memtable), I could see in the logs that all the data disappeared (compacted to 0 byte).
I changed the mode to 'timeout' and it flushed normally again.
https://github.com/scylladb/scylladb/blob/scylla-5.1.6/tombstone_gc.cc#L76-L79
According to the above code, if
tombstone_gc
isimmediate
,gc_before
is set to 'clock max'. It uses thisgc_before
value to determine whether sstable is fully expired or not, and my guess is that the expiration time of unexpired records is treated as "current time + remaining TTL" (in future), and since gc_before is very far in the future (clock max), so it is treated as expired.I think the
gc_before
value should be 'query time', not 'clock max'.To Reproduce
Steps to reproduce the behavior:
INSERT INTO test_table JSON ? USING TTL ?
Expected behavior
Logs
compaction logs with 'immediate' mode
after changed to 'timeout' mode
Installation details
The text was updated successfully, but these errors were encountered: