CNDB-15141: add LogTransaction#validate API to perform validation before txn prepare #2008
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Checklist before you submit for review
|
jasonstack
force-pushed
the
cndb-15141-main
branch
from
6757397 to
db0d670
Compare
jasonstack
force-pushed
the
cndb-15141-main
branch
from
d34a49c to
0033493
Compare
blambov
reviewed
Sep 23, 2025
| return; | ||
|
|
||
| // Early-open modifies obsolete SSTables' start keys, making validation unreliable | ||
| if (DatabaseDescriptor.getSSTablePreemptiveOpenIntervalInMB() > 0) |
There was a problem hiding this comment.
Outside of CNDB this is a very significant limitation as the database is expected to always run with early open enabled. Would it work to look at the matching entry for inputs in the CANONICAL sstable set?
(If not, we need to issue a warning somewhere when the combination of these two options is set.)
There was a problem hiding this comment.
Good point, I think it's possible to get the original sstables with original start position from LifecycleTransaction
There was a problem hiding this comment.
pushed e96829ba514609e3ca4cf3e4361bedb962cca284 to use the original sstables instead of logged.obsolete which are modified by early-open
blambov
approved these changes
Sep 24, 2025
jasonstack
force-pushed
the
cndb-15141-main
branch
2 times, most recently
from
021d5f8 to
8a98d05
Compare
blambov
reviewed
Sep 25, 2025
| /** | ||
| * Whether to enable compaction validation to detect potential data loss and abort compaction | ||
| */ | ||
| COMPACTION_VALIDATION_ENABLED("cassandra.compaction_validation_enabled", "true"), |
There was a problem hiding this comment.
Do we want this to be turned on by default everywhere? And do we want it in non-abort mode?
Related to this, I would change the flag to be an enum (e.g NONE/WARN/ABORT) to make the configuration cleaner.
There was a problem hiding this comment.
Pushed 3ab5de9f134 to change config to enum (NONE/WARN/ABORT).
For CC, the default is NONE.
For CNDB side, it would be ABORT for compactor and NONE for all other services.
wdyt?
There was a problem hiding this comment.
can you review cndb PR? https://github.com/riptano/cndb/pull/15431
jasonstack
force-pushed
the
cndb-15141-main
branch
2 times, most recently
from
b5d44cf to
3ab5de9
Compare
blambov
approved these changes
Sep 26, 2025
…erlapping info for validation
jasonstack
force-pushed
the
cndb-15141-main
branch
from
3ab5de9 to
c114914
Compare
|
✔️ Build ds-cassandra-pr-gate/PR-2008 approved by ButlerApproved by Butler |
michaelsembwever
pushed a commit
that referenced
this pull request
Oct 9, 2025
…lidation before txn prepare (#2008) riptano/cndb#15141 Added compaction task verification before transaction commit to validate if all source sstables' min/max keys are present in output sstables or expired if missing from output sstables. This is mainly to detect data loss caused by compaction strategy skipping some subranges of source sstables HCD-130. It works as following: 1. Extract all boundary keys (min/max) from source sstable 2. Check if boundary keys exist in output sstables a If all present, validation passes b. If any keys are missing, continue validation 3. For missing keys, read partition data from source sstables 4. Apply tombstone purging by using `gc_grace_seconds=0` 5. Check purged content a If there is no live data, validation passes. Keys are properly obsoleted b If there is live data, validation fails and throws to abort compaction if it's configured to abort Configs: * `cassandra.compaction_validation_mode=NONE`, available options: NONE/WARN/ABORT
michaelsembwever
pushed a commit
that referenced
this pull request
Oct 10, 2025
…lidation before txn prepare (#2008) riptano/cndb#15141 Added compaction task verification before transaction commit to validate if all source sstables' min/max keys are present in output sstables or expired if missing from output sstables. This is mainly to detect data loss caused by compaction strategy skipping some subranges of source sstables HCD-130. It works as following: 1. Extract all boundary keys (min/max) from source sstable 2. Check if boundary keys exist in output sstables a If all present, validation passes b. If any keys are missing, continue validation 3. For missing keys, read partition data from source sstables 4. Apply tombstone purging by using `gc_grace_seconds=0` 5. Check purged content a If there is no live data, validation passes. Keys are properly obsoleted b If there is live data, validation fails and throws to abort compaction if it's configured to abort Configs: * `cassandra.compaction_validation_mode=NONE`, available options: NONE/WARN/ABORT
michaelsembwever
pushed a commit
that referenced
this pull request
Oct 14, 2025
…lidation before txn prepare (#2008) riptano/cndb#15141 Added compaction task verification before transaction commit to validate if all source sstables' min/max keys are present in output sstables or expired if missing from output sstables. This is mainly to detect data loss caused by compaction strategy skipping some subranges of source sstables HCD-130. It works as following: 1. Extract all boundary keys (min/max) from source sstable 2. Check if boundary keys exist in output sstables a If all present, validation passes b. If any keys are missing, continue validation 3. For missing keys, read partition data from source sstables 4. Apply tombstone purging by using `gc_grace_seconds=0` 5. Check purged content a If there is no live data, validation passes. Keys are properly obsoleted b If there is live data, validation fails and throws to abort compaction if it's configured to abort Configs: * `cassandra.compaction_validation_mode=NONE`, available options: NONE/WARN/ABORT
michaelsembwever
pushed a commit
that referenced
this pull request
Oct 15, 2025
…lidation before txn prepare (#2008) riptano/cndb#15141 Added compaction task verification before transaction commit to validate if all source sstables' min/max keys are present in output sstables or expired if missing from output sstables. This is mainly to detect data loss caused by compaction strategy skipping some subranges of source sstables HCD-130. It works as following: 1. Extract all boundary keys (min/max) from source sstable 2. Check if boundary keys exist in output sstables a If all present, validation passes b. If any keys are missing, continue validation 3. For missing keys, read partition data from source sstables 4. Apply tombstone purging by using `gc_grace_seconds=0` 5. Check purged content a If there is no live data, validation passes. Keys are properly obsoleted b If there is live data, validation fails and throws to abort compaction if it's configured to abort Configs: * `cassandra.compaction_validation_mode=NONE`, available options: NONE/WARN/ABORT
michaelsembwever
pushed a commit
that referenced
this pull request
Oct 15, 2025
…lidation before txn prepare (#2008) riptano/cndb#15141 Added compaction task verification before transaction commit to validate if all source sstables' min/max keys are present in output sstables or expired if missing from output sstables. This is mainly to detect data loss caused by compaction strategy skipping some subranges of source sstables HCD-130. It works as following: 1. Extract all boundary keys (min/max) from source sstable 2. Check if boundary keys exist in output sstables a If all present, validation passes b. If any keys are missing, continue validation 3. For missing keys, read partition data from source sstables 4. Apply tombstone purging by using `gc_grace_seconds=0` 5. Check purged content a If there is no live data, validation passes. Keys are properly obsoleted b If there is live data, validation fails and throws to abort compaction if it's configured to abort Configs: * `cassandra.compaction_validation_mode=NONE`, available options: NONE/WARN/ABORT
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the issue
https://github.com/riptano/cndb/issues/15141
What does this PR fix and why was it fixed
Added compaction task verification before transaction commit to validate if all source sstables' min/max keys are present in output sstables or expired if missing from output sstables.
This is mainly to detect data loss caused by compaction strategy skipping some subranges of source sstables HCD-130.
It works as following:
a If all present, validation passes
b. If any keys are missing, continue validation
gc_grace_seconds=0a If there is no live data, validation passes. Keys are properly obsoleted
b If there is live data, validation fails and throws to abort compaction if it's not dry run.
Configs:
cassandra.compaction_validation_mode=NONE, available options: NONE/WARN/ABORT