Parquet crash testing unit testing hooks#3028
Merged
Conversation
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3028 +/- ##
==========================================
- Coverage 58.33% 58.27% -0.06%
==========================================
Files 2079 2077 -2
Lines 171723 171262 -461
==========================================
- Hits 100168 99810 -358
+ Misses 62630 62563 -67
+ Partials 8925 8889 -36
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
blindchaser
reviewed
Mar 6, 2026
| // file descriptors and locks so the test process can reopen the same directory. | ||
| func (s *Store) SimulateCrash() { | ||
| if s.pruneStop != nil { | ||
| close(s.pruneStop) |
Contributor
There was a problem hiding this comment.
set pruneStop to nil after close? otherwise Close() will do double close on a closed channel
blindchaser
reviewed
Mar 6, 2026
| // be recoverable via WAL replay. | ||
| func TestCrashRecoveryStress(t *testing.T) { | ||
| seed := int64(42) | ||
| t.Logf("random seed: %d (change to reproduce a specific run)", seed) |
Contributor
There was a problem hiding this comment.
it looks the seed is always 42 not random?
blindchaser
approved these changes
Mar 6, 2026
yzang2019
approved these changes
Mar 6, 2026
Kbhat1
approved these changes
Mar 9, 2026
yzang2019
pushed a commit
that referenced
this pull request
Mar 19, 2026
## Describe your changes and provide context This PR adds test-only fault-injection hooks to the parquet receipt store so we can simulate crashes at specific points in the write pipeline and validate recovery behavior. The hooks cover the key stages of persistence: after WAL write, before parquet flush, after parquet flush, after closing writers during file rotation, and after WAL clear during rotation. It also adds a SimulateCrash() helper that intentionally abandons the store without the normal flush/finalization path, which lets the tests mimic abrupt process termination and then reopen the same store directory to verify recovery. On top of that, this PR adds parquet receipt crash-recovery coverage that: verifies recovery at each hook point, including file-rotation scenarios runs randomized multi-crash stress tests to ensure WAL-committed blocks remain readable after reopen verifies concurrent readers can still read committed receipts and logs while writes are artificially slowed The goal is to increase confidence in parquet receipt durability and crash recovery behavior without changing normal production behavior outside of tests. ## Testing performed to validate your change go test ./sei-db/ledger_db/receipt -run 'TestCrashRecoveryAtEachHookPoint|TestCrashRecoveryStress|TestSlowFlushWithConcurrentReads' -count=1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe your changes and provide context
This PR adds test-only fault-injection hooks to the parquet receipt store so we can simulate crashes at specific points in the write pipeline and validate recovery behavior. The hooks cover the key stages of persistence: after WAL write, before parquet flush, after parquet flush, after closing writers during file rotation, and after WAL clear during rotation.
It also adds a SimulateCrash() helper that intentionally abandons the store without the normal flush/finalization path, which lets the tests mimic abrupt process termination and then reopen the same store directory to verify recovery.
On top of that, this PR adds parquet receipt crash-recovery coverage that:
verifies recovery at each hook point, including file-rotation scenarios
runs randomized multi-crash stress tests to ensure WAL-committed blocks remain readable after reopen
verifies concurrent readers can still read committed receipts and logs while writes are artificially slowed
The goal is to increase confidence in parquet receipt durability and crash recovery behavior without changing normal production behavior outside of tests.
Testing performed to validate your change
go test ./sei-db/ledger_db/receipt -run 'TestCrashRecoveryAtEachHookPoint|TestCrashRecoveryStress|TestSlowFlushWithConcurrentReads' -count=1