Implement WAL group commits#547
Merged
Merged
Conversation
Contributor
Author
|
Cc: @loganpowell |
Contributor
Author
|
Contributor
Author
|
Postgres and SQLite handle fsync() failures differently. Postgres poisons the WAL and blocks any future commits. SQLite continues with an "uncertain durability" state. We're taking the Postgres approach. |
17d1afd to
63e8ad1
Compare
The group-commit review fix made TransactionManager::commit() clear the active transaction and decrement activeWriteTransactionCount whenever commit failed after a write transaction had entered the committing state. That is too aggressive for failures in publishCommit(). At that point localStorage::commit() may have performed partial in-memory publish work before throwing, and the normal query error path still needs the active Transaction object so TransactionContext::rollback() can undo those partial changes. Clearing the transaction inside commit() leaves TransactionContext with a dangling activeTransaction pointer; the outer error handler then tries to roll it back and can report "Invalid transaction type to rollback" or crash. CI exposed this in CopyTest.RelInsertBMExceptionDuringCommitRecovery. Keep the transaction active on commit failure and only drop committingWriteTransactionCount / notify waiters. The caller's rollback path remains responsible for clearing activeTransactions and activeWriteTransactionCount. Successful commits still clear both counters in commit(), and rollback already notifies cvForCommittingWriteTransaction.
9f88fab to
5733320
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Benchmark
C++ benchmark source:
node_write_bench.cpp. Optimal was 64 threads at 6k writes/sec.Setup:
/private/tmpenableMultiWrites=trueautoCheckpoint=falseforceCheckpointOnClose=falseCREATE (:person {id, name})per writeConnectionper writer thread1,000 writes per thread
2,000 writes per thread
Main remains fsync-bound at roughly 250-290 writes/sec as writer concurrency increases. With group commits, throughput scales to roughly 1,017 writes/sec at 8 writers and 2,116 writes/sec at 16 writers for this benchmark.