Conversation
The WAL logs logical kv-batch frames: the exact serialized WriteBatch the engine later applies, so what is durable and what is applied are byte-identical and redo shares one code path with normal operation. A chained, salted xxHash64 over each frame lets recovery find the exact durable tail without trusting any external pointer. The first frame that breaks the chain or carries a stale salt ends the durable log; a batch counts as committed only when a valid commit frame for it is reached, so an uncommitted trailing batch is dropped. Redo needs no page-LSN bookkeeping. Every mutation is keyed by a unique internal key of user-key, inverted version, and kind, so replaying a committed batch re-inserts keys already present and is a no-op. That makes recovery idempotent and restartable for free. Synchronous levels mirror PRAGMA synchronous: Off, Normal (defer the per-commit sync to checkpoint), Full (fdatasync every commit), and Extra (also sync the inode on growth). A sync error is fatal and not retried. WriteBatch gains Encode/DecodeBatch so the batch owns its wire form and the WAL stores it verbatim. Group commit, physical page-image torn-write frames, and the replay driver that feeds engine.Apply on open are reserved for later slices.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second slice of M1: the WAL that both engines commit through, plus the recovery scan that finds the durable tail.
What this adds
wal.WAL: an append-only log over a-walsidecar.LogBatch+Commitis the durability path;Checkpointedrecords a fold boundary and rotates the salt.kv-batchframes carry the serializedWriteBatchverbatim, so what is durable and what is applied are byte-identical and redo reuses the runtimeApplypath.wal.Recoverwalks frames, finds the durable tail, and returns committed batches in LSN order.engine.WriteBatch.Encode/engine.DecodeBatch: the batch owns its wire form.Off/Normal/Full/ExtramirroringPRAGMA synchronous; sync errors are fatal and non-retryable (fsyncgate).Design notes
user_key || ^version || kindinternal key, so replaying a committed batch re-inserts already-present keys as a no-op. Redo is idempotent and restartable for free. The package doc records this as a deliberate substitution for ARIES page LSNs.page-imageframe type is reserved (torn-write protection wires in with the checkpoint fold). Group commit and the on-open replay driver are reserved for later slices.Tests
go test -race ./...green. WAL tests cover commit/recover round-trip, uncommitted-tail drop, torn-tail detection, crash durability by sync level (Fullsurvives,Normalunsynced is lost, no corruption), salt rotation across a checkpoint, and sync accounting.Implementation doc:
notes/Spec/2059/implementation/05-wal-and-recovery.md.