DST depth: write-back crash model, 3 caught planted bugs, 14 scenarios - #54
Closed
gburd wants to merge 4 commits into
Closed
DST depth: write-back crash model, 3 caught planted bugs, 14 scenarios#54gburd wants to merge 4 commits into
gburd wants to merge 4 commits into
Conversation
…os_io Make a simulated crash actually DROP bytes written-but-not-fsynced so an ack-before-fsync durability bug is detectable after crash+recover: - __db_sim_wb_crash() truncates every tracked real file back to its durable frontier (last fsync), keyed by a stable file-name hash so the frontier tracks a logical file across libdb's close/reopen; the WB entry now stores the on-disk name for the truncation. - __os_io write fast path consults __db_sim_io_write_fault_hook: returns ENOSPC (whole write fails, nothing persists) or a torn-prefix length (persist a strict prefix, report full success) -- the previously unconsumed ENOSPC and torn-write knobs are now load-bearing. - __os_io read fast path is offset-aware (__db_sim_io_read_hook now takes fhp+offset) and consults a ported stale-read ring plus the corrupt bit-flip; a per-I/O __db_sim_io_latency_hook consumes the latency knob (a tiny capped sleep when armed). All behind HAVE_DST; the OFF library still exports zero __db_sim_* symbols and compiles the hooks to the stock path.
The FoundationDB/TigerBeetle 'DST finds real bugs' proof. Each planted bug lives at a real library site, gated by -DDB_DST_INJECT_BUG=<n> and compiled out of every normal build: - 1 NODURABLE: __log_flush_int skips the log fsync but still acks the commit. The write-back durable frontier never advances, so the crash drops the un-synced log and every 'committed' txn is lost. - 2 NOCKSUM: __db_check_chksum ignores a checksum mismatch and accepts the page, so a corrupt page flows silently into the tree. - 3 LOSTUPDATE: __memp_pgwrite skips a dirty-page write but reports success, so a checkpoint believes a page is durable when it never reached disk. Each is caught by a specific scenario's invariant within K=1 seeds. Nested #if defined(HAVE_DST)/#if DB_DST_BUG(n) so the OFF build (where DB_DST_BUG is undefined) still preprocesses and links cleanly.
…rios Each runs a seeded workload, injects its fault, recovers, and asserts the safety invariant (committed survives / uncommitted doesn't / checksum catches corruption / no silent bad data). All build+run+pass; each crash/fault scenario passes across a multi-seed sweep and replays bit-identically per seed. New (11): test_sim_hash_crash / _recno_crash / _queue_crash (per-access- method op+crash+recover); test_sim_ckp_crash (page-flush durability with DB_PRIVATE, catches LOSTUPDATE); test_sim_torn_log (recovery safe past a torn log tail); test_sim_enospc (graceful ENOSPC, no corruption); test_sim_abort_atomic (committed present, aborted leave no trace); test_sim_recover_idempotent (recover twice -> identical state hash); test_sim_dup_crash (DB_DUPSORT dups, exact multiplicity); test_sim_overflow_torn (overflow-record corrupt read caught); test_sim_split_crash (btree split/merge churn survives). crash_recover reworked to use the real write-back drop so the NODURABLE planted bug is a hard catch. sim_scenario.h factors out the fork+crash+recover boilerplate. dist/Makefile.in gains the 11 targets and dst_tests builds all 14.
…ADME - dst-sweep.sh: run one scenario over a seed range, report pass count and the failing seeds (each a reproducer). - dst-bug-inject.sh: the bug-detection-latency yardstick -- build a dedicated library per planted bug (-DDB_DST_INJECT_BUG=n), build the guarding scenario, and assert each bug is caught within K seeds (all three caught at K=1). - .gitignore: TESTDIR_sim_* scratch env dirs. - DESIGN.md / README.md: reflect the landed write-back drop, the three real-site planted bugs, the 14-scenario catalog, and the scripts; honest notes on what is deferred (stale-read scenario, guard planting, os_aio/physwrite hooks, meson wiring).
Coccinelle convention checksNo new violations. ✅ Resolved since baseline (2) -- update dist/cocci/baseline.txt to lock these in. |
ABI diff vs
|
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DST depth: real write-back crash model, 3 caught planted bugs, 14 scenarios
Deepens the Deterministic Simulation Testing framework (Tier A1+A2+A3 of the
maturity plan). Everything is behind
--enable-dst; the OFF library exports0
__db_sim_*symbols and compiles the__os_*hooks to the stock path.A1 — write-back durable-frontier crash model, fully wired
A simulated crash now genuinely drops bytes written-but-not-fsynced:
__db_sim_wb_crash()truncates each tracked real file back to its durablefrontier (last fsync), so a commit acked without its log being fsync'd is
detectably lost after crash+recover. The write-side ENOSPC and torn-write knobs
are consumed by
__os_io's write fast path; the read fast path is offset-aware(corrupt bit-flip + a ported stale-read ring); a per-I/O latency hook consumes
the latency knob.
A2 — planted-bug harness (the "DST finds real bugs" proof)
Three known durability bugs planted at real library sites, each caught by a
scenario within K=1 seeds (
test/sim/dst-bug-inject.shbuilds a dedicatedlibrary per bug and asserts the catch):
__log_flush_intskips the log fsync, still ackstest_sim_crash_recover— 64 "committed" txns lost__db_check_chksumignores a checksum mismatchtest_sim_torn— SILENT-BAD reads__memp_pgwriteskips a dirty-page write, ackstest_sim_ckp_crash— flushed records lostA normal build compiles all three out and every scenario passes.
A3 — scenario catalog grown 3 → 14
btree/hash/recno/queue op+crash+recover, checkpoint/page-flush durability, torn
log, ENOSPC degradation, txn abort atomicity, recovery idempotency, sorted
dups, overflow records, btree split/merge churn. Each: seeded workload → inject
fault → recover → assert invariant; deterministic (same seed → same outcome,
verified). 14/14 pass from a fresh build; each crash/fault scenario passes
an 8-seed sweep.
A4-lite
dst-sweep.sh(seed-range swarm) anddst-bug-inject.sh(bug-detection-latencyyardstick).
Honest gaps (documented in DESIGN.md §4)
Stale-read ring is consumed on read but has no dedicated scenario (needs
offset-threaded write-side snapshot). Determinism-guard planting at
__os_gettime/__os_iddeferred (read on legitimate recovery paths; earns itskeep with the v2 scheduler).
os_aio*/__os_physwritehooks and meson wiringremain follow-ups.
Verified: fresh
--enable-dstbuild → 14/14 scenarios PASS; OFF build → 0 DSTsymbols;
dst-bug-inject.sh 16→ all 3 bugs caught at seed 1.