Skip to content

v0.2.2 — spurious corrupt fixed at the read; Django 4.2 catalog lookups; mpedb License 1.0

Choose a tag to compare

@punnerud punnerud released this 01 Aug 22:54
· 136 commits to main since this release

Verified on Linux x86-64, macOS/Apple Silicon and Windows x86-64 — tests, clippy and the crash harnesses green on all three. Prebuilt binaries attached (CLI for all three platforms + the sqlite3 drop-in shim; sha256sums below).

A spurious "database corrupt" under load, found by our own fuzz harness

The map-collide SIGKILL harness caught a reader dying with no valid meta page (both checksums invalid) on a perfectly healthy file — about 4 % of runs on a loaded box at durability = none. The adversarial review pinned the mechanism, and it was not the obvious suspect:

  • The commit protocol was not at fault: no ordering, fence, or publication change was needed, and the durability gate plus crash recovery came out of the review exactly as design/DESIGN.md contracts them.
  • The read tears, not the write: the meta reader loads a slot's checksum, then its body. Commits alternate between the two meta slots and cost microseconds at durability = none, so a reader descheduled inside one read can straddle two consecutive commits — slot A torn by commit N+1, slot B by commit N+2 — and a single-pass read reports both slots invalid.
  • The gated (durable) read path has always carried a bounded retry loop for its own version of this window, documented as load-bearing. The ungated path (durability = none/async) had a single pass. It now has the twin loop, with the same termination argument: retry while the observed slot state moves; a pass that sees the same state as the pass before admits nothing new, so the failure is genuine corruption.

Measured: 5 of 120 runs failed before the fix, 0 of 240 after, same loaded box (p < 1e-4 under the old rate). No SIGKILL is actually required to trigger it — the race is reader-vs-two-commits — so long-running read-heavy workloads at durability = none could hit it in production. Upgrading is recommended.

Django 4.2's catalog lookups answer now

The first Django 4.2 project switched onto the shim stalled at migration: 4.2's get_constraints() queries sqlite_master with a double-quoted name (WHERE type='table' and name="tblname") — sqlite's double-quote misfeature, where a quoted string that names no column falls back to a string literal. Django 5.x parameterizes the same lookup, which is why the 18,214-test Django suite never met the form. The shim's catalog evaluator now implements exactly the measured rule: literal fallback when no column matches (tables, indexes, IN-lists), and a refusal — never a guess — for the identifier half (name="name" is sqlite's always-true column self-compare, with no real-world consumer).

Two more Django 4.2 catalog forms are measured and queued, not yet closed: the recursive references_graph CTE its sql_flush walks, and UPDATE sqlite_master under writable_schema.

Also in this release

  • The pypi workflow got ~10 minutes cheaper per ordinary push: the wheel fan (including QEMU-emulated ARM builds) now runs only when the version is new to PyPI; the Python suite still runs on every push.
  • README: download, don't build — the Install section now leads with the prebuilt release binaries and spells out the LD_PRELOAD route for unchanged code (Django's manage.py test in one line), versus pip install mpedb for new code.

License change: the mpedb License 1.0

From this release the project is licensed under the mpedb License 1.0 (previously MIT): source-available, free of charge for every person and every organization, with one exception — a corporate group whose consolidated revenue or valuation exceeds USD 5 billion (2026 dollars, whichever is higher) owes a one-time USD 0.07 (seven US cents) per physical device (per connected device for server deployments, whichever count is higher), with a five-year tail once the threshold is crossed. See LICENSE for the exact terms.

Attached binaries

file contents
mpedb-0.2.2-linux-x86_64.tar.gz mpedb CLI, Linux x86-64
mpedb-0.2.2-macos-arm64.tar.gz mpedb CLI, macOS Apple Silicon
mpedb-0.2.2-windows-x86_64.zip mpedb.exe, Windows x86-64
libmpedb_sqlite3-0.2.2-linux-x86_64.tar.gz sqlite3 drop-in shim, Linux
libmpedb_sqlite3-0.2.2-macos-arm64.tar.gz sqlite3 drop-in shim, macOS
SHA256SUMS checksums for everything above