libdb 2026.09.6

libdb 2026.09.6 acts on a second round of architectural review. Three of the
items below are real defects that the existing test suite did not catch, and one
is a retraction of a claim made in the previous release's own benchmark notes.

Fixed: a self-deadlock at lk_partitions=1, reachable by default

LOCK_SYSTEM_LOCK and TXN_SYSTEM_LOCK are not two different latches. The lock,
transaction, and log regions all assign the same physical mutex
(renv->mtx_regenv — "We share the region so we need the same mutex", at
lock_region.c:179, txn_region.c:118 and log.c:224), and when
lk_partitions == 1 the object-partition lock is that same region mutex. The
SSI commit path took it at lock.c:801 and then re-took it at lock.c:1119:
a single thread deadlocking against itself.

This was worse than a corner case. lock_method.c:42 computes
lk_partitions = ncpu > 1 ? 10 * ncpu : 1, so 1 is the default on any
single-CPU machine — a container with one core, a small VM, an embedded target.
It also explains hangs logged against lk_partitions=1 across earlier sessions.

The fix guards the nested acquisition on part_t_size == 1, matching three
neighbouring sites that already use exactly that guard. The #136 commit-window
property is preserved by interval containment: the caller's existing hold
[801, 803] strictly contains the window [1119, 1148] that was removed, so the
state published under the latch is unchanged. Both premises are asserted under
--enable-diagnostic.

A regression gate ships with it: test/c/lock_order_check.c now requires a clean
completion at lk_partitions=1, and treats a checker abort as failure.

Added: a lock-order checker (DIAGNOSTIC only) — it found the above

A mutex acquisition-order checker that builds the observed order graph at runtime
and reports a cycle. It found the deadlock above on its first run.

One design note worth recording: node identity had to be the mutex index, not
the MTX_* rank, precisely because the lock, txn and log regions alias one
mutex — a rank-based model cannot express "these three names are one object", and
would have reported the self-deadlock as three well-ordered acquisitions.

The checker is honest about its limits: mtx_buf is a pin rather than an ordered
latch, so it would not have caught the os_aio deadlock still open below.
Production builds are untouched — the hooks expand to do { } while (0) and no
__db_lo_* symbol appears in a production shared library.

Added: db_get_multiple() — 1.79–2.47× on concurrent point reads

Profiling a concurrent read workload put 85.3% of all cycles in the cursor
lifecycle mutex (42.95% __db_cursor_int + 42.09% __dbc_close). Not the tree
descent (__bam_cmp, 0.69%), and not the buffer pin (1.92%). The cost of a point
read at scale was the bookkeeping around the read.

db_get_multiple() fetches N scattered keys per API crossing, amortizing one
environment entry/exit, one replication check, one transaction-consistency check
and — above all — one cursor allocate/free pair instead of N. Each key still
gets its own descent, its own page read lock or SIREAD marker, and its own return
code, so isolation semantics are identical to the equivalent DB->get() loop; a
differential test asserts that, including abort behaviour under
DB_TXN_SERIALIZABLE.

At 96 threads: 2.21M → 5.45M ops/s (2.47×), p99 4× better, cursor mutex
85.32% → 0.70%. Individual DB->get() peaks at 32 threads while the batched path
is still climbing at 96.

It is a free function, not a DB method, deliberately: a new method pointer
would change sizeof(DB) (public ABI) and __env_struct_sig(), and the latter
makes existing environments refuse to attach.

Retracted: the explanation of the write-path gap

The previous release's cross-engine notes said libdb lacked group commit. That
was wrong, and the error was mine. The claim came from grepping for the
feature name; the mechanism has been there since Sleepycat.
__log_flush_int implements leader/follower group commit with a lp->commits
queue, an in_flush baton, and per-waiter DB_MUTEX_SELF_BLOCK mtx_txnwait
wakeups filtered by LOG_COMPARE(&lp->s_lsn, &commit->lsn) > 0.

It also works: measured flush-to-commit ratio 1.0 → 0.12 (up to 31 commits per
fsync), and at peak 380 waiters to a single __os_fsync.

The real limiter is elsewhere: throughput ceilings near 4,100 ops/s at 32 threads
and regresses to 3,623 at 96; ~507 fsync rounds/s (~2.0 ms/round) against a
~1.2 ms device fsync implies ~0.7 ms/round of software handoff, and p99 is
272 ms against a flat 3.8 ms p50 (68×) — an unfair wait queue plus per-round
baton handoff, not a missing feature. Corrected in
test/bench/CROSS-ENGINE-2026-09.md, with the analysis in
test/bench/WRITE-PATH-2026-09.md. Leader-handoff latency and queue fairness are
the next write-path lever.

The ALREADY_FLUSHED follower re-check added here is defensive hardening, not
a bug fix; no missed wakeup was observed.

Harness integrity: two gates, and what they immediately found

Vacuous greens — tests that report success without running — have been this
project's signature defect. Two gates now attack it structurally.

test/MANIFEST + test/check_manifest.sh. One flat list of every verdict the
suite promises, because every historical trap here was two lists drifting. The
runners read their work from the manifest, so a test cannot be silently dropped.
It found immediately:

• test/c/leak-run.sh was referenced by zero workflows — it had never run in
  CI. The #138 purge gate and the os_aio durability gate that the previous
  release "fixed" were never executed by CI at all.
• It was mode 100644 — not executable: the lost-exec-bit trap, verbatim.

dist/env_sig_print.sh + a region-signature gate. Adding a struct field is a
three-gate question: public ABI (abidiff sees it), region layout
(major/minor version sees it), and __env_struct_sig() — which hashes ~137
structs including process-private ones and makes env_region.c refuse to attach
an existing environment. Nothing in CI covered the third until now.

Both gates were themselves adversarially checked, and this mattered: the
region-signature gate, on first review, had never measured a base on any
branch — a path bug made it compare empty to empty and announce "unchanged"
with exit 0. The vacuous-green gate was vacuous. Fixed to require both values
match 0x%08x, and to measure the base with the head's instrument (the base
predates the script). The leak tier likewise had two drivers printing PASS to the
console while emitting no verdict line; the gate caught that too (11 → 14
verdicts).

Signature this release: unchanged from v2026.09.5 — struct __db_mpool fields
byte-identical, sizeof(DB/DBC/DB_ENV/DB_TXN) = 1744/552/2088/336.

Measured: essentially every SSI false abort is a page-granularity artifact

RFC 0005 proposed row-level conflict tracking. Before building it, the number was
measured: 585 runs, 117 configurations, holding the logical conflict graph
empty and varying only physical co-location.

11.37% aborts when keys share a leaf page; 0.000% when they do not. The abort
rate spans ~630× purely from records-per-leaf (0.000% at 512 B pages → 11.35%
at 32 KB). So ~100% of SSI aborts on a realistic read-modify-write are artifacts
of page granularity, not real serialization conflicts.

The recommendation is not to build RFC 0005. Users already control this with
set_pagesize, and on Zipfian workloads page-level write-write deadlocks
exceed false aborts roughly 3× (40.1% vs 12.8%) — key-precise read edges would
not remove those. A cheap knob beats a large mechanism.

Also fixed here: a field-order bug in run_bench.sh that made previously
published NOISE.md / GATE-VERIFICATION.md rows unreproducible.

Documented: phantom prevention is invariant D10

There is no predicate, next-key, gap or range lock anywhere in src/ —
verified, not assumed. Phantom prevention under DB_TXN_SERIALIZABLE is
emergent: SR_INSERT includes SR_WRITE, and SIREAD markers hang off
obj->sireaders for the page an insert lands on, so an insert into a range
someone read conflicts through page-level co-location. That is load-bearing
behaviour arising from an implementation detail, so it is now recorded as
invariant D10 in docs/design/global-invariants.md. Note the tension with the
result above: the same page granularity that causes ~100% of the false aborts is
what makes phantom prevention work.

Known issues

• os_aio deadlock (opt-in DB_MPOOL_AIO, default OFF). Concurrent
  checkpoint/trickle/sync deadlocks in roughly 2–5 of 40–67 runs. The trylock
  winner blocks acquiring mtx_buf for a new buffer while holding deferred-write
  pins, because the deferred path only drains at nflight >= MEMP_AIO_WINDOW —
  hold-and-block across a window that only drains when full. No data loss observed
  (lost=0); recovery and db_verify are clean in every occurrence. The
  lock-order checker does not cover this, as mtx_buf is a pin, not an ordered
  latch.
• A second, independent lk_partitions=1 failure remains in multi-process locker
  teardown (ssi009 / BDB2047), tracked as S5; it is not fixed by this
  release's latch fix.
• ssi_gc_pressure@serializable remains tuning-sensitive: the passing window is
  roughly 300–310 markers before BDB4525 ENOMEM, under 7% margin (tracked as T1).
• Pre-existing and unchanged: lockmatrix ASan bad-free (T2), the
  run_upgrade hash-v5 fixture (T3), a cutest abort (exit 134, BDB2032),
  and a log_put.c zero-length-memcpy UBSan note.

The T labels above are test-suite / harness issues, tracked in
test/KNOWN-ISSUES.md. They are deliberately NOT the G labels used in
docs/design/global-invariants.md, which number invariant-coverage gaps —
G4/G5/G6 there mean something entirely different.

Compatibility

No on-disk, log, region or public-ABI change. The ABI compatibility level stays
frozen at 5.3.37 (libdb-5.3.so, _5003 symbol mangling); 2026.09.6 is the
release identity. Existing environments attach unchanged — the build signature is
identical to v2026.09.5.