libdb 2026.09.7

A read-path performance release, plus two negative results that redirected the work
and one correctness test that turned out to be fake.

Faster reads: RFC 0007 phase 1, optimistic read-path validation

Profiling after the previous release put 71.9% of self time in __memp_fget and
its pin atomics at 96 threads, against 0.69% for the actual key comparison
(__bam_cmp). libdb was not slow at being a B-tree; it was slow at the bookkeeping
around touching a page. The cost was atomic_inc(&bhp->ref) — a read-modify-write on
a cacheline every other core also wants, once per page per level of every descent.

Read descents now validate instead of pinning: a reader takes no shared write at
all, then confirms the frame did not change underneath it. This is the LeanStore/OLC
design, adapted to the one constraint LeanStore does not have — a buffer pool shared
by unrelated processes.

path	t=1	t=8	t=32	t=96
batched db_get_multiple(32)	1.12×	1.64×	1.84×	2.05×
per-key DB->get	1.09×	1.26×	1.71×	0.77×

Profile after: __memp_fget 18.4% → 2.3%, __memp_fput 11.1% → 2.1%,
__os_atomic_read 28.7% → 8.7%.

The t=96 per-key regression is disclosed rather than hidden, and it is not this
change: 82% of that time is the pre-existing cursor-lifecycle mutex
(__dbc_close/__db_cursor_int), the one db_get_multiple() exists to amortize. A
faster descent simply arrives at that spinlock more often. The identical code on the
batched API is 2.05×. If you are doing high-thread-count point reads, use
db_get_multiple().

Why this was cheap: the hazard list already existed

struct __db_thread_info already carried dbth_pincount/dbth_pinlist/
dbth_pinarray in shared memory, and __memp_fget already wrote a pin record on
every fetch — consumed only by failchk. libdb was doing the work twice, and the
expensive contended copy was the redundant one. This release makes the cheap,
thread-local, already-present record authoritative for eviction eligibility. The
change is best understood as finishing a hazard-pointer scheme libdb already had
two-thirds of.

Format compatibility, which was the hard gate

env_sig.c hashes struct __bh, so a new field would normally change
__env_struct_sig() and make every existing environment refuse to attach
(BDB1539) — while abidiff stays green, because BH is not public ABI. That is
the class of break that nearly shipped in v2026.09.5.

sizeof(BH) is 96 on LP64 with a 1-byte hole at offset 15, between wired and
the 4-byte-aligned priority. BH.gen goes there:

• sizeof(BH) = 96, unchanged; gen at offset 15; no other member moved.
• __env_struct_sig() identical — 0xb86f77f0 production, 0xeae0caa0
  diagnostic, on both arms.
• Three-way environment join verified, not inferred: an environment created by
  v2026.09.6 attaches under this release and vice versa, with a working negative
  control (a production-then-diagnostic join correctly fails).
• Public ABI 1744/552/2088/336 unchanged.

BH.gen is a one-byte seqlock — bit 0 "in flux", bits 1–7 an episode counter bumped
once per exclusive acquisition — validated as the full (gen, pgno, mf_offset) tuple.
The design is fail-closed by construction: a missing ENTER would be a correctness
bug, but a missing EXIT only leaves the frame looking permanently in flux, so readers
fall back to pinning forever. All eight exclusive-acquisition sites route through one
macro. Residual ABA is stated exactly: 128·k exclusive acquisitions ending on the same
page and file inside one reader's window.

One caveat worth knowing: the optimistic path needs the per-thread pin list, which
only exists if the application calls DB_ENV->set_thread_count(). Without it the path
silently declines to engage — correct, but it means the win is opt-in in practice.

Also fixed

• T1 — SSI marker GC now has a real margin (≥95×, was under 7%). Root cause was
  not the GC trigger: one long-lived transaction pins __txn_oldest_reader, so the
  LSN gate retained every later committed reader's marker and GC fired constantly
  while reclaiming zero (a forced checkpoint moved the population 807 → 807). The
  fix reclaims committed read-only markers, which are provably interchangeable
  rather than merely old. Ships with a tamper control showing each condition of the
  new predicate is load-bearing.
• S1 — the os_aio deadlock (opt-in DB_MPOOL_AIO, still default OFF). There
  were two variants, and the more common one had been missed: a hold-and-spin
  case where the winner never blocks, spinning __os_yield while RUNNABLE with no
  mutex wait in its own backtrace — which is why it read as ordinary throttling.
  Both named candidate fixes were required, not either alone. Measured 5.7% (11/192)
  → 0/384, p = 1.2e-05. A caution for anyone testing this area: shrinking
  MEMP_AIO_WINDOW suppresses the bug, so testing that way would falsely
  conclude it was fixed; the sharpener is a larger window.

Two negative results, both of which redirected the work

Read-descent locks are already minimal. The premise that a 3-level point read
costs three lock acquisitions was false: it is one lock, on the leaf,
depth-independent (1.000 locks/read at 2, 3 and 4 levels), because interior pages are
latch-coupled through buffer pins and never locked. The "skip locks on internal pages"
optimization has existed since Berkeley DB. Plain snapshot-isolation readers with MVCC
take zero lock objects. The one remaining lock under DB_TXN_SERIALIZABLE is the
DB_LOCK_SIREAD marker whose entry on obj->sireaders is the only thing forming
the rw-antidependency edge — that lock is invariant D10, and removing it would
reproduce a phantom bug we documented in a competing engine. Verdict: nothing left to
remove. The real read-path cost is LOCK_PART = ndx % part_t_size — one hot page is
one partition mutex regardless of partition count — so future work should target
marker contention, never marker absence.

The write convoy is PGNO_BASE_MD, not the leaf. At 96 threads under bulk insert,
400 of 400 lock waits are on page 0; the leaf and split-time ancestor locks held
across commit have zero waiters. __db_new takes the metadata page
DB_LOCK_WRITE with LCK_ALWAYS and releases via __TLPUT, which is a no-op for a
write lock inside a transaction — so one page allocation holds the database's
metadata page until commit, across its own ~3.7 ms fsync. Utilisation 0.89 and
Little's law (333 ms) independently predict the measured 282 ms put p99. This convoy
had been misattributed twice — first to the log, then to leaf/split lock scope — both
times by reasoning about which locks are held rather than measuring which are
waited on. Tracked as P1; the fix is a genuine two-phase-locking question,
because the metadata page records the free list.

Test integrity

• A phantom-prevention test was fake. The existing D10 test put both markers on
  the same leaf, so an ordinary page write lock stopped the second transaction
  identically under snapshot and serializable isolation. It passed at both levels —
  and still passed on a build with phantom prevention destroyed. Replaced with
  phantom_pages, which puts the inserts on different leaves, XFAILs under
  DB_TXN_SNAPSHOT (the anti-vacuity control, proving the scenario is a real anomaly)
  and passes under DB_TXN_SERIALIZABLE, with a tamper build that makes it fail.
• 82 scripts had lost their exec bit, including the test/db/run_*.sh regression
  runners and RFC 0007's own sabotage test. This is the third time a lost exec bit has
  hidden a test here. There is now a gate (test/check_execbits.sh) keyed on the
  shebang, so files meant to be sourced are not flagged. On its first CI run it caught
  itself — it had a shebang and no exec bit.
• RFC 0007's sabotage test is wired into CI and emits a harness verdict, so "printed
  success" can no longer be confused with "ran".

Known issues

Unchanged from v2026.09.6 and tracked in test/KNOWN-ISSUES.md: P1 above; T1's
tuning sensitivity on high-core machines; the lockmatrix ASan bad-free; the
run_upgrade hash-v5 fixture; a cutest abort; a log_put.c zero-length-memcpy
UBSan note; and ssi009/BDB2047 at lk_partitions=1. B1 records that an
unmerged branch (perf/bhpin-r1) carries a latent latch defect and must not be
merged.

Compatibility

No on-disk, log, region, or public-ABI change. The ABI compatibility level stays
frozen at 5.3.37 (libdb-5.3.so, _5003 symbol mangling); 2026.09.7 is the release
identity. Existing environments attach unchanged, verified by three-way join rather
than by hash comparison alone.

The --enable-handoff-trace build option added this cycle deliberately changes the
log region layout and the environment signature; it is off by default and no
shipped build enables it.