Two fixes, and the more important one is a live correctness bug we introduced
ourselves — a self-deadlock that every test in the suite was structurally incapable
of catching.
Fixed: a broken cursor-walk retry protocol (self-deadlock on DB_THREAD)
Commit f9937fab5 ("shard per-handle cursor free/active queues", June) moved
__db_walk_cursors to hold cq_parts[i].mutex across its callback — but left all three
drop-and-rescan callbacks releasing dbp->mutex, a mutex the walk no longer holds:
src/btree/bt_curadj.c:360__bam_ca_dup_func→__bam_opd_cursorsrc/btree/bt_curadj.c:447__bam_ca_undodup_func→__dbc_closesrc/hash/hash_rec.c:1524__ham_chgpg_recover_func→__dbc_close
On a DB_THREAD handle that is two defects at once. It unlocks a mutex the thread does
not hold — DIAGNOSTIC panics; production glibc NORMAL-type unlock of an unlocked mutex
is undefined behaviour and can leave two holders. And the partition mutex stays held
across __db_cursor_int/__dbc_close → CQ_LOCK on the same handle, which is a
self-deadlock on a non-recursive latch whenever that resolves to the partition being
walked. A third bug on the same lines leaked a stale DB_LOCK_NOTGRANTED in ret to
callers as a real error.
Verified in both directions: unfixed, it hangs (rc=124, with a gdb backtrace in the
report); fixed, it passes.
Why the entire test suite was green
dbp->mutex and cq_parts[].mutex are both allocated only under DB_THREAD
(db.c:497, db.c:508). Non-threaded handles leave both MUTEX_INVALID, so CQ_LOCK
and MUTEX_UNLOCK compile to no-ops and both defects are invisible — and the TCL cursor
tests use non-threaded handles. It shipped green for three months and bites only
DB_THREAD combined with off-page-duplicate conversion, duplicate undo, or hash
chgpg recovery.
A new DB_THREAD test (test/db/curadj_dup_partition.c) closes that hole. Worth
recording how close this came to being missed again: the first attempt reported 16/16
cursor tests passing, and instrumenting the retry path then measured zero hits — the
tests had never executed the code under repair.
Fixed: DB_LOG_DIRECT could not complete a transactional open
O_DIRECT requires the buffer address, the file offset and the transfer length all
block-aligned; the log write path satisfied none of them, so DB->open inside a
transaction failed EINVAL.
__log_write_direct now restages each write into 4096-byte aligned whole blocks —
base = w_off & ~(B-1), read back the leading partial block, zero-pad the trailing one —
using ALIGNP_INC over a stack buffer, so there is nothing to free on an error path.
lp->w_off still advances by exactly len, so no offset or LSN arithmetic changes
meaning, and db_log_verify succeeds on a log written entirely through the new path.
Two failing sites, not one. The tracked diagnosis named __log_write; the observed
1-byte EINVAL actually came from __db_file_extend in log preallocation. Fixing only
the documented site would have left the flag broken. And the offset is aligned
essentially never — instrumented, head != 0 on 1,999 of 2,000 writes — so the
read-back is the hot path, not a corner case.
Alternatives were rejected for stated reasons: a split descriptor would put the durable
frontier at the mercy of which of two fds last touched a block (the same failure class as
the __memp_aio_drain "fast liar"), and a persistent staging buffer would need a DB_LOG
field, changing __env_struct_sig() and making every existing environment refuse to
attach — an unacceptable price for an optional flag.
Durability, proven rather than asserted
- 300 commits acked under
DB_TXN_SYNC, processSIGKILLed with no clean shutdown, no
close and no checkpoint — all 300 present after recovery.db_verifyclean. - 43,982 of 43,982 log
pread/pwritecalls 4096-aligned in offset and length. - Group commit intact:
st_maxcommitperflushunchanged.
A sabotage test initially passed — removing the read-back did not lose data, because
the stack staging buffer still happened to hold the right block. Poisoning the buffer
first makes the mutant fail properly. That poison is load-bearing and commented as such;
without it the test would have certified a broken fix.
Also: mtx_dblist no longer defeats the cursor sharding
__db_walk_cursors held env->mtx_dblist exclusively across the whole walk, so the
8-way cq_parts[] sharding bought nothing and one environment-wide mutex serialized
every B-tree insert. It is now taken shared: the latch guards the shape of
env->dblist, the walk is a pure reader, and __db_refresh unlinks under the same latch
exclusively — so a handle cannot be unlinked while a reader holds it shared. That makes
the handle-lifetime question vanish rather than requiring a pin that does not exist.
__bam_ca_di: 30.74% → 2.89% of profile at t=64.
The throughput effect is modest, and we disclose a disagreement about it. The
implementing measurement found a 9.5% regression at t=64 over 14 reps; an independent
re-measurement (3 alternating reps, 96 vCPU, production builds, fresh directories) found
the opposite sign: t=64 +10.7% (median 125,890 → 139,396), t=32 −5.3% at 6.4% CV,
i.e. inside noise. Opposite signs on the same point means the honest claim is "not a
large throughput win". This change is merged on its correctness merits and the
unambiguous profile shift, not on a throughput number.
Why the scaling ceiling did not move much: __log_put (P5)
With P1 (locker stripes) and P4 (cursor-walk latch) both fixed, profiling at t=64 puts
56% of all time in __log_put, 87% of that on a single log-region latch. Every
committing transaction must append to the log, and the append is serialized.
That is the general lesson from this round: removing contention upstream of a saturated
serial stage moves the queue rather than shortening it. Recorded as P5, with the
constraint that the log is the durability frontier, so any fix there needs a
crash/recovery argument at least as strong as P3's.
Also corrected: the previously reported "cliff after 8 threads" is not a cliff. Measured
on 96 vCPU, throughput peaks at t=2 and declines smoothly with no knee — which is
the signature of one global lock, not a partitioned structure saturating.
Known issues
P5 above is the open scaling item. Unchanged: T1–T6, S1 (opt-in DB_MPOOL_AIO),
S5, U1–U6, W1, B1, F1. P1 remains "fixed, partially" — its shape concern is
reduced, not retired. See test/KNOWN-ISSUES.md.
Disclosed: DB_LOG_DIRECT costs more fsyncs than the buffered path (347 vs 173) — a
throughput effect, not a correctness one, with the upgrade path noted in the source.
test_sim_torn and a TCL subset fail, both reproduced on pristine master in fresh build
dirs, so pre-existing.
Compatibility
No on-disk, log, region, or public-ABI change. __env_struct_sig() is 0xb86f77f0,
verified on both arms in fresh build dirs; public sizes remain 1744/552/2088/336; the
compatibility triplet stays 2026.0.9 and db_version() returns 2026,0,9, checked
against a built library. Existing environments attach unchanged.
Qualified from a pristine clone on a dedicated 96-vCPU EC2 instance: build, version and
ABI, environment signature, exec-bit gate, manifest self-check, db tier, leak tier, and
the flag tier at 14 verdicts with direct_log PASS and O_DIRECT confirmed on the
log descriptor.