Repository navigation
Release v0.0.331
Description
This PR fixes two interconnected contention/deadlock sources in storage:
-
PreSave lock-free metadata read:
- In
ContainerProfileProcessor.PreSave(line 147), time-series profile processing checks the consolidated profile's metadata. It was callingGetContainerProfileMetadatawhich takes a per-key read lock (s.locks.RLock). - When the consolidated profile is being updated or consolidated (holding
s.locks.Lock), or when nested in a write lock, this attempt to acquireRLockcauses a self-deadlock / blocks forlockTimeout(5s) until context deadline exceeded. - Fixed by switching line 147 and
getAggregatedData(line 814) toGetContainerProfileMetadataNoLock, matching line 193 which was previously updated for non-TS profiles. - Added regression test
TestPreSave_TimeSeries_NoDeadlockUnderHeldWriteLockwhich fails in 2.0s without the fix and passes in 0.00s with the fix.
- In
-
Single-writer lock ordering (connection-then-lock):
- In
singleWriter.commit,s.locks.Lock(job.key)was acquired before callings.pool.Take(poolCtx). - If pool connections were constrained or held by long-running operations like
WithConnection,commitpinned the per-key write lock for up topoolTimeout(15s), blocking all readers on that key and creating a lock-order inversion deadlock against operations that take a connection first and then request an RLock. - Since the single writer goroutine is strictly serialized and holds at most 1 connection at a time, taking the pool connection before acquiring the per-key lock ensures the per-key lock is only held for sub-millisecond SQLite metadata write + payload rename, eliminating lock-order inversion.
- In
Signed-off-by: Matthias Bertschy matthias.bertschy@gmail.com