Skip to content

Release v0.0.331

Choose a tag to compare

@github-actions github-actions released this 03 Sep 16:51
· 70 commits to main since this release
255e263

Description

This PR fixes two interconnected contention/deadlock sources in storage:

  1. PreSave lock-free metadata read:

    • In ContainerProfileProcessor.PreSave (line 147), time-series profile processing checks the consolidated profile's metadata. It was calling GetContainerProfileMetadata which takes a per-key read lock (s.locks.RLock).
    • When the consolidated profile is being updated or consolidated (holding s.locks.Lock), or when nested in a write lock, this attempt to acquire RLock causes a self-deadlock / blocks for lockTimeout (5s) until context deadline exceeded.
    • Fixed by switching line 147 and getAggregatedData (line 814) to GetContainerProfileMetadataNoLock, matching line 193 which was previously updated for non-TS profiles.
    • Added regression test TestPreSave_TimeSeries_NoDeadlockUnderHeldWriteLock which fails in 2.0s without the fix and passes in 0.00s with the fix.
  2. Single-writer lock ordering (connection-then-lock):

    • In singleWriter.commit, s.locks.Lock(job.key) was acquired before calling s.pool.Take(poolCtx).
    • If pool connections were constrained or held by long-running operations like WithConnection, commit pinned the per-key write lock for up to poolTimeout (15s), blocking all readers on that key and creating a lock-order inversion deadlock against operations that take a connection first and then request an RLock.
    • Since the single writer goroutine is strictly serialized and holds at most 1 connection at a time, taking the pool connection before acquiring the per-key lock ensures the per-key lock is only held for sub-millisecond SQLite metadata write + payload rename, eliminating lock-order inversion.

Signed-off-by: Matthias Bertschy matthias.bertschy@gmail.com