Release v0.0.335

Summary

Fixes a consolidation goroutine self-stalling on its own held SQLite write lock for up to the full busy-timeout (60s in production) via an unrelated settings-cache refresh — confirmed as the dominant driver of kubescape/node-agent's component-tests CI flakiness (real-CI validation dropped failures from a 15-19/31 baseline to 0-1/31 with a workaround alone).

Overview

A consolidation goroutine holding SQLite's write lock (inside its own deferred transaction) could self-stall for up to the full busy-timeout (DefaultBusyTimeout, 60s in production) via an unrelated settings-cache refresh.

Root cause: NewCRDCollapseSettingsProvider's cache (10s TTL) is refreshed from inside ContainerProfileProcessor.PreSave, which itself runs inside consolidation's open transaction on connection A. The refresh calls Get on a second pool connection B. When no CollapseConfiguration custom resource exists, that read's underlying get() finds no payload file and issues a DeleteMetadata on connection B — which needs the same SQLite write lock connection A already holds, on the same goroutine. SQLite's busy handler retries for the full timeout before giving up.

This fires on essentially every consolidation tick with data whenever no CollapseConfiguration CR exists (node-agent's REST Create returns from PreSave before this point, so only consolidation saves trigger a refresh; the 30s consolidation interval exceeds the 10s TTL, so nearly every tick hits an expired cache). Live since v0.0.291 (2026-06-26) — this predates and is independent of the write-shard/single-writer work in #397/#399.

Real-world confirmation: adding a CollapseConfiguration resource to kubescape/node-agent's component-tests chart (zero storage code changes) dropped CI failures from a 15-19/31 baseline to 0-1/31 across three independent runs — this is very likely the dominant driver of the flakiness that motivated the broader write-path investigation.

Fix

Two independent commits, each with its own fail-before/pass-after tests:

• 957af4cd — NewCRDCollapseSettingsProvider becomes stale-while-revalidate: primed synchronously at construction, hot path is a single atomic load, refresh runs in a CAS-guarded background goroutine. No pool/lock/statement work ever happens on the caller's goroutine again.
• e3429c2f — get() checks ReadMetadata before issuing DeleteMetadata on a missing payload, so a read of an absent key is a SELECT, never a write-lock-requiring DELETE (defense in depth; orphan-pruning behavior for a key whose row does exist is unchanged).
• bf690e36 — test hygiene follow-up from independent review: removes a latent -race flake (a shrunk lockTimeout package var the background refresh could read unsynchronized) by joining the refresh explicitly instead.

How to Test

go test -race -count=20 ./pkg/registry/file/ -run 'Collapse|Consolidate|TestGet_'

All new tests fail deterministically on unfixed code at ~the busy-timeout duration and pass after each corresponding fix; AC-B1 still fails with only Fix A applied, confirming both fixes are independently necessary.

Additional Information

Went through an independent adversarial code review (verdict: SHIP, no blocking findings) that re-derived every claim from source rather than trusting the design doc, including tracing the CAS-guarded refresh for genuine race-freedom and checking all six callers of get() individually. Full root-cause chain, design rationale, and review are recorded in docs/features/collapse-settings-self-stall.md.

git merge-tree against current main (which now includes #399) reports zero conflicts.

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01LCMGT6Po2tSr1VEDVrbbYd

https://claude.ai/code/session_01LCMGT6Po2tSr1VEDVrbbYd

AI Review

Independent code-review subagent (Claude, Fable model) · verdict: SHIP, no blocking findings. Re-derived every claim from source rather than trusting the design doc: reproduced the fail-before timings independently in scratch worktrees, traced the CAS-guarded background refresh for genuine race-freedom, and checked all six callers of get() individually to confirm none depend on the removed DELETE side effect. Two minor test-hygiene items found (both latent -race flakes in test code, not production code) — addressed in bf690e36.

Ticket

None