SBOM generation stalls node-wide when a short-lived container blocks the single-worker SbomManager pool

[Silenthunt0]

Summary

A single short-lived container can silently halt SBOM generation node-wide. When a container exits before the container watcher populates its "shared container data" (pod/workload metadata), SbomManager.processContainer blocks the manager's single worker in a ~15-minute uncancelled backoff. Because the pool size is 1, this head-of-line-blocks SBOM generation for every other container on that node — no sbomsyft CRs are written until the wait unblocks or the node-agent pod restarts. A frequently-firing short-lived workload (e.g. a CronJob every few minutes) sustains the stall indefinitely.

This is related to but distinct from #848 (closed as benign log noise). The SbomManager - container not found in shared data ERROR line is harmless on its own, but the same underlying condition also stalls SBOM generation node-wide, which is not documented anywhere and is not benign.

Version

Observed on v0.3.142. The relevant code paths are unchanged through the latest tag v0.3.147 and current main.

Mechanism (code refs, as of v0.3.147 / main)

• pkg/sbommanager/v1/sbom_manager.go: the manager uses a single-worker pool (pool: workerpool.New(1)); ContainerCallback submits processContainer to that pool.
• processContainer blocks in waitForSharedContainerData, which polls GetSharedContainerData via backoff.Retry(context.Background(), ...) with a default ExponentialBackOff. Because the context is context.Background() (never cancelled), the retry only terminates at cenkalti/backoff's DefaultMaxElapsedTime = 15m.
• For a container that exits almost immediately, shared data may never be populated (it is deleted on the container-remove event in pkg/containerwatcher/v2/containercallback.go), so the single worker waits the full 15m — during which all other containers' SBOMs on that node are stuck in the queue.
• The correlated pkg/containerprofilemanager/v1/lifecycle.go message "timeout while adding container to the container profile manager" (hardcoded MaxWaitForSharedContainerData = 10m in helpers.go) shares the same root cause, but it runs in a per-container goroutine and does not itself block other containers — so it is a correlated symptom, not the node-wide cause.

Impact (observed)

On a cluster with a linkerd-meshed CronJob firing every 10 minutes (two ultra-short-lived containers per run: the job container plus its injected proxy sidecar), node-agent silently stopped writing sbomsyft for newly-deployed long-lived workloads that landed on the affected nodes. Downstream, those images never produced a vulnerabilitymanifest and never reached our vulnerability-management pipeline — while all job-liveness / pod-health checks stayed green. Diagnosis required correlating the missing SBOMs with node-agent uptime and the timeout log lines.

Workaround

Exclude the short-lived pods via excludeLabels / excludeNamespaces — both gate Config.IgnoreContainer at the top of the SBOM callback, before the shared-data wait, so an excluded pod never occupies the worker. This is an effective per-workload mitigation but not a fix for the general case (any short-lived container in a monitored namespace can trigger it).

Suggested fix

Make the SBOM shared-data wait resilient to short-lived containers, e.g.:

• cancel waitForSharedContainerData when the container-remove event fires (tie the wait to a context cancelled on removal), and/or
• bound the wait with a short timeout instead of the 15m backoff default, and/or
• process containers concurrently (worker pool > 1) so a single stuck container cannot head-of-line-block SBOM generation for the whole node.