Problem
When a Kubernetes pod restarts within the same ReplicaSet, the new worker gets the same name (e.g., 1@pulp-worker-abc123-xyz). The new worker creates a fresh AppStatus but Redis still holds locks from the previous incarnation under the same name. These locks block resources indefinitely.
During the 2026-07-24 incident, worker 8qc25 was killed without graceful shutdown. Its 543 task locks and 524 resource locks persisted in Redis. When new workers came online, they had different names (different ReplicaSet), so they couldn't release the old locks. The app_worker_cleanup mechanism also missed them because the AppStatus record was already gone.
Proposed Fix
Add release_stale_locks_for_self() to RedisWorker.__init__(), called after redis_conn.ping() and before startup_hook():
- Guard: skip if another live AppStatus has our name (should not happen under normal K8s semantics, but defensive)
- Scan Redis for all
task:* and pulp:resource_lock:* keys owned by our worker name
- Release them (delete string keys, srem from set keys)
- Log a warning with counts if any stale locks were found
This handles pod-restart-with-same-name immediately at startup, before accepting any tasks.
References
Problem
When a Kubernetes pod restarts within the same ReplicaSet, the new worker gets the same name (e.g.,
1@pulp-worker-abc123-xyz). The new worker creates a fresh AppStatus but Redis still holds locks from the previous incarnation under the same name. These locks block resources indefinitely.During the 2026-07-24 incident, worker
8qc25was killed without graceful shutdown. Its 543 task locks and 524 resource locks persisted in Redis. When new workers came online, they had different names (different ReplicaSet), so they couldn't release the old locks. Theapp_worker_cleanupmechanism also missed them because the AppStatus record was already gone.Proposed Fix
Add
release_stale_locks_for_self()toRedisWorker.__init__(), called afterredis_conn.ping()and beforestartup_hook():task:*andpulp:resource_lock:*keys owned by our worker nameThis handles pod-restart-with-same-name immediately at startup, before accepting any tasks.
References