fix(daemon): clean up Kubernetes runtime plane hygiene - #970
Conversation
There was a problem hiding this comment.
Found one blocking Kubernetes lifecycle regression. Scoping the runtime-probe claim per active daemon member fixes the simultaneous-start collision, and the explicit sandbox namespace change is internally consistent. However, the documented member identity is the Pod UID, which changes on every Pod replacement, while probe claims have no owner or stale-member cleanup path. If a daemon exits after creating the probe claim but before its finally deletion completes—or if that deletion fails—the replacement hashes a different UID and can never reuse or remove the old claim. Because the claim retains its Sandbox and workspace volume, interrupted probes and rollouts can accumulate permanent resources; the previous fixed name at least let a later process converge on and delete the same claim. Please retain member isolation while adding restart-safe ownership or garbage collection for abandoned probe claims.
sent by review-bot (Codex · gpt-5.6-sol) · open in session
There was a problem hiding this comment.
The new label-and-expiry sweep bounds abandoned probe resources and addresses the previous permanent-orphan finding, but one blocking stale-delete race remains. The sweeper decides expiry from a LIST snapshot and then deletes only by claim name. A container restart within the same Pod keeps the same member ID, so probeRuntimes() can delete and recreate that deterministic claim name with a fresh expiry while another member is processing an older snapshot. The delayed GC delete then removes the fresh live claim, tearing down the Sandbox during the daemon’s startup runtime probe; that probe is not retried, so the member can remain advertising no runtimes. Please fence GC deletion to the listed object incarnation, for example with a Kubernetes UID/resourceVersion delete precondition.
sent by review-bot (Codex · gpt-5.6-sol) · open in session
There was a problem hiding this comment.
Approved. The explicit sandbox namespace and member-scoped probe identity are applied consistently, the orphaned drain-watch machinery is removed cleanly, and probe cleanup now has a bounded recovery path. The follow-up correctly fences each expired-claim delete with the UID/resourceVersion from the LIST snapshot and treats a precondition conflict as a superseded object, so a stale sweep cannot delete a freshly recreated live probe claim. I found no remaining blocking issues in the complete patch.
sent by review-bot (Codex · gpt-5.6-sol) · open in session
Part of #955.
What changed
AC_K8S_MEMBER_IDAC_K8S_SANDBOX_NAMESPACEand use it instead of the daemon Pod namespaceWhy
The deleted per-org operator was the only producer of
agentconnect.md/drain-requested, so the daemon watch had become dead machinery. A fixed probe claim made simultaneous pool members race on one SandboxClaim, while member-scoped claims needed restart-safe garbage collection. Deriving the resource namespace from in-cluster config also targeted the daemon namespace rather than the shared sandbox namespace.Impact
--k8snow fails fast unless deployments provideAC_K8S_SANDBOX_NAMESPACEand a member-uniqueAC_K8S_MEMBER_ID(normally the Pod UID via the Downward API). Abandoned probe claims are bounded without touching ordinary agent claims, and stale sweeps cannot delete a same-name replacement.Validation
pnpm --filter @agentconnect.md/daemon typecheckeslint .GIT_ASKPASScontamination)