Skip to content

fix(daemon): clean up Kubernetes runtime plane hygiene - #970

Merged
spacedragon merged 3 commits into
mainfrom
dev/yulong/filefish
Aug 14, 2026
Merged

fix(daemon): clean up Kubernetes runtime plane hygiene#970
spacedragon merged 3 commits into
mainfrom
dev/yulong/filefish

Conversation

@spacedragon

@spacedragon spacedragon commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Part of #955.

What changed

  • remove the orphaned Sandbox drain-annotation watch and its stale launch outcome/tests
  • scope runtime-probe claim names to a daemon member using a hash of AC_K8S_MEMBER_ID
  • label probe claims with a 15-minute expiry and periodically reap abandoned claims using UID/resourceVersion-fenced deletes
  • require AC_K8S_SANDBOX_NAMESPACE and use it instead of the daemon Pod namespace
  • update the cluster design docs for the two-namespace pool layout and probe lifecycle

Why

The deleted per-org operator was the only producer of agentconnect.md/drain-requested, so the daemon watch had become dead machinery. A fixed probe claim made simultaneous pool members race on one SandboxClaim, while member-scoped claims needed restart-safe garbage collection. Deriving the resource namespace from in-cluster config also targeted the daemon namespace rather than the shared sandbox namespace.

Impact

--k8s now fails fast unless deployments provide AC_K8S_SANDBOX_NAMESPACE and a member-unique AC_K8S_MEMBER_ID (normally the Pod UID via the Downward API). Abandoned probe claims are bounded without touching ordinary agent claims, and stale sweeps cannot delete a same-name replacement.

Validation

  • 83 related daemon tests passed across the K8s runtime plane, client, driver, metrics, ACP E2E, credentials, and tunnel suites
  • pnpm --filter @agentconnect.md/daemon typecheck
  • targeted ESLint and Prettier checks
  • repository pre-push eslint .
  • full daemon suite on the initial revision: 3510 passed; 3 unrelated environment/live-runtime failures (provider quota/model behavior and host GIT_ASKPASS contamination)

@agentconnect-md-test agentconnect-md-test Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found one blocking Kubernetes lifecycle regression. Scoping the runtime-probe claim per active daemon member fixes the simultaneous-start collision, and the explicit sandbox namespace change is internally consistent. However, the documented member identity is the Pod UID, which changes on every Pod replacement, while probe claims have no owner or stale-member cleanup path. If a daemon exits after creating the probe claim but before its finally deletion completes—or if that deletion fails—the replacement hashes a different UID and can never reuse or remove the old claim. Because the claim retains its Sandbox and workspace volume, interrupted probes and rollouts can accumulate permanent resources; the previous fixed name at least let a later process converge on and delete the same claim. Please retain member isolation while adding restart-safe ownership or garbage collection for abandoned probe claims.

sent by review-bot (Codex · gpt-5.6-sol) · open in session

Comment thread packages/daemon/src/k8s/runtime-plane.ts

@agentconnect-md-test agentconnect-md-test Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new label-and-expiry sweep bounds abandoned probe resources and addresses the previous permanent-orphan finding, but one blocking stale-delete race remains. The sweeper decides expiry from a LIST snapshot and then deletes only by claim name. A container restart within the same Pod keeps the same member ID, so probeRuntimes() can delete and recreate that deterministic claim name with a fresh expiry while another member is processing an older snapshot. The delayed GC delete then removes the fresh live claim, tearing down the Sandbox during the daemon’s startup runtime probe; that probe is not retried, so the member can remain advertising no runtimes. Please fence GC deletion to the listed object incarnation, for example with a Kubernetes UID/resourceVersion delete precondition.

sent by review-bot (Codex · gpt-5.6-sol) · open in session

Comment thread packages/daemon/src/k8s/runtime-plane.ts Outdated

@agentconnect-md-test agentconnect-md-test Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. The explicit sandbox namespace and member-scoped probe identity are applied consistently, the orphaned drain-watch machinery is removed cleanly, and probe cleanup now has a bounded recovery path. The follow-up correctly fences each expired-claim delete with the UID/resourceVersion from the LIST snapshot and treats a precondition conflict as a superseded object, so a stale sweep cannot delete a freshly recreated live probe claim. I found no remaining blocking issues in the complete patch.

sent by review-bot (Codex · gpt-5.6-sol) · open in session

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant