fix(agentbox): self-heal runtime/agentbox mTLS cert mismatch via CA fingerprint

[jacoblee-io]

Summary

Make a runtime↔AgentBox mTLS cert mismatch self-healing: when the runtime CA changes, stale AgentBox pods are now automatically recycled with a fresh cert instead of staying Running-but-unreachable (403 / handshake failure) until someone manually deletes them.

Problem

mTLS trust anchors on the siclaw-runtime-ca CA. A clean helm upgrade / kubectl set image does not change it (the runtime only reads it; cert-manager.ts loads from SICLAW_CA_CERT). But it does change when the CA Secret is regenerated — helm template | kubectl apply (the chart's lookup is a no-op in render contexts → genCA), a lookup-less helm op, or a manual delete. When that happens:

• every already-running AgentBox pod has a server cert signed by the old CA and a mounted ca.crt trusting the old CA;
• the new runtime presents/verifies with the new CA → mTLS fails both directions → 403;
• there is no detection or recovery — the pod stays Running but dead until manually deleted.

(Observed on siclaw-inner: orphaned agentbox-*-cert Secrets from before the last CA regeneration.)

Solution

Stamp each pod and its -cert Secret with a <prefix>/ca-fp label = a fingerprint of the signing CA. The runtime reuses a running pod only if that fingerprint matches its current CA; a mismatch (or a legacy pod with no label) is stale → delete + respawn with a cert from the current CA. One respawn fixes both directions at once.

• Pure label comparison — no TLS probe, no per-request verification.
• No-op when the CA is unchanged (normal releases unaffected; matching pods keep serving).
• Self-heals on any CA change (rotation, accidental regen), turning a permanent stuck mismatch into one transparent respawn.

Keeping the CA stable (persisted Secret / generateCa=false + external caSecret) remains the primary recommendation; this is the safety net for when it isn't.

Changes

• cert-manager.ts: caFingerprint() (sha256 of CA PEM, 16 hex chars)
• k8s-spawner.ts: stamp pod + Secret label; recycle stale pod in the reuse branch; expose caFingerprint(); return it from get()
• manager.ts: isCertFresh() gate on the getOrCreateK8s reuse path
• types.ts / spawner.ts: AgentBoxInfo.caFingerprint + optional BoxSpawner.caFingerprint()
• docs/design/invariants.md §1.3: document the self-heal contract

Test Plan

• npx tsc --noEmit clean
• manager.test.ts: reuse on fp match; recycle on fp mismatch / missing label / legacy; back-compat when spawner reports no CA
• k8s-spawner.test.ts: reuse on match (no delete); recycle stale running pod; recycle unlabeled legacy pod; pod+secret stamped with current fp; caFingerprint() reflects cert manager
• Full src/gateway/agentbox/ + server suites green (113 tests)

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
scitix-e92a9e77	🟢 Ready	View Preview	Jun 23, 2026, 1:37 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

[LikiosSedo]

Reviewed the full diff plus the secret-creation and reuse paths. This is correct and well-targeted — pure label comparison, no TLS probe, a genuine no-op when the CA is unchanged, and it turns a permanently-stuck 403 into one transparent respawn.

I specifically verified the heal is complete end to end: on the stale path the pod is deleted, and the -cert Secret create hits the existing 409 → delete + recreate branch, so the new pod mounts a cert + ca.crt signed by the current CA. mTLS recovers in both directions in a single respawn — no "pod recreated but still serving the old cert" gap. Moving the if (!certManager) throw to the top of spawn() is correct now that the reuse branch needs caFp. Back-compat (no-CA spawner → treated fresh; unlabeled legacy pod → recycled once) and the two-layer check (manager.isCertFresh + spawner reuse branch) are consistent. LGTM.

One low-priority note inline.

Merge coordination: this PR and #350 both edit the same lines of docs/design/invariants.md §1.3 and conflict with each other — verified via git merge-tree. Each merges cleanly into main alone; the second to land needs to hand-merge that section: keep #350's idle-window paragraph + this PR's CA-fingerprint bullet + a combined Source line.

[LikiosSedo]

[Low — multi-replica caveat] The self-heal compares each replica's current CA fingerprint against the pod label. If two runtime replicas ever hold different CAs simultaneously (e.g. mid-rollout right after a CA regen, where replicas loaded different PEM versions), they'd each judge the other's freshly-spawned pods as stale and recycle them → respawn thrash. The "keep the CA stable" guidance avoids this, and the behavior is strictly better than today's stuck 403 regardless. Might be worth one clause here noting the self-heal assumes a single consistent CA across replicas.

[jacoblee-io]

Thanks @LikiosSedo. Addressed the multi-replica note in 53e95fd: added a clause to the §1.3 CA-fingerprint bullet spelling out that the self-heal assumes a single consistent CA across runtime replicas, that the runtime Deployment is a singleton (replicas: 1, Recreate) so the thrash can't arise today, and that even in that pathological case it's strictly better than the pre-self-heal stuck 403.

Re merge coordination with #350 (invariants.md §1.3): confirmed — they conflict on that section but each merges cleanly into main alone. The second to land resolves by keeping #350's idle-window paragraph + this PR's CA-fingerprint bullet + a combined Source line (already validated on the shared inner test branch).

[LikiosSedo]

Approving — re-reviewed after the doc commit (53e95fd).

• Code is correct and self-contained. Verified the heal is complete end to end: on a stale pod the pod is deleted, the -cert Secret is re-minted via the existing 409 → delete + recreate path, and the new pod serves a cert + ca.crt signed by the current CA — both directions recover in a single respawn, no "recreated but still serving the old cert" gap. Two-layer check (manager.isCertFresh + the spawner reuse branch) is consistent; back-compat (no-CA spawner → treated fresh, unlabeled legacy pod → recycled once) is handled.
• My only review note (multi-replica thrash) is addressed in 53e95fd, with the correct added point that the runtime Deployment is a singleton (replicas: 1, Recreate) so it can't arise today.
• CI green (Type Check / Test / Portal Web Test); merges cleanly into main.

Merge-order suggestion: this conflicts with #350 only in docs/design/invariants.md and src/gateway/agentbox/manager.test.ts (both trivial — see my comment on #350). Since #352 is the smaller, self-contained change, it's the better one to land first; #350 then resolves those two conflicts. src/gateway/agentbox/manager.ts auto-merges correctly either way (verified).