Summary
Agent.Analyze/Execute/Verify/Escalate are synchronous, minutes-long calls
executed inside the reconcile loop (sandbox pod creation, readiness wait,
HTTP call to the agent, response handling — all within one Reconcile
invocation). This architecture produced three distinct failure modes during
testing, all with the same root cause.
Failure modes observed
-
Operator restart orphans in-flight proposals. The handler sets the
step condition to Unknown ("Analysis agent is running") and then blocks
in the agent call. If the operator restarts (upgrade, rollout, OOM), the
new process sees Analyzed=Unknown and waits forever — there is no
timeout, no lease, no recovery. Observed live: a proposal permanently
stuck after a routine oc set image rollout mid-analysis; the only remedy
was deleting and recreating it.
-
Proposal deletion blocks on the finalizer until the blocked call
completes. The rbac-cleanup finalizer is processed by the same worker
that is blocked inside the agent call, so oc delete proposal on a
proposal mid-analysis hangs for minutes. Observed live: an external
client's delete appeared to succeed but the CR persisted for ~4 minutes
until the analysis finished, causing the client (the lightspeed-evaluation
driver) to interleave cleanup with a still-live proposal and delete the
test workload out from under the following run.
-
Stale-cache duplicate step execution. After a status patch decides a
retry, the next reconcile can start from a not-yet-updated informer cache
and re-run the entire just-completed step. Observed live: verification ran
twice for the same attempt (two sandbox pods, two LLM calls, both logging
"attempt 2/3"), 40 seconds apart. Blocking calls stretch the stale-cache
window from microseconds to minutes, and the step handlers have no
already-decided guard against it.
Suggested direction
Run agent interactions asynchronously with respect to reconciliation — e.g.
persist step state and drive the sandbox interaction from a separate
goroutine/workqueue keyed by (proposal, step, attempt), or model each step as
a child resource whose completion triggers reconciliation. Short of that:
resumable in-progress detection (re-attach to a running sandbox on restart),
a step-level timeout/lease on the Unknown condition, and idempotency guards
in each handler against stale-cache re-entry.
Context
Found while validating the operator end-to-end against a self-hosted vLLM
endpoint (Gemma 4 31B) with the lightspeed-evaluation proposal suite, and
A/B-comparing against Claude Opus 4.6 on Vertex. Full testing report with
setup details and all findings:
https://gist.github.com/harche/1cb5e80a1156c75cad53b44fe3976099
cc @mrunalp
Summary
Agent.Analyze/Execute/Verify/Escalateare synchronous, minutes-long callsexecuted inside the reconcile loop (sandbox pod creation, readiness wait,
HTTP call to the agent, response handling — all within one
Reconcileinvocation). This architecture produced three distinct failure modes during
testing, all with the same root cause.
Failure modes observed
Operator restart orphans in-flight proposals. The handler sets the
step condition to
Unknown("Analysis agent is running") and then blocksin the agent call. If the operator restarts (upgrade, rollout, OOM), the
new process sees
Analyzed=Unknownand waits forever — there is notimeout, no lease, no recovery. Observed live: a proposal permanently
stuck after a routine
oc set imagerollout mid-analysis; the only remedywas deleting and recreating it.
Proposal deletion blocks on the finalizer until the blocked call
completes. The rbac-cleanup finalizer is processed by the same worker
that is blocked inside the agent call, so
oc delete proposalon aproposal mid-analysis hangs for minutes. Observed live: an external
client's delete appeared to succeed but the CR persisted for ~4 minutes
until the analysis finished, causing the client (the lightspeed-evaluation
driver) to interleave cleanup with a still-live proposal and delete the
test workload out from under the following run.
Stale-cache duplicate step execution. After a status patch decides a
retry, the next reconcile can start from a not-yet-updated informer cache
and re-run the entire just-completed step. Observed live: verification ran
twice for the same attempt (two sandbox pods, two LLM calls, both logging
"attempt 2/3"), 40 seconds apart. Blocking calls stretch the stale-cache
window from microseconds to minutes, and the step handlers have no
already-decided guard against it.
Suggested direction
Run agent interactions asynchronously with respect to reconciliation — e.g.
persist step state and drive the sandbox interaction from a separate
goroutine/workqueue keyed by (proposal, step, attempt), or model each step as
a child resource whose completion triggers reconciliation. Short of that:
resumable in-progress detection (re-attach to a running sandbox on restart),
a step-level timeout/lease on the
Unknowncondition, and idempotency guardsin each handler against stale-cache re-entry.
Context
Found while validating the operator end-to-end against a self-hosted vLLM
endpoint (Gemma 4 31B) with the lightspeed-evaluation proposal suite, and
A/B-comparing against Claude Opus 4.6 on Vertex. Full testing report with
setup details and all findings:
https://gist.github.com/harche/1cb5e80a1156c75cad53b44fe3976099
cc @mrunalp