fix(simulator): worker-roll safety and bounded Kubernetes calls - #978
Merged
Conversation
Every submission installed the run worker before it connected to Temporal, so a submission carrying a different controller image rolled the singleton Deployment while other runs were mid-flight. Rolling it deletes the only Pod heartbeating their controller activities: each attempt fails its 60-second heartbeat deadline, the workflow's shielded cleanupRun then deletes the run's namespace, and the run is gone with no failure the operator asked for. Connect the client first, and only then decide. When the image the cluster already runs differs from the one being submitted, the queue is asked what it still owes; if any run is open — or if the queue could not be listed at all, which is not the same as empty — the submission is refused with the images, the run ids, and the override that proceeds anyway. Submitting the image the cluster already runs applies a Pod template it already has, rolls nothing, and therefore asks nothing: that path must never refuse, or every ordinary submission made during a run would. Reading the installed image and applying the new one remain two calls, so a run submitted between them is still rolled over. That window is accepted for these single-operator clusters; closing it needs a lock the cluster does not offer. The worker Pod also now holds SIGTERM for longer than one heartbeat interval and has a grace period that outlasts that delay, so a roll that is allowed gives an in-flight attempt one more beat rather than cutting it off. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The client's request had no deadline, so an API server that accepted the connection and then answered nothing left the submitter waiting with nothing on stdout to say what it was waiting for — observed once as fourteen silent minutes before the operator gave up. Every call now ends: it either answers, is refused, or fails naming the operation it was making, within a bound that MOLTZAP_KUBERNETES_CALL_TIMEOUT_MS overrides and that defaults to 30 seconds. Being cut off by the bound reads as never answered rather than as failed. The distinction matters to the callers that tolerate a 404 as "already gone": an unanswered delete must not be swallowed as a namespace that is no longer there. The controller's own society calls keep their unbounded shape. Those run inside a cohort that already carries a startup deadline, so an unanswered call there fails the run on that budget rather than silently; the submitter had no such outer bound, which is what made this failure invisible. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
chughtapan
added a commit
that referenced
this pull request
Aug 6, 2026
…ound, one image-producer fact Every Kubernetes call in the simulator now goes through the bounded `kubernetesCall` family, so PR #978's stated invariant holds for the society API's workload, secret, and sandbox calls too. The society seam keeps `ClusterError` as its public error type through one adapter. The evals-side diagnostic bound was structurally dead — the submitter already publishes at most 8192 UTF-8 bytes, so a naive 8192-code-unit slice could only split surrogate pairs — and is gone. The submitter's byte-aware bound is the single authority; the worker's two layers now name their units. `ledgerAllocationFailed` and `runInfrastructureFailed` were the same function twice; they merge into one constructor keyed on the summary `_tag`, and the sequential string-compare dispatch reduces to one. The image-producer fact is one descriptor with the controller path written once, and `EvaluationImageKey` is derived from it. Also: the invisible-workspace-file list is computed once at definition time; the `McpServer` structural narrow is one exported matcher rather than three `"url" in server` sites; the worker-roll visibility query quotes the operator-supplied task queue and stops interpolating a constant status; `INSTALL_ORDER` derives its membership from `RunWorkerManifests` so a new object fails compile; and the cluster.sh prelude assertion checks definition and use rather than an occurrence count. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two observed cluster failure modes closed:
MOLTZAP_FORCE_WORKER_ROLL=1overrides). Same-image submissions stay a no-op (regression-tested). Grace period + preStop sized to the heartbeat./simplify applied. No new failure types: refusals use the existing stage-tagged RunSubmissionError channel.
🤖 Generated with Claude Code