JobReconciliationSweep fails healthy ephemeral jobs: the 60s heartbeat grace period is unreachable on a cold-start executor #3519
Closed
luis-balam
started this conversation in
Ideas
Replies: 1 comment
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Since 2.33.0,
JobReconciliationSweepfails jobs whose executor is alive and working.With ephemeral agents (
api.ephemeralExecution.enabled: true) on an autoscaled cluster, a fresh podcannot write its first heartbeat inside
HEARTBEAT_GRACE_PERIOD, so the sweep concludes theexecutor is gone and fails the job.
Environment
api.ephemeralExecution.enabled: true)Cause
The sweep measures age from
job.getUpdatedDate()and fails the job when the heartbeat key is absent:The executor writes that key only after it has picked up a job, on a 15s schedule:
An ephemeral pod on a new node cannot beat 60s. Measured: node provisioning 25-33s, image pull
~10s (388 MB, and far worse when many nodes pull at once), Spring Boot start plus job pickup
15-20s, then 0-15s of
fixedDelayphase before the first write. First heartbeat lands atroughly 50-78s.
HEARTBEAT_GRACE_PERIODisprivate static finalwith no@Valueand no chart value, so there isno way to tune it. (The source itself notes it is "duplicated as a literal in executor's
JobExecutionWatchdog - separate Spring Boot apps, no shared constants".)
Note also that
JobReconciliationSweepServiceschedules the sweep with cron*/30 * * ? * *, so ajob is failed at the first 30s tick past 60s, the effective window is [60s, 90s). That phase
race means a fraction of a batch fails rather than all of it, which makes the symptom look
intermittent and random.
Reproduction
The larger the concurrent batch, the more reliably it fails: more nodes to provision and more
simultaneous image pulls push every pod past the deadline.
Evidence the executor was alive
The sweep declared this executor gone at 68s. The pod was
Running, connected to Valkey with noerrors, and its plan step went on to complete normally:
The apply step is destroyed before it ever runs
Because the sweep evaluates the job rather than the step, a slow plan start kills the still-
pendingapply step. This job was the only one running:Final state:
step 100 Terraform Plan = completed,step 200 Terraform Apply = failed, withoutputandoutputErrorboth empty. Terraform never ran for the apply , the step was failedadministratively while pending.
Impact on our instance
110 of those 111 had a failed step with zero output and zero error text, killed by the sweep, not
by Terraform. Roughly 100 workspaces silently stopped applying their changes for two days.
Suggested fix
Either of these alone would resolve it:
job.updatedDate. The API creates the Job/pod, so dispatch and running can be recordedseparately.
api.properties.heartbeatGracePeriodSeconds). Simplest change, and lets operators set it fromtheir measured cold-start time.
Two smaller improvements that would also help:
remove the 0-15s jitter.
pendingand has never been assigned an executor. Theabsence of a heartbeat for work that has not started is not evidence of a dead executor.
Workaround for others hitting this
Keep autoscaled nodes warm so pods do not pay node provisioning plus image pull. On Karpenter,
raising the executor node pool's
consolidateAfterfrom60sto15mdropped pod start from~45s to ~1s , after which concurrent applies on a warm node succeed.
This is a mitigation, not a fix, the first job after scale-to-zero still races the deadline.
All reactions