Ephemeral runner survives a lost job assignment (409 Conflict) and becomes a ghost

[LouisCuvelier]

Describe the bug

An ephemeral runner that fails to acquire an assigned job with HTTP 409 Conflict calls continue and re-enters its message loop. By then the service has already removed its registration, because assigning a job to an ephemeral runner consumes it.

The result is a ghost runner: the process is alive, the container reports healthy, and the last line on stdout is Listening for Jobs — but GET /orgs/{org}/actions/runners does not list it, and it will never be assigned work again. Nothing surfaces the failure. Only a manual restart recovers it.

To Reproduce

Hard to reproduce on demand — it is a race on job assignment. Observed on a pool of 5 org-scoped ephemeral runners handling two active monorepos. Each occurrence permanently removes one runner from the pool.

1. Run several ephemeral runners (config.sh --ephemeral) against one org.
2. Queue enough concurrent jobs that assignments race.
3. Wait. Runners drop out one by one, with no error on stdout.

Expected behavior

An ephemeral runner whose job assignment is lost should exit. The supervising process (Docker restart policy, systemd, ARC) then replaces it with a freshly registered runner.

Actual behavior

It stays in the listen loop forever, deregistered service-side.

Runner Version and Platform

Runner: 2.334.0
OS: Linux x64, Docker (myoung34/github-runner:latest)
Scope: organization runners, --ephemeral --disableupdate

What's not working?

_diag/Runner_*.log, last lines before the runner went silent:

[11:55:53Z INFO Terminal] WRITE LINE: 2026-08-07 11:55:53Z: Listening for Jobs
[11:55:53Z INFO JobDispatcher] Set runner/worker IPC timeout to 30 seconds.
[11:59:45Z INFO BrokerMessageListener] Acknowledging runner request '5f7c2a52-****'.
[11:59:46Z ERR  GitHubActionsService] POST request to https://run-actions-1-azure-eastus.actions.githubusercontent.com/37/acquirejob failed. HTTP Status: Conflict
[11:59:46Z INFO Runner] Skipping message Job. Job message already acquired '5f7c2a52-****'. job assignment is invalid: MissingKey

Nothing after that. The process was still running 12 minutes later, and the runner was absent from the org runners API the whole time.

Root cause

src/Runner.Listener/Runner.cs (main, ~L737):

catch (Exception ex) when (
    ex is TaskOrchestrationJobNotFoundException ||          // HTTP status 404
    ex is TaskOrchestrationJobAlreadyAcquiredException ||   // HTTP status 409
    ex is TaskOrchestrationJobUnprocessableException)       // HTTP status 422
{
    Trace.Info($"Skipping message Job. {ex.Message}");
    await _acquireJobThrottler.IncrementAndWaitAsync(messageQueueLoopTokenSource.Token);
    continue;
}

The handler never checks whether the runner is ephemeral. Skipping is the right call for a persistent runner, but for an ephemeral one the registration is already gone, so continue loops on a dead session.

The analogous case ~30 lines above already handles ephemeral correctly:

Trace.Info($"Acknowledge returned job-not-found for ephemeral runner request '{messageRef.RunnerRequestId}'. Exiting runner.");
runOnceJobCompleted = true;
return Constants.Runner.ReturnCode.Success;

So the intent exists for a lost acknowledge, but not for a lost acquire.

Still present on main as of today, i.e. after v2.336.0 — none of 2.335.0 / 2.335.1 / 2.336.0 touch this path.

Related issues

• Ephemeral runner hangs indefinitely after GitHub outage and does not recover #4076 reports the same user-visible symptom (ephemeral runner alive, Listening for Jobs, never picks up work) but attributes it to a GitHub outage and has no _diag trace. The trace above may be one concrete cause behind it. That issue asks for a generic --exit-after-idle flag; the fix below is narrower and does not need a new flag.
• Runner.Worker hangs before reading the job message (30s IPC timeout), ignores SIGINT/SIGTERM; ephemeral runner exits 0 and the assigned job is orphaned in queued #4598 is the mirror case: the ephemeral runner exits and orphans the assigned job. Here it survives an assignment it never got.
• "Runner listener exit with retryable error" breaks ephemeral behaviour #2286 and Intermittent runner communication loss on AWS Fargate (ephemeral, ~2% failure rate) #4498 involve a listener restart and a mid-job communication loss respectively — different paths.

Impact

Silent, cumulative pool drain. Each hit costs one runner with no visible error — containers stay Up and healthy, stdout still reads Listening for Jobs. Our pool reached zero available runners with a job queued for 68 minutes before anyone noticed.

Suggested fix

Mirror the existing ephemeral handling, but only on the two statuses that mean the assignment is gone:

if (settings.Ephemeral &&
    (ex is TaskOrchestrationJobNotFoundException || ex is TaskOrchestrationJobAlreadyAcquiredException))
{
    _term.WriteLine("The job assigned to this ephemeral runner is no longer available. Cleaning up local configuration.");
    Trace.Info($"Ephemeral runner lost its job assignment. Exiting runner. {ex.Message}");
    skipSessionDeletion = true;
    runOnceJobCompleted = true;
    return Constants.Runner.ReturnCode.Success;
}

Trace.Info($"Skipping message Job. {ex.Message}");
await _acquireJobThrottler.IncrementAndWaitAsync(messageQueueLoopTokenSource.Token);
continue;

422 (TaskOrchestrationJobUnprocessableException) is excluded on purpose: it says the job cannot be processed, not that the assignment moved, and exiting would delete the local config of a runner that could still serve. skipSessionDeletion avoids a 30 s stall deleting a session whose registration the service already consumed. A persistent runner keeps the current skip-and-retry behaviour.

PR: #4618.

Workaround

External watchdog: read agentName from /actions-runner/.runner in each running container, compare against GET /orgs/{org}/actions/runners, and restart any container whose name is absent for more than two minutes.