Jobs wait ~60 s before the broker offers them to idle self-hosted runners, then go to several runners at once

[adinballew]

Why are you starting this discussion?

Bug

What GitHub Actions topic or product is this about?

Actions Runner

Discussion Details

Jobs sometimes wait about 60 seconds before GitHub offers them to any self-hosted runner, even though idle runners with matching labels are online and long-polling the broker the whole time. The job then starts by itself, so it is not the indefinitely-stuck case in #186811, but the wait is the same length every time and nothing on the runner side accounts for it.

What the runner _diag/Runner_*.log files show in both traced cases:

1. Several runners with the job's labels are idle, with no errors, back-offs or reconnects logged after they went idle.
2. No runner logs Acknowledging runner request for the job for about 60 s after the job's created_at.
3. The same request ID is then offered to several runners in the same second. All but one fail with RunnerRequestJobNotFoundException: Job not found on /acknowledge, then get HTTP 409 on acquirejob. One runner wins and runs the job.

The ~60 s gap matches the documented re-queue for an assigned job that is not picked up within 60 seconds, but none of our runners logged an earlier offer of the job, so if a first assignment happened, no runner received it.

Environment

• Runner version 2.337.0 (latest release), Linux X64, persistent (non-ephemeral) runners, organization-level, all on one host.
• 7 runners share the job's label set (self-hosted, Linux, X64, <lane label>). GET /orgs/{org}/actions/runners lists only our runners, all online, so there are no stale registrations that could have taken the first assignment.
• Jobs use runs-on with that label set. The delayed jobs had no pending needs: dependency.

Incidents (UTC)

Case 1, job ID 111313202003, runner request ea0da1be-acb4-5d02-8246-12da9e1579b2

• Job created_at 2026-10-03T23:06:00Z, no needs:.
• Runner A idle from 23:03:25Z; runner B idle from 23:05:47Z. Neither logged anything until the offer.
• 23:07:01Z: runner C logs Acknowledging runner request 'ea0da1be-…', then Job not found on /acknowledge, then 409 on acquirejob.
• 23:07:02Z: runner A acknowledges the same request and runs it. Job started_at 23:07:03Z.

Case 2, job ID 111327411034, runner request 547006ec-a918-5c71-862d-71c784fb6d06

• The job's only needs: dependency completed at 2026-10-04T00:36:58Z. Job created_at 00:36:59Z.
• 00:38:09Z: runners A and D both acknowledge the same request; A gets Job not found and 409 on acquirejob, D wins. Job started_at 00:38:11Z.

In other runs on the same runners the broker offers a job within 1–2 s of creation.

Ruled out

• Capacity: idle runners with matching labels were online throughout.
• needs: dependencies: the run's jobs API timings show both jobs were ready at created_at.
• Stale or offline registrations: the org runner list holds only our online runners.
• Runner version: 2.337.0 is the latest release.
• The TaskCanceledException / Back off block logged when a job finishes: it is the listener cancelling its status=Busy poll to re-poll as Online (as in Runner Fails to Connect to GitHub Actions Broker - TaskCanceledException / SocketException actions/runner#3904), and it does not line up with either gap.

What we need

Confirmation of whether this ~60 s hold is expected broker behavior (for example, a first assignment to a runner session whose message is never delivered, followed by the 60 s re-queue), and whether anything on the runner or configuration side avoids it. The request IDs and timestamps above should locate both jobs in the broker logs.

[github-actions[bot]]

Thank you for your interest in contributing to our community! We currently only accept discussions created through the GitHub UI using our provided discussion templates. Please re-submit your discussion by navigating to the appropriate category and using the template provided.

This discussion has been closed because it was not submitted through the expected format. If you believe this was a mistake, please reach out to the maintainers.