An error occurred: Runner not found

[theihor]

Describe the bug

Hello. I am maintaining Linux Kernel BPF CI.

We use self-hosted runners on various hardware. Recently they started failing with the following vague error: An error occurred: Runner not found. It affects all architectures that we use (x86_64, aarch64 and s390x). This then causes the runner systemd service to restart, and then the GH App quickly runs out of tokens due to frequent restarts.

Sometimes some runners are able to pick up a job when the tokens reset, but most fail with this error.

As far as I can tell, runner not found message indicates RunnerNotFoundException caught at https://github.com/actions/runner/blob/main/src/Runner.Listener/Program.cs#L148-L153

The only place it is thrown from is https://github.com/actions/runner/blob/main/src/Sdk/WebApi/WebApi/BrokerHttpClient.cs#L118-L119
Which means that github broker endpoint returns this error.

However, there is no clue what it means exactly. The runners are ephemeral with no reuse, and "replace" themselves on each run.

There were no recent changes to the self-hosted infra, or runner updates, or in the workflows.

From systemd logs on the hosts, it appears that these errors first started to appear some time in April, but in the last few days they became much more frequent to the point of blocking the testing pipeline.

To Reproduce
N/A

Expected behavior

Runners should pick up and run jobs normally.

Runner Version and Platform

v2.323.0 on Linux x86_64, aarch64 and s390x

What's not working?

Runners fail to pick up jobs after successful authentication.

Job Log Output

May 13 00:07:18 bpf-ci-runner-s390x-01 systemd[1]: Started actions-runner-kernel-patches@01.service - Ephemeral GitHub Actions Runner Container for kernel-patches - 01.
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: Runner reusage is disabled
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: Obtaining the token of the runner
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: Ephemeral option is enabled
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: Disable auto update option is enabled
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: Configuring
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: --------------------------------------------------------------------------------
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: |        ____ _ _   _   _       _          _        _   _                      |
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: |       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: |      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: |      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: |       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: |                                                                              |
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: |                       Self-hosted runner registration                        |
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: |                                                                              |
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: --------------------------------------------------------------------------------
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: # Authentication
May 13 00:07:21 bpf-ci-runner-s390x-01 docker[5099]: √ Connected to GitHub
May 13 00:07:21 bpf-ci-runner-s390x-01 docker[5099]: # Runner Registration
May 13 00:07:22 bpf-ci-runner-s390x-01 docker[5099]: A runner exists with the same name
May 13 00:07:22 bpf-ci-runner-s390x-01 docker[5099]: √ Successfully replaced the runner
May 13 00:07:22 bpf-ci-runner-s390x-01 docker[5099]: √ Runner connection is good
May 13 00:07:22 bpf-ci-runner-s390x-01 docker[5099]: # Runner settings
May 13 00:07:22 bpf-ci-runner-s390x-01 docker[5099]: √ Settings Saved.
May 13 00:07:24 bpf-ci-runner-s390x-01 docker[5099]: √ Connected to GitHub
May 13 00:07:24 bpf-ci-runner-s390x-01 docker[5099]: Current runner version: '2.323.0'
May 13 00:07:24 bpf-ci-runner-s390x-01 docker[5099]: 2025-05-13 00:07:24Z: Listening for Jobs
May 13 00:07:25 bpf-ci-runner-s390x-01 docker[5099]: An error occurred: Runner not found
May 13 00:07:25 bpf-ci-runner-s390x-01 systemd[1]: actions-runner-kernel-patches@01.service: Main process exited, code=exited, status=1/FAILURE
May 13 00:07:25 bpf-ci-runner-s390x-01 systemd[1]: actions-runner-kernel-patches@01.service: Failed with result 'exit-code'.

Runner and Worker's Diagnostic Logs

Didn't have a chance to capture these. Will add a comment when I do.

[TingluoHuang]

Can you share more detail about your runner?
ex: what's the name of your runner and how it get setup and replaced?

[kargerova]

Hello, we are facing the same Issue at our organization with two of three our runners.
All three runners are:

• self hosted
• organization scoped
• in the same group
• x64 running as docker container on Linux virtual machine
• using image myoung34/github-runner

The two failing runners are ephemeral, registered via GitHub personal access token (token is valid until 2026),
started by system service running docker run

ExecStartPre=-/usr/bin/docker stop %N
ExecStartPre=-/usr/bin/docker rm %N
ExecStart=/usr/bin/docker run --rm \
                              --env-file /etc/ephemeral-github-actions-runner.env \
                              -e RUNNER_NAME=%H \
                              -v /var/run/docker.sock:/var/run/docker.sock \
                              -v /tmp/runner:/tmp/runner \
                              --name %N \
                              --security-opt label=disable \
                              myoung34/github-runner:latest

The one functional runner is not emphemeral,
started directly by docker compose up

services:

  github-runner:
    image: myoung34/github-runner:latest
    container_name: github-runner
    restart: unless-stopped
    security_opt:
      - label:disable
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /tmp/runner:/tmp/runner
    env_file:
      - .env

The issue occurs regardless of the runner version - with latest as with older '2.322.0'

All runners have been working properly until today. Nothing changed on our infrastructure.

Job log output

May 13 08:06:10 docker[4073]: Runner reusage is disabled
May 13 08:06:10 docker[4073]: Obtaining the token of the runner
May 13 08:06:11 docker[4073]: Ephemeral option is enabled
May 13 08:06:11 docker[4073]: Configuring
May 13 08:06:11 docker[4073]: --------------------------------------------------------------------------------
May 13 08:06:11 docker[4073]: |        ____ _ _   _   _       _          _        _   _                      |
May 13 08:06:11 docker[4073]: |       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
May 13 08:06:11 docker[4073]: |      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
May 13 08:06:11 docker[4073]: |      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
May 13 08:06:11 docker[4073]: |       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
May 13 08:06:11 docker[4073]: |                                                                              |
May 13 08:06:11 docker[4073]: |                       Self-hosted runner registration                        |
May 13 08:06:11 docker[4073]: |                                                                              |
May 13 08:06:11 docker[4073]: --------------------------------------------------------------------------------
May 13 08:06:11 docker[4073]: # Authentication
May 13 08:06:12 docker[4073]: √ Connected to GitHub
May 13 08:06:12 docker[4073]: # Runner Registration
May 13 08:06:12 docker[4073]: A runner exists with the same name
May 13 08:06:12 docker[4073]: √ Successfully replaced the runner
May 13 08:06:14 docker[4073]: √ Runner connection is good
May 13 08:06:14 docker[4073]: # Runner settings
May 13 08:06:14 docker[4073]: √ Settings Saved.
May 13 08:06:14 docker[4073]: √ Connected to GitHub
May 13 08:06:15 docker[4073]: Current runner version: '2.322.0'
May 13 08:06:15 docker[4073]: 2025-05-13 08:06:15Z: Listening for Jobs
May 13 08:06:16 docker[4073]: An error occurred: Runner not found
May 13 08:06:17 systemd[1]: ephemeral-github-actions-runner.service: Main process exited, code=exited, status=1/FAILURE
May 13 08:06:17 systemd[1]: ephemeral-github-actions-runner.service: Failed with result 'exit-code'.
May 13 08:06:17 systemd[1]: ephemeral-github-actions-runner.service: Scheduled restart job, restart counter is at 2.

[kargerova]

I removed the runners from organization settings and they (as they are ephemeral) registered again. Now runners are working properly.

So this is not an issue anymore, but don't know what really happened.
Hope this helps.

[theihor]

@TingluoHuang

Can you share more detail about your runner?

Very similar setup to what @kargerova described above.

self hosted
organization scoped
in the same group
x64 running as docker container on Linux virtual machine

All checks. Only we customize myoung34/github-runner image, see Dockerfile.

Runners are provisioned with ansible scripts. In short, a systemd service is set up to maintain runner containers:

[Unit]
Description=Ephemeral GitHub Actions Runner Container for kernel-patches - %i
After=docker.service
Requires=docker.service

[Service]
TimeoutStartSec=0
Restart=always
EnvironmentFile=/etc/actions-runner/runner_unit.env
# Optionally loaded file. Use this to override per runner environment
EnvironmentFile=-/etc/actions-runner/runner_unit-%i.env
ExecStartPre=-/usr/bin/docker stop %p-%i
ExecStartPre=-/usr/bin/docker rm %p-%i
ExecStartPre=-/usr/bin/docker pull ghcr.io/kernel-patches/runner:${DOCKER_TAG}
ExecStartPre=-/etc/actions-runner/gh_token_generator.sh 250536 "/etc/actions-runner/kernel-patches-runner_250536_priv.pem" "/etc/actions-runner/actions-runner-kernel-patches-worker-%i-ghtoken.env"
ExecStart=/usr/bin/docker run --device=/dev/kvm \
            --tmpfs /tmp/work:exec \
            --rm \
            --env-file "/etc/actions-runner/actions-runner-kernel-patches-worker-%i.env" \
            --env-file "/etc/actions-runner/actions-runner-kernel-patches-worker-%i-ghtoken.env" \
            --name %p-%i \
            ghcr.io/kernel-patches/runner:${DOCKER_TAG}

[Install]
WantedBy=multi-user.target

ex: what's the name of your runner and how it get setup and replaced?

Runner names have a format like this: <prefix>-<host>-worker-<i>. Examples: bm-i-0a77d6f5721623905-worker-01, bpf-ci-runner-s390x-03-worker-01.

When I say "replaced", I mean that normally I'd see Runner successfully added in the log, which I presume means that technically a new "runner" from the pov of github is created every time the container starts. Log snippet from a successful job run:

May 13 11:05:00 ip-10-0-0-97.us-west-1.compute.internal docker[16128]: # Authentication
May 13 11:05:01 ip-10-0-0-97.us-west-1.compute.internal docker[16128]: √ Connected to GitHub
May 13 11:05:01 ip-10-0-0-97.us-west-1.compute.internal docker[16128]: # Runner Registration
May 13 11:05:02 ip-10-0-0-97.us-west-1.compute.internal docker[16128]: √ Runner successfully added
May 13 11:05:02 ip-10-0-0-97.us-west-1.compute.internal docker[16128]: √ Runner connection is good
May 13 11:05:02 ip-10-0-0-97.us-west-1.compute.internal docker[16128]: # Runner settings
May 13 11:05:02 ip-10-0-0-97.us-west-1.compute.internal docker[16128]: √ Settings Saved.
May 13 11:05:03 ip-10-0-0-97.us-west-1.compute.internal docker[16128]: √ Connected to GitHub
May 13 11:05:04 ip-10-0-0-97.us-west-1.compute.internal docker[16128]: Current runner version: '2.323.0'
May 13 11:05:04 ip-10-0-0-97.us-west-1.compute.internal docker[16128]: 2025-05-13 11:05:04Z: Listening for Jobs
May 13 11:05:07 ip-10-0-0-97.us-west-1.compute.internal docker[16128]: 2025-05-13 11:05:07Z: Running job: x86_64 gcc-14 / test (test_progs_no_alu32, false, 360) / test_progs_no_alu32 on x86_64 with gcc-14
May 13 11:11:31 ip-10-0-0-97.us-west-1.compute.internal docker[16128]: 2025-05-13 11:11:31Z: Job x86_64 gcc-14 / test (test_progs_no_alu32, false, 360) / test_progs_no_alu32 on x86_64 with gcc-14 completed with result: Succeeded
May 13 11:11:31 ip-10-0-0-97.us-west-1.compute.internal docker[16128]: √ Removed .credentials
May 13 11:11:31 ip-10-0-0-97.us-west-1.compute.internal docker[16128]: √ Removed .runner

[theihor]

I removed the runners from organization settings and they (as they are ephemeral) registered again. Now runners are working properly.

So this is not an issue anymore, but don't know what really happened. Hope this helps.

@kargerova I will try this, thanks for the tip.

[theihor]

For anyone interested, a working mitigation is to use unique runner names when runner container starts.

If you're using myoung34/github-runner image, you can achieve this by:

RUNNER_NAME_PREFIX=your-runner-name
RANDOM_RUNNER_SUFFIX=true
# instead of
# RUNNER_NAME=your-runner-name

These are handled by entrypoint.sh.

Given that this helps, I suspect the reason github broker might return this error is when a runner name is reused (maybe with "unexpected" access token?).

[djtarazona]

We're experiencing this issue was well with Tartelet. Manually deleting the runner in GitHub settings fixes the issue for a while. It will eventually happen again.

[patrickvinograd]

I believe this is the same or similar as #3819 - e.g. see this comment with the "Runner not found" error in the self-hosted runner logs.

[atsu85]

Hi
It seems to me, that it doesn't matter if you're using some fork of runner base image or the official one from this repository, because

• same symptoms happen with both official and forks of base images:
  • @theihor and @kargerova are using myoung34/github-runner
  • @djtarazona mentioned Tartelet.
  • I'm using ghcr.io/actions/actions-runner:2.324.0 (with Actions Runner Controller (ARC) Kubernetes operator to auto-scale number of ephemeral runners in k8s as needed)
• symptoms started happening without significant changes besides the base image version, as mentioned above after we updated :

There were no recent changes to the self-hosted infra, or runner updates, or in the workflows.

The first Failed EphemeralRunners was created couple of hours after after we updated the runner base image!

Although I'm not using fork of the runner, I'm seeing very similar issues and logs myself as well from runner:

[GChanathip]

I got the same issue using tartelet with runner version 2.324.0.

[austinorth]

Same here, version 2.324.0.