Failure during orphan termination processing

[sdarwin]

The scale-down lambda fails to terminate orphan AWS self-hosted runner instances, and throws errors.

Background:

The latest versions of start-runner.sh and start-runner.ps1 both intended to terminate the EC2 instance after the job has finished. You can see this by searching for "terminate" in the files. If those succeed in terminating the instance after the job is done, then everything is fine. No orphans. But what happens if that termination doesn't occur?

Termination will not occur if...

• you are using an older version of start-runner.ps1, which did not yet call "terminate". The problem would always happen.
• currently I am trying to re-design start-runner.ps1 (to set the user) and if the script doesn't yet manage to call "terminate", the problem happens.
• most importantly - the whole terraform system should be robust to unexpected failures. If start-runner.sh or start-runner.ps1 glitch, or github itself has problems, and you may end up with an orphan instance. Once you get an orphaned instance, the scale-down should handle the situation gracefully, and terminate it. Any time you encounter an "orphan", scale-down should terminate it correctly. But that isn't happening now, it seems.

A stack trace from cloudwatch is shown below. Consider:

GET /repos/myorg/myrepo/actions/runners/653 - 404

This api call was added recently in https://github.com/github-aws-runners/terraform-aws-github-runner/pull/4595/files . It's a new feature, right?

type OrgRunnerList = Endpoints['GET /orgs/{org}/actions/runners']['response']['data']['runners'];
type RepoRunnerList = Endpoints['GET /repos/{owner}/{repo}/actions/runners']['response']['data']['runners'];
type RunnerState = OrgRunnerList[number] | RepoRunnerList[number];

By basic code inspection, ask yourself, how will this function react if the runner doesn't exist. If the job finished, Github Actions doesn't know about the ephemeral runner any more. That means, the endpoint "/orgs/{org}/actions/runners" returns nothing, because Github thinks the runner isn't there. Will that cause the 404? Then, if the scale-down process gets a 404, is it able to handle it, and terminate the orphan?

Cloudwatch log message 1:

2025-09-22T20:31:03.167Z	e1066279-d200-4500-bc75-3f7a6274b4e9	ERROR	GET /repos/myorg/myrepo/actions/runners/653 - 404 with id B82E:10BDA9:D22A39:D6CC52:68D1B206 in 246ms

Cloudwatch log message 2:

{
    "level": "WARN",
    "message": "Failure during orphan termination processing.",
    "timestamp": "2025-09-22T20:32:03.223Z",
    "service": "runners-scale-down",
    "sampling_rate": 0,
    "xray_trace_id": "1-68d1b242-4965478d1f42877e3fcd62a6",
    "region": "us-west-2",
    "environment": "gha-ubuntu-noble",
    "module": "scale-down",
    "aws-request-id": "7212ccf6-8d5c-4ce8-98af-493f59b0eda3",
    "function-name": "gha-ubuntu-noble-scale-down",
    "error": {
        "name": "HttpError",
        "location": "file:///var/task/index.js:158569",
        "message": "Not Found - https://docs.github.com/rest/actions/self-hosted-runners#get-a-self-hosted-runner-for-a-repository",
        "stack": "HttpError: Not Found - https://docs.github.com/rest/actions/self-hosted-runners#get-a-self-hosted-runner-for-a-repository\n    at fetchWrapper (file:///var/task/index.js:158569:11)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at async Job.doExecute (file:///var/task/index.js:126562:18)",
        "status": 404,
        "request": {
            "method": "GET",
            "url": "https://api.github.com/repos/myorg/myrepo/actions/runners/653",
            "headers": {
                "accept": "application/vnd.github.v3+json",
                "user-agent": "github-aws-runners octokit-rest.js/22.0.0 octokit-core.js/7.0.2 Node.js/22",
                "authorization": "token [REDACTED]"
            },
            "request": {}
        },
        "response": {
            "url": "https://api.github.com/repos/myorg/myrepo/actions/runners/653",
            "status": 404,
            "headers": {
                "access-control-allow-origin": "*",
                "access-control-expose-headers": "ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset",
                "content-encoding": "gzip",
                "content-security-policy": "default-src 'none'",
                "content-type": "application/json; charset=utf-8",
                "date": "Mon, 22 Sep 2025 20:32:03 GMT",
                "referrer-policy": "origin-when-cross-origin, strict-origin-when-cross-origin",
                "server": "github.com",
                "strict-transport-security": "max-age=31536000; includeSubdomains; preload",
                "transfer-encoding": "chunked",
                "vary": "Accept-Encoding, Accept, X-Requested-With",
                "x-accepted-github-permissions": "administration=read",
                "x-content-type-options": "nosniff",
                "x-frame-options": "deny",
                "x-github-api-version-selected": "2022-11-28",
                "x-github-media-type": "github.v3; format=json",
                "x-github-request-id": "833A:2C77BC:D42741:D8CCE8:68D1B242",
                "x-ratelimit-limit": "5000",
                "x-ratelimit-remaining": "4840",
                "x-ratelimit-reset": "1758574803",
                "x-ratelimit-resource": "core",
                "x-ratelimit-used": "160",
                "x-xss-protection": "0"
            },
            "data": {
                "message": "Not Found",
                "documentation_url": "https://docs.github.com/rest/actions/self-hosted-runners#get-a-self-hosted-runner-for-a-repository",
                "status": "404"
            }
        }
    }
}

How to replicate the issue:

Modify /modules/runners/templates/start-runner.sh , comment out trap 'cleanup $? $LINENO $BASH_LINENO' EXIT so that cleanup doesn't happen.
Modify /modules/runners/templates/start-runner.ps1 , comment out aws ec2 terminate-instances --instance-ids "$InstanceId" --region "$Region" so that cleanup doesn't happen.

Run github actions jobs. Instances will become "orphans". And then scale-down will fail to terminate the orphans.

What should happen:

In the past, orphans would get scaled-down without errors, and that was the right result.

@stuartp44 , @npalm

[stuartp44]

The code you mention above where added to stop premature termination due a pagination issue when you operate runners at scale. We have a lot of data slippage, on the GitHub API when we query the volume of runners we operate at. Unfortunately, this means we get runners that are classed as orphan because the data of the said runner was on page x and then slipped, to a lower page, while we were paginating over it, which then means that it gets terminated even though it's in a healthy state and still exists and is still registered to GitHub.

What I did with this pull request was implement logic so that when termination is called and an EC2 instance is in a state of orphanisation, it doesn't end up in a one-way street where the termination is just immediate. Instead, what we do is we do a last check to ensure that the state of the runner is actually non-existent on GitHub rather than what we had before, which was relying on the pagination to see if the runner exists in that pagination and then tag it accordingly.

You are right to point out, though, that the behavior from GitHub at the time was not documented that a 404 is a valid response. We have since fed that back to GitHub and had it confirmed that 404 does indeed mean it's missing/no longer registered. Now, what we've also done now is handled 404 issues within the actual function to ensure that if we do get a 404, it immediately is removed but this was available/fixed as of version 6.7.4

Hope this helps!

[sdarwin]

@stuartp44 thanks. I'm testing now with 6.7.8. but assume it will pass.

• Since this issue was about PR 4595 - more feedback about that. It adds a file "scale-down-state-diagram.md", that mentions "quota" and "quota full". Could there be a footnote in the file "scale-down-state-diagram.md" at the end of the page, such as "* Quotas are implemented via the runners_keep_minimum_instances variable, in the multi-runner module" (or whichever variable it is). Since you mention "quota", but it's difficult to search for information about that word. I tried grep in the repository.

• "GitHub confirmed that 404 does indeed mean it's missing/no longer registered.". If you have a contact at Github, how about replying to them, with a suggestion. On the page https://docs.github.com/en/enterprise-cloud@latest/rest/using-the-rest-api/troubleshooting-the-rest-api?apiVersion=2022-11-28#404-not-found-for-an-existing-resource where it says:

"If you make a request to access a private resource and your request isn't properly authenticated, you will receive a 404 Not Found response. GitHub uses a 404 Not Found response instead of a 403 Forbidden response to avoid confirming the existence of private repositories."

It could be preceded by one additional sentence:

"Usually if you receive a 404 Not Found error from the api, that is confirmation the runner does not exist, and it's the expected return code after a runner has been deactivated. However... if your request isn't properly authenticated, you will also receive a 404 Not Found response. "

Well, that's optional, if you don't think it's worth the trouble, but it could help to improve github's own documentation.

---

Edit:

6.7.8 succeeded. "Runner 'i-0784377318d4388b7' is judged to be orphaned." (instead of the errors). Great!

[sdarwin]

This issue can be closed.
It's not necessary to bother Github about their documentation. that was a random suggestion.
The only reason I have not closed the issue, was a question:
What does 'quota' refer to in "scale-down-state-diagram.md"? The only place the concept of 'quota' seems to exist is in that file. (But I might be wrong).

[pgbezerra]

We've been facing this problem for two days, and I updated the module to the latest version, but the problem persists.

A runner executing a job may be killed in the middle of the execution.

The log during the down-scale lambda execution is:

{
  "level": "ERROR",
  "message": "github-launch service unavailable - https://docs.github.com/rest/actions/self-hosted-runners#list-self-hosted-runners-for-an-organization",
  "timestamp": "2025-09-26T14:12:44.668Z",
  "service": "runners-scale-down",
  "sampling_rate": 0,
  "xray_trace_id": "1-68d69f56-5728f311595736733a4f2577",
  "region": "us-west-2",
  "environment": "[REDACTED]-ci-amazon-linux-2023",
  "aws-request-id": "0064f084-a9cb-4af1-adef-f0de0e92f01b",
  "function-name": "[REDACTED]-ci-amazon-linux-2023-scale-down",
  "module": "lambda.ts",
  "error": {
    "name": "HttpError",
    "location": "file:///var/task/index.js:161370",
    "message": "github-launch service unavailable - https://docs.github.com/rest/actions/self-hosted-runners#list-self-hosted-runners-for-an-organization",
    "stack": "HttpError: github-launch service unavailable - https://docs.github.com/rest/actions/self-hosted-runners#list-self-hosted-runners-for-an-organization\n    at fetchWrapper (file:///var/task/index.js:161370:11)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at async Job.doExecute (file:///var/task/index.js:129540:18)",
    "status": 503,
    "request": {
      "method": "GET",
      "url": "https://api.github.com/organizations/28880063/actions/runners?per_page=100&page=2",
      "headers": {
        "accept": "application/vnd.github.v3+json",
        "user-agent": "github-aws-runners octokit-rest.js/22.0.0 octokit-core.js/7.0.2 Node.js/22",
        "authorization": "token [REDACTED]"
      },
      "request": {}
    },
    "response": {
      "url": "https://api.github.com/organizations/28880063/actions/runners?per_page=100&page=2",
      "status": 503,
      "headers": {
        "access-control-allow-origin": "*",
        "access-control-expose-headers": "ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset",
        "content-length": "186",
        "content-security-policy": "default-src 'none'",
        "content-type": "application/json; charset=utf-8",
        "date": "Fri, 26 Sep 2025 14:12:44 GMT",
        "referrer-policy": "origin-when-cross-origin, strict-origin-when-cross-origin",
        "server": "github.com",
        "strict-transport-security": "max-age=31536000; includeSubdomains; preload",
        "vary": "Accept-Encoding, Accept, X-Requested-With",
        "x-accepted-github-permissions": "organization_self_hosted_runners=read",
        "x-content-type-options": "nosniff",
        "x-frame-options": "deny",
        "x-github-api-version-selected": "2022-11-28",
        "x-github-media-type": "github.v3; format=json",
        "x-github-request-id": "E056:D126D:2A3AE5E:2B3AF80:68D69F5B",
        "x-ratelimit-limit": "15000",
        "x-ratelimit-remaining": "14405",
        "x-ratelimit-reset": "1758897786",
        "x-ratelimit-resource": "core",
        "x-ratelimit-used": "595",
        "x-xss-protection": "0"
      },
      "data": {
        "message": "github-launch service unavailable",
        "documentation_url": "https://docs.github.com/rest/actions/self-hosted-runners#list-self-hosted-runners-for-an-organization",
        "status": "503"
      }
    }
  }
}

I still didn't dig into the lambda code to see what happens, but I wonder if a 503 would lead to the instance's termination.

[sdarwin]

I wonder if a 503 would lead to the instance's termination.

Good question. This is a similar topic, but different from what I was facing, where a 404 failed to lead to an instance's termination. It's almost the opposite. Now there are extra terminations. But it might be in the same section of code.

[sdarwin]

@pgbezerra wrote "I updated the module". There are at least two things to update. And they could be done separately (unless it's all automated now). Were they both done?

1. The lambdas. Download newer lambda zip files. which will be uploaded to AWS.
2. The main terraform code, the runner modules, etc.

[pgbezerra]

@pgbezerra wrote "I updated the module". There are at least two things to update. And they could be done separately (unless it's all automated now). Were they both done?

1. The lambdas. Download newer lambda zip files. which will be uploaded to AWS.
2. The main terraform code, the runner modules, etc.

I updated the terraform modules, and all the lambdas were recreated, I assumed the code was updated. Let me double check it.

[pgbezerra]

After digging more, I understand that these are different problems. I was facing the eviction issue, which was solved with the terraform upgrade.

Now I have an internal issue that is killing my containers from the inside, so it's completely unrelated.

Thank you all! 🙏🏻