A lot of random "A task was cancelled" errors

[omer2500]

Describe the bug

We've encountered intermittent "A task was cancelled" errors during various phases of our GitHub Actions workflows. These errors occur inconsistently—sometimes during the setup-job phase, and other times mid-execution within a job step.

In the logs, we didn't find anything interesting...
And we are looking for a direction to investigate
To Reproduce

Steps to reproduce the behavior:

1. run a lot of matrix jobs in self self-hosted runner

Runner Version and Platform

Version of your runner?

linux 2.323.0

What's not working?

we are running the runner on a self hosted aws eks cluster with ARC

[patrickvinograd]

We are also observing this for roughly the last week.

[Seggan]

Can confirm, for example https://github.com/pylonmc/pylon-core/actions/runs/14648537973/job/41108671805

[omer2500]

Can confirm, for example https://github.com/pylonmc/pylon-core/actions/runs/14648537973/job/41108671805

im not sure its the same issue

Error: The operation was canceled. vs Error: A task was canceled

[Seggan]

Ah, maybe not. I'll make a separate issue

[mjeszenkavalstro]

Our organization is having the same issue.

[adel-s]

Ours also.

[Noamshmueli]

Our organization also experience the same issue

[YevhS]

The same for me as well

[m0ana]

Same in our org

[Storm-BE]

Same thing here :(

[Storm-BE]

I did some investigation before on the logs of our runners, and found a pattern for jobs that get cancelled while being started.

The runner shuts down immediately because it gets a 'Runner not found' response from some API.
This seems related to following change: #3536
I suspect that this 'runner not found' is a transient issue and a retry fixed it before this change got merged

[patrickvinograd]

We see the same pattern in the runner logs. If we put them side-by-side with the gha-scale-set-controller logs, we see:

May 15 06:47:02.249 | gha-runner-scale-set-controller | Checking if runner exists in GitHub service
May 15 06:47:02.249 | gha-runner-scale-set-controller | Runner exists in GitHub service

And then immediately in the runner logs:

An error occurred: Runner not found

The scale-set-controller tries up to 5 times before failing the ephemeral runner, and the pattern is consistent throughout - the controller thinks the runner resource exists on the GHA service side, the runner starts up and thinks it does not. I have not dug through the code to see if they are invoking the exact same API/referring to the same thing.

[patrickvinograd]

The runner code is generating the RunnerNotFound exception here invoking a /message endpoint and getting some error payload with "RunnerNotFound" from that GHA broker API. I.e. it's not a connection failure or anything like that, it's getting a response back from the service and parsing it.

The scale-set-controller is invoking a GetRunner method here, seems to be a totally different endpoint _apis/distributedtask/pools/0/agents/<runner_id> that's returning successfully.

Without knowing any internal details of the Github Actions service APIs I don't see what else affected parties outside of Github can do to resolve or even mitigate this.

[mihstaub]

Our org have the same issue....