Actions time out after some amount of time

[Kampe]

When utilizing the actions runner controller in our clusters within GKE (1.18.20), we've noticed that long running jobs (usually ones over half an hour) will eventually kill themselves while in the process of completing and responds with no exit code. The job seemingly stops reporting to GitHub entirely. Is this known behavior and is there anything we can do to alleviate this?

[mumoshu]

@Kampe Perhaps you're affected by this? #466

Also, do you use preemptive instances on GKE for your worker nodes that are running your runner pods?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[FalconerTC]

I also tend to see this issue and am running on GKE preemptive instances. These failures don't align with preemptive node reclaims though.

[mumoshu]

@FalconerTC Hey! Could you perhaps check if the pods are OOM-killed? I occasionally see the behavior on runner pods getting OOM-killed.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[FalconerTC]

Looking into this again with some more details. #466 does seem related. I often (once or twice a day with maybe 100-200 various workflow runs a day) see the following in the UI

The self-hosted runner: ubuntu-20.04-ppmfq-84ds4 lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

Looking at the pods, they don't appear to be hitting an OOM. Here are some logs from one pod that was killed

Starting Runner listener with startup type: service
Started listener process
Started running service

√ Connected to GitHub

2021-10-06 15:59:04Z: Listening for Jobs
Runner update in progress, do not shutdown runner.
Downloading 2.283.1 runner
Waiting for current job finish running.
Generate and execute update script.
Runner will exit shortly for update, should back online within 10 seconds.
Runner listener exited with error code 4
Runner listener exit with undefined return code, re-launch runner in 5 seconds.
Starting Runner listener with startup type: service
Started listener process
Unrecognized command-line input arguments: 'once'. For usage refer to: .\config.cmd --help or ./config.sh --help

√ Connected to GitHub

2021-10-06 16:00:09Z: Listening for Jobs
2021-10-06 16:00:16Z: Running job: Lint
2021-10-06 16:04:37Z: Job Lint completed with result: Succeeded
Runner listener exited with error code 0
Runner listener exit with 0 return code, stop the service, no retry needed.

In this case, it looks like a job successfully completed in a runner, the runner failed to restart correctly, was assigned a new workflow, and then the pod completed and the workflow run was abandoned. The runner was also marked offline in the Github UI.

I should also note I'm seeing this both on a RunnerGroup on preemptibles and on a RunnerGroup running on regular GKE nodes. I'll add more information as I find it.

[mumoshu]

@FalconerTC Thanks for the update!

Which kind of runners are you using, persistent or ephemeral?
If you're using persistent ones, you'd better try ephemeral as you seem to use preemptible instances.

We also recently introduced support for the new GA version of ephemeral runners https://github.com/actions-runner-controller/actions-runner-controller#ephemeral-runners. It addresses a known race issue that a runner tries to grab job even while it's being stopped and that results in an error like that. If you haven't tried it, I'd recommend doing so.

Also, are you using cluster-autoscaler?

[FalconerTC]

I am using ephemeral runners, as well as the GKE cluster-autoscaler. I'm also using RunnerDeployments. I'll try out RUNNER_FEATURE_FLAG_EPHEMERAL and see if it improves things

[mumoshu]

@FalconerTC Thanks for confirming. Yes, try the new ephemeral runners. It should improve things.

Also- when you're using cluster-autoscaler, you should block cluster-autoscaler from trying to terminate a node in which any runner pod is running a job.
I think that can be done by adding "cluster-autoscaler.kubernetes.io/safe-to-evict": "false" annotation to every runner pod.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.