Unable to register Arm64 runners

[welteki]

Which Github Action / Prow Jobs are flaking?

We noticed an issue with Arm64 jobs running on Actuated. Jobs stay queued because we have insufficient permissions to get a JIT config for registering the runners.

If any changes were made recently, permissions for the Actuated app should be checked at the organisation and repo level.

cc @alexellis

Which tests are flaking?

Arm64 tests running on managed runners.

Github Action / Prow Job link

No response

Reason for failure (if possible)

Insufficient permissions for the Actuated app at the organisation or repo level to create a JIT config for registering runners.

Anything else we need to know?

No response

[welteki]

Looking at one of the queued Arm64 jobs we see a message that self-hosted runners are disabled.

Self-hosted runners in the repository are disabled by your administrator

[jmhbnz]

Hi @welteki - Thanks for raising this, I am not aware of any changes at our end. Checking our actions queue I can see a number of arm64 jobs have been sitting in queue for a while https://github.com/etcd-io/etcd/actions?query=is%3Aqueued++.

Has something changed on the Actuated side or is there a GitHub change/outage potentially?

If we think it is a permissions change on the etcd side I will need to reach out to Kubernetes project folks as etcd-io org is now under the Kubernetes GitHub Enterprise account.

/assign @jmhbnz

[alexellis]

We've seen the message "Self-hosted runners in the repository are disabled by your administrator" once or twice before - this happens when people are fiddling with the settings at the org/enterprise level without realising the impact the change would have.

[welteki]

Hi @jmhbnz,

Has something changed on the Actuated side or is there a GitHub change/outage potentially?

We did not make any changes on the Actuated side. There is no outage reported for GitHub and we are not seeing any issues with jobs for other organisations.

If we think it is a permissions change on the etcd side I will need to reach out to Kubernetes project folks as etcd-io org is now under the Kubernetes GitHub Enterprise account.

I can see jobs queued for other repos in the etcd-io org as well: https://github.com/etcd-io/bbolt/actions?query=is%3Aqueued+. All jobs display the same message Self-hosted runners in the repository are disabled by your administrator so I think the cause is a permissions change at the org level.

[jmhbnz]

Thanks team - I've started a slack chat with the Kubernetes GitHub management folks, https://kubernetes.slack.com/archives/C01672LSZL0/p1717015195019279 so we can check from that side 🙏🏻

[jmhbnz]

Confirmed issue was on Kubernetes GitHub org admins side and has been corrected:

> @jmhbnz I was reviewing some enterprise-level settings related to a non-etcd related security issue. I’ve reverted the setting change.. can you try again?

Jobs are now being picked up again. Thanks for raising the alarm @welteki.

[alexellis]

Good 👍

We've requested that the pending 64 jobs that were still pending get started now on new VMs. Let us know if you see any further issues with jobs not starting etc.

[alexellis]

In this circumstance, it looks like the jobs that are pending are stuck and in deadlock because their definitions disallow self-hosted runners. They will need to be cancelled/restarted.