Allow HA for the gha-runner-scale-set listener

[braun1928]

🏷️ Discussion Type

Question

💬 Feature/Topic Area

ARC (Actions Runner Controller)

Discussion Details

We had issues with the listener today, that it was running in a node that went on a spiral and stopped working. Checking what to do, we weren't able to set the listener to have more than one pod to have a little more availability. I know the docs say you can deploy the same runner set in different clusters and create a group in GitHub, but no idea if you can create two different runner sets in the same cluster (with different names), and create that group to mimic that set up.
At least we were able to add nodeSelector/toleration to choose where the listener pod will be, but ideally it should work with at least 2 pods to allow some disruption.

[Luke-Corwin]

The listener is currently designed as a singleton component for a runner scale set, so scaling the listener itself for high availability isn’t generally supported in the same way as scaling runner pods.

One possible approach is what you mentioned: creating multiple runner scale sets and placing them in the same GitHub runner group. GitHub will schedule jobs across all available runners in that group regardless of which scale set they belong to.

Running multiple scale sets in the same cluster should be possible as long as they have unique names and are configured independently. This can provide some redundancy compared to relying on a single scale set.

For the listener specifically, many teams mitigate risk by:

• Scheduling it onto dedicated nodes using nodeSelector and tolerations
• Applying PodDisruptionBudgets where appropriate
• Using cluster autoscaling and node health monitoring
• Running multiple runner scale sets rather than relying on a single listener

Out of curiosity, when the listener stopped working, did Kubernetes restart the pod and it remained unhealthy, or did the node itself become unavailable? The root cause may influence whether multiple scale sets would provide sufficient redundancy.