Missing documentation for AutoscalingRunnerSet

[jaskolla]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.6.1

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

Missing Documentation - Not being able to use AutoscalingRunnerSet

Describe the bug

IMO, Github runner scaled set helm chart is a not a great design ( see bottom on why I think so) and it's mostly due to helm tool's own restrictions and design choices.

AutoscalingRunnerSet is pretty simple and mostly self explanatory, I think community should be able to write their own helm chart to suit their purposes. But, I couldn't find any official documentation for AutoscalingRunnerSet, and all the docs point to using ARSS helm chart.

Reason on why helm chart is limited:

• Template Spec containers such as runner is made as array instead of map, which means even to override Image we need to define the entire runner. This means even if we subchart ARSS helm chart, a lot of details leak into our users.
• Containers assume infinite resource available to them. dind-container is hardcoded and and it can't be deployed in k8s cluster with OPA validations around resources.
• Can't add labels to resources created by the helm chart
• Doesn't support Multiple Runners per helm chart

PS: Above strong opinions are my own, not related to my employer.

Describe the expected behavior

Documentation for AutoscalingRunnerSet

Additional Context

NA

Controller Logs

NA

Runner Pod Logs

NA

[github-actions[bot]]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[nikola-jokic]

Hey @jaskolla, I will try my best to answer all questions ☺️

As you pointed out, helm has some drawbacks that we need to adhere to. Helm was picked as one of the most widely used tools. There could be an argument for kustomize, but we had to make a judgement call. Some people would be happy with helm, others would be happy with kustomize.

I'm glad that you find the design of AutoscalingRunnerSet simple. However, there are some internal variables that we are using while applying a chart. They could be documented, but instead of that, a simple --dry-run could show you the exact resource that could be applied. Also, some of them are internal and may change from one version to another.

The community is always welcome to write their charts. The support poses an issue here. We can offer support for our officially supported charts, but not for custom-built ones. Resources can be created in many ways, and there is just no way we can support every single one of them.

To answer the question for template runners, you can provide any PodSpecTemplate. If you are building your runner image, there is just no way for us to know the entrypoint, or other configuration variables that should be exposed. ARC sets appropriate configuration variables with the assumption that the container named runner contains and will execute the runner binary. You can apply any constraints on your image, define security context, or provide volumes that you'd like. At the end of the day, the runner is just a pod that needs to execute something and shut down. The definition of the pod is up to you, based on your constraints.

Any container mode that we provide contains only the off-the-shelf implementation. It does not affect the controller in any way. Customizations are very well described in this discussion post.

Adding labels to resources is available, and you can control most of the visible resources. However, we do create some internal resources that we need to fully control. It would be great if you could provide an example of why would someone pick special annotations to the EphemeralRunnerSet for example. It is an internal detail and not part of the public API.

As for the multiple runners, all scale sets are homogenous. All runners that belong to that scale set should behave the same way. A single installation of the chart creates a single instance of the scale set.

You can always use sub-charts to manage multiple installations at once.

That being said, what kind of documentation would you be interested in?

[jaswanthikolla]

However, there are some internal variables that we are using while applying a chart. They could be documented, but instead of that, a simple --dry-run could show you the exact resource that could be applied

I already did run dry-run before and found it simple, just to make it clear I am talking about following k8s kind. Can you give examples of the internal variables you are talking about, I am asking that because I don't see any in the helm generated kind: AutoscalingRunnerSet or in this autoscalingrunnerset.yaml tempalte.

apiVersion: actions.github.com/v1alpha1
kind: AutoscalingRunnerSet
metadata:
  name: arc-runner-set

We can offer support for our officially supported charts, but not for custom-built ones.

And

That being said, what kind of documentation would you be interested in?

Yes, Of course. I am only asking enable others to write their own chart. Specifically, I meant that make the k8s underlying resource kind AutoscalingRunnerSet documented and supported as a public API so that community can write their own charts.

Why I think Github ARSS chart is limited, and community needs their own charts:

Adding labels to resources is available,

One example I can give is the way role is created. , It doesn't allow labels to be defined by values.yaml.

Another example of not being able to define resource limits to dind containers. If you see this template function , It hardcodes everything and doesn't allow defining resource limit such as following

  limits:
    cpu: "1"
    memory: "1Gi"
  requests:
    cpu: "1"
    memory: "1Gi"

You can always use sub-charts to manage multiple installations at once.

Yes, But the problem is, ARSS chart creates resources without giving option to disable it such as service accounts and roles. While a individual repo needs multiple runners but it doesn't make sense to create multiple service accounts.

To answer the question for template runners, you can provide any PodSpecTemplate.

Sorry for not being clear. I meant that containers are declared as array instead of map, which means if I want to override one property I need to declare the whole thing. Reason for that is, Helm merges the map and replaces the arrays.

[nikola-jokic]

Hey, sorry for the delay.

Your request does make sense to me, but we cannot prioritize it at this moment and documenting everything would take a considerable amount of time in my opinion. I agree that writing your own chart would be somewhat difficult, considering that we do have some "unusual" ways of configuring stuff, such as applying roles during installation to limit cluster level permissions.

The way I would personally do it today is start here, to see all fields that are associated with the autoscaling runner set. You can see all types for each field, which should be enough to understand how to create the top level resource. As for the rest of it, I don't have a better advice, except for reading adrs like limiting role permissions to see why and how we applied roles.
Kubernetes mode is particularly difficult since it requires a service account applied to the runner in order for the hook to create pods. That is why I said that documenting everything would take considerable amount of time, because we do try to hide some complexity associated with configuring different modes.

Until we are able to document it, what I can offer is help answer questions if you have any issues developing your own chart ☺️

[jaskolla]

Just checking if you can take a look at the above follow up comment. Thanks so much!