Multi-tenancy deployment with helm chart

[aajith-arista]

Hi,
I wanted to try out the multi-tenancy model in actions-runner-controller-0.21.0 / v0.26.0.
I am using the helm chart to deploy and I use the github app authentication and webhook server.
So I set githubWebhookServer.enabled=true.

I got rid of the global secret authSecret.name="controller-manager". I understand that instead the secret needs to be specified in the spec of RunnerDeployment and HorizontalAutoscaler.

However, the helm chart fails to apply cleanly.
The deployment <release>-actions-runner-controller fails to come up. The pods are still looking for the global secret.

> kubectl describe -n actions-runner-system pod/arc-release-actions-runner-controller-594cc69d-fb79h
...
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age               From               Message
  ----     ------       ----              ----               -------
  Normal   Scheduled    33s               default-scheduler  Successfully assigned actions-runner-system/arc-release-actions-runner-controller-594cc69d-fb79h to k8s-worker23.sjc.aristanetworks.com
  Warning  FailedMount  3s (x7 over 34s)  kubelet            MountVolume.SetUp failed for volume "secret" : secret "controller-manager" not found

If I run with authSecret.enabled=false, the pods end up in a crash loop.

> kubectl  logs -n actions-runner-system pod/arc-release-actions-runner-controller-59b7d857b-8tx5k manager
Error: Client creation failed. authentication failed: using private key of size 0 (...): could not parse private key: invalid key: Key must be a PEM encoded PKCS1 or PKCS8 key

How do I get this working for the helm chart ?

Thanks,
Arun

[aajith-arista]

@mumoshu Can you please help me here ?

[some-natalie]

👋🏻 Random drive-by opinion here. 😄

First, controller-manager is the secret (PAT or GitHub app) that ARC needs to authenticate to GitHub (cloud or server). Probably need to put that back.

I'm unclear what you mean by "multi-tenancy", but am assuming you're trying to run multiple controllers in the same cluster. While that's technically possible, it's probably simpler to have 1 controller manage lots of deployments.

[aajith-arista]

Hi,

I'm talking about this: https://github.com/actions-runner-controller/actions-runner-controller/#multitenancy.
Earlier, we had to run multiple instances of the controllers per organization/installation since we have secret per organization/installation. IIUC, with the new feature, this becomes possible for multiple installations to share the same controller by decoupling the secret from the controller and moving it to the RunnerDeployment.

Thanks,
Arun

[bm1216]

Hey,

With multi-tenancy you can define secrets per RunnerDeployment. But the controller still needs the original secret. I have a question here #1844 to find out WHY it needs it. But as far as I can see, it still needs the original secret otherwise the controller won't run.

[bm1216]

👋🏻 Random drive-by opinion here. 😄

First, controller-manager is the secret (PAT or GitHub app) that ARC needs to authenticate to GitHub (cloud or server). Probably need to put that back.

I'm unclear what you mean by "multi-tenancy", but am assuming you're trying to run multiple controllers in the same cluster. While that's technically possible, it's probably simpler to have 1 controller manage lots of deployments.

There's a cool new "multi-tenancy" feature in ARC 😄 - #1371

[some-natalie]

OMG, how'd I even miss this? 🤦🏻‍♀️

Thanks for the heads up @bm1216 !

[mumoshu]

I got rid of the global secret authSecret.name="controller-manager"

@aajith-arista Hey! No, you can't do that. You still need the default auth secret in case the HRA controller needs to call GitHub API before deciding which HRA to scale(ARC can't decide which githubAPICredentialsFrom.configMapName to read until the HRA controller decides which HRA to scale!), and to prepare for cases where there's no githubAPICredentialsFro field specified for a particular HRA/RunnenrDeployment/RunnerSet/etc.

[mumoshu]

@bm1216 Hey! See this- https://github.com/actions-runner-controller/actions-runner-controller/blob/5fd6ec4bc82ed8ad50c87340cfafe2cf1a4cbd93/controllers/horizontal_runner_autoscaler_webhook.go#L521-L543 here HRA controller fetches visible runner groups for the repository to decide which runner group hence HRA to scale.

[aajith-arista]

Couple more things that wasn't clear from the documentation.

1. I understand githubAPICredentialsFrom.secretRef.name refers to the secret for this particular runner/scaler. Should this secret be deployed in the runner/scaler namespace or in the controller's(system) namespace ?

2. Also is the global secret identical to the per-installation secrets ? I mean do you need to specify all of app_id, app_installation_id and app_private_key for per-installation secrets or just the app_installation_id because that's the only thing that needs to be different from the global secret ?

[bm1216]

@aajith-arista It should be in the runner/scaler's namespace.
https://github.com/actions-runner-controller/actions-runner-controller/blob/5fd6ec4bc82ed8ad50c87340cfafe2cf1a4cbd93/controllers/multi_githubclient.go#L227-L229

[bm1216]

Couple more things that wasn't clear from the documentation.

1. I understand githubAPICredentialsFrom.secretRef.name refers to the secret for this particular runner/scaler. Should this secret be deployed in the runner/scaler namespace or in the controller's(system) namespace ?
2. Also is the global secret identical to the per-installation secrets ? I mean do you need to specify all of app_id, app_installation_id and app_private_key for per-installation secrets or just the app_installation_id because that's the only thing that needs to be different from the global secret ?

The global secret is pretty much identical to the per-installation secret. It parses those 3 values as ENV vars from the secret. While on the per-installation secret, it reads those values in here -
https://github.com/actions-runner-controller/actions-runner-controller/blob/5fd6ec4bc82ed8ad50c87340cfafe2cf1a4cbd93/controllers/multi_githubclient.go#L269-L303

As you can see, it's reading all 3 of those values (and more).

[mike-chen-samsung]

@mumoshu But how do I know which secret to give the controller in a multi-tenant setting? I have two GitHub Organizations and each one has their own GitHub App. Or is multi-tenancy only valid when installing Enterprise Runners with a PAT?

[mstory21]

We're hitting an issue using this in practice. We have two private github apps. One for the global controller that is in the arista-eos-external org and one that's for the barney-ci org. I'm referring to the latter secret in my runner and scaler. I'm getting this error in the controller's logs:

2023-03-07T11:43:01Z	DEBUG	events	Warning	{"object": {"kind":"Runner","namespace":"barney","name":"github-runner-rgsnr-zg6nw","uid":"8551008d-0257-499d-9613-4ad28fd0ec95","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"2248545439"}, "reason": "FailedUpdateRegistrationToken", "message": "Updating registration token failed"}

The barney-ci app has actions (read), metadata (read), and self-hosted runners (read/write) permissions.

Does the controller's app need to be installed on the barney-ci org as well? Unfortunately if that's the case then we'd have to make it public meaning anyone could install it.

[mstory21]

Nevermind, this is working now. Yaml indentation problem.