fix(scale-set-chart): Added missing deployments permissions to role

[willejs]

If you are running in kubernetes mode, the buildx builder fails to start as it cannot create a deployment.
This fixes the permissions issue.

[santosr2]

I'm experiencing the same issue. Any plan for when it will be reviewed?

[davepgreene]

This is still an issue. Any maintainers interested in taking a look?

[davepgreene]

@nikola-jokic it looks like you're the primary contributor these days. This is a tiny change and fixes an issue that's been persistent for a year. Can we get it merged or at least addressed?

[nikola-jokic]

Hey @davepgreene,

I don't think we should include this one in the release. The hook and the config that we provide through the chart contain the bare minimum required for it to run the basic workflow. If we were to include the deployment for this use case, where would we draw the line then? Why should the hook have this permission out of the box?

Although it is not as easy as it should be to provide a custom config for the container mode, until we can prioritise work related to customising container modes, you will have to provide your own custom config and not use the containerMode field.

[davepgreene]

Edit: I figured out the issue. The buildx kubernetes driver uses deployments: https://github.com/docker/buildx/blob/master/driver/kubernetes/factory.go#L146

Might be useful to call it out in docs as the wording I noted below seems to apply to any/all docker actions.

Hey @davepgreene,

I don't think we should include this one in the release. The hook and the config that we provide through the chart contain the bare minimum required for it to run the basic workflow. If we were to include the deployment for this use case, where would we draw the line then? Why should the hook have this permission out of the box?

Although it is not as easy as it should be to provide a custom config for the container mode, until we can prioritise work related to customising container modes, you will have to provide your own custom config and not use the containerMode field.

I'm not sure I understand your apprehension. The docs suggest that kubernetes mode is equivalent to dind but in-cluster whereas the reality is that it doesn't work because the scale set controller can't create the deployment that does the build. Note specifically the docs say:

When the runner detects a workflow run that uses a container job, service container, or Docker action, it will call runner-container-hooks to create a new pod.

I can't tell if this is a bug where the runner should be creating a pod rather than a deployment or this particular PR where the permissions just need to be updated but either way trying to run scale-sets that build images in kubernetes fails without those permissions. This is my scale set configuration:

---
githubConfigUrl: https://github.com/davepgreene/<REPO>
githubConfigSecret: runner-app-secret
containerMode:
  type: kubernetes
  kubernetesModeWorkVolumeClaim:
    accessModes: ["ReadWriteOnce"]
    storageClassName: longhorn
    resources:
      requests:
        storage: 1Gi
maxRunners: 2
template:
  spec:
    initContainers:
      - name: kube-init
        image: ghcr.io/actions/actions-runner:latest
        command: ["sudo", "chown", "-R", "1001:123", "/home/runner/_work"]
        volumeMounts:
          - name: work
            mountPath: /home/runner/_work
    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:latest
        command: ["/home/runner/run.sh"]
        env:
          - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
            value: "false"

And this is my action (some steps removed because they aren't relevant):

name: Build and Push Docker Image

on:
  push:
    branches: [main]

jobs:
  build-and-push:
    runs-on: arc-runner-set
    permissions:
      contents: read
      packages: write
      id-token: write

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
        with:
          driver: kubernetes
          driver-opts: |
            replicas=1
            limits.memory=128Mi
            limits.ephemeral-storage=2G
            qemu.install=true

      - name: Build and push Docker image
        id: build-and-push
        uses: docker/build-push-action@v5
        with:
          context: .
          build-args: GIT_HASH=${{ github.sha }}
          push: true
          cache-from: type=gha
          cache-to: type=gha,mode=max
          sbom: true

This fails during the docker/setup-buildx-action step with the following error:

Error: ERROR: error for bootstrap "builder-30c6328a-d5c7-4703-9a1a-c6cade4df1580": deployments.apps "builder-30c6328a-d5c7-4703-9a1a-c6cade4df1580" is forbidden: User "system:serviceaccount:arc-systems:arc-runner-set-gha-rs-kube-mode" cannot get resource "deployments" in API group "apps" in the namespace "arc-systems"

I've spent literal hours poring through docs and code to try to figure out why this is failing and the only thing I've done that solves the issue is to update the role permissions. I'm sure I'm missing something but it's not clear at all to me what it is.

Happy to file an issue if you think that's useful.