Skip to content

Annotations are not set to EphemeralRunner pods #4143

Open
@andrey-petrenko-develeap

Description

Checks

Controller Version

0.11.0

Deployment Method

Helm

Checks

  • This isn't a question or user support case (For Q&A and community support, go to Discussions).
  • I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

1. Install `actions-runner-controller` and `gha-runner-scale-set` Helm charts with Terraform `helm_release` resource
2. Update `gha-runner-scale-set` values with:

annotations:
  ad.datadoghq.com/exclude: "true"

3. Update `gha-runner-scale-set` Helm release with Terraform
4. Annotation is added to `AutoscalingRunnerSet` resource, but any EphemeralRunner` is created without it

Describe the bug

I am installing actions-runner-controller and gha-runner-scale-set Helm charts with Terraform.
I need to add an annotation ad.datadoghq.com/exclude: "true" to all EphemeralRunner pods. I set it in the values file, and after the apply, any EphemeralRunner is still created without it.

Now looking again at values.yaml at the chart repo I see the line "## Optional annotations and labels applied to all resources created by helm installation" which would make sense, as EphemeralRunners are not directly created by Helm.

But then I'd expect all the custom annotations being applied to the downstream resources, controlled by the ones, that were created with Helm, i.e.
AutoscalingRunnerSet -> EphemeralRunnerSet -> EphemeralRunner

Describe the expected behavior

Annotations that are set at annotations: section of the Helm values should propagate to every created resource, as documented, or there should be another way of setting runner pod annotations with Helm chart.

Additional Context

Terraform code for scale set Helm release:

resource "helm_release" "runners_aws_4g_scale_set" {
  count = var.github_runners_enabled ? 1 : 0

  name       = "runner-aws-${var.aws_profile}-4g"
  namespace  = local.github_runners_namespace
  repository = "oci://ghcr.io/actions/actions-runner-controller-charts"
  chart      = "gha-runner-scale-set"
  version    = "0.11.0"

  depends_on = [helm_release.actions_runner_controller]

  values = [
    file("${path.module}/../runners.yaml"),
  ]
  set {..}

}


Helm values after apply:

helm -n github-runners get values runner-aws-dev-4g

USER-SUPPLIED VALUES:
affinity: {}
annotations:
  ad.datadoghq.com/exclude: "true"
containerMode:
  type: dind
githubConfigSecret:
[...]
labels: {}
maxRunners: 16
minRunners: 1
podLabels: {}
podSecurityContext: {}
priorityClassName: ""
resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi
runnerGroup: default
runnerLabels: []
securityContext: {}
serviceAccount:
  create: true
  name: gha-runner-scale-set
template:
  spec:
    containers:
    - command:
      - /home/runner/run.sh
      image: *redacted*/runner:0.1.2
      imagePullPolicy: IfNotPresent
      name: runner
      resources:
        limits:
          memory: 4Gi
      volumeMounts:
      - mountPath: /runner/_work
        name: runner-cache
    imagePullSecrets:
    - name: github-token
      namespace: github-runners
    volumes:
    - emptyDir: {}
      name: runner-cache
topologySpreadConstraints: []
volumeMounts: []
volumes: []


AutoscalingRunnerSet CRD after apply - the annotation is in the list:

kubectl describe AutoscalingRunnerSet runner-aws-dev-4g -n github-runners
Name:         runner-aws-dev-4g
Namespace:    github-runners
Labels:       actions.github.com/organization=***
[...]
              app.kubernetes.io/part-of=gha-rs
              app.kubernetes.io/version=0.11.0
              helm.sh/chart=gha-rs-0.11.0
Annotations:  actions.github.com/cleanup-github-secret-name: runner-aws-dev-4g-gha-rs-github-secret
              actions.github.com/cleanup-manager-role-binding: runner-aws-dev-4g-gha-rs-manager
              actions.github.com/cleanup-manager-role-name: runner-aws-dev-4g-gha-rs-manager
              actions.github.com/cleanup-no-permission-service-account-name: runner-aws-dev-4g-gha-rs-no-permission
              actions.github.com/runner-group-name: Default
              actions.github.com/runner-scale-set-name: runner-aws-dev-4g
              actions.github.com/values-hash: 98612cfa3c2e56aec9d889a91a633ca6c455c80df366d637ad800e27c9aba5c
              ad.datadoghq.com/exclude: true
              meta.helm.sh/release-name: runner-aws-dev-4g
              meta.helm.sh/release-namespace: github-runners
              runner-scale-set-id: 26
API Version:  actions.github.com/v1alpha1
Kind:         AutoscalingRunnerSet
[...]
Status:
  Current Runners:            1
  Pending Ephemeral Runners:  1
Events:                       <none>


Example of the runner pod:

kubectl describe pod runner-aws-dev-4g-f8q97-runner-ljvlv -n github-runners

Name:             runner-aws-dev-4g-f8q97-runner-ljvlv
Namespace:        github-runners
Priority:         0
Service Account:  runner-aws-dev-4g-gha-rs-no-permission
Node:             ***
Start Time:       Tue, 24 Jun 2025 14:20:13 +0300
Labels:           actions-ephemeral-runner=True
[...]
                  app.kubernetes.io/name=runner-aws-dev-4g
                  app.kubernetes.io/part-of=gha-runner-scale-set
                  app.kubernetes.io/version=0.11.0
                  helm.sh/chart=gha-rs-0.11.0
                  pod-template-hash=5dbbbc8d65
Annotations:      actions.github.com/patch-id: 0
                  actions.github.com/runner-group-name: Default
                  actions.github.com/runner-scale-set-name: runner-aws-dev-4g
                  actions.github.com/runner-spec-hash: 5ffc57fc6
                  cluster-autoscaler.kubernetes.io/safe-to-evict-local-volumes: datadog
Status:           Running
[...]
Controlled By:  EphemeralRunner/runner-aws-dev-4g-f8q97-runner-ljvlv
[...]
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
[...]
Events:
[...]

Controller Logs

no relevant logs

Runner Pod Logs

no relevant logs

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinggha-runner-scale-setRelated to the gha-runner-scale-set modeneeds triageRequires review from the maintainers

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions