Skip to content

upstream github-action-runners dockerhub authentication #726

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 20, 2023
Merged

upstream github-action-runners dockerhub authentication #726

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions modules/eks/actions-runner-controller/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,21 @@ components:
github_app_id: "REPLACE_ME_GH_APP_ID"
github_app_installation_id: "REPLACE_ME_GH_INSTALLATION_ID"

# use to enable docker config json secret, which can login to dockerhub for your GHA Runners
docker_config_json_enabled: true
# The content of this param should look like:
# {
# "auths": {
# "https://index.docker.io/v1/": {
# "username": "your_username",
# "password": "your_password
# "email": "your_email",
# "auth": "$(echo "your_username:your_password" | base64)"
# }
# }
# } | base64
ssm_docker_config_json_path: "/github_runners/docker/config-json"
Copy link
Member

@osterman osterman Jun 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Document how to set this value in SSM

Copy link
Member

@osterman osterman Jun 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally with an atmos workflow

Copy link

@batkins-carecloud batkins-carecloud Jun 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@osterman here's the Atmos workflow I created for setting the value in SSM

  save/docker-config-json:
    description: Prompt for uploading Docker Config JSON to the AWS SSM Parameter Store
    steps:
      - type: shell
        command: |-
          echo "Please enter the Docker Config JSON file path"
          echo "See https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry for information on how to create the file"
          read -p "Docker Config JSON file path: " -r DOCKER_CONFIG_JSON_FILE_PATH
          if [ -z "DOCKER_CONFIG_JSON_FILE_PATH" ]
          then
              echo 'Inputs cannot be blank please try again!'
              exit 0
          fi

          DOCKER_CONFIG_JSON=$(<$DOCKER_CONFIG_JSON_FILE_PATH);
          ENCODED_DOCKER_CONFIG_JSON=$(echo "$DOCKER_CONFIG_JSON" | base64 -w 0 );

          echo $DOCKER_CONFIG_JSON
          echo $ENCODED_DOCKER_CONFIG_JSON

          AWS_PROFILE=cch-core-gbl-auto-admin

          set -e

          chamber write github_runners/docker config-json -- "$ENCODED_DOCKER_CONFIG_JSON"

          echo 'Saved Docker Config JSON to the AWS SSM Parameter Store'


# ssm_github_webhook_secret_token_path: "/github_runners/github_webhook_secret_token"
# The webhook based autoscaler is much more efficient than the polling based autoscaler
webhook:
Expand Down Expand Up @@ -410,6 +425,7 @@ Consult [actions-runner-controller](https://github.com/actions-runner-controller
| Name | Type |
|------|------|
| [aws_eks_cluster_auth.eks](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/eks_cluster_auth) | data source |
| [aws_ssm_parameter.docker_config_json](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ssm_parameter) | data source |
| [aws_ssm_parameter.github_token](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ssm_parameter) | data source |
| [aws_ssm_parameter.github_webhook_secret_token](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ssm_parameter) | data source |

Expand All @@ -430,6 +446,7 @@ Consult [actions-runner-controller](https://github.com/actions-runner-controller
| <a name="input_create_namespace"></a> [create\_namespace](#input\_create\_namespace) | Create the namespace if it does not yet exist. Defaults to `false`. | `bool` | `null` | no |
| <a name="input_delimiter"></a> [delimiter](#input\_delimiter) | Delimiter to be used between ID elements.<br>Defaults to `-` (hyphen). Set to `""` to use no delimiter at all. | `string` | `null` | no |
| <a name="input_descriptor_formats"></a> [descriptor\_formats](#input\_descriptor\_formats) | Describe additional descriptors to be output in the `descriptors` output map.<br>Map of maps. Keys are names of descriptors. Values are maps of the form<br>`{<br> format = string<br> labels = list(string)<br>}`<br>(Type is `any` so the map values can later be enhanced to provide additional options.)<br>`format` is a Terraform format string to be passed to the `format()` function.<br>`labels` is a list of labels, in order, to pass to `format()` function.<br>Label values will be normalized before being passed to `format()` so they will be<br>identical to how they appear in `id`.<br>Default is `{}` (`descriptors` output will be empty). | `any` | `{}` | no |
| <a name="input_docker_config_json_enabled"></a> [docker\_config\_json\_enabled](#input\_docker\_config\_json\_enabled) | Whether the Docker config JSON is enabled | `bool` | `false` | no |
| <a name="input_eks_component_name"></a> [eks\_component\_name](#input\_eks\_component\_name) | The name of the eks component | `string` | `"eks/cluster"` | no |
| <a name="input_enabled"></a> [enabled](#input\_enabled) | Set to false to prevent the module from creating any resources | `bool` | `null` | no |
| <a name="input_environment"></a> [environment](#input\_environment) | ID element. Usually used for region e.g. 'uw2', 'us-west-2', OR role 'prod', 'staging', 'dev', 'UAT' | `string` | `null` | no |
Expand Down Expand Up @@ -461,6 +478,7 @@ Consult [actions-runner-controller](https://github.com/actions-runner-controller
| <a name="input_resources"></a> [resources](#input\_resources) | The cpu and memory of the deployment's limits and requests. | <pre>object({<br> limits = object({<br> cpu = string<br> memory = string<br> })<br> requests = object({<br> cpu = string<br> memory = string<br> })<br> })</pre> | n/a | yes |
| <a name="input_runners"></a> [runners](#input\_runners) | Map of Action Runner configurations, with the key being the name of the runner. Please note that the name must be in<br>kebab-case.<br><br>For example:<pre>hcl<br>organization_runner = {<br> type = "organization" # can be either 'organization' or 'repository'<br> dind_enabled: false # A Docker sidecar container will be deployed<br> image: summerwind/actions-runner # If dind_enabled=true, set this to 'summerwind/actions-runner-dind'<br> scope = "ACME" # org name for Organization runners, repo name for Repository runners<br> group = "core-automation" # Optional. Assigns the runners to a runner group, for access control.<br> scale_down_delay_seconds = 300<br> min_replicas = 1<br> max_replicas = 5<br> busy_metrics = {<br> scale_up_threshold = 0.75<br> scale_down_threshold = 0.25<br> scale_up_factor = 2<br> scale_down_factor = 0.5<br> }<br> labels = [<br> "Ubuntu",<br> "core-automation",<br> ]<br>}</pre> | <pre>map(object({<br> type = string<br> scope = string<br> group = optional(string, null)<br> image = optional(string, "")<br> dind_enabled = bool<br> node_selector = optional(map(string), {})<br> pod_annotations = optional(map(string), {})<br> tolerations = optional(list(object({<br> key = string<br> operator = string<br> value = optional(string, null)<br> effect = string<br> })), [])<br> scale_down_delay_seconds = number<br> min_replicas = number<br> max_replicas = number<br> busy_metrics = optional(object({<br> scale_up_threshold = string<br> scale_down_threshold = string<br> scale_up_adjustment = optional(string)<br> scale_down_adjustment = optional(string)<br> scale_up_factor = optional(string)<br> scale_down_factor = optional(string)<br> }))<br> webhook_driven_scaling_enabled = bool<br> webhook_startup_timeout = optional(string, null)<br> pull_driven_scaling_enabled = bool<br> labels = list(string)<br> storage = optional(string, null)<br> pvc_enabled = optional(bool, false)<br> resources = object({<br> limits = object({<br> cpu = string<br> memory = string<br> ephemeral_storage = optional(string, null)<br> })<br> requests = object({<br> cpu = string<br> memory = string<br> })<br> })<br> }))</pre> | n/a | yes |
| <a name="input_s3_bucket_arns"></a> [s3\_bucket\_arns](#input\_s3\_bucket\_arns) | List of ARNs of S3 Buckets to which the runners will have read-write access to. | `list(string)` | `[]` | no |
| <a name="input_ssm_docker_config_json_path"></a> [ssm\_docker\_config\_json\_path](#input\_ssm\_docker\_config\_json\_path) | SSM path to the Docker config JSON | `string` | `null` | no |
| <a name="input_ssm_github_secret_path"></a> [ssm\_github\_secret\_path](#input\_ssm\_github\_secret\_path) | The path in SSM to the GitHub app private key file contents or GitHub PAT token. | `string` | `""` | no |
| <a name="input_ssm_github_webhook_secret_token_path"></a> [ssm\_github\_webhook\_secret\_token\_path](#input\_ssm\_github\_webhook\_secret\_token\_path) | The path in SSM to the GitHub Webhook Secret token. | `string` | `""` | no |
| <a name="input_stage"></a> [stage](#input\_stage) | ID element. Usually used to indicate role, e.g. 'prod', 'staging', 'source', 'build', 'test', 'deploy', 'release' | `string` | `null` | no |
Expand Down
94 changes: 71 additions & 23 deletions modules/eks/actions-runner-controller/charts/actions-runner/templates/runnerdeployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,16 @@ spec:
# save space.
storage: 100Gi
{{- end }}
{{- if .Values.docker_config_json_enabled }}
---
apiVersion: v1
kind: Secret
metadata:
name: regcred
type: kubernetes.io/dockerconfigjson
data:
.dockerconfigjson: {{ .Values.docker_config_json }}
{{- end }}
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
Expand All @@ -34,16 +44,32 @@ spec:
{{- toYaml . | nindent 8 }}
{{- end }}
spec:
{{- if .Values.docker_config_json_enabled }}
# secrets volumeMount are always mounted readOnly so config.json has to be copied to the correct directory
# https://github.com/kubernetes/kubernetes/issues/62099
# https://github.com/actions/actions-runner-controller/issues/2123#issuecomment-1527077517

initContainers:
- name: docker-config-writer
image: {{ .Values.image | quote }}
command: [ "sh", "-c", "cat /home/.docker/config.json > /home/runner/.docker/config.json" ]
volumeMounts:
- mountPath: /home/.docker/
name: docker-secret
- mountPath: /home/runner/.docker
name: docker-config-volume
{{- end }}

# As of 2023-03-31
# Recommended by https://github.com/actions/actions-runner-controller/blob/master/docs/automatically-scaling-runners.md
terminationGracePeriodSeconds: 100
env:
# RUNNER_GRACEFUL_STOP_TIMEOUT is the time the runner will give itself to try to finish
# a job before it gracefully cancels itself in response to a pod termination signal.
# It should be less than the terminationGracePeriodSeconds above so that it has time
# to report its status and deregister itself from the runner pool.
- name: RUNNER_GRACEFUL_STOP_TIMEOUT
value: "90"
# RUNNER_GRACEFUL_STOP_TIMEOUT is the time the runner will give itself to try to finish
# a job before it gracefully cancels itself in response to a pod termination signal.
# It should be less than the terminationGracePeriodSeconds above so that it has time
# to report its status and deregister itself from the runner pool.
- name: RUNNER_GRACEFUL_STOP_TIMEOUT
value: "90"

# You could reserve nodes for runners by labeling and tainting nodes with
# node-role.kubernetes.io/actions-runner
Expand Down Expand Up @@ -89,6 +115,10 @@ spec:
dockerdWithinRunnerContainer: {{ .Values.dind_enabled }}
image: {{ .Values.image | quote }}
imagePullPolicy: IfNotPresent
{{- if .Values.docker_config_json_enabled }}
imagePullSecrets:
- name: regcred
{{- end }}
serviceAccountName: {{ .Values.service_account_name }}
resources:
limits:
Expand All @@ -105,29 +135,47 @@ spec:
{{- end }}
{{- if and .Values.dind_enabled .Values.storage }}
dockerVolumeMounts:
- mountPath: /var/lib/docker
name: docker-volume
- mountPath: /var/lib/docker
name: docker-volume
{{- end }}
{{- if .Values.pvc_enabled }}
{{- if or (.Values.pvc_enabled) (.Values.docker_config_json_enabled) }}
volumeMounts:
- mountPath: /home/runner/work/shared
name: shared-volume
{{- if .Values.pvc_enabled }}
- mountPath: /home/runner/work/shared
name: shared-volume
{{- end }}
{{- if .Values.docker_config_json_enabled }}
- mountPath: /home/.docker/
name: docker-secret
- mountPath: /home/runner/.docker
name: docker-config-volume
{{- end }}
{{- end }}
{{- if or (and .Values.dind_enabled .Values.storage) (.Values.pvc_enabled) }}
{{- if or (and .Values.dind_enabled .Values.storage) (.Values.pvc_enabled) (.Values.docker_config_json_enabled) }}
volumes:
{{- if and .Values.dind_enabled .Values.storage }}
- name: docker-volume
ephemeral:
volumeClaimTemplate:
spec:
accessModes: [ "ReadWriteOnce" ] # Only 1 pod can connect at a time
resources:
requests:
storage: {{ .Values.storage }}
- name: docker-volume
ephemeral:
volumeClaimTemplate:
spec:
accessModes: [ "ReadWriteOnce" ] # Only 1 pod can connect at a time
resources:
requests:
storage: {{ .Values.storage }}
{{- end }}
{{- if .Values.pvc_enabled }}
- name: shared-volume
persistentVolumeClaim:
claimName: {{ .Values.release_name }}
- name: shared-volume
persistentVolumeClaim:
claimName: {{ .Values.release_name }}
{{- end }}
{{- if .Values.docker_config_json_enabled }}
- name: docker-secret
secret:
secretName: regcred
items:
- key: .dockerconfigjson
path: config.json
- name: docker-config-volume
emptyDir:
{{- end }}
{{- end }}
16 changes: 13 additions & 3 deletions modules/eks/actions-runner-controller/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
locals {
enabled = module.this.enabled

webhook_enabled = local.enabled ? try(var.webhook.enabled, false) : false
webhook_host = local.webhook_enabled ? format(var.webhook.hostname_template, var.tenant, var.stage, var.environment) : "example.com"
runner_groups_enabled = length(compact(values(var.runners)[*].group)) > 0
webhook_enabled = local.enabled ? try(var.webhook.enabled, false) : false
webhook_host = local.webhook_enabled ? format(var.webhook.hostname_template, var.tenant, var.stage, var.environment) : "example.com"
runner_groups_enabled = length(compact(values(var.runners)[*].group)) > 0
docker_config_json_enabled = local.enabled && var.docker_config_json_enabled
docker_config_json = one(data.aws_ssm_parameter.docker_config_json[*].value)

github_app_enabled = length(var.github_app_id) > 0 && length(var.github_app_installation_id) > 0
create_secret = local.enabled && length(var.existing_kubernetes_secret_name) == 0
Expand Down Expand Up @@ -100,6 +102,12 @@ data "aws_ssm_parameter" "github_webhook_secret_token" {
with_decryption = true
}

data "aws_ssm_parameter" "docker_config_json" {
count = local.docker_config_json_enabled ? 1 : 0
name = var.ssm_docker_config_json_path
with_decryption = true
}

module "actions_runner_controller" {
source = "cloudposse/helm-release/aws"
version = "0.7.0"
Expand Down Expand Up @@ -225,6 +233,8 @@ module "actions_runner" {
pvc_enabled = each.value.pvc_enabled
node_selector = each.value.node_selector
tolerations = each.value.tolerations
docker_config_json_enabled = local.docker_config_json_enabled
docker_config_json = local.docker_config_json
}),
each.value.group == null ? "" : yamlencode({ group = each.value.group }),
local.busy_metrics_filtered[each.key] == null ? "" : yamlencode(local.busy_metrics_filtered[each.key]),
Expand Down
12 changes: 12 additions & 0 deletions modules/eks/actions-runner-controller/variables.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,18 @@ variable "s3_bucket_arns" {
default = []
}

variable "docker_config_json_enabled" {
type = bool
description = "Whether the Docker config JSON is enabled"
default = false
}

variable "ssm_docker_config_json_path" {
type = string
description = "SSM path to the Docker config JSON"
default = null
}

variable "runners" {
description = <<-EOT
Map of Action Runner configurations, with the key being the name of the runner. Please note that the name must be in
Expand Down