Skip to content

v1.6.0

Latest

Choose a tag to compare

@github-actions github-actions released this 26 Aug 23:34
· 16 commits to main since this release
Immutable release. Only release title and notes can be modified.
7794b6e

Zero-idle self-hosted GitHub Actions runners for multi-tenant Kubernetes.

Warning

GitHub's minimum runner version is being enforced right now, not at some point in the future. GHEC brownouts began 2026-08-24 and run to 2026-09-18, with full enforcement on 2026-09-25; the registration floor is runner 2.329.0 (changelog). A brownout is a four-hour window, so a tenant whose workerImage is behind fails intermittently first, which reads as a flake rather than a version problem. This release is what tells you which image is behind: RunnerVersionTooOld reported it per runner set in v1.5.0, and v1.6.0 adds the version a worker pod actually ran for the images the tag could not answer for.

Note

spec.scaleUp changes what it does when the bucket is empty, and it is off by default, so a tenant that never configured a ramp sees nothing. A throttled job used to be claimed and then held on a GitHub job lock until a token freed; it is now not claimed at all, and stays queued at GitHub for redelivery. The ramp is unchanged, but time-to-pickup for a throttled job becomes a redelivery interval. Read Upgrading below.

Highlights

The runner version a worker pod actually ran is now reported, read from the runner's own dependency manifest by the injected wrapper and handed back on the pod's termination message. It answers what a digest-only or custom workerImage tag ships, which is the case RunnerVersionTooOld reports as WorkerImageVersionUnknown because the AGC cannot read a version off the reference at all. It is a self-report, not an attestation: the job's own steps run inside that container, so it is a diagnostic for an operator debugging their own image and it deliberately does not move the condition.

"Why is my runner set not being offered jobs?" is answerable from kubectl describe. RunnerSet status now carries the capacity the set advertised to GitHub and the rung of the admission ladder that took each withheld slot, so a tenant can tell their own maxWorkers from the namespace ResourceQuota from the cluster being unable to place their workers. The same accounting was previously reachable only through Prometheus, which a tenant without metrics access cannot read.

spec.scaleUp no longer spends a GitHub job lock to enforce a ramp. The token bucket became a rung of the pre-acquisition admission ladder. A lock that lapses cancels its job rather than redelivering it, so a large burst at a low maxPerSecond could destroy the jobs the ramp existed to smooth.

Workers that bind a job and never start are reported, and can gate intake. An image that will not pull used to consume capacity silently; the set now reports PodsNotStarting and, where the gate is enabled, declines further intake rather than accumulating stuck workers.

The egress proxy can write a per-connection audit record, one structured line per accepted CONNECT at tunnel close, carrying the destination host and port, bytes each way, and duration. It is off by default and opt-in per pool, because the record is data about a tenant's egress and retaining it should be a decision rather than a side effect. It changes nothing about what the proxy forwards: enforcement stays destinationFQDNs/destinationCIDRs plus the pod-egress NetworkPolicy.

This is the release that finishes the ARC migration checklist, and the last item on it closed as a documented decline rather than a feature. ARC's containerMode: kubernetes runs container: and services: steps as separate pods. Reproducing it needs a pod-create grant that RBAC cannot scope below a namespace, which would let one job take another job's credentials, so the mechanism is declined permanently and Docker-in-Docker under Kata is the supported answer. Kata needs nested virtualization, so if you run GKE Autopilot, AMD or Arm nodes, or an AWS fleet outside the selected Intel families, this is the one place a migration from ARC costs you something: privileged-dind where ARC needed no pod privilege. The reasoning, and what ARC pays instead, are in D.15.

Upgrading

No breaking change, and nothing is removed. The upgrade is the ordinary one:

  1. Apply the chart's CRDs, as every upgrade does. The command is unconditional and idempotent, and the chart preflights it and fails with the same command if you skip it.
  2. helm upgrade the release. The RunnerSet and EgressProxy schema changes in this release are additive fields on CRDs the chart ships as templates, so they carry with the release.

The full command and its options are in Upgrading.

Two label values appear on existing metric series, so widen any selector that enumerates reason exhaustively: actions_gateway_jobs_admission_rejected_total{reason="scaleup"} and actions_gateway_scaleset_capacity_withheld{reason="scaleup"}.

actions_gateway_worker_scaleup_throttled_total keeps its name and changes what it counts, from delayed pod creations to withheld intake. On the scale-set tier it stays a pod-creation measure and falls close to zero, because the advertisement withholds the work before GitHub assigns it; a non-zero rate there now means the advertisement was stale rather than that the ramp is engaging.

If you raised maxPerSecond because claimed jobs were being held, the original value is now safe to restore.

Deprecations

Nothing is removed in this release, and both notices are unchanged from v1.5.0.

v1alpha1 and v2alpha1 are both deprecated, and the apiserver warns on every apply of either. Both are removed at v2.0.0, together with the Classic acquisition protocol. Tenants on v1alpha1 migrate with gag-migrate (Migrating a tenant to v2); moving from v2alpha1 to v2beta1 is a re-apply rather than a migration (Before you upgrade past v2.0.0).

Everything since v1.5.0

166 commits. These 12 change the shipped binaries and chart, as scripts/release/semver-floor.sh reads them off the released surface rather than off commit subjects; build, CI, test, tooling, and docs work is left out and summarised under Project and tooling below. Eleven are listed here. The twelfth (#1706) adds scale-set listing to a maintainer diagnostic that no release publishes, so it moves a binary without giving an operator anything to run.

Features (9)
  • Report the runner version a worker pod actually ran (#1721)
  • Publish advertised capacity and withheld reasons on RunnerSet status (#1711)
  • Fold the scale-up token bucket into the admission ladder (#1702)
  • Gate intake on workers that bind and never start (#1626)
  • Report workers that bind and never start, gate or no gate (#1708)
  • Write a per-connection egress audit record on the proxy, off by default (#1705)
  • Publish GitHub's queued-job count as a scale-set gauge (#1699)
  • Say plainly that a scale set's labels are fixed at creation, because the Actions Service accepts a labels PATCH and discards it (#1700)
  • Validate ReadWriteMany worker storage and write the reference architecture (#1713)
Fixes (2)
  • Send the agent's registered runner name on the broker session (#1710)
  • Clear a session-sourced RunnerVersionTooOld once the runner version is accepted (#1703)

API and metric surface

Four status fields, one spec field, four condition reasons, one Event reason and two metrics are published for the first time. No field, enum value, default, metric, condition reason, or Event reason was removed or renamed, no existing enum constraint or default changed, and the chart's values keys are unchanged. Field descriptions are in the generated API reference, which covers v2beta1; a deprecated-version field still needs kubectl explain.

New spec fields (1)

EgressProxy.spec.auditLogging takes Off (the default) or Connections. Selects the per-connection egress audit record the pool writes. Changing it rolls the pool, the same as logLevel.

New status fields (4)

RunnerSet.status.advertisedCapacity is the X-ScaleSetMaxCapacity most recently advertised. Absent on a classic-tier set, which decides per delivered job and never states a capacity, and before the first poll.

RunnerSet.status.withheldCapacity[] carries one entry per admission-ladder rung, keyed by reason, with the slots that rung removed. Every rung evaluated appears, including those withholding nothing, so an absent reason was not evaluated rather than not binding.

RunnerSet.status.observedRunnerVersion is the runner version a worker pod reported running. Sticky and newest-wins; empty until a worker has terminated and reported.

New condition reasons (4) and Event reasons (1)

Condition reasons: PodsNotStarting, WorkersNotStarting, WorkersStarting, VersionAccepted.

Event reason: WorkersNotStarting.

VersionAccepted is the healthy baseline the classic listener publishes on session start, and it exists to clear a RunnerVersionTooOld a previous instance left behind. The version is the AGC's own compile-time pin, so the fix is a gateway upgrade and the condition would otherwise survive it in status.

New metrics (2)
  • actions_gateway_scaleset_jobs_available: GitHub's queued-job count for the scale set.
  • actions_gateway_runnerset_workers_not_starting: workers that bound a job and never started.

Validation

v1.6.0-rc.2 passed its dogfood validation, exit 0, against a tree whose released surface is byte-identical to this tag's. check-artifact-unchanged.sh v1.6.0-rc.2 reports nothing on the released surface changed between the candidate and this commit, so the verdict below covers what ships here.

It is the second candidate. rc.1 was superseded before it was ever validated: three dependency bumps merged on top of it and moved 11 files of the released surface, including k8s.io/* from 0.36.3 to 0.36.4 inside the shipped AGC binary. The rule is byte-identical artifacts rather than "docs only", and that is what caught it.

The legs, and what the artifact checks discriminate

Run against the signed release artifacts rather than a build (run):

Leg Result
deploy release candidate deployed and CI routed to it
e2e matrix on project-provisioned runners 75 passed, 0 failed, 13 skipped
sizing, NodeShare Active on ci-e2e, deriving the envelope's per-worker share
sizing, Throughput Active on ci, 211 samples
signed v2 CRD manifest blob signature Verified OK, all five CRDs applied and registered

The suite covered cross-tenant network isolation, worker preemption and recovery, a drained worker that must trigger no rerun, a ceiling-held job cancelled rather than redelivered, the v1 to v2 migration dry run, and Vault workload identity with no PEM Secret.

The provenance is the check that discriminates: signatures prove who built an image, and only the digest proves what was built. Its signer URI ends publish.yml@refs/tags/v1.6.0-rc.2 and its sourceRepositoryDigest equals the tagged commit. Re-run against a deliberately wrong signer workflow, it exits 1, so the pass discriminates rather than merely exiting 0.

Cluster preflight passed with two warnings that are properties of the dogfood cluster rather than the candidate: no cert-manager and no metrics-server.

Project and tooling

This release's parity work concluded in a decline rather than a build, and the reasoning is published because the alternative is a gap nobody can audit.

ARC's containerMode: kubernetes runs container: and services: steps as separate pods on a provisioned volume. Reproducing it needs a pod-create grant that RBAC cannot scope below a namespace, and a per-job Secret holding a job's jitconfig sits in that namespace, so the grant would let one job take another job's credentials: an escalation between jobs of one tenant. GAG refuses that token at RunnerTemplate admission and again when the provisioner overwrites the tenant template, so the mechanism is declined permanently rather than deferred.

The decline is honest exactly where Kata is available, and Kata needs nested virtualization, which GKE Autopilot does not allow at all and which is absent on E2, C2D and N2D. A team on Autopilot, on AMD or Arm nodes, or on most AWS fleets therefore has no Kata, and the decline hands them privileged-dind where their ARC setup needed no pod privilege. That population is named as the decline's cost on every comparison surface rather than left for them to discover.

The storage half closed the other way: a ReadWriteMany volume was validated across two nodes against a live storage class, in the pod the provisioner really builds, and the result is a reference architecture rather than an assertion.

Security

No security advisory accompanies this release, no reported vulnerability is patched in this project's own code, and no dependency or toolchain moved for a CVE. The Go toolchain stays at 1.26.6.

It does carry security-relevant work. The egress audit record makes a tenant's outbound destinations observable for the first time, and it ships off by default and opt-in per pool, because the safe direction for a record of who reached what is not to retain it until someone decides to. On a pool shared via spec.sharing the record attributes to the pool rather than the consumer, which is stated on the field rather than left to be inferred.

The containerMode: kubernetes decline above is a security decision recorded as one: the mechanism is refused because a namespace-wide pod-create grant breaks a boundary between jobs, and that reasoning is published so it can be argued with.

Every pull request runs govulncheck, a Trivy image scan, and a Polaris posture check; the same scans run on a schedule against the default branch.

Container images

Pin these immutable multi-arch index digests rather than the floating v1.6.0 tag. Each index serves both linux/amd64 and linux/arm64, so one pinned digest schedules on either.

  • gmc: ghcr.io/actions-gateway/gmc@sha256:f4c89859a18aa5f4fa92e68b3ff6e9a392a05527d23d8fac8170c86195f9fd8a
  • agc: ghcr.io/actions-gateway/agc@sha256:2a0da5083b3405c6c5324cf25febbae1d42204586bd1e7eb3424cf5cb3341019
  • proxy: ghcr.io/actions-gateway/proxy@sha256:389f21f6ebd061693d810f84efac8bb1d5a193d306c1b12987b0e2f536c44518
  • worker: ghcr.io/actions-gateway/worker@sha256:92971fc95fc242f6853b501acd913f6279f782f8630aef4ef070196d3319ea27
  • wrapper: ghcr.io/actions-gateway/wrapper@sha256:f38609cdd039e2854f2e8d58c93ba05c7168ada06586d216bd846c85bef5c4ae

Verifying this release

We sign every image and both charts.

make verify-release VERSION=v1.6.0

The signed v2 CRD manifest and SHA256SUMS ship as release assets with detached cosign bundles, and the publish workflow attests build provenance.