Skip to content

v1.5.0

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 15 Aug 19:29
· 6 commits to main since this release
Immutable release. Only release title and notes can be modified.
v1.5.0
251d4db
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Zero-idle self-hosted GitHub Actions runners for multi-tenant Kubernetes.

Warning

Two upgrade steps, and one of them can reject a configuration this release's predecessor accepted. Apply the chart's CRDs before upgrading.
Skip it and the GMC now refuses to start, rather than provisioning tenants against a runner-group boundary the cluster pruned without saying so.
Then check that no two ScaleSet runner sets under one GitHub org share a first runnerLabel: that pair is refused on write, and a pair carried in from an older release is reported on the gateway at the first reconcile.
Read Upgrading below.

Note

GitHub starts enforcing a minimum runner version on GHEC during this release's lifetime. Brownouts run 2026-08-24 to 2026-09-18, with full enforcement on 2026-09-25, and the registration floor is runner 2.329.0.
A tenant whose workerImage is behind will start failing at GitHub, intermittently at first.
This release is what tells you: the new RunnerVersionTooOld condition reports it per runner set, on both acquisition tiers.

Highlights

A GitHub runner group boundary per tenant. The runner group is the GitHub-side authorization point for which repositories may target which runners, and every scale set this project created registered into the installation's default group.
A repository outside a tenant could previously name that scale set in runs-on and route work into the tenant's namespace, quota, and egress IP.
spec.runnerGroup on a RunnerSet, or spec.defaultRunnerGroup once on the gateway, now binds it.
An unresolvable name fails the set closed at Ready=False/RunnerGroupNotFound rather than falling back to the wider group, and an adopted scale set is moved into its declared group instead of quietly keeping the old one (#1391).

Multi-label runner sets, so a runs-on array migrates unedited. A ScaleSet set previously registered only its first label at GitHub while the CRD accepted a list, so a workflow asking for any other label in that list waited forever.
Every label is registered now.
This is the one gap that previously broke the zero-edit migration this project claims for teams coming from Actions Runner Controller (#1393).
It also closes the last capability v1alpha1 allowed that v2beta1 refused.
Every other v1-to-v2 gap had closed by v1.4.0, which is why the Deprecations note below says a v2alpha1 Classic set has no reason left to stay.

Job duration and pod-creation latency now exist on the tier every new tenant runs. v1.4.0's notes carried a warning that actions_gateway_job_duration_seconds and actions_gateway_pod_creation_latency_seconds were classic-tier only, leaving two SLOs, a severity-critical alert, four recording rules and both shipped dashboards blank on the default tier.
Both observations moved onto the shared pod informer, which cannot tell the tiers apart.
job_duration_seconds is now worker pod lifetime on both tiers, which is also a correction: a pod is billed from creation, and the classic tier's old span started at job acquisition, charging the staging, quota-retry and spec.scaleUp throttle window during which no pod existed, so cost attribution was reading high (#1390).
That span changed without a rename, so classic-tier percentiles and anything costed off them step down at this upgrade; the upgrade note has the detail, and both shipped dashboards now mark it.

A worker lost while the controller was down is re-run automatically. On the scale-set tier the worker pod was the only record of which run to re-run, so a controller that was down for the pod's teardown issued nothing.
The run identity is now persisted behind every worker, and any run whose pod is gone at startup is re-run.
This was found by asking whether a parity claim was actually true, rather than by a gate or a bug report (#1487).

The default acquisition tier no longer lacks anything the classic one has. ScaleSet is the default tier and the only one v2beta1 exposes, so a capability that reached only the classic tier was one most tenants could not use.
This release closed the last of them: job duration and pod-creation latency (#1390), and restart-safe disruption recovery (#1487).
No capability on the inventory carries a classic-only badge now.

The two halves are not checked the same way.
The metric surface below is derived from the source and gated; the capability half is a walk someone does by hand.
Restart-safe recovery is why that distinction stays in these notes: it was classic-only, carried no badge, and left the inventory reading clean for as long as it was open (#1487).

Acquisition-tier parity on the metric surface, with a gate that keeps it. All 54 actions_gateway_* series the AGC defines are now reconciled against the tier that emits them: 27 reach both tiers, 16 are classic-only, 10 are scale-set-only, one is tier-neutral.
make metric-tiers-check holds that ledger to the source in both directions, so a series added on one tier fails until someone answers the tier question (#1494).

Upgrading

Two steps.
Full procedure in the upgrade guide, which documents rollback per component.

1.
Apply the chart's CRDs. Part of every upgrade of this chart, not a one-off: Helm installs the chart-root crds/ directory on a fresh install only.

helm show crds oci://ghcr.io/actions-gateway/charts/actions-gateway \
  --version 1.5.0 | kubectl apply -f -

Charts carry no leading v.
Images are tagged v1.5.0, charts 1.5.0.

Skipping this step used to be silent, and is not any more. A structural schema prunes an undeclared field on write with no error, so a cluster left on the previous release's CRD accepts a manifest setting spec.runnerGroup and stores nothing: a tenant boundary going away rather than a feature going missing.
The GMC now probes the installed schema at startup and exits naming the field, the version, and the command to fix it (#1525).
Every field added from here has the same exposure.

2.
Check for scale-set name collisions across namespaces. A ScaleSet set's first runnerLabel is its scale set's name at GitHub, so two sets sharing it drive one scale set from two controllers and each tenant's runners can acquire the other's jobs.
The guard existed but was namespace-scoped, which is the one boundary that does not match how GitHub adopts a scale set by name.

Admission runs on create and update, so nothing re-validates a pair that already exists: left to itself, the rejection would arrive at someone's next unrelated apply, long after the upgrade and on a tenant who changed nothing.
The GMC reports it instead. Each affected gateway reconciles into an advisory ScaleSetNameCollision=True with a Warning Event, and actions_gateway_scale_set_name_collision reads 1, so a pair carried in from an older release surfaces on the first reconcile after the upgrade.
It does not gate Ready and does not stop the AGC: GAG cannot pick which tenant loses the name, and refusing to provision would take down the tenant that was there first as well (#1526).

One command finds it before you upgrade:

kubectl get runnersets -A -o jsonpath='{range .items[?(@.spec.acquisitionProtocol=="ScaleSet")]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.spec.runnerLabels[0]}{"\n"}{end}'

Any first label appearing twice under one GitHub scope needs one of the two sets given a distinct first label.
Labels after the first may be shared freely (#1484).

Five changes that need no action, but will be visible.

  • A runner set reports when its worker image is below GitHub's minimum — RunnerVersionTooOld with WorkerImageBelowMinimum, WorkerImageCurrent, or WorkerImageVersionUnknown.
    Read at each reconcile off the effective worker image, so it needs no session and reaches both tiers (#1392).
  • A controller restart can now spend re-run budget — each recovered run consumes one slot of the shared maxEvictionRetries budget under a new cause="vanished", and dashboards enumerating that label should add it (#1487).
  • A lost claim no longer costs a disrupted worker its re-run, and reports itself when the evidence is genuinely gone via EvictionRecoveryEvidenceLost (#1441).
  • An alert fires when a capacity gate is throttling job intake — the gate is opt-in and off by default, so this is silent unless you use it (#1438).
  • Cancelling a run and then deleting its stuck worker no longer re-queues the job you just stopped. Deleting the pod is what the cancel runbook tells you to do, and it left the deletion mark that recovery treats as a disruption.
    GitHub accepts a re-run for a cancelled conclusion where it refuses one for a success, so the remedy re-ran the job on both tiers.
    Recovery now reads the run's conclusion before each attempt and stands down on cancelled, recording eviction_rerun_withheld_total{reason="run_cancelled"} and an EvictionRerunWithheld Event.
    An unreadable conclusion is retried rather than assumed, and a window that closes on one asks for a manual re-run instead of firing blind (#1515).

Deprecations

Nothing is removed in this release, and both notices are unchanged from v1.4.0.

v1alpha1 and v2alpha1 are both deprecated, and the apiserver warns on every apply of either. Both are removed at v2.0.0, together with the Classic acquisition protocol.
Tenants on v1alpha1 migrate with gag-migrate (Migrating a tenant to v2); moving from v2alpha1 to v2beta1 is a re-apply rather than a migration (Before you upgrade past v2.0.0).

One thing worth knowing if you are still on v2alpha1 Classic: that was the documented migration path for a runner set needing more than one label, and this release removes the reason to take it.

Everything since v1.4.0

160 commits.
These 13 change the shipped binaries and chart, as scripts/release/semver-floor.sh reads them off the released surface rather than off commit subjects; build, CI, test, tooling, and docs work is left out and summarised under Project and tooling below.

Features (6)
  • Pin each scale set to its declared GitHub runner group (#1391)
  • Register every runnerLabel on a runner set's scale set (#1393)
  • Warn when a worker image ships a runner below GitHub's minimum (#1392)
  • Report an existing scale-set name collision at reconcile (#1526)
  • Refuse to start when an installed CRD schema drops a boundary field (#1525)
  • Take the acquisition-tier ledger to label-value granularity, correcting seven metric Help strings an operator reads off /metrics (#1531)
Fixes (7)
  • Emit job duration and pod-creation latency on both acquisition tiers (#1390)
  • Re-run a scale-set worker that vanished while the controller was down (#1487)
  • Scope scale-set label uniqueness to the GitHub org, not the namespace (#1484)
  • Stand down a disruption re-run when the run was cancelled (#1515)
  • A lost claim must not cost a disrupted worker its re-run (#1441)
  • Floor the empty-poll interval so a non-blocking 202 cannot spin a poll loop (#1409)
  • Bump the Go toolchain to 1.26.6 to clear stdlib CVEs (#1488)

API and metric surface

Two wire fields, two condition types, eight condition reasons, three metrics and five Event reasons are published for the first time.
No field, enum value, default, metric, or Event reason was removed or renamed, no enum constraint or default changed, and the chart's values keys are unchanged.
Field descriptions are in the generated API reference, which covers v2beta1; a deprecated-version field still needs kubectl explain.

New spec fields (2)

RunnerSet

  • spec.runnerGroup: the GitHub runner group this set's scale set registers into.
    Unset inherits the gateway's default (#1391)

ActionsGateway

Both are served in v2alpha1 and v2beta1.
An undeclared group leaves an existing scale set where it is, so widening is always explicit.

New condition types (2) and reasons (8)

Both types are abnormal-is-True and neither gates Ready: a configuration mismatch is not an outage.

  • RunnerLabelsIncomplete, on a RunnerSet: a live scale set registered fewer labels than the set declares, which happens when a label is appended to an existing set.
    Reasons LabelsRegistered / LabelsNotRegistered (#1393)
  • ScaleSetNameCollision, on an ActionsGateway: a bound ScaleSet runner set claims a scale-set name another already holds in the same GitHub scope, which admission refuses on write but never re-checks for a pair that predates the guard.
    Reasons ScaleSetNameShared / ScaleSetNamesUnique.
    The message names a conflicting set only when it sits in this gateway's own namespace; a cross-tenant holder goes to the GMC log, the same non-enumeration rule the admission error follows (#1526)
  • RunnerGroupNotFound, on Ready: the declared runner group does not resolve.
    Fails the set closed rather than falling back to the installation default (#1391)
  • WorkerImageBelowMinimum / WorkerImageCurrent / WorkerImageVersionUnknown, on RunnerVersionTooOld: the runner version read off the effective worker image, against GitHub's registration floor (#1392)
New metrics (3) and Event reasons (5)

All metrics prefixed actions_gateway_, listed without it.
The controller now defines 72 series, up from 69.

  • eviction_recovery_evidence_lost_total: a disruption whose recovery evidence was gone before it could be read (#1441)
  • scale_set_name_collision: a gauge reading 1 while a gateway holds ScaleSetNameCollision=True, so the pair is alertable rather than only visible in kubectl describe (#1526)
  • eviction_rerun_withheld_total: a disruption re-run the AGC declined to issue, by reason (#1515)

Event reasons go from 32 to 37:

  • OrphanedWorkerRecovered (Warning) on the RunnerSet, naming the pod and the run (#1487)
  • EvictionRecoveryEvidenceLost (#1441)
  • RunnerLabelsNotRegistered (#1393)
  • ScaleSetNameShared (Warning) on the ActionsGateway, on the transition into a collision including the first observation after an upgrade (#1526)
  • EvictionRerunWithheld (Normal) on the RunnerSet, naming why a re-run was not issued (#1515)

Which tier emits which series is now written down. The acquisition-tier ledger covers all 54 actions_gateway_* series, and make metric-tiers-check holds it to the source (#1494).
Two counters it settled: renew_job_errors_total and renew_job_teardowns_total are classic-only by construction, since a scale-set runner renews and completes its own job.

Validation

v1.5.0-rc.3 passed its dogfood validation, exit 0, against the exact commit this tag is cut from.

It is the third candidate.
rc.1 validated and was then superseded by eight further gating rows; rc.2 was tagged at a stale commit and could not be corrected, because a published immutable Release locks its tag (postmortem).

The five legs, and what the artifact checks discriminate

Run against the signed release artifacts rather than a build (run):

Leg Result
deploy release candidate deployed and CI routed to it
e2e matrix on project-provisioned runners 75 passed, 0 failed, 12 skipped in 7m47s
sizing, NodeShare Active; worker CPU request derived to 1500m where the templates ask 2 and 3
sizing, Throughput Active on 199 samples, actuating — this candidate ran CI on derived sizing
signed v2 CRD manifest blob signature Verified OK, all five CRDs applied and registered

Throughput actuating was not expected: it needs roughly 20 samples per template container and this matrix runs about seven jobs.
The samples were already there because the sampler tracks every worker pod whatever spec.sizing holds.

Both identity checks were re-run against deliberately wrong identities and both failed: a refs/heads/ regexp took cosign to exit 12 naming the real subject, and a wrong signer workflow took gh attestation verify to exit 1.
So the passes discriminate rather than merely exiting 0.

Project and tooling

Two defects in the release tooling itself surfaced while cutting this release, both fixed before the tag.

The dogfood gate's CPU-budget preflight could not read a node pool's autoscale ceiling, because GKE omits minNodeCount when it holds its 0 default and the projected row was split on a tab, which read discards (#1498).
It had merged three days after the previous dogfood run, so it had never executed against a real cluster; both test doubles modelled a minimum the API never sends, and nine assertions downstream stayed green.

The release sentinel then reported a passing verdict belonging to a previous release candidate, because the gate wrote no event until after its preflight and a spent stream still held its terminal event (#1500).
The stream is now emptied before preflight, so no reader can meet it.

A third came out of asking why either had needed interpreting: the gate had no retry anywhere, so one HTTP 401 at 645s of a settle wait killed a candidate run after 43 good polls, and a 22-minute leg reported nothing while it passed.
Both are fixed, along with a runbook line that called a normal sizing result a surprise (#1528).

Elsewhere: both end-to-end lanes moved to merge-queue time rather than per-PR (#1415), the gate list is reconciled against the workflows and found five unwired gates (#1489), and a competitor-side verdict on the comparison page now carries the version and date it was read at or renders as unverified (#1440, #1475).

Security

No security advisory accompanies this release, and no reported vulnerability is patched in this project's own code.

It does carry security-relevant work.
The Go toolchain moved to 1.26.6 to clear standard-library CVEs (#1488).
Two changes harden trust boundaries without fixing a reported vulnerability: the runner group binding closes a path by which a repository outside a tenant could route work into that tenant's namespace, quota, and egress IP (#1391), and scale-set name uniqueness is now enforced across the whole GitHub scope rather than per namespace, closing a cross-tenant job-acquisition path between two tenants under one org (#1484).

Declaring a boundary and keeping it enforced are different problems, and two more changes address the second.
A skipped CRD apply left spec.runnerGroup accepted in a manifest and pruned in the cluster, so the boundary read as configured and was not; the GMC now refuses to start on it (#1525).
And admission can only refuse a colliding scale-set name on write, so a pair predating the guard kept running unreported; the GMC now reports one on every reconcile, with a gauge to alert on (#1526).

Every pull request runs govulncheck, a Trivy image scan, and a Polaris posture check; the same scans run on a schedule against the default branch.

Container images

Pin these immutable multi-arch index digests rather than the floating v1.5.0 tag.
Each index serves both linux/amd64 and linux/arm64, so one pinned digest schedules on either.

  • gmc: ghcr.io/actions-gateway/gmc@sha256:5052bfdecae57a334ea8920efa7b3705d574bece4f9c45ec1b78a279dcdfb822
  • agc: ghcr.io/actions-gateway/agc@sha256:817a586fc6f659ec6af696dc6ed6f52372512c69a4fcc279ae5078cd18c78315
  • proxy: ghcr.io/actions-gateway/proxy@sha256:cbabaedabf50da64dd69ee4bfda7e37d50937d42ffa96dbecef81ce0cccbfd19
  • worker: ghcr.io/actions-gateway/worker@sha256:862522b0ea739347a8c92f244ab7f27e5c233eafc34737a248196e8bb2b8460f
  • wrapper: ghcr.io/actions-gateway/wrapper@sha256:968114d70310570657736db956fc6c422a9cc8960e8ea3dd39726bd25b0555e4

Verifying this release

We sign every image and both charts.

make verify-release VERSION=v1.5.0

The signed v2 CRD manifest and SHA256SUMS ship as release assets with detached cosign bundles, and the publish workflow attests build provenance.

Assets 12
Loading