v1.10.0

Zero-idle self-hosted GitHub Actions runners for multi-tenant Kubernetes.

Warning

Every actions-gateway.com kind now stores v2, so the only release you can roll back to is v1.9.0. Any write after the upgrade, a controller's status update included, stores the object at v2, and a release that does not serve v2 cannot read it. v1.9.0 serves v2 and reads what this release stores. If you are on v1.8.0 or earlier, upgrade to v1.9.0 first if you want a rollback target.

Note

v1.9.0's default worker image runs actions/runner 2.337.0, which falls outside GitHub's 30-day update window around 2026-11-05. After that, GitHub stops queueing jobs to those runners while each runner keeps logging Listening for Jobs, so nothing looks wrong. This release moves the default to 2.338.1. If you pin your own workerImage, read Upgrading below.

Highlights

v2 is the storage version, and gag-migrate storage-version rewrites what is already stored. This is the release that exists so v2.0.0 can be removals only. v2.0.0 stops serving v2beta1, v2alpha1 and v1alpha1, and the apiserver refuses to drop a version a CustomResourceDefinition (CRD) still lists in status.storedVersions. The new subcommand of the signed gag-migrate binary writes every object back unchanged and prunes storedVersions to ["v2"]. Run it any time between this upgrade and v2.0.0. Every version stays served meanwhile, so no client notices.

A job whose worker's node shuts down is now re-run. A Spot or preemptible node that the kubelet shuts down gracefully, or a Node object deleted under a running worker, used to leave the job failed for someone to re-run by hand. The Actions Gateway Controller (AGC) now re-runs it on both acquisition tiers, Classic and scale-set, counted under actions_gateway_eviction_retries_total{cause="node_shutdown"}. So is a job on a dead node an admin taints node.kubernetes.io/out-of-service, Kubernetes' confirmation that the node is not coming back. A node that goes NotReady and keeps its Node object, with no such taint, is not covered: recovery starts only once the taint is applied or the Node object is deleted.

The scale-set re-run budget survives an AGC restart, and one run attempt is re-run once. maxEvictionRetries on a scale-set RunnerSet was counted in AGC memory, so a restart refilled it, and each disrupted worker of one run spent a slot and re-ran the run. The budget now lives in a scaleset-failures-<set> ConfigMap per set, and disruptions of one run attempt share one re-run, so a drain taking three workers of one run re-runs it once. The budget is now per set rather than shared across tiers.

Scale-set recovery follows the job a runner actually took. GitHub gives a scale-set job to whichever of the set's runners asks first, so the worker created for one job can end up running another. Five fixes follow from that one fact:

• An evicted, preempted or drained worker re-runs the run it was serving, not the one it was created for.
• A worker that never started a job no longer cancels or re-runs a run that is running elsewhere.
• An idle worker left behind by a cancelled queued job is reclaimed after five minutes, where it used to hold a node until maxWorkerLifetime (12 hours by default).
• A worker's just-in-time (JIT) runner config Secret, which holds a runner credential, is deleted with the worker pod instead of lingering until the RunnerSet is deleted.
• A replayed job whose old worker is gone gets a fresh JIT-config Secret instead of the previous runner's used one.

The Upgrading guide has a section for each of the first four. None of these five was reproduced on a cluster, and the release gate does not assert them. Each has unit tests that fail with its fix removed.

Egress-audit attribution is judged per RunnerSet. The gateway-level EgressAuditUnattributed condition reads only the gateway's defaultProxyRef, so for a set naming its own proxyRef it could miss a set whose egress records cannot be attributed, or flag one whose records can. Each v2 RunnerSet now carries the condition itself, with the gauge actions_gateway_runnerset_egress_audit_unattributed{namespace,runner_set,reason}, and the security dashboard plots it.

An egress proxy pool can tie its failures to the job that hit them, as an opt-in. With failureAttribution: PerConsumerPort on an EgressProxy and proxyFailureAttribution: PerJob on the consuming ActionsGateway, the pool gives that namespace a CONNECT port of its own. The AGC then reads back each connection the pool refused or could not dial, and attributes it to the worker pod that made it. Each attributed record is counted in actions_gateway_proxy_failure_records_total, and a policy denial is also reported as an EgressDestinationDenied Warning Event on the worker's RunnerSet, naming the blocked host, once per job and destination. No retry decision uses the records. Both fields default to Disabled. Turning them on rolls the pool once and moves each new job's workers onto the namespace's port. Turning either off moves new jobs back at once, while the pool keeps the port open for running jobs for up to the longest maxWorkerLifetime of the namespace's RunnerSets. The per-port isolation is a NetworkPolicy, so it holds only on a Container Network Interface (CNI) plugin that enforces one. The end-to-end suite measures it on Calico: a pod dialling another namespace's port gets no answer (#2131).

The Gateway Manager Controller (GMC) reads Secrets only in tenant namespaces. Its ClusterRole no longer grants get, list or watch on Secrets cluster-wide. It binds a gmc-tenant-secret-reader ClusterRole in each namespace labelled as a tenant instead, so a compromised GMC can no longer read Secrets in kube-system or any other namespace outside the tenants. Installs whose admission guard runs in Audit mode, or with admission policies disabled, have a step to take first, under Upgrading below.

The worker-image registry read presents the worker ServiceAccount's pull secrets. On the air-gapped wiring, which puts the pull Secret on the ServiceAccount, the read behind RunnerVersionTooOld failed authentication on every reconcile while the pull itself worked. It now presents the ServiceAccount's imagePullSecrets when the pod template names none.

Upgrading

Nothing is removed, and one step is required on some installs. The upgrade is the ordinary one, with a namespace check first:

1. Label every namespace holding an ActionsGateway or EgressProxy that carries neither tenant marker, actions-gateway.github.com/tenant=true or actions-gateway.com/tenant=managed (the commands). Only an install whose gmc-tenant-resource-guard binding runs in Audit mode, or with admissionPolicy.enabled=false, can have such a namespace. The default Deny mode has already marked them all. A gateway in an unmarked namespace stops reconciling after the upgrade, and the GMC logs secrets is forbidden for it on every retry.
2. Apply both charts' CRDs, as every upgrade does: the actions-gateway-crds-v2 render, and the main chart's PriorityClassAllowlist CRD from the pulled chart, since on Helm 4.2.3 helm show crds oci://… prints a pull banner kubectl rejects (the commands). Both now mark v2 as the storage version.
3. helm upgrade the release. The agc-tenant-role ClusterRole gains get on serviceaccounts, with no list, watch or write. The GMC's ClusterRole drops get, list and watch on secrets and gains bind on the gmc-tenant-secret-reader ClusterRole alone.

A GMC deployed from your own manifests needs one new env entry. It reads its ServiceAccount name from POD_SERVICE_ACCOUNT, set by the downward API (fieldRef: {fieldPath: spec.serviceAccountName}), and refuses to start without it. The chart sets it.

Removing a namespace from the tenants now takes a second step: delete its gmc-tenant-secret-reader RoleBinding after removing the marker. The admission guard denies the GMC that delete once the namespace is no longer a tenant, so the binding, and the GMC's read access, stay until you remove it.

The full command and its options are in Upgrading.

Then, before v2.0.0, run the storage sweep. It is not needed for this release to work, and v2.0.0's CRD apply fails without it.

1. Migrate any EgressProxy still on CiliumFQDN or CalicoFQDN (the check). The sweep refuses to start while one exists.
2. Preview it with gag-migrate storage-version --context <your-cluster-context>, which writes nothing.
3. Run it with --apply, as a cluster administrator and with the GMC up, since every write passes through its webhooks.
4. Confirm every CRD reads [v2] with the storedVersions check.

A second run is safe: a CRD already reading ["v2"] is skipped.

If you apply a library template through your own kustomize overlay, retarget its patches to v2. The ClusterRunnerTemplates under deploy/templates/ are now written at actions-gateway.com/v2, so a JSON 6902 patch whose target still says version: v2beta1 matches nothing, and kustomize skips it without an error (what to change). Your workers then run the library template's default image and land on whatever nodes it allows.

Check a custom workerImage's runner version. The move to 2.338.1 changes only the default, so a RunnerTemplate pinned to its own image keeps the runner it was built on, and nothing on the cluster names the cause when jobs stop arriving (what to compare).

Expect more orphaned_running reaps on busy scale-set RunnerSets that cancel queued jobs. Each WorkerPodOrphanedRunning Event and actions_gateway_worker_pods_reaped_total{reason="orphaned_running"} increment is an idle worker that used to sit until the lifetime cap.

A worker evicted for exceeding its own storage limit is no longer re-run. A re-run gets the same limit and fails the same way, so the AGC now withholds it, on both tiers, and emits an EvictionRerunWithheld Event quoting the kubelet's message. A node-pressure eviction is re-run as before (what to do).

A run that spans two RunnerSets, or a Classic RunnerGroup and a RunnerSet, now has a re-run budget in each. maxEvictionRetries used to bound re-runs per run across both tiers.

Scale-set recovery now reads the run from GitHub before re-running it. A run the AGC cannot read is not re-run, and counts under actions_gateway_eviction_rerun_failures_total as conclusion_unknown or api_error.

Rolling back reaches v1.9.0 and no further, as the warning above says.

Deprecations

Nothing is removed in this release, and no notice is new. Every removal below was announced in v1.9.0 or earlier.

v2beta1, v2alpha1 and v1alpha1 are removed at v2.0.0, together with the Classic acquisition protocol. The storage sweep above is what lets that release's CRD apply succeed. Tenants on v1alpha1 migrate with gag-migrate (Migrating a tenant to v2).

The CiliumFQDN and CalicoFQDN egressPolicyMode aliases are removed at v2.0.0. A new write naming one has been rejected since v1.9.0.

Everything since v1.9.0

95 commits. 27 feat, fix and perf commits touch the released surface, as scripts/release/semver-floor.sh reads it rather than reading commit subjects. Build, CI, test, tooling and documentation work is left out and summarised under Project and tooling below.

Features (14)

• Store v2, and rewrite what is stored with gag-migrate storage-version (#2060)
• Scope the GMC's Secret reads to tenant namespaces (#2110)
• Re-run jobs whose worker's node shut down (#2108)
• Re-run a worker whose dead node an admin tainted out-of-service (#2123)
• Keep the scale-set re-run budget in a per-set ledger (#2109)
• Withhold the re-run of a worker evicted for its own storage limit (#2117)
• Record CONNECT failures per consumer namespace in the egress proxy (#2112), provision the per-namespace ports, certificates and API fields for it (#2122), move an opted-in namespace's workers onto its port (#2125), read and attribute the records in the AGC (#2127), and report each attributed policy denial as an Event (#2134)
• Hold a departing namespace's attribution port until its running jobs can have ended (#2126)
• Per-RunnerSet egress-audit attribution condition and gauge (#2058)
• Present the worker ServiceAccount's pull secrets to the registry read (#2077)

Fixes (10) and performance (1)

• Start the AGC of a v2 gateway whose defaultProxyRef names a shared pool in another namespace (#2130). It pointed at a proxy Service and mounted a CA Secret that exist only in the pool's namespace, so it could not start. It now mounts the CA from the share ConfigMap the GMC copies into the gateway's namespace, and the gateway reports ProxyShareNotGranted until that copy arrives.
• Write the GMC's own Secrets with an update rather than a patch, which its role does not grant (#2151). An EgressProxy with failure attribution that gained a consumer namespace after its first reconcile went Degraded, and its pool never rolled out with the new namespace's port.
• Give the PriorityClassAllowlist template's install-order failure message the command that works, pulling the chart before reading its CRDs (#2149)
• Key the out-of-service re-run on the runner container, and exclude the AGC's own deletions from every node-shutdown reason (#2124)
• Key scale-set recovery on the job a runner started (#2059)
• Spare a never-started scale-set worker's namesake run (#2103)
• Reclaim the idle scale-set worker a runnerless completion frees (#2062)
• Hand a scale-set worker's JIT Secret to its pod (#2063)
• Replace a replayed scale-set job's Secret left by a gone worker (#2101)
• Bump actions/runner to 2.338.1, and refresh base-image digests (#2114)
• Share one cluster read across RunnerSet admission guards (#2102). A scale-set RunnerSet write makes four uncached cluster-wide List calls instead of six, and a List failure is now reported by whichever guard reads first.

The two Go toolchain bumps are under Security.

Operator-visible work that ships in no image or chart (5)

• Reference architectures, a new page naming the supported deployment profiles and whether each is gated every release or validated once (#2106)
• A Dragonfly node-client recipe, deploy/dragonfly-node/, so containerd on the nodes running Kata Containers (microVM-isolated) workers pulls worker images through Dragonfly's peer-to-peer image distribution (#2094). Its peer ports are unauthenticated on every node address, and the page says who can reach them.
• The registry mirrors with the Dragonfly back end, validated once under the privileged Docker-in-Docker (DinD) worker (#2115). A DinD tenant's allow-all egress policy gives its jobs a path to the node client's unauthenticated upload port, and the page records it.
• The security dashboard plots the per-set egress-audit gauge (#2078)
• Tenant onboarding writes its objects at actions-gateway.com/v2, and so do the library templates under deploy/templates/ (#2147)

API and metric surface

Two new spec fields, three new metrics and two new Event reasons lead the additions listed below. No field, enum value, default, metric, condition reason or Event reason was removed or renamed, and no CLI flag or chart values key was added or changed. Beyond the two fields, the CRDs change only in which version is stored and in description text. Metric names declared outside tests go from 74 to 77.

What is new

• EgressProxy.spec.failureAttribution (Disabled by default, or PerConsumerPort) and ActionsGateway.spec.proxyFailureAttribution (Disabled by default, or PerJob), in v2, v2beta1 and v2alpha1. While a pool is PerConsumerPort, a validation rule caps spec.sharing.allowedNamespaces at 32.
• AttributionCertificateIssued, a Normal Event on an EgressProxy when the GMC issues its attribution certificates.
• EgressDestinationDenied, a Warning Event on a RunnerSet, related to the worker pod, when the AGC attributes a pool's policy denial to one of the set's jobs. It names the pool and the blocked host, once per job and destination and for at most eight destinations a job.
• gmc-tenant-secret-reader, a ClusterRole the chart ships, and a RoleBinding of the same name the GMC creates in each tenant namespace.
• POD_SERVICE_ACCOUNT, an environment variable the GMC requires, set by the chart from the downward API.
• EgressAuditUnattributed on RunnerSet. The condition type already existed on ActionsGateway, and a v2 RunnerSet now carries it too, with the existing reasons and one new one, WorkerAuditWithoutProxy: the gateway records worker addresses but the set resolves no EgressProxy.
• actions_gateway_runnerset_egress_audit_unattributed{namespace,runner_set,reason}, a gauge, 1 while that condition is True.
• cause="node_shutdown", a new value of the existing cause label on actions_gateway_eviction_retries_total and actions_gateway_eviction_recovery_evidence_lost_total.
• reason="budget_unrecorded" on actions_gateway_eviction_rerun_failures_total: a scale-set re-run withheld because its budget decision could not be written.
• reason="own_limit" on the existing actions_gateway_eviction_rerun_withheld_total: a worker evicted for exceeding its own storage limit.
• PROXY_ATTRIBUTION_PORTS and PROXY_ATTRIBUTION_PORT, environment variables the egress proxy reads to open per-namespace CONNECT listeners and a mutual TLS (mTLS) read endpoint for their failures. The GMC sets them only on a pool with failure attribution turned on, so a pool left at Disabled behaves as before.
• actions_gateway_proxy_failure_records_total{namespace,outcome,result} and actions_gateway_proxy_failure_reads_total{namespace,result}, counters an AGC with proxyFailureAttribution: PerJob emits: the failure records it read, attributed or discarded, and its reads of each pool replica.
• proxy-attribution-<pool-namespace>-<pool>, a ConfigMap and a Secret the GMC writes into an opted-in tenant namespace: the namespace's port and read endpoint, and its client certificate for the read.
• actions-gateway.com/attribution-held-until, an annotation on the pool's Deployment naming when a departing namespace's held port is released.
• scaleset-failures-<set>, a ConfigMap owned by each scale-set RunnerSet, holding its re-run budget. RBAC is unchanged.
• actions-gateway.com/surplus-for-job, an annotation the AGC sets on an idle scale-set worker it reclaims for a job no runner started.
• Scale-set worker annotations now follow the started job. actions-gateway.com/run-id, /repository and /job-name are rewritten when the pod's runner starts a job, and the OrphanedWorkerRecovered Event names the job rather than a pod.

Validation

v1.10.0 ships the same images, charts and CRDs as v1.10.0-rc.2, and that candidate passed every leg of the release gate in CI (gate run 40407403332) on the dogfood cluster, the project's own GKE cluster. The end-to-end suite ran three times, on Kata Containers, on privileged DinD, and on Kata with the Dragonfly mirror back end, and each run passed 76 tests with 0 failures and 17 skipped. The sizing, capacity, signed-CRD and soak legs passed too.

The storage advance was read on a cluster that had run 1.x. A run of the same gate against the same candidate rewrote 9 stored objects with the signed gag-migrate storage-version and left all six CRDs at storedVersions: ["v2"]. That run was local, so it records nothing the release pipeline reads. The CI run that does found the cluster already at ["v2"] and passed the leg without rewriting anything.

v1.10.0-rc.1 was not promoted. It failed only the storage leg, on a defect in the gate's own script (#2148), and was superseded by rc.2, which adds the fixes in #2149 and #2151.

Project and tooling

The release gate gains a storage leg, which applies the chart's PriorityClassAllowlist CRD, runs the sweep and reads storedVersions (#2076). Only the first candidate's run rewrites anything, since it leaves the dogfood cluster at ["v2"].

The DinD e2e tenant is now wired to the registry mirrors (#2115). Its egress stays open, so a pull that skips the mirrors still succeeds. That is why the gate's Dragonfly leg stays on Kata, whose tenant can pull images only through the mirrors.

Security

No security advisory accompanies this release, and no reported vulnerability in this project's own code is patched.

Every binary is rebuilt on Go 1.27.3 and golang.org/x/net v0.60.0, which fix vulnerabilities present in v1.9.0's build:

• CVE-2026-111943, rated HIGH by Trivy: crypto/x509 can accept a chain through untrusted intermediate certificates, letting a man-in-the-middle through. Fixed in Go 1.27.3 (#2111).
• GO-2026-6603, -6611, -6612, -6613 and -6617, in golang.org/x/net v0.58.0 and net/http in Go 1.27.1, which govulncheck reported reachable from the api module. Fixed by Go 1.27.2 and golang.org/x/net v0.60.0 (#2061).

The GMC no longer reads Secrets outside tenant namespaces (#2110), as described under Highlights. This hardens the controller's own privileges rather than patching a reported vulnerability.

Two scale-set fixes tighten handling of runner credentials without patching a reported vulnerability: a worker's JIT-config Secret is now deleted with its pod (#2063), and a replayed job no longer starts on the previous runner's already-used JIT config (#2101).

A pull request that changes code runs govulncheck and a Trivy image scan of every built image. One that changes the chart runs a Polaris posture check of it.

Container images

Pin these immutable multi-arch index digests rather than the floating v1.10.0 tag. Each index serves both linux/amd64 and linux/arm64, so one pinned digest schedules on either.

• gmc: ghcr.io/actions-gateway/gmc@sha256:70c2989b164f3a880b173f591a590cc538066133870b31daa54ca00f91ecd84e
• agc: ghcr.io/actions-gateway/agc@sha256:17efb187446ba1e1b05afd6de22fc28b02799ec16286e301973bd4a0b01b2800
• proxy: ghcr.io/actions-gateway/proxy@sha256:939c42a2702d8b7f90b094ed40b6fca90afc4fed5a9d3112f5f70a64e72316a1
• worker: ghcr.io/actions-gateway/worker@sha256:7abbf1edef3e7bd17e17ac7274afd959ca69d801248c66805a0c3a184429a8cd
• wrapper: ghcr.io/actions-gateway/wrapper@sha256:c20631ccb95ea2cdd85412eae83a344d881dfea080b8af7b42d084c263b0dc10

Verifying this release

We sign every image and both charts.

make verify-release VERSION=v1.10.0

The signed v2 CRD manifest and SHA256SUMS ship as release assets with detached cosign bundles, and the publish workflow attests build provenance.