Zero-idle self-hosted GitHub Actions runners for multi-tenant Kubernetes.
Note
The tight-egress posture is a recipe you apply, not something an upgrade turns on. Installing v1.7.0 changes no network policy and no worker's reachable set. The mirrors, their policies and the worker wiring are manifests under deploy/registry-mirror/, and the step that makes them enforcement is deleting your own allow-all egress policy. Read Untrusted pull requests before assuming the version alone bought you anything.
Note
A set that configured spec.scaleUp may now ramp faster than it has been, and condition-transition Events get more frequent on every set. Neither is a knob you changed: both are defects fixed underneath you, so the observed behaviour moves without the spec moving. Read Upgrading below.
Highlights
A reference architecture for an in-cluster container registry mirror, so you can tighten worker egress to almost nothing. Worker egress was default-deny already; pulling images was the reason you had to open it up. With the mirrors in place a pull request's code reaches exactly three things: cluster DNS, GitHub, and the mirrors. The allow-all rule comes out, and the default-deny is the whole story. Every Kata end-to-end run on this project's own cluster carries the closed posture, with negative probes confirming it.
A mirror rather than an allowlist, because the allowlist is the wrong shape. Registries are CDN-fronted, so a CIDR list rots, and fully-qualified-domain-name policy is not enforceable on GKE Dataplane V2. Even where a host allowlist is enforceable, permitting docker.io permits docker.io/<anyone>/<anything> in both directions, so the exfiltration surface stays open. Each mirror instance is instead pinned to one upstream in its pod spec and refuses uploads, giving one auditable chokepoint instead of five open hostnames. Both topologies ship, shared or per-tenant, and the choice is a platform administrator's: it turns on tenant count, disk budget, and whether your tenants are mutually hostile.
The egress proxy's audit log can now say which tenant and which job made each connection. The proxy writes one log entry per outbound connection a worker makes, recording the destination, the time, and the bytes each way. Until now that entry named the proxy pool and nothing finer: on a pool shared between tenants it could not say which of them reached a destination, and on any pool it could not say which job did. Closing both takes two opt-ins joined on the source address: EgressProxy.spec.auditLogging: ConnectionsWithSource adds the client address to the proxy's entry, and the new ActionsGateway.spec.auditLogging: WorkerAddresses has the gateway record which job holds each address while its pod lives. Both default Off and both are needed, because a worker pod is deleted when its job ends, so an address resolves to nothing after the fact and the binding has to be recorded live.
A scale set that stops withholding capacity now says so promptly, instead of taking up to 10 hours. A set bound at zero headroom recomputed its advertisement every long-poll and recorded it, but only a reconcile published it, and an idle set had nothing to requeue on. Status kept reporting intake as withheld long after the quota that withheld it was restored. This is the defect the v1.7.0-rc.1 candidate found (see Validation).
Two pools whose names collide no longer fight over one GitHub runner record. A v1alpha1 RunnerGroup named rs-<x> derives the same agent Secret and the same runner name as a RunnerSet named <x>, and neither pool's label selector sees the other's Secret. The loser kept finding its index missing, registering, taking the 409, and resolving it by deregistering the live incumbent, so the pool that was serving jobs lost its registration to the one that was not. Registration now checks the derived Secret's owner first, and the 409 path re-checks before deleting anything, with a regression test asserting the incumbent's record survives that path.
Upgrading
No breaking change, and nothing is removed. The upgrade is the ordinary one:
-
Apply the chart's CRDs, as every upgrade does. The command is unconditional and idempotent, and the chart preflights it and fails with the same command if you skip it.
-
helm upgradethe release. TheActionsGatewayandEgressProxyschema changes here are additive fields on CRDs the chart ships as templates, so they carry with the release. -
Wait for the GMC rollout to complete before applying any v2 object. Applying while the conversion webhook has no endpoints fails with
no endpoints available for service "webhook-service", which reads like a broken install but only means the apply came too early (what to do).
The full command and its options are in Upgrading.
A spec.scaleUp token is now charged per worker pod, not per delivered job, which affects every classic-tier pool that sets it. It is off by default, so a pool that never configured a ramp sees nothing, and ScaleSet pools never charged per delivery. GitHub fans a queued job out to several sibling sessions at once and only one provisions; each of the others took a token and never returned it, so an N-way fan-out spent N tokens for one pod and cut the effective ramp to maxPerSecond / N. A delivery that returns before asking for a worker pod now hands its token back, so an affected pool ramps at the rate it configured, which may be faster than what it has been doing. The refund stops at pod creation and never fires on completion, so the bucket stays a rate limit rather than becoming a second concurrency ceiling. actions_gateway_worker_scaleup_throttled_total and actions_gateway_jobs_admission_rejected_total{reason="scaleup"} should fall for an affected pool. Details.
Condition-transition Events now fire on every flip rather than once per object, so expect more of them on sets that flap. Five guards held a pointer into the conditions slice across the write that mutates it, so each compared a value against itself and could only fire on a first observation. Measured before the fix: Ready going True to False to True emitted one Event rather than three, and a workerImage downgraded below the minimum on a set already carrying the condition emitted none. If you alert on Event volume rather than on conditions, re-baseline it.
Nothing else moves. No metric was added, removed or renamed, no chart values key changed, and no command-line flag changed on any binary.
Deprecations
Nothing is removed in this release, and both notices are unchanged from v1.6.0.
v1alpha1 and v2alpha1 are both deprecated, and the apiserver warns on every apply of either. Both are removed at v2.0.0, together with the Classic acquisition protocol. Tenants on v1alpha1 migrate with gag-migrate (Migrating a tenant to v2); moving from v2alpha1 to v2beta1 is a re-apply rather than a migration (Before you upgrade past v2.0.0).
Everything since v1.6.0
68 commits, 29 of them carrying a feat, fix or perf subject. Nine change the shipped binaries and chart, as scripts/release/semver-floor.sh reads them off the released surface rather than off commit subjects; build, CI, test, tooling and docs work is left out and summarised under Project and tooling below.
This release's headline is in neither list below. The tight-egress posture ships as manifests, worker wiring and documentation, so it reaches no image and no chart and appears in none of the nine. That is a property of how it is delivered, not a measure of its weight.
Features (2)
Fixes (7)
- Publish a scale-set capacity change instead of waiting for the 10h resync (#1799)
- Say when the recovery scan judged a terminating worker and declined (#1793)
- Emit condition transition Events on every flip, not once per object (#1786)
- Refund the scale-up token when an admitted job never provisions (#1763)
- Stop a name-colliding agent pool deregistering the incumbent (#1760)
- Make the shutdown drain one-way so a session cannot outlive it (#1754)
- Give a projected proxy share a bounded event source (#1750)
Operator-visible work that ships in no image or chart (6)
Each of these moved an operator's surface with no binary behind it:
- Refuse
GET /v2/_catalogon every mirror instance (#1794) - Ship both mirror topologies and the choice between them (#1789)
- Add the in-cluster registry pull-through cache (#1753)
- Refuse a stable tag no candidate validated (#1788)
- Warn when a candidate stops covering
main, at the merge (#1752) - Verify the docs artifact carries the version, and that the site serves it (#1751)
API and metric surface
One spec field, one new enum value on an existing field, and nothing else. No field, enum value, default, metric, condition reason or Event reason was removed or renamed, no existing default changed, and the chart's values keys are unchanged. Field descriptions are in the generated API reference, which covers v2beta1; a deprecated-version field still needs kubectl explain.
New spec field (1) and new enum value (1)
ActionsGateway.spec.auditLogging takes Off (the default) or WorkerAddresses. Under WorkerAddresses the gateway writes a second stream on its own stdout, selected on msg == "worker address audit", binding each worker pod's address to the job holding it while the pod lives. Changing it rolls the workload, the same as logLevel.
EgressProxy.spec.auditLogging gains a third value, ConnectionsWithSource, alongside the Off and Connections shipped in v1.6.0. It writes the same audit log entry as Connections with the client's source address added, read from the accepted connection and never from a request header, so a worker cannot forge it.
Both default Off, and each alone records something nothing asks about: the join of the two is what attributes a destination to a job.
No new condition types, condition reasons, Event reasons, labels or annotations, and no metric added, removed or renamed: 75 metric names declared in the Go source, an identical set on both sides of the window.
Validation
v1.7.0-rc.3 passed its dogfood validation, exit 0, run against the signed release artifacts rather than a build. check-artifact-unchanged.sh v1.7.0-rc.3 reports nothing on the released surface changed between the candidate and this commit, so the verdict below covers what ships here.
It is the third candidate, and the three tell the story of the release. v1.7.0-rc.1 was tagged, published and artifact-verified, and its e2e matrix passed 75/75. It then failed the admission ladder's quota rung: the rung bound correctly at zero headroom and, 300 seconds after the ResourceQuota was restored, was still withholding. That is the fix shipped here as #1799. The leg that caught it had been added hours before the cut and found the defect on its first outing.
v1.7.0-rc.2 passed with that fix in place. v1.7.0-rc.3 was cut because the shared mirror set's ingress policy was narrowed to worker pods afterwards, and that policy ships in no image and no chart, so the artifact check cannot see it and only a live Kata run exercises it. A candidate is validated against a tree, not against a digest.
The legs, and what the artifact checks discriminate
Run against the signed release artifacts (e2e run):
| Leg | Result |
|---|---|
| deploy | release candidate deployed and CI routed to it |
| e2e matrix on project-provisioned runners | 75 specs, 62 passed, 0 failed, 13 skipped |
sizing, NodeShare |
Active on ci-e2e; worker runner CPU request 1500m, derived, where the templates ask 2 and 3 |
sizing, Throughput |
Active on ci, 258 samples |
| admission ladder, quota rung | bound at zero headroom (withheldCapacity[quota]=2, advertisedCapacity=0), then released (0 and 2) after the quota was restored |
| admission ladder, placeability rung | verdict published: WorkerCapacityDeclined=False, reason CapacityAvailable |
| signed v2 CRD manifest | blob signature Verified OK, all five CRDs applied and registered |
The suite covered cross-tenant network isolation, worker preemption and recovery, a drained worker that must trigger no rerun, a ceiling-held job cancelled rather than redelivered, the v1 to v2 migration dry run, and Vault workload identity with no PEM Secret.
The provenance is the check that discriminates: signatures prove who built an image, and only the digest proves what was built. Its signer URI ends publish.yml@refs/tags/v1.7.0-rc.3 and its sourceRepositoryDigest equals the tagged commit. Re-run against a deliberately wrong signer workflow it exits 1, so the pass discriminates rather than merely exiting 0.
Cluster preflight passed with two warnings that are properties of the dogfood cluster rather than the candidate: no cert-manager and no metrics-server.
Project and tooling
The mirror recipe was validated on the cluster that runs this project's own CI, in four phases each gated on the one before: manifests serving, the job's image clients riding them, the allow-all egress policy deleted with negative probes confirming enforcement, and the documentation rewritten from a caveat into a how-to. Every phase booked a live cluster session, which was this release's schedule risk and the part planning could not compress. The wiring phase had to run with image caches cold so the quay.io and registry.k8s.io prepulls were exercised rather than skipped; nothing had to be arranged for it, because the self-hosted lane has had no actions/cache step since that lane was tightened.
The release tooling gained two gates out of this cycle, both of which exist because a release can otherwise ship something no candidate saw: one refuses a stable tag no candidate validated, and one warns at merge time when a candidate stops covering main. A third checks that the published documentation artifact carries the version it claims and that the site actually serves it.
Security
No security advisory accompanies this release, no reported vulnerability is patched in this project's own code, and no dependency or toolchain moved for a CVE. No dependency moved at all in this window. The Go toolchain stays at 1.26.6.
It carries security-relevant work, and the tight-egress posture is the substance of it. The posture is opt-in because it is a recipe applied to your own cluster, not because the safe direction was traded away: its final step is deleting an additive allow-all policy, so the enforced state is what an operator gets by not acting, while the privileged pod shape stays a deliberate templateRef. Nothing ships as a cluster default.
A shared mirror set no longer hands one tenant the list of what the others pulled. GET /v2/_catalog names every repository in the cache and answers on the same port the worker policy admits, so every instance is now fronted by a proxy that refuses it with 403 under both topologies. It takes a fronting proxy rather than a setting, because Distribution registers that route unconditionally and its one catalog knob is raised to a default whenever it is set to zero. What a shared set still gives away is timing: whether a repository is already warm says some other tenant pulled it. That channel is narrower and is not something tuning closes, and whether a hit is distinguishable from a miss from inside a Kata guest is not measured. Tenants who must not learn what each other build should take the isolated topology rather than wait for that reading.
The egress attribution above is the other half. It ships off by default and needs two separate opt-ins, because a per-job record of who reached what is data about a tenant, and retaining it should be a decision rather than a side effect.
Every pull request runs govulncheck, a Trivy image scan of every built image, and a Polaris posture check of the shipped chart. All three run again on the merge queue's candidate commit and on every push to main that touches something other than documentation.
Container images
Pin these immutable multi-arch index digests rather than the floating v1.7.0 tag. Each index serves both linux/amd64 and linux/arm64, so one pinned digest schedules on either.
- gmc:
ghcr.io/actions-gateway/gmc@sha256:7712f4199efe3d80fe5f0de624e5eb594311c5c4f4b9fc97ca33d3c6758e97f7 - agc:
ghcr.io/actions-gateway/agc@sha256:3dcca9eebc9240b54ec5bb42a9afdca84a0fe0f04ecd69032ef2d7cf119b189b - proxy:
ghcr.io/actions-gateway/proxy@sha256:1028aca7c8e8663e9addc2641e8152ea50ee5949b1c6eb6a91406590ebbcc695 - worker:
ghcr.io/actions-gateway/worker@sha256:3a93603e29eb45c679762e273a9de9e77e86e96f237c39c63a533d69baa6b50e - wrapper:
ghcr.io/actions-gateway/wrapper@sha256:3069040c3b886a253ec9287f7b44f7f1eeaf812e1494892aaefb7f7479b36a4a
Verifying this release
We sign every image and both charts.
make verify-release VERSION=v1.7.0The signed v2 CRD manifest and SHA256SUMS ship as release assets with detached cosign bundles, and the publish workflow attests build provenance.