Skip to content

v1.3.0

Latest
Latest

Choose a tag to compare

View all tags
@karlkfi karlkfi released this 03 Aug 11:30
v1.3.0
3519137
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Zero-idle self-hosted GitHub Actions runners for multi-tenant Kubernetes.

Warning

Two required upgrade steps, and one rollback that is not symmetric. Read Upgrading below before deploying. Rolling back past the PriorityClass allowlist migration re-arms a cluster-wide apiserver outage, and upgrading correctly does not protect you from it.

Highlights

Measured worker right-sizing. The AGC sizes workers from what jobs actually use, not what the template asks for. Two opt-in profiles: NodeShare divides a node's allocatable capacity, Throughput derives from observed usage. Validated on GKE: a worker took a derived 1500m where its templates asked for 2 and 3 CPU.

A capacity gate that stops claiming jobs the cluster cannot place. A RunnerSet refuses work it has nowhere to run instead of acquiring a job and stranding it. Whether the cluster can grow is now a gateway-level fact, not something each tenant asserts. Opt-in via spec.capacityGate.mode, default Off.

Automatic re-run for interrupted workers. Evicted, preempted, drained, and hand-deleted workers get their job re-run instead of vanishing. The eviction counter's new cause label tells the four apart.

GitHub Enterprise Server gateways address their own appliance. They never did: the GMC set no GITHUB_API_BASE_URL, so token exchange fell back to api.github.com and a GHES tenant failed before acquiring any job. A private CA in front of the appliance is now trusted via spec.githubCABundleRef.

Note

Untested against a real GHES appliance. The flow and the plumbing have test coverage. No GHES host has ever run it.

Upgrading

Two steps, both guarded: skip either and the upgrade stops with a message naming the fix. Full procedure in the upgrade guide, which also documents rollback per component: GMC, AGC, proxy, and worker image.

1. Apply the chart's CRDs. Part of every upgrade of this chart, not a one-off: Helm installs the chart-root crds/ directory on a fresh install only.

helm show crds oci://ghcr.io/actions-gateway/charts/actions-gateway \
  --version 1.3.0 | kubectl apply -f -

Charts carry no leading v. Images are tagged v1.3.0, charts 1.3.0.

2. If you set priorityClassAllowlist.configMapName, move those names into allowedPriorityClasses. The allowlist is now a cluster-scoped PriorityClassAllowlist CR. It fails closed, so names you do not carry across start being rejected.

Caution

Rolling back past step 2 re-arms a cluster-wide outage, and upgrading correctly does not protect you from it. The allowlist moved off a ConfigMap because a ValidatingAdmissionPolicy paramKind on a core type stays broken for the life of a kube-apiserver process once its bindings go empty, which helm uninstall does. Read Rolling back past this change before reverting.

Three changes that need no action, but will be visible. Each links its own section of the upgrade guide.

Deprecations

Nothing is removed in this release. Both notices below point at v2.0.0.

v2alpha1 is deprecated, and as of this release the apiserver warns on every apply. It stays served until v2.0.0, which removes it. Moving to v2beta1 is a re-apply rather than a migration tool: Before you upgrade past v2.0.0.

v1alpha1 and Classic are removed at v2.0.0 too. Tenants still on v1alpha1 migrate with gag-migrate: Migrating a tenant to v2. Unlike v2alpha1, v1alpha1 emits no apiserver warning today, so nothing will remind you it is going away.

Everything since v1.2.0

Over 450 commits. These 63 are the ones that ship in the product; build, CI, test, and docs work is left out.

New features (22)

Worker right-sizing

  • Per-RunnerSet worker usage sampling (#741)
  • Measured sizing recommendations in RunnerSet status (#744)
  • Opt-in sizing profiles applied at pod-build time (#746)
  • Report a Throughput profile cancelled by an injected CPU limit (#956)

Capacity-aware job intake

  • Opt-in capacity gate: refuse jobs the cluster cannot place (#902)
  • Gate intake on the node autoscaler's own declination (#913)
  • Refuse to claim jobs when namespace quota has no headroom (#793)
  • Pre-claim quota rung ported to the scale-set tier (#868)

Worker recovery

  • Recover preempted workers off PreemptionByScheduler (#972)
  • Recover drained workers on the deletion mark (#1011)
  • Eviction recovery ported to the scale-set tier (#851)
  • Bound a worker orphaned while the AGC is down with a lifetime cap (#914)

GitHub Enterprise Server (untested against a real appliance)

  • Trust a private CA in front of a GHES appliance (spec.githubCABundleRef) (#1137)

Autoscaling and observability

  • Bring-your-own proxy autoscaler via EgressProxy managedAutoscaling (#758)
  • Managed VPA opt-in for the GMC and per-gateway AGC (#761)
  • Export AGCAutoscalingUnavailable as a Prometheus gauge (#763)
  • Export GitHubEgressIncomplete as a gauge for fleet alerting (#1038)
  • Count scale-set poll errors in the shared counter (#878)
  • Alias runner_set onto worker_pods_reaped_total so scale-set series join (#1020)

API

  • Mark v2alpha1 deprecated so the apiserver warns (#828)
  • Rename sizingRecommendation.windowStart to windowStartTime (breaking, pre-GA) (#943)
  • Exempt deletion-only updates from PriorityClass allowlist re-validation (#1017)
Fixes (41)

Scale-set listener

  • Deregister scale-set runner records on reap, and sweep unclaimed ones (#1052)
  • Re-offer a scale-set job whose runner name will not register (#1048)
  • Measure and fix the AGC restart queue replay (#1138)
  • Flush the listener's pending message deletes on shutdown (#1148)
  • Stop stranding a worker pod on a reclaimed job Secret (#1119)
  • Reclaim the per-job JIT-config Secret (#727)
  • Latch the capacity gate past the reap with a one-probe slot (#1000)
  • One autoscaler loop's two verdicts must not gate intake (#949)
  • Give up on assignments GitHub no longer holds (#1108)
  • Defer ceiling-blocked jobs instead of spinning on redelivery (#1105)

Shutdown and lifecycle

  • Delete broker sessions reliably on SIGTERM (#739)
  • Forward SIGTERM to Runner.Worker so cancelled jobs report (#747)
  • Arm the worker termination relay before starting the child (#872)
  • Drain in-flight CONNECT tunnels on proxy shutdown (#748)
  • Linger for endpoint removal inside the drain budget (#755)
  • Give a Running scale-set worker a reap deadline (#824)
  • Delete the worker pod of a job the listener abandons (#1032)
  • Reap tenant worker pods when a v2 gateway is deleted (#1140)

Quota accounting

  • Count native sidecars and RuntimeClass overhead in the worker footprint (#883)
  • Count the storage keys in the worker footprint (#887)

Naming and identity

  • Keep a truncated worker pod name valid and unique (#921)
  • Bound the derived RunnerGroup name via one shared helper (#922)
  • Keep v1 and v2 agent pools apart during coexistence (#915)
  • Key a v2 proxy pool solely on its own identity label (#1117)
  • A gateway-scoped AGC no longer reconciles v1 RunnerGroups (#1036)

GitHub Enterprise Server (untested against a real appliance)

  • Make GHES gateways address their own appliance (#1034)
  • Address GITHUB_API_BASE_URL for the re-run call (#973)
  • Retry an evicted job's re-run until GitHub accepts it (#1010)

Admission and API

  • Move the PriorityClass VAP paramKind off a core type (#958)
  • Reject a nodeShare.allocatable declaring neither cpu nor memory (#946)
  • Ratchet the v2beta1 single-runnerLabel rule onto the field (#847)
  • Generate the missing ClusterCapacity deepcopy (#928)
  • Dedupe apiserver deprecation warnings to one log line per process (#1021)

Tenant migration (gag-migrate)

  • Dual-read the privileged grant across both label domains (#911)
  • Retry a transiently unreachable webhook in gag-migrate --apply (#904)
  • Make a Docker-in-Docker tenant actually migratable (#856)

GMC and proxy

  • Let a rollout restart of a managed Deployment take effect (#1104)
  • Derive the AGC NO_PROXY apiserver exemption from the cluster (#912)
  • Own every namespaced v1 child so a stripped finalizer cannot leak them (#848)
  • Fail AGC startup on an unreadable proxy CA cert (#1133)
  • Read classic-tier run identity from the github context (#967)

API and metric surface

One new CRD: PriorityClassAllowlist, cluster-scoped, the only kind added since v1.2.0. It is also the only CRD the chart ships in its chart-root crds/ directory, which Helm installs on a fresh install and skips on every upgrade. That is why applying CRDs is step 1 of Upgrading above, and why the upgrade hard-fails without it.

Thirteen spec fields and two status fields are added, and nothing is removed — verified against the CRD schemas at both tags, not assembled by hand. The rest are new fields, metrics, and settings this tag publishes for the first time. There is no generated API reference yet, so the CRDs themselves are the authority: kubectl explain, or the signed actions-gateway-crds-v2.yaml attached below.

Sixteen metrics are added and none removed, so existing queries keep working. The shipped Grafana dashboards were rebuilt around them and the PrometheusRule gains an ActionsGatewayGitHubEgressIncomplete alert — both in deploy/monitoring/, described in observability-dashboards.

New spec fields (13)

RunnerSet

  • spec.sizing.profile: Static, Binpack, Throughput, or NodeShare (default Static)
  • spec.sizing.nodeShare.allocatable / .workersPerNode: the NodeShare envelope
  • spec.sizing.minRequests / .maxRequests / .limitHeadroomPercent: bounds on what a profile may derive
  • spec.capacityGate.mode: Off or Observe (default Off)
  • spec.maxWorkerLifetime: cap on a worker pod's lifetime

ActionsGateway

  • spec.clusterCapacity.nodeAutoscaling: Present or Absent (default Present)
  • spec.agcAutoscaling.mode: managed VPA for the AGC, Off, Initial, or Recreate
  • spec.githubCABundleRef.name: ConfigMap holding a private CA for a GHES appliance

EgressProxy

  • spec.managedAutoscaling: bring-your-own proxy autoscaler opt-in (default true)

PriorityClassAllowlist (new cluster-scoped kind)

  • spec.allowedPriorityClasses: the platform allowlist, replacing the watched ConfigMap
New status fields (2)

RunnerSet.status

  • sizingProfileState: Active or AwaitingSamples
  • sizingRecommendation[]: per-container observedPeak, observedP95, requests, limits, sampleCount, windowStartTime
New configuration: 4 chart values, 3 tunables, 1 renamed flag

Nothing here is required — every item defaults to the previous behaviour. Listed because none of it is discoverable from the CRDs.

Chart values — the GMC's own managed VPA, off by default

  • vpa.enabled (default false)
  • vpa.updateMode (default "Off")
  • vpa.minAllowed / vpa.maxAllowed (default {})

Environment tunables — documented with their defaults in troubleshooting and observability-metrics

  • PROXY_SHUTDOWN_DRAIN_TIMEOUT (default 45s) — the proxy's whole shutdown budget, covering endpoint linger then in-flight CONNECT drain
  • PROXY_SHUTDOWN_LINGER (default 10s, -1s to skip) — how long to wait for endpoint removal before draining
  • AGC_QUOTA_ADMISSION (default on) — set false to turn the namespace-quota rung off AGC-wide

Renamed flag — only affects an install that passes GMC arguments directly; the chart handles it

  • --priority-class-allowlist-configmap → --priority-class-allowlist-name (#958)
New metrics (16)

All prefixed actions_gateway_, listed here without it. Their remedies and alerting guidance are in observability-metrics.

Worker usage sampling

  • worker_usage_cpu_peak_cores
  • worker_usage_memory_peak_bytes
  • worker_usage_job_cpu_peak_cores
  • worker_usage_job_memory_peak_bytes
  • worker_usage_jobs_sampled_total
  • worker_usage_jobs_unsampled_total
  • worker_usage_poll_errors_total

Scale-set capacity and intake

  • scaleset_advertised_capacity
  • scaleset_capacity_withheld
  • scaleset_jobs_deferred
  • scaleset_jobs_abandoned_total

Worker recovery

  • eviction_rerun_failures_total
  • eviction_recovery_identity_unknown_total

Gateway health

  • agc_autoscaling_unavailable
  • github_egress_incomplete
  • broker_session_leaks_total
New Kubernetes Event reasons (8)

A surface separate from the condition reasons below — these land in kubectl get events, and six have their own section in troubleshooting. None was removed.

  • WorkerPodLifetimeExceeded — the lifetime cap killed a worker
  • WorkerPodOrphanedRunning — a Running worker was reaped
  • WorkerPodCompletedPending — a worker was reaped while Pending after its job finished
  • WorkerPodsReapedOnGatewayTeardown — workers cleared as a gateway was deleted
  • WorkerDrainTimeout — a drain did not finish in budget
  • WaitingForWorkerDrain — deletion is blocked pending drain
  • AssignmentAbandoned — the listener gave up an assignment
  • WorkerCapacityDeclined — also a condition reason, listed below
New condition reasons (28)

Most are documented with their remedies in observability-metrics and troubleshooting. Five are not yet: SizingWithinRange, SizingDriftDetected, InsufficientSamples, JobsProvisioning, and GitHubEgressAllowed. Until the operator docs catch up, their godoc in api/apiconditions/conditions.go is the reference.

Sizing

  • SizingProfileActive
  • SizingProfileOverridden
  • SizingWithinRange
  • SizingDrift
  • SizingDriftDetected
  • InsufficientSamples
  • CPULimitInjected
  • NoCPULimitInjected

Capacity and job intake

  • CapacityAvailable
  • WorkerCapacityDeclined
  • WorkerCeilingReached
  • ScaleUpDeclined
  • GateModeUnsupported
  • AwaitingProbe
  • AwaitingWorkerPods
  • JobsProvisioning
  • JobProvisionStalled
  • RunnerNameConflict

AGC autoscaling

  • AGCAutoscalingActive
  • AGCAutoscalingDisabled
  • AGCAutoscalingUnavailable
  • VPACRDNotInstalled

GHES, egress, and CA trust

  • GitHubEgressAllowed
  • GitHubEgressIncomplete
  • ApplianceRangesRequired
  • CABundleNotFound
  • CABundleInvalid

Lifecycle

  • GatewayTerminating

Validation

This is the first release whose tag is gated by a passing end-to-end run on real infrastructure, rather than advised by one. The gate predates v1.2.0. Getting a verdict out of it took five candidates: rc.1 through rc.3 aborted before finishing, rc.4 passed but never caught a live worker pod, and v1.3.0-rc.5 passed with the pod-level check rc.4 missed. rc.5 needed its own verdict regardless — it carries product changes past rc.4, so rc.4's pass does not transfer.

73/73 e2e specs on Kata microVM workers, on live GKE — the four legs, and what none of them assert

Run against the signed release artifacts rather than a build (run):

  • The full e2e matrix on gateway-provisioned runners, 73/73 specs, on Kata microVM-isolated workers
  • Both sizing profiles actuating on real worker pods, with the derived value confirmed at the pod: 1500m where the templates asked for 2 and 3 CPU
  • The signed v2 CRD artifact downloaded, signature-verified, applied, and all five CRDs registered
  • Teardown back to zero nodes

That is what the gate itself asserts. It does not assert the capacity gate, the namespace-quota rung, or anything GHES — the run targets github.com, and it builds an autoscaling cluster with headroom rather than a constrained one. Those rest on the unit, envtest, and e2e suites instead — several of which grew this cycle to reach a live cluster, listed under Testing and internal quality below.

Every PR now upgrades from the last released chart, not just HEAD to HEAD (#1008). It immediately caught what became step 1 above: #958 shipped a CRD into chart-root crds/, breaking every v1.2.0 upgrade while CI stayed green.

This tag adds no product code beyond that candidate.

Full changelog: v1.2.0...v1.3.0

Project and tooling

Documentation is now versioned. Since this release cycle the site publishes a build per stable tag, so these notes link 1.3.0 pages rather than whatever main says today, and an operator on an older release reads that release's docs. Older versions were seeded, so 1.0.0 through 1.3.0 are all live.

Three operator guides are new, and one grew by a third. Worker right-sizing and Sizing the platform-owned ResourceQuota are linked from their features above. Node shutdown budgets across platforms is reference material with no feature of its own: terminationGracePeriodSeconds is a request, not a guarantee, and it collects what each platform actually grants — the proxy asks for 60s and gets 15 on a GKE Spot node. Troubleshooting gained 36 sections, most of them the remedy for a condition or event reason listed above.

The rest is contributor-facing, listed because a release is also a claim about how the project is built.

Release, docs-site, CI, and tooling (25)

Documentation site

  • Versioned docs via mike — stable default, dev opt-in (#768)
  • Publish docs from stable release tags (#766)
  • Derive the site announce bar from the git tags (#775)
  • Gate the release on the docs-site announce bar naming the tag (#801)
  • Publish the backlog on the dev docs site, filterable (#1059)
  • Gate site-side link resolution, fix 13 site-only 404s (#1062)

Release process

  • Release delta report + scope-ledger convention (#1058)
  • Surface operator caveats in the pre-flight, don't rely on memory (#992)
  • A status file and a sentinel for the release gate (#1161)
  • Relay e2e spec progress into the release-validation gate (#1159)
  • Make the release gate able to fail on a dead sizing profile (#867)

Test and CI speed

  • Show live spec progress and render the JUnit report (#1152)
  • Report unit-run progress from the go test -json stream (#1162)
  • Compile the e2e images' dependency tree once, and cache it across runs (#870)
  • Overlap the e2e runner disk cleanup with job setup (#871)
  • Cut the suite's serial tail (#874)
  • Measure coverage in one workspace-wide invocation (#952)
  • Run workspace unit tests as one multi-module invocation (#722)
  • Change-scope local golangci-lint to the modules a diff affects (#724)

Repository tooling

  • Allocate Q-IDs from a ref claim, not a counter line (#823)
  • Merge driver resolving STATUS.md Queue rows by ID (#879)
  • Repo-wide leftover conflict-marker gate (#732)
  • Auto-rebase stranded Dependabot Go PRs (#827)
  • Parse workflow YAML with a real parser, in a new devtools/ module (#1114)
  • Add nolintlint + a ratcheted funlen gate (#771)
Testing and internal quality (13)

Coverage that reaches real infrastructure rather than a fake, and the consolidation work behind it. 52 test and 15 refactor commits landed in this window; these are the ones that changed what the suites can observe.

Tests that drive a live cluster

  • Assert the capacity gate's autoscaler matcher against a live cluster-autoscaler (#929)
  • Extend the live-autoscaler drift gate to Karpenter (#1024)
  • Gate day-2 helm upgrade over a live release (#935)
  • Validate the v1→v2 migration live on GKE (#909)
  • Prove a quota-blocked job redelivers to a sibling gateway (#1028)
  • Scale-set disruption recovery under the chart's real RBAC, and the ordering fix it caught (#1018)
  • Measure the node-drain path on both tiers (#895)
  • Execute the cosign download path and verify-release itself (#1145)

Consolidation

  • Code-restraint sweep: trim narrative comments, dedup helpers, drop dead weight (#1013)
  • Merge the twin worker-quota evaluators into one shared core (#1022)
  • Decompose the GMC main() and AGC run() god functions (#765)
  • Unify the broker/scale-set error taxonomy in githubapp/httpx (#794)
  • Share the version-neutral v2 API code, gate the identical remainder (#760)

Security

No security advisory accompanies this release, and no fix here is embargoed or backported. What it carries is dependency maintenance: a security bump of gRPC to v1.82.1 and, in the build-tools module, x/net to v0.56.0 (#749), plus six routine dependency-group updates.

Three fixes below harden handling of credentials and trust material rather than patching a reported vulnerability: an AGC that now fails startup on an unreadable proxy CA cert (#1133), and two that stop a per-job JIT-config Secret being stranded or reused (#727, #1119).

Every pull request — with no path filter, deliberately — runs govulncheck across the workspace and a Trivy scan of the images, and main re-runs both on every push.

Container images

Pin these immutable multi-arch index digests rather than the floating v1.3.0 tag. Each index serves both linux/amd64 and linux/arm64, so one pinned digest schedules on either.

  • gmc — ghcr.io/actions-gateway/gmc@sha256:e29c56f3bcc7bbaeace702a6011469b9ed7995d80abb1993b454c369b868ca3a
  • agc — ghcr.io/actions-gateway/agc@sha256:326c5498618c7bc3ac1f31d834f355dd178d95003071c704a4a17a3c28efcc33
  • proxy — ghcr.io/actions-gateway/proxy@sha256:8f1b832bd5b98cdf7d13b306c6b563197bb55b6602782081bdd4a0cb4706c90c
  • worker — ghcr.io/actions-gateway/worker@sha256:3954b73f158f1f7d41a4d8b8362c9f3b916fd8b447c83f7d84b8356f9c5ddf17
  • wrapper — ghcr.io/actions-gateway/wrapper@sha256:f0b5eabb1e9c4aef5c6d26e84da3c3bcd7f3e27500990d4dd90f15df74d23d8d

The chart's digest fields ship empty on purpose — an unconfigured install fails closed rather than running a floating tag. These are the values to put in them.

Verifying this release

We sign every image and both charts.

make verify-release VERSION=v1.3.0

The signed v2 CRD manifest and SHA256SUMS ship as release assets with detached cosign bundles, and the publish workflow attests build provenance.

Assets 11
Loading