Allow deprecated runners to self-update, even while refused jobs

[sdarwin]

🏷️ Discussion Type

Bug

💬 Feature/Topic Area

Actions Runner

Discussion Details

Allow deprecated runners to self-update, even while refused jobs

Problem

We are launching self-hosted github actions runners from AWS AMIs, where the runner is baked in.

When a self-hosted runner version is deprecated, the service denies it at the
message-listener layer:

GitHub.DistributedTask.WebApi.AccessDeniedException: Runner version v2.334.0
is deprecated and cannot receive messages.
   at ...BrokerHttpClient.GetRunnerMessageAsync(...)

Self-update instructions are delivered through that same message queue. So a
deprecated runner cannot receive the instruction that would let it stop being
deprecated. It is too old to be told how to stop being too old, and requires
manual intervention on every affected host.

This recurs constantly: actions/runner#3381, actions/runner#3405,
actions/runner#3466, actions/runner#3651, actions/runner#3767,
actions/runner#4019, actions/runner#4203, actions/runner#4305,
actions/runner#4392, actions/runner#4442 — spanning v2.317 through v2.336, over
two years. Each is the same deadlock rediscovered by a different operator.

Proposal

Let a deprecated runner complete a self-update, while continuing to refuse jobs.
Concretely: keep the update path reachable when the job path is denied, so the
runner can pull the current release, restart, and rejoin normally.

Enabled by default, on the same terms as normal auto-update:

• If auto-update is enabled (the default), deprecation should not be what stops
  it. The runner heals itself.
• If --disableupdate is set, the runner stays refused. The operator asked to
  pin a version; silently overriding that would violate their intent.

Nothing new is granted. The runner already authenticates successfully — this is
a policy denial, not an auth failure — and the update fetches a signed GitHub
release. No user code runs on a deprecated runner under this proposal.

Important: warn when the update cannot persist

Self-update writes to the instance's local disk. For ephemeral runners baked
into a VM image (AMI, container image), that write dies with the instance: every
new instance boots at the stale image version and updates again, forever.

Without a warning, this proposal would convert a visible outage into permanent
invisible waste — an update download prepended to every job, with no signal that
the image needs rebaking. That would be a worse outcome than today's hard
failure, which at least tells the operator what to fix.

So the update should be paired with an explicit, loud diagnostic when the runner
is --ephemeral and started deprecated. Something like:

WARNING: This runner started at deprecated version v2.334.0 and self-updated
to v2.337.0. This runner is ephemeral, so the update will NOT persist — every
new instance from this image will repeat it. Rebuild your runner image with a
current version.

That makes the failure self-describing and points at the real root cause, which
is usually a pinned RUNNER_VERSION in a Packer template or user-data script.

Prior art

#4285 (merged 2026-03-13) added
ACTIONS_RUNNER_RETURN_VERSION_DEPRECATED_EXIT_CODE=1, making a deprecated
runner exit with code 7 so orchestrators can detect it. That helps operators
detect the deadlock cleanly, and is a good addition — but it still requires
manual remediation. This proposal is the natural next step: let the runner
resolve it without a human, and make the non-persistent case obvious.

Secondary request

Claude Opus is requesting this, not me. (Actually it wrote the whole message.)

There is no published deprecation schedule. The docs say only that a runner
"will not be able to process jobs if it falls below a certain version threshold"
— no threshold, no timeline, no grace period. Observed behavior suggests roughly
three releases / four months behind, enforced shortly after a new release ships,
but operators are reverse-engineering this from outages.

Even publishing the rule ("versions more than N releases behind are refused")
would let people schedule image rebuilds instead of discovering the cutoff in
production. If the self-update change lands, this matters much less.

[github-actions[bot]]

💬 Your Product Feedback Has Been Submitted 🎉

Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users.

Here's what you can expect moving forward ⏩

• Your input will be carefully reviewed and cataloged by members of our product teams. 
  • Due to the high volume of submissions, we may not always be able to provide individual responses.
  • Rest assured, your feedback will help chart our course for product improvements.
• Other users may engage with your post, sharing their own perspectives or experiences. 
• GitHub staff may reach out for further clarification or insight. 
  • We may 'Answer' your discussion if there is a current solution, workaround, or roadmap/changelog post related to the feedback.

Where to look to see what's shipping 👀

• Read the Changelog for real-time updates on the latest GitHub features, enhancements, and calls for feedback.
• Explore our Product Roadmap, which details upcoming major releases and initiatives.

What you can do in the meantime 💻

• Upvote and comment on other user feedback Discussions that resonate with you.
• Add more information at any point! Useful details include: use cases, relevant labels, desired outcomes, and any accompanying screenshots.

As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities.

Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐

[sdarwin]

Upon further reflection, you don't want to DDoS yourself with a million updates at 200MB each.

But it's still a problem.

Other ideas,
Claude says: so I'd rank the asks: publish the schedule (free, fixes the surprise) > shrink or delta the package (fixes the economics) > lengthen the cadence (helps, but treats the symptom) > self-update on deprecation (now looks like the weakest of the four, given what the egress numbers show).

If the forced deprecation cycle was much slower, 1-2 years, and the updates could find some way to "delta the package", these would also help solve the issue.

[jackl57]

This makes a lot of sense. Letting deprecated runners update themselves while still refusing jobs seems like a much cleaner solution, especially for self hosted setups. The warning for ephemeral runners is also a good idea since otherwise the same issue would keep coming back with every new instance.

[tomm1990]

the deadlock is real - and your follow-up ranking is right.

deprecated → AccessDeniedException on the same message listener that would deliver self-update = “too old to be told how to stop being too old.” for baked AMIs that recurs every deprecation wave (the issue list you linked is the same bug with new version numbers).

today (ops), before any product change:

1. treat runner version as image content, not something ephemeral instances heal:
   • pin/latest in Packer/user-data to a current release (right now that’s around v2.337.0)
   • rebuild + roll the ASG/Launch Template when github ships - don’t expect in-place update on --ephemeral
2. make deprecation fail loud for orchestrators:

# already shipped (actions/runner#4285)
export ACTIONS_RUNNER_RETURN_VERSION_DEPRECATED_EXIT_CODE=1
# deprecated listener denial → exit 7 → replace the instance / fail the bootstrap

3. if a long-lived host is already refused: don’t wait for a job message - pull the release on the box (or just terminate and let a fresh AMI join). the queue won’t save you.

product (your ranked asks): +1 on publish the deprecation rule first (N releases / grace window). that alone stops the surprise outage for AMI bakers. delta packages / slower cadence help the economics. “self-update while refused” is still the right end-state for non-ephemeral, but only with the loud --ephemeral warning you described - otherwise you turn a hard fail into silent 200MB download tax on every boot.

so: bake current, exit-7 on deprecate, rebuild on schedule once github publishes one. the platform ask is “stop making operators reverse-engineer the cutoff from production.”