Skip to content

Let the listener own the EphemeralRunner Running phase transition - #4646

Open
nikola-jokic wants to merge 1 commit into
nikola-jokic/lazy-copyfrom
nikola-jokic/listener-owns-running-phase
Open

Let the listener own the EphemeralRunner Running phase transition#4646
nikola-jokic wants to merge 1 commit into
nikola-jokic/lazy-copyfrom
nikola-jokic/listener-owns-running-phase

Conversation

@nikola-jokic

Copy link
Copy Markdown
Collaborator

Stacked on #4580. First of two PRs that split #4582 apart.

af55427f in #4582 bundled two unrelated things: a behavioural change to who
owns the Running phase, and a workqueue optimisation. This PR is the
behavioural half. The predicates follow separately so that an optimisation is
never reviewed together with a semantic change.

The problem

updateRunStatusFromPod derived EphemeralRunner.Status.Phase straight from
the pod phase, so a runner flipped to Running the moment its pod started,
whether or not it had picked up a job. Running therefore meant "the pod is
up", not "the runner is busy".

That matters for scale-down. newEphemeralRunnersByStates groups runners into
pending and running, and deleteIdleEphemeralRunners walks pending first
and running second. Because every started runner was Running, the two
buckets carried no information about which runners were safe to remove.

The change

The listener already knows exactly when a job is assigned to a named runner, so
the transition moves there:

  • HandleJobStarted reads the runner first and only promotes it to Running
    when it is not Failed, Succeeded or Outdated and is not being deleted.
    The phase goes out in the same status merge patch as the job fields it
    already writes.
  • The listener role gains get on ephemeralrunners for that read, split out
    from the existing ephemeralrunners/status rule so status keeps only
    patch.
  • updateRunStatusFromPod still publishes the initial Pending phase while the
    pod starts, and no longer promotes to Running.

Runners waiting for work now stay Pending, so scale-down drains genuinely idle
runners before ones executing a job.

Note for review

The read and the patch are two calls, so a runner can reach a terminal phase in
between and still receive Running. The window is small and the ephemeral
runner set reaps terminal runners regardless, but it is worth a look if you want
this tightened with a resource version precondition.

Testing

go test ./controllers/... ./cmd/... with envtest, all green. New coverage:
TestHandleJobStarted in the scaler, plus envtest cases asserting the
controller does not set Running from pod status and that readiness still
tracks the pod.

@nikola-jokic
nikola-jokic force-pushed the nikola-jokic/listener-owns-running-phase branch from 0f307dd to cc67813 Compare September 10, 2026 12:07
@nikola-jokic
nikola-jokic added this pull request to stack #4645 September 10, 2026 12:07
Copilot AI lite review requested due to automatic review settings September 10, 2026 12:52
@nikola-jokic
nikola-jokic force-pushed the nikola-jokic/listener-owns-running-phase branch from cc67813 to 82a24f7 Compare September 10, 2026 12:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Address the phase-transition race conditions and update the Running phase documentation.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Lite
Findings: 1 High severity · 1 Low severity

New issues introduced by this change (2)
Severity Finding
High severity cmd/​ghalistener/​scaler/​scaler.go — The GET and this status merge patch are not tied by a resourceVersion precondition. If the runner…
Low severity cmd/​ghalistener/​scaler/​scaler.go — This changes the public meaning of Running to indicate an assigned/busy runner, but the API type…
What changed in this PR

This PR moves EphemeralRunner’s Running phase transition from pod reconciliation to listener job assignment.

Changes:

  • Adds listener reads and phase-aware status updates.
  • Keeps pod reconciliation from promoting runners to Running.
  • Updates RBAC and test coverage.
File Review summary
controllers/​actions.github.com/​resourcebuilder.go Adds listener read access for ephemeral runners.
controllers/​actions.github.com/​ephemeralrunnerset_controller_test.go Covers listener-owned phase transitions.
controllers/​actions.github.com/​ephemeralrunner_controller.go Retains Pending initialization; its phase write needs resource-version protection.
controllers/​actions.github.com/​ephemeralrunner_controller_test.go Tests pod phase and readiness behavior.
cmd/​ghalistener/​scaler/​scaler.go Sets Running on assignment; requires conditional updates to prevent terminal-state races and updated phase documentation.
cmd/​ghalistener/​scaler/​scaler_test.go Tests listener phase transitions and job updates.
Suppressed comments (1)

controllers/actions.github.com/ephemeralrunner_controller.go:858

  • This guard only observes the reconciler's cached object. If that read sees an empty phase, the listener can patch Running before this pod reconcile writes its status; the subsequent merge patch then includes Pending and can overwrite the listener-owned Running phase because it is not an optimistic-lock patch. Protect this initial phase write with a resource-version/refetch strategy so the controller cannot regress a phase that the listener has already advanced.
	phase := ephemeralRunner.Status.Phase
	if pod.Status.Phase == corev1.PodPending && phase == "" {
		phase = v1alpha1.EphemeralRunnerPhasePending
	}

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +177 to +181
if currentRunner.DeletionTimestamp == nil &&
currentRunner.Status.Phase != v1alpha1.EphemeralRunnerPhaseFailed &&
currentRunner.Status.Phase != v1alpha1.EphemeralRunnerPhaseSucceeded &&
currentRunner.Status.Phase != v1alpha1.EphemeralRunnerPhaseOutdated {
patchRunner.Status.Phase = v1alpha1.EphemeralRunnerPhaseRunning
Comment on lines +177 to +181
if currentRunner.DeletionTimestamp == nil &&
currentRunner.Status.Phase != v1alpha1.EphemeralRunnerPhaseFailed &&
currentRunner.Status.Phase != v1alpha1.EphemeralRunnerPhaseSucceeded &&
currentRunner.Status.Phase != v1alpha1.EphemeralRunnerPhaseOutdated {
patchRunner.Status.Phase = v1alpha1.EphemeralRunnerPhaseRunning
@nikola-jokic
nikola-jokic force-pushed the nikola-jokic/listener-owns-running-phase branch from 82a24f7 to 3cad613 Compare September 10, 2026 16:20
@nikola-jokic
nikola-jokic force-pushed the nikola-jokic/listener-owns-running-phase branch from 3cad613 to 4afa7e1 Compare September 10, 2026 20:56
The controller derived Status.Phase directly from the pod phase, so a runner
became Running as soon as its pod started, whether or not it had picked up a
job. That made Running mean "the pod is up" instead of "the runner is busy",
and it left the EphemeralRunnerSet scale-down path unable to tell an idle
runner from one that is executing a job.

The listener already knows when a job is assigned to a specific runner, so
move the transition there. HandleJobStarted now reads the runner first and
only promotes it to Running when it is not terminal (Failed, Succeeded or
Outdated) and not being deleted, then patches the phase alongside the job
fields it already writes. The listener role gains "get" on ephemeralrunners
for that read.

On the controller side updateRunStatusFromPod keeps publishing the initial
Pending phase while the pod is starting, and no longer promotes to Running.
Runners waiting for work now stay Pending, so scale-down picks them before
runners that are actually executing a job.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Critical and moderate correctness and concurrency issues remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Lite
Findings: 3 High severity · 1 Low severity

New issues introduced by this change (2)
Severity Finding
High severity cmd/​ghalistener/​scaler/​scaler.go — Use the lowercase EphemeralRunner resource name
High severity controllers/​actions.github.com/​ephemeralrunner_controller.go — Stale pod reconciliation can reset the listener-owned phase
Pre-existing issues (2)
Severity Finding
High severity cmd/​ghalistener/​scaler/​scaler.go — The GET and this status merge patch are not tied by a resourceVersion precondition. If the runner… View comment
Low severity cmd/​ghalistener/​scaler/​scaler.go — This changes the public meaning of Running to indicate an assigned/busy runner, but the API type… View comment

Get().
Prefix("apis", v1alpha1.GroupVersion.Group, v1alpha1.GroupVersion.Version).
Namespace(w.config.EphemeralRunnerSetNamespace).
Resource("EphemeralRunners").
Comment on lines +855 to +858
phase := ephemeralRunner.Status.Phase
if pod.Status.Phase == corev1.PodPending && phase == "" {
phase = v1alpha1.EphemeralRunnerPhasePending
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants