Skip to content

feat(fleet): make draining a member an operation, not a remembered sequence - #279

Merged
rldyourmnd merged 1 commit into
mainfrom
feat/member-drain
Aug 29, 2026
Merged

feat(fleet): make draining a member an operation, not a remembered sequence#279
rldyourmnd merged 1 commit into
mainfrom
feat/member-drain

Conversation

@rldyourmnd

Copy link
Copy Markdown
Contributor

Closes #262.

Taking a member out of service worked, but only if you knew three things in the right order — and getting it wrong looked like it had worked.

The member's gate is owned by the pressure publisher, which reasserts it every eleven seconds. Closing the gate — or writing scheduler.instance by hand — without first stopping gha-pressure-gate.timer is undone within one cycle: observed live, set at 17:02:27 and read back open at 17:02:52.

gha-fleet drain-member does the sequence: stop the timer, close the gate through its owner, then wait for the jobs already running to finish.

  • It never stops a worker. If they outlast the deadline it reports the member is still occupied, names what is on it, and exits non-zero so a caller scripting a reboot sees it. Ending someone's build is not this command's decision.
  • --restore hands the member back by republishing from live pressure rather than forcing the gate open, so a member genuinely under pressure stays closed on its own merits.
  • Without --apply it reports what it would do and what the member is carrying, and changes nothing.

One honesty detail worth reading. The restore result reports the scheduler value the publisher actually wrote, which is manual — hysteresis holds a just-closed member shut for a cycle or two. gate_republished says what happened; claiming the gate was open there would have been the more convenient field and the false one.

Verified on live hardware, twice on gha-runner-3: gate allmanual with the timer stopped and occupancy reported, then republished and open again at t+45s and t+60s under its owner. make verify green; nine unit tests cover the ordering (timer before gate), the wait, the timeout that ends nobody's job, the dry run, and the refusal to close the gate if the timer cannot be stopped.

https://claude.ai/code/session_01NpzpgiRaxi5mHVTMoRYndt

…quence

Taking a member out of service worked, but only if you knew three things in
the right order, and getting it wrong looked like it had worked.

The member's gate is owned by the pressure publisher, which reasserts it every
eleven seconds. Closing the gate -- or writing scheduler.instance by hand --
without first stopping gha-pressure-gate.timer is undone within one cycle:
observed live, set at 17:02:27 and read back open at 17:02:52.

`gha-fleet drain-member` does the sequence: stop the timer, close the gate
through its owner, then wait for the jobs already running to finish. It never
stops a worker. If they outlast the deadline it reports that the member is
still occupied and names what is on it, and exits non-zero so a caller
scripting a reboot sees it -- ending someone's build is not this command's
decision. `--restore` hands the member back, republishing from live pressure
rather than forcing the gate open, so a member genuinely under pressure stays
closed on its own merits.

Without --apply it reports what it would do and what the member is carrying.

The restore result says gate_republished and reports the scheduler value the
publisher actually wrote, which is "manual": hysteresis holds a just-closed
member shut for a cycle or two. Claiming the gate was open there would have
been the more convenient field and the false one. Proven twice on gha-runner-3:
gate all -> manual with the timer stopped, then republished and open again at
t+45s and t+60s under its owner.

Closes #262.

Claude-Session: https://claude.ai/code/session_01NpzpgiRaxi5mHVTMoRYndt
@rldyourmnd
rldyourmnd merged commit d94d75c into main Aug 29, 2026
10 checks passed
@rldyourmnd
rldyourmnd deleted the feat/member-drain branch August 29, 2026 20:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

image build has no supported way to free a cluster member

1 participant