What happens
reconcile-image --apply refuses unless the target member has zero instances:
gha-fleet: reconcile golden image: image build requires cluster member "gha-runner-1"
to be empty; active instances there: nddev-ajedtqvsnlch,nddev-r5fjtqnrr0iu
The isolation itself is right — the build should own the machine. What is missing is any supported way to get the machine empty.
Why waiting is the only option today
The placement scriptlet skips a member unless:
if member.config.get("user.gha_pressure.state", "") != PRESSURE_OPEN:
continueso setting that key would drain a member — but the pressure observer owns the field and overwrites it on its next publish. There is no maintenance flag the scriptlet honours and no drain command.
In practice a build means polling for a natural gap and racing new placements. On 2026-08-28 that took 37 polls (~28 minutes) on a moderately loaded fleet, and a job can still land between the check and the build starting.
Why this was invisible until now
Builds used to run on a dedicated host: direct-jit-image-stage-audit.json and sccache-image-stage-audit.json both record "host": "server-nddev-amsterdam". That host no longer has incus or gha-fleet installed, so the procedure quietly moved onto cluster members, where the emptiness precondition suddenly matters — and no record names where the current b13 image was actually built.
Options
- Restore a dedicated build host. Isolation for free, no drain needed, no production capacity consumed.
server-nddev-amsterdam has 8 cores, 16 GB and 293 GB free.
- Add a maintenance state the placement scriptlet honours and the pressure observer does not clobber, plus a command to enter and leave it.
The first is closer to how this already worked and does not add a state machine to placement.
Found while building image b14 (Flutter SDK).
https://claude.ai/code/session_01HMPUfMTrzqwrcgDCFvjnFG
What happens
reconcile-image --applyrefuses unless the target member has zero instances:The isolation itself is right — the build should own the machine. What is missing is any supported way to get the machine empty.
Why waiting is the only option today
The placement scriptlet skips a member unless:
so setting that key would drain a member — but the pressure observer owns the field and overwrites it on its next publish. There is no maintenance flag the scriptlet honours and no drain command.
In practice a build means polling for a natural gap and racing new placements. On 2026-08-28 that took 37 polls (~28 minutes) on a moderately loaded fleet, and a job can still land between the check and the build starting.
Why this was invisible until now
Builds used to run on a dedicated host:
direct-jit-image-stage-audit.jsonandsccache-image-stage-audit.jsonboth record"host": "server-nddev-amsterdam". That host no longer hasincusorgha-fleetinstalled, so the procedure quietly moved onto cluster members, where the emptiness precondition suddenly matters — and no record names where the currentb13image was actually built.Options
server-nddev-amsterdamhas 8 cores, 16 GB and 293 GB free.The first is closer to how this already worked and does not add a state machine to placement.
Found while building image
b14(Flutter SDK).https://claude.ai/code/session_01HMPUfMTrzqwrcgDCFvjnFG