[P1][Fleet] Add and prove post-pilot two-runner capacity promotion

[Nickfost]

Summary

Add a reviewed, fail-closed post-pilot capacity-promotion procedure and prove one consumer workload at exactly two-runner capacity.

Context

• The existing pilot preflight intentionally hardcodes MIN=0 and MAX=1.
• Host-addition documentation permits later capacity growth only after measurement.
• No reviewed operational MAX>1 promotion procedure currently exists.
• Preserve the one-runner pilot gate; add a separate post-pilot capacity preflight.

Related:

• downstream-consumer repository implementation: private downstream evidence
• Completed downstream-consumer fleet contract: private downstream evidence
• Completed isolated Phase 3 proof: Phase 3: deploy one isolated live runner and prove one-job destruction #7

Required repository change

• Keep the strict pilot preflight at MIN=0, MAX=1.
• Add a separate explicit post-pilot capacity validator for a requested target MAX.
• Validate CPU, memory, Docker-filesystem, controller, credential-file metadata, scale-set identity, routing label, runner group, and idle/drained state.
• Fail closed for active managed runners, active fleet jobs, inadequate headroom, disk pressure, OOM evidence, or unrelated workloads.
• Distinguish configured MAX from effective controller MAX.
• Never print secrets or arbitrary environment contents.
• Add deterministic tests and a documented drain/change/recreate/verify procedure.

Live proof

• Back up the exact host-local configuration as a root-only file.
• Change only CI_FLEET_MAX_RUNNERS=1 to CI_FLEET_MAX_RUNNERS=2.
• Recreate only the controller and verify effective MIN=0, MAX=2.
• Dispatch exactly one separately authorized two-runner consumer proof: the downstream-consumer experimental Fast matrix.
• Record resource-budget, concurrency, runner lifecycle, cleanup, health, and retain-or-restore evidence.
• Restore the prior MAX when any capacity, isolation, cleanup, or health criterion fails.

Boundaries

• No unrelated controller or runner mutation.
• No runner-group, repository-access, routing-label, runner-limit, or MIN change.
• No MAX above two.
• No unrestricted Docker prune.
• No production, development, deployment, package, or image publication capability.

Acceptance criteria

• Pilot MAX=1 hard gate remains unchanged and tested.
• Separate post-pilot MAX=2 preflight is reviewed and tested.
• Resource accounting and objective rollback thresholds are documented before the live proof.
• Drain, backup, exact edit, controller-only recreation, effective-state verification, retain, and restore procedures are documented.
• Exactly one downstream consumer Fast matrix is observed at a maximum of two runners.
• Actual two-runner and two-job overlap is proven without a third runner.
• Every ephemeral runner and project resource is destroyed automatically.
• Controller and Docker health remain good with no OOM or residue.
• Final MAX retain-or-restore decision is recorded.

[Nickfost]

downstream-consumer proof issue: private downstream evidence

[Nickfost]

Live proof stopped at the pre-mutation capacity gate.

• Capacity PR: feat(ops): add post-pilot capacity promotion checks #31, merged as 3d118432dce86d702fa1b78ecadcfcee750bfd9a
• Host: rd-ci-fleet-01.localdomain
• Strict pilot preflight: PREFLIGHT_OK warnings=0
• First actionable failure: the reviewed MAX=2 CPU budget requires 11,200 millicpus, but the isolated host has 8,000 millicpus (3,200 millicpu shortfall).
• Controller remained healthy at configured/effective MIN=0, MAX=1.
• Host checkout, environment file, controller, runner group/access, routing label, and limits were not changed.
• No backup was needed, no runner or task resource existed, cleanup dry-run was empty, and no workflow was dispatched.

Rollback state: not required; the host never left the healthy MAX=1 baseline. This issue remains open for separately authorized host-capacity remediation or a revised resource model.

[Nickfost]

Reopened because the repository PR merged successfully but the separately tracked live MAX=2 proof stopped at the pre-mutation CPU-capacity gate. The capacity issue must remain open until that live criterion is satisfied under separate authorization.

[Nickfost]

The resumed capacity proof stopped before mutation at the authenticated Proxmox management-path gate.

• PR feat(ops): add post-pilot capacity promotion checks #31 remains merged on current clean main; deterministic capacity tests pass.
• Required guest policy remains two 4-CPU / 8,192-MiB runners plus controller, Docker, and OS reserves.
• Preferred VM 130 target is 16 vCPU / 32,768 MiB; minimum is 12 vCPU / 24,576 MiB.
• This Hermes session cannot inspect the physical node or VM: batch guest SSH is denied, no fleet/Proxmox SSH alias or key is available, no Proxmox client/auth environment exists, and retained evidence does not identify the node.
• Physical headroom therefore cannot be proven safely; VM 130 was not resized.
• No host checkout, environment, controller, runner-group, label, runner limit, or repository-access change occurred.
• MAX=2 was not applied. Last verified configured/effective state remains MIN=0, MAX=1.
• No workflow was dispatched and no cleanup was required.

One exact operator/OpenClaw resize handoff is preserved in private operational records. Resume the proof only after that bounded admission/resize handoff reports objective before/after node and VM evidence.

[Nickfost]

Capacity proof remains blocked; the controller has been returned to the safe pilot state.

• VM 130 is still 8 vCPU / 15,930 MiB. No authenticated Proxmox management path or hosting-node identity is available, so physical admission and resize were not attempted.
• One Fast matrix run appeared after the prior blocker: private downstream evidence. It ran while the guest remained undersized and configured/effective MAX was 2.
• After fast-forwarding /opt/ci-fleet to merged main 3d118432dce86d702fa1b78ecadcfcee750bfd9a, strict pilot preflight passed exactly PREFLIGHT_OK warnings=0; the target-MAX=2 pre-change capacity gate failed closed on CPU.
• No pre-promotion environment backup existed. A protected recovery backup of the discovered MAX=2 file was created at /etc/ci-fleet/backups/ci-fleet.env.before-forced-max1-rollback.20260718T014849Z (root:root, 0600), then only MAX was changed from 2 to 1.
• Only the controller was recreated. Its ready record is MIN=0/MAX=1; configured/effective values are 0/1; restart count and OOM are zero.
• Healthcheck and scoped cleanup dry-run pass. Managed runners and project containers/networks/volumes/run images/workspace-or-JIT-name residue are all zero; no manual cleanup or unrestricted prune ran.

Because the physical capacity gate was never satisfied and the one-shot proof lacks a prestarted continuous resource observer, MAX=2 cannot be retained. This issue remains open. No source defect or new capacity PR was found.

[Nickfost]

Beginning the operator-authorized replacement capacity proof.

Prior run 29622216381 is retained and is not being erased. It passed all 29/29 Fast task jobs, but it is non-qualifying because VM 130 was still undersized (8 vCPU / approximately 15.9 GiB) and the required continuous pre-dispatch observer was absent.

VM 130 has now been externally resized. Guest verification on the unchanged rd-ci-fleet-01 machine identity reports 32 online logical CPUs and approximately 62.73 GiB visible memory. Complete physical-node/OpenClaw admission documentation has not been found and will not be overstated.

This task authorizes exactly one replacement observed CI Fleet Experimental Matrix Fast proof after the reviewed MAX=1 and MAX=2 capacity gates pass. There will be no automatic retry. Any proof or observer failure restores the new protected MAX=1 backup and leaves this issue open.

[Nickfost]

The single authorized replacement proof is terminal and MAX=2 was not retained.

Positive evidence: VM 130 is guest-visible at 32 logical CPUs / 64,235 MiB; strict MAX=1 baseline passed; pre- and post-change MAX=2 capacity preflights passed; the protected backup is /etc/ci-fleet/backups/ci-fleet.env.before-observed-max2.20260718T055250Z; only the controller was recreated. A replacement downstream-consumer run, retained in private evidence, tested 19538d0a588c06209fc75e482f2ac21f44bac724, plan succeeded, and all 29/29 task jobs succeeded. Observed peaks were exactly two task jobs and two managed runners; 29 runner creates and 29 destroys were recorded; lifecycle peak was two and no third runner event or sample occurred. Final project and runner residue was zero without manual cleanup. Host peaks stayed within resource thresholds: CPU 14.64%, minimum available memory 62,536 MiB, Docker/root disk 5%, no OOM, controller and Docker healthy.

The run is nevertheless non-qualifying. The required five-second host sampler recorded a material 32.757-second gap (12 gaps above 7.5 seconds, four above 10 seconds), so observer continuity failed. Its conservative workload classifier also recorded one or two running containers it could not attribute in 56 samples, so the no-unrelated-workload criterion was not proven. Separately, the complete OpenClaw physical-node admission/reserve report remains unavailable; only the guest resize was independently observed.

Per the no-retry policy, no second run was dispatched. The new protected backup was restored, only the controller was recreated, and configured/effective MIN/MAX are now 0/1; strict preflight returned PREFLIGHT_OK warnings=0, healthcheck passed, cleanup dry-run was empty, and managed runners remain zero. This issue remains open. No Full matrix, fixed pilot, development, production, deployment, VM, node, or other guest operation was performed by Hermes.

[Nickfost]

Repository capacity-gate reconciliation is complete; the live proof remains blocked and no infrastructure was mutated.

• Generic reviewed-target preflight merged in fix(capacity): validate reviewed runner targets #47 as 84859c53c8d8fa5a0e2dfef568b61df805536088; exact PR head 2b00989c5c308b6e6ab7abfad9d348f2d0871914 had green CI, a clean exact-head Codex review, and all three accepted review threads resolved.
• Private desired state pinned that engine in a reviewed private desired-state change as fa94728 after green strict validation and clean exact-head Codex review.
• The MAX=6 private reconciliation runbook added explicit pre/post capacity gates in a reviewed private desired-state change as d1ddc49 after green strict validation and clean exact-head Codex review.

No VM 130, controller, runner, runner group, secret, or production change was performed. The remaining gate is an explicitly authorized live reconciliation/proof with recovery access, continuous full-target observation, and rollback on any failed criterion.

[Nickfost]

Closing as completed and superseded by the implemented single-controller reconciliation architecture tracked through #53. The acceptance target—one active controller managing six concurrent ephemeral runners—was demonstrated by the six-way canary (private pilot evidence), followed by successful scale-down and scale-to-zero proof (private pilot evidence). The rollout also proved SSH-disabled reconciliation, rollback, and power-cycle survival.