Phase 3: deploy one isolated live runner and prove one-job destruction

[Nickfost]

Outcome

Deploy one controller on one isolated Docker host, register one organization-level experimental scale set, run one manually dispatched read-only job, and prove that the runner and job resources disappear afterward.

sequenceDiagram
  participant O as Operator
  participant G as GitHub
  participant C as ci-fleet controller
  participant R as ephemeral runner
  O->>C: start with MIN=0 MAX=1
  O->>G: dispatch read-only pilot
  G->>C: desired runners = 1
  C->>R: create with one JIT config
  R->>G: execute pilot
  G->>C: job completed
  C->>R: capture logs and destroy
  O->>C: verify no residue

Loading

Prerequisites

• Select one isolated Docker host and assign a stable fleet instance ID
• Create an organization-owned GitHub App with Metadata read-only and Self-hosted runners read/write
• Install the App on only the target organization
• Store the PEM only at /etc/ci-fleet/secrets/github-app.pem
• Create/restrict the experimental runner group to the pilot repository
• Complete the committed preflight without warnings

Pilot

• Start exactly one unique host scale set with MIN=0, MAX=1
• Confirm zero runner containers before dispatch
• Dispatch the committed read-only pilot workflow
• Confirm exactly one runner appears and accepts the job
• Confirm the job has only contents: read
• Confirm the runner container is destroyed after completion
• Confirm no job containers, networks, volumes, workspaces, or credentials remain
• Save controller and host-health evidence without secret values

Rollback

• Stop the controller
• Verify/delete only this host's experimental scale set
• Run scoped cleanup in dry-run mode, inspect, then apply if needed
• Remove the App installation and local PEM if the pilot is abandoned
• Confirm existing downstream-consumer runners remain unchanged

Hard gates

• Do not target downstream consumers yet.
• Do not allow public repositories or fork-originated code to use the group.
• Do not raise maximum concurrency above one.
• Do not use a PAT.
• Stop immediately if preflight, permissions, lifecycle cleanup, or residue checks fail.

Repository preparation evidence

• Unique scale-set identity per host plus shared routing-label model implemented
• Public ci-fleet repository excluded from live runner access
• Private-repository pilot workflow template committed
• GitHub App permissions and secret-storage runbook committed
• Host preflight, maintenance timers, and rollback procedure committed
• Inert controller and runner image validation passed

Evidence: commit 8447ef8, validation run 29287297659.

The issue remains open at the host, private pilot repository, runner-group, and GitHub App authorization gate.

[Nickfost]

Phase 3 live-pilot preflight update (rd-ci-fleet-live-pilot-v1):

• ci-fleet is clean current main at c5a6bd4cdb3add2bc7ae9a0fcea27c9cf11814b1.
• A dedicated private pilot repository now contains exactly one active, manual-only workflow. It uses only contents: read, requests only docker-ci-experimental, and pins actions/checkout to an immutable commit. Dispatch count remains zero.
• The existing organization-owned rd-ci-fleet-01 GitHub App installation has Metadata read-only, organization Self-hosted runners read/write, and repository Administration no access.
• Stopped at two hard gates: the authenticated GitHub CLI OAuth token lacks the admin:org scope required to inspect/create/restrict the organization runner group, and the designated pilot host is ICMP-reachable but refuses SSH on port 22. Host isolation, services, containers, mounts, configuration, PEM ownership/mode, preflight, and image state therefore cannot yet be verified.

No controller was started, no workflow was dispatched, no runner or scale set was created by this session, and no cleanup apply was run. Existing downstream-consumer runners, development, and production were not modified. Issue #7 remains open.

Resume only after an operator approves admin:org for the existing GitHub CLI session and restores an authorized secure management path to the designated isolated host. Do not use the Default runner group or broaden repository access.

[Nickfost]

Phase 3 live lifecycle proof passed with one permitted root-cause retry.

• GitHub authorization includes admin:org; trusted-private-ci-experimental is selected-repositories-only, allows no public repositories, and authorizes only the private pilot repository. Default and downstream consumers are excluded.
• Existing isolated controller was reconciled without duplication: docker-ci-rd-ci-fleet-01, label docker-ci-experimental, MIN=0, MAX=1. Committed preflight ended exactly PREFLIGHT_OK warnings=0.
• Initial run 29451460747 exposed a root-owned volume/non-root proof-writer defect; its runner and resources were destroyed. The one-line root-cause fix merged in fix: allow live pilot to write Docker volume #29 and was copied to the pilot.
• The single permitted retry succeeded; its evidence remains in the private pilot record: one queued job, one JIT-backed ephemeral runner, one accepted job, contents: read, Docker capability pass, scoped container/network/volume proof pass, then runner destruction.
• Final state: zero runner/job containers, pilot networks, pilot volumes, workspaces, persisted JIT/token artifacts, or expired cleanup candidates. Scoped cleanup dry-run listed nothing; apply was unnecessary. Healthcheck passed with the controller running and disk at 5%.
• Rollback is documented. Existing downstream-consumer runners remained unchanged; neither workflow was dispatched by this task.

Current ci-fleet main: 674fe318f851e750dcc31f538ac927331b8e4df4.