feat(broker): credential broker laptop CLI + Ansible roles (spec 005)

[pofallon]

Summary

• Removes long-lived developer credentials from Remo instances. Provisioning creds now come from laptop fnox (FR-006); per-instance bootstrap tokens are delivered via provider-specific transport (SSH push / IMDS / bind-mount); a per-project allowlist manifest plus on-instance broker socket gates secret access from devcontainers.
• New CLI surface: remo init --backend …, remo {incus,proxmox} add-node, remo audit, remo rotate-bootstrap, pre-destroy bootstrap-token revoke on remo destroy (FR-020, exit 5), passive overdue-rotation reminder on every invocation.
• Cross-repo dep: BROKER_PINNED_VERSION = "0.1.0" — Rust broker daemon owned by get2knowio/remo-broker, consumed as a signed binary release. This PR is the laptop CLI + Ansible half.

Scope

Phases delivered (all tasks in specs/005-credential-broker/tasks.md):

• 1 — Setup: jsonschema dep, broker version constants, vendored manifest schema baseline
• 2 — Foundational: core/fnox.py, core/nodes.py (0600 enforced), core/manifest.py, core/broker_install.py, models/node.py, models/manifest.py, providers/broker.py skeleton, three Ansible roles (broker_install + 3× bootstrap_token_*), group_vars migrated to lookup('pipe', 'fnox get …')
• 3 — US1 (bootstrap delivery): per-provider transport — Hetzner SSH stdin push, AWS per-dev-per-region IAM instance profile with scoped secretsmanager:GetSecretValue, Incus lxc config device add … readonly=true, Proxmox pct set -mp0 …,ro=1. add-node CLI for self-hosted nodes. core/devcontainer.py socket helper + language auto-synthesis.
• 4 — US2 (multi-device + reboot survival): systemd unit with Restart=on-failure, RestartSec=5s, LoadCredential=bootstrap-token:…; post-install verify; no device-bound state.
• 5 — US3 (no creds on instance): remo init with backend selection, fnox-missing rejection (exit 3), age-git downgrade warning (exit 2), interactive-identity rejection (exit 4); Hetzner env reads → _get_hetzner_api_token via fnox.
• 6 — US4 (manifest + audit): manifest discover/synthesize/validate with JSON-Schema + TOML position info, .gitignore ensure-block, remo audit CLI with --tail/--since/--json.
• 7 — US5 (rotation + revocation): all four backend mint/revoke impls (1Password SCIM, Vault accessor, AWS IAM deny-all + teardown, age-git warn-only), remo rotate-bootstrap with cadence reading + 1-hour fresh-skip + partial-success exit 7, pre-destroy revoke hook wired into Hetzner, passive overdue reminder.
• 8 — US6 (devcontainer auto-synth): language-marker priority table (Node/Python/Rust/Go/Ruby/Ubuntu), socket mount in every synthesized json, exit-warning state.yml plumbing.
• 9 — Polish: docs/credential-broker.md (threat model + operator runbook), README updates, scripts/grep-credential-leaks.sh enforcing FR-005/FR-006, version bump 2.0.0rc4 → 2.1.0rc1.

Spec & supporting work:

• Authoritative spec moved from docs/remo-fnox-spec.md to specs/005-credential-broker/ alongside plan/research/data-model/contracts/quickstart/tasks.
• Pre-existing 005-provider-snapshots specs renumbered to 007 to free the slot.

Deferred / partial ([~] in tasks.md)

• T039 / T063 / T086 — laptop-side cli/shell.py hooks are a no-op because project workspaces live on the instance; helpers (core.devcontainer.ensure_socket_mount, core.manifest.synthesize_default, core.devcontainer.synthesize_devcontainer_json) are reachable for instance-side automation when remo-broker ships its hooks.
• T074 — pre-destroy revoke wired into Hetzner; AWS/Incus/Proxmox destroy() flows can adopt the same core/broker_revoke.py hook.
• T078 — cadence reader works for Hetzner labels; --cadence-days N writer on remo {provider} create not yet exposed.
• T088 — explicit "exit to instance shell" menu option lives in the server-side picker script; state.yml plumbing is laptop-side.
• T097 — quickstart end-to-end runs against real cloud accounts deferred to pre-GA validation.

Test plan

• uv run pytest — 711 passed (was 644; +67 new tests)
• uv run ruff check src/remo_cli — all checks passed
• bash scripts/grep-credential-leaks.sh — credential-leak grep gate: ok
• uv run mypy src/remo_cli — clean modulo pre-existing missing third-party stubs (jsonschema/yaml/boto3)
• Live quickstart on real AWS + Hetzner + Incus + Proxmox (gated on remo-broker daemon release)
• Verify remo init --backend 1password round-trip on a fresh devcontainer

🤖 Generated with Claude Code

[pofallon]

Update: three follow-up commits landed since the review

Pushed 0daf172, c326727, 795e086 resolving the 15 review findings in #33 and pushing the credential-broker rotation lifecycle to completion. Branch now ships:

1. 0daf172 — Resolve 15 code-review findings (closes Code review findings: 15 defects from /code-review of #32 (005-credential-broker) #33). Token-leak in mint exceptions, Hetzner label keys swap (: → _), per-instance AWS IAM role (not per-developer), result_callback hook now actually fires past sys.exit, broker_config bridge so remo init isn't a no-op, JSONC string-aware stripper, aware-UTC datetime normalization, SSH host-key verification before bootstrap-token push.
2. c326727 — Admin-socket client + complete rotation lifecycle. New core/broker_admin.py (NDJSON-over-SSH bridge to /run/remo-broker/admin.sock); remo rotate-bootstrap now pushes the fresh token + tells the broker to reload before revoking the old token. Previously the daemon ended up with a revoked credential.
3. 795e086 — Per-instance rotation cadence at create time (T078). --cadence-days N on remo {hetzner,aws,incus,proxmox} create. Hetzner + AWS fully wired (label/tag at create + last_rotation/token_id on rotate + AWS-tag reader). Incus persists cadence; Proxmox emits a clear deferred warning.

Coordination: companion PR get2knowio/remo-broker#6 adds the release workflow that unblocks the broker_install Ansible role's current 404.

Tests: 764 passing, ruff clean.

Deferred for Phase 3: Incus + Proxmox rotation flow (cadence write lands here; admin-socket reload via container exec is next).

[pofallon]

Phase 3 landed — `644c131`: Incus + Proxmox rotation lifecycle

Rotation flow now complete on all four providers. Same shape as Hetzner:

• Incus: _push_bootstrap_token_to_container via incus exec <container> -- install … (localhost fast-path + remote SSH); admin-socket reload via incus exec … -- sudo python3 … (broker_admin.rotate_bootstrap_via_incus). Metadata in host-side user.remo.* config keys.
• Proxmox: mirror via pct exec <vmid> -- … (always over SSH; no localhost flavour). Metadata in in-container files under /etc/remo-broker/ — Proxmox LXC has no host-side user.* primitive, but the in-container file has the same lifetime as the bootstrap-token file the broker already consumes there.

broker_revoke._lookup_token_id now reads both stores. Empty stdout = "no token minted yet" (None, success). Non-zero exit = transport failure (TokenLookupError, blocks destroy unless --force).

providers/{incus,proxmox}.destroy() now build the pre-revoke candidate KnownHost with the right fields so _lookup_token_id can resolve the exec target.

Tests: 791 passing, ruff clean. 13 new tests across test_broker_admin.py, test_rotate.py, test_broker_revoke.py, and two new files test_incus_token_push.py / test_proxmox_token_push.py. The "unsupported provider" warning test is retired (no unwired provider remains).

[pofallon]

`deb72a3`: un-gate broker_install now that remo-broker v0.1.0 ships

remo-broker #6 merged and tag `v0.1.0` pushed — release workflow currently running (will publish the cross-arch binaries + `.sha256` files + `remo-broker.v1.json` schema to https://github.com/get2knowio/remo-broker/releases/tag/v0.1.0).

This commit drops the temporary `REMO_BROKER_BACKEND` env-gate that Finding 1 added on the `broker_install` Ansible role include in the four `*_configure.yml` plays. The systemd unit installs enabled-but-stopped when no token is present (Finding 9), so running `broker_install` on every `create` is harmless even without a backend configured.

The `bootstrap_token_{file,imds,mount}` assertion roles stay gated — those correctly only run when the user has opted in to a backend and minted a token.

Tests: 791 passing.

[pofallon]

Closing without merge.

End-to-end testing on 2026-05-29 against a real Proxmox container surfaced a categorical mismatch between this branch's architecture (external secret backend + bootstrap-token-on-instance + broker-fetches-on-demand) and the threat model it set out to defend.

Specifically: the bootstrap token at /etc/remo-broker/bootstrap-token is itself the kind of "credential laying around" the origin-story principle (@nateberkopec) was specifically scrubbing. An AI agent or supply-chain attacker in the devcontainer who escalates to read that file gets every secret behind it, bypassing the per-project manifest gate.

Six findings from the test session reinforced this — most notably that age-git was advertised as a downgrade path but errors on first use; that no create-time initial-token mint+push was wired (rotate-bootstrap was doing double duty); and that broker_install was never being included in the *_site.yml playbooks the create flow actually runs (fixed in 803667a for branch-integrity, but moot given the supersede).

Superseded by spec 006: laptop-push model.

• remo: specs/006-credential-broker-laptop-push/ (spec, plan)
• remo-broker: specs/002-laptop-push-secrets/ (spec) — supersedes 001-broker-daemon

The redesign: laptop encrypts a project-scoped secret bundle (age, X25519 + ChaCha20-Poly1305) and pushes it to the instance over SSH; broker decrypts in memory using a key loaded via systemd's LoadCredentialEncrypted= (TPM2 → host-key → plaintext-mode-0600 fallback ladder); devcontainers fetch via the existing per-project Unix socket protocol with manifest allowlist enforcement. No external backend, no on-disk bootstrap token, no fnox-core dependency (which cascade-deletes Cross.toml, all 6 deny.toml advisory ignores, and shrinks the binary from ~32 MiB toward the 15 MiB NFR target).

~60% of the laptop-side code in this PR carries forward to the new branch (broker chassis, manifest model, install machinery, fnox-as-laptop-store, per-provider create/destroy plumbing, SSH/incus exec/pct exec bridges). On the broker side, ~80% of the chassis carries forward; src/backend.rs and src/bootstrap.rs delete entirely.

The 005-credential-broker branch is being kept intact as historical reference; not deleting.