docs(mpc): consensus-mode installation + operator runbook + smoke test (closes #1)

[abhicris]

Closes #1.

Follow-up to the issue I filed: rewrites docs/INSTALLATION.md to lead with consensus-mode (ZAP transport, no NATS/Consul deps), moves the legacy NATS+Consul transport to an appendix with a deprecation banner + migration checklist, and lands the three companion docs / scripts the issue asked for.

Files

docs/INSTALLATION.md (rewrite)

• Leads with the consensus-mode quick start that matches the README (mpcd start --node-id ... --peer ..., no external message bus).
• Full flag reference for mpcd start — node id, listen/api/api-listen ports, threshold, peers, --hsm-provider, --hsm-signer, --hsm-attest, log level — cross-checked against cmd/mpcd/main.go on main.
• Peer manifest + identity generation via the mpc CLI (matches cmd/mpc/), event initiator setup, Docker Compose example aligned with the actual compose.yml (SQLite default, Postgres+Valkey commented), Kustomize pointer for k8s/.
• Production hardening checklist: environment: production + cloud HSM provider (the startup guard in resolveZapDBPassword rejects env/file in production), encrypted identities, share rotation cadence, off-host backups.
• Appendix A — legacy NATS+Consul transport retained for back-compat, including a step-by-step migration to consensus mode.

docs/RUNBOOK.md (new)

Operator checklist for the day-to-day:

• Daily / periodic health cadence table.
• Kubernetes liveness + readiness probe YAML with rationale on thresholds and initialDelaySeconds.
• Key rotation: share refresh (public key stays, via POST /v1/wallets/{id}/reshare) vs. wallet rotation (new key); operator preconditions / post-conditions / checklist for each.
• Node add / remove without resharing (new wallets can use the expanded cohort; existing wallets are cohort-bound).
• Backup + restore from age-encrypted ZapDB archives; the important caveat that a backup is share-bound to identity + HSM password, not a key backup.
• Incident triage: ready: false, signing timeouts, suspected node compromise (isolate → reshare → rotate password → destroy backups), HSM/KMS outage.
• Rollback policy (no mixed-version cohorts for signing).
• Useful one-liners for cluster-wide status, audit tailing, key counts.

docs/HEALTH.md (new)

Spec for GET /healthz on the internal API listener:

• Response schema pulled from the healthHandler in cmd/mpcd/main.go — status, node_id, mode, expected_peers, connected_peers, ready, threshold, version.
• Status code contract: 200 when ready: true, 503 when ready: false.
• Recommended probe intervals + initialDelaySeconds and the rationale for each.
• Sampling cadence table by consumer (K8s probes, external monitoring, incident triage) and a don't-poll-faster-than note.
• Example Prometheus alert rules.
• Explicit what /healthz does NOT check section so operators add canaries (see scripts/smoke-test.sh) instead of over-instrumenting the probe.

scripts/smoke-test.sh (new, executable)

End-to-end canary:

• Generates a peer manifest (mpc generate-peers -n 3), per-node identities (mpc generate-identity), an event initiator (mpc generate-initiator).
• Boots 3 mpcd nodes on loopback (:19651-3 P2P, :19800-2 API).
• Waits for /healthz ready=true on each node (60 s timeout, configurable).
• Fires POST /keygen with a random org + wallet id, requires result_type == "success" and a non-empty ecdsa_pub_key.
• Signs a random 32-byte nonce, structurally verifies signature length (64 or 65 bytes).
• SKIP_BOOT=1 points the same test at a running staging cluster for periodic canary use.
• Traps to tear down processes and WORKDIR on exit.
• Requires: mpcd, mpc, curl, jq, openssl.

Cross-checks done

• Flag names + env var names verified against cmd/mpcd/main.go (--node-id, --listen, --api, --api-listen, --hsm-provider, MPC_HSM_PROVIDER, --hsm-attest, MPC_HSM_ATTEST, etc.).
• Endpoint paths verified against mux.HandleFunc calls on main: /healthz, /keys, /backup, /keygen. Response fields verified against the healthHandler map literal.
• config.yaml schema verified against the existing config.yaml in the repo (mode, environment, mpc_threshold, max_concurrent_keygen, db_path, backup_enabled, backup_period_seconds, backup_dir, event_initiator_pubkey).
• Build targets verified against the Makefile (mpcd and mpc binaries).
• Docker Compose example mirrors the actual compose.yml (SQLite default, Postgres + Valkey commented out).
• K8s manifest reference matches k8s/mpc-statefulset.yaml (5 replicas, readOnlyRootFilesystem, node{0..4}@mpc-node-{0..4}.mpc-node-headless.lux-mpc.svc:9651).
• Production guard mentioned in INSTALLATION.md matches the explicit fatal in resolveZapDBPassword(): production + env/file provider is rejected.

Happy to split any of these into smaller PRs if preferred — e.g. land INSTALLATION + HEALTH first and RUNBOOK + smoke-test as a follow-up. LMK.