fix(kiloclaw): prevent 401s after API key rotation via SecretRef

[pandemicsyn]

Summary

KiloClaw controller rotates the KILOCODE_API_KEY env var and signals the gateway, but rotations silently no-op'd due to two compounding bugs:

1. Plaintext on disk shadows env. openclaw onboard persisted the literal key to agents/<id>/agent/auth-profiles.json. OpenClaw's auth resolver prefers configured auth-profiles over env vars, so the stale on-disk key kept winning.
2. The gateway child process can't see controller env changes. The controller spawns the gateway with env: process.env at spawn time, so subsequent process.env.KILOCODE_API_KEY = <new> in the controller never reaches the gateway. openclaw secrets reload re-resolves SecretRefs from the gateway's OWN (frozen) env. SIGUSR1 with OPENCLAW_NO_RESPAWN=1 takes an in-process restart branch that keeps the same env. The rotation's "success" signal was cosmetic.

Net result: users saw 401s against the Kilo AI gateway after every key rotation until the next image redeploy.

Fix has three parts, all in the controller:

1. New installs — onboard with --secret-input-mode ref. No plaintext key on disk; openclaw stores an env-backed keyRef instead.
2. Legacy installs — migrate auth-profiles.json on boot. Idempotent rewrite of any plaintext kilocode key to the same keyRef shape, atomic-written at 0o600. Runs at the end of runOnboardOrDoctor (so the gateway's first read already sees keyRef) and defensively on rotation.
3. Rotation — supervisor.restart(). POST /_kilo/env/patch updates process.env, runs the migration, then calls supervisor.restart(). SIGTERM → gateway exits → the controller supervisor respawns with the controller's current env. The respawned gateway reads the migrated file and resolves the keyRef against the fresh env.

Architectural notes:

• The response field on /_kilo/env/patch stays named signaled for wire compatibility with the worker (EnvPatchResponseSchema, reconcile.ts). Semantics unchanged — the controller delivered the env change to the running gateway — only the mechanism changed from SIGUSR1 to a full restart.
• openclaw secrets reload looks attractive but is unusable for env-backed SecretRefs in our architecture: the server re-resolves against the gateway's OWN frozen env and returns stale values with ok: true. Two viable zero-downtime follow-ups exist but are out of scope: switch to { source: "file", ... } SecretRef (controller-owned file the controller updates on rotation), or remove OPENCLAW_NO_RESPAWN=1 so SIGUSR1 triggers a clean exit + respawn.

Verification

Smoke-test scripts have been extended to guard the new behavior end-to-end, and both passed locally on an x86_64 macOS Docker build:

• scripts/controller-smoke-test.sh (fresh / onboard path) asserts:
  • auth-profiles.json stores a keyRef with no plaintext key after onboard (catches reverted --secret-input-mode ref).
  • POST /_kilo/env/patch response has { ok: true, signaled: true } (catches a rename that would silently break reconcile.ts).
  • The gateway child PID changes after a rotation request (catches a revert to SIGUSR1, which would leave the PID unchanged under OPENCLAW_NO_RESPAWN=1).
• scripts/controller-entrypoint-smoke-test.sh (volume / doctor path) seeds a legacy plaintext auth-profiles.json and asserts it's rewritten to keyRef form with the plaintext literal removed.

Run locally (use non-default ports if the local dev stack already holds 18789/18790):

cd services/kiloclaw
docker build --progress=plain -t kiloclaw:controller .
PORT=19789 bash scripts/controller-smoke-test.sh
PORT=19790 bash scripts/controller-entrypoint-smoke-test.sh

Recommended manual checks on a staging Fly instance:

• Fresh install: /root/.openclaw/agents/main/agent/auth-profiles.json contains keyRef and no key.
• Legacy install: same file, pre-existing plaintext gets rewritten on next controller boot.
• Force rotation by setting PROACTIVE_REFRESH_THRESHOLD_HOURS=10000 (worker env) so every alarm tick (≤5 min) rotates. Remove the override after testing.
• Confirm reconcile.ts logs api_key_refreshed once per rotation (not every alarm), i.e., the new expiry persists after a successful push.
• Additional manual verification: (reviewer/author to fill in)

Visual Changes

N/A

Reviewer Notes

• All changes scoped to services/kiloclaw/controller/ and services/kiloclaw/scripts/. No worker, DB, or openclaw-fork changes.
• The wire contract { ok, signaled } is preserved. migratedProfiles is added as an additive field (zod strips unknown keys).
• Self-healing rollout: every instance migrates its own auth-profiles.json on next controller boot (which also restarts the gateway → picks up the migrated file).
• routes/env.ts accepts a deps parameter for the migration call so tests can stub the filesystem walk.
• bootstrap.ts::BootstrapDeps gained a statSync field; all fake harnesses were updated.
• A previous iteration of this branch attempted openclaw secrets reload + SIGUSR1 fallback; reviewer @kilo-code-bot correctly pointed out that neither delivers a new env to a running gateway in our setup (frozen child-process env). The commit history shows that analysis and the subsequent pivot to supervisor.restart().

[kilo-code-bot]

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Files Reviewed (3 files)

• .plans/kiloclaw-kilocode-key-secretref.md
• services/kiloclaw/controller/src/bootstrap.ts
• services/kiloclaw/controller/src/config-writer.ts

---

_{Reviewed by gpt-5.4-20260305 · 458,905 tokens}

[pandemicsyn]

Smoke test run:

❯ PORT=19789 bash scripts/controller-smoke-test.sh
waiting for controller on port 19789 ...
  [1] waiting...
  [2] state=bootstrapping
  ready after 3s

--- health endpoints ---
PASS: /_kilo/health -> 200 (got 200)
PASS: /health -> 200 (got 200)

--- gateway status ---
PASS: gateway status (no auth) -> 401 (got 401)
PASS: gateway status (bearer auth) -> 200 (got 200)

--- proxy token ---
PASS: root without proxy token (REQUIRE_PROXY_TOKEN=true) -> 401 (got 401)

--- env patch endpoint ---
PASS: env patch (no auth) -> 401 (got 401)
PASS: env patch (non-patchable key) -> 400 (got 400)
PASS: env patch (empty body) -> 400 (got 400)

--- auth-profiles SecretRef mode ---
PASS: auth-profiles.json stores keyRef (got 1)
PASS: auth-profiles.json has no plaintext key (got 1)

--- env patch rotation semantics ---
PASS: env patch response ok=True (got True)
PASS: env patch response signaled=True (got True)
PASS: gateway pid changed after env patch (got 1)

--- version endpoint ---
PASS: version (no auth) -> 401 (got 401)
PASS: version (bearer auth) -> 200 (got 200)

=== Results: 15 passed, 0 failed ===

cloud/services/kiloclaw florian/fix/401-prevention ≡9s
❯ PORT=19790 bash scripts/controller-entrypoint-smoke-test.sh
waiting for /_kilo/health on port 19790 ...
  [1] waiting...
  [2] state=bootstrapping
  ready after 3s

--- health endpoints ---
PASS: /_kilo/health -> 200 (got 200)
PASS: /health -> 200 (got 200)

--- gateway status ---
PASS: gateway status (no auth) -> 401 (got 401)
PASS: gateway status (bearer auth) -> 200 (got 200)

--- proxy token ---
PASS: root without proxy token (REQUIRE_PROXY_TOKEN=true) -> 401 (got 401)

--- auth-profiles migration (doctor path) ---
PASS: legacy plaintext migrated to keyRef (got 1)
PASS: legacy plaintext removed from disk (got 1)

=== Results: 7 passed, 0 failed ===

and what a deploy looks like on disk now:

root@02f254a998c5:~/.openclaw/agents/main/agent# cat auth-profiles.json | jq
{
  "version": 1,
  "profiles": {
    "kilocode:default": {
      "type": "api_key",
      "provider": "kilocode",
      "keyRef": {
        "source": "env",
        "provider": "default",
        "id": "KILOCODE_API_KEY"
      }
    }
  }
}