Stage 2: ship credentials-service worker (arch.md §15.1) — Lambda + mTLS to signer

[hanwencheng]

Deferred from: stage 1 (#89). The bucket + role separation (arch.md §17 per-data-class layer) is shipped in this PR; the worker itself is deferred to stage 2.

Background

Per arch.md §15.1 credentials-service:

• Worker takes a cap-token + (service, plaintext_or_ciphertext)
• Derives the per-actor KEK via mTLS to the signer (derive_cred_kek(operator_omni, k3_epoch))
• AES-256-GCM encrypt/decrypt under that KEK
• PUTs / GETs the blob at s3://$VAULT_BUCKET/bots/<actor_omni>/credentials/<service>.enc
• Independent re-verify of the cap against the on-chain ScopeContract before any S3 / signer touch

What's already in place (no work needed)

Piece	State	Source
Per-actor KEK derivation	shipped (client-side)	crates/agentkeys-core/src/s3_backend.rs:222 — derives via signer's /dev/sign-message
AES-256-GCM envelope (v1 + v2 shapes)	shipped (client-side)	same file, store_credential / read_credential
OIDC AssumeRoleWithWebIdentity → STS session w/ agentkeys_actor_omni tag	shipped in this PR	crates/agentkeys-broker-server/src/handlers/oidc.rs
$VAULT_BUCKET separate from $EMAIL_BUCKET per arch.md §17	shipped in this PR	scripts/provision-vault-bucket.sh + scripts/provision-vault-role.sh + scripts/apply-vault-bucket-policy.sh
agentkeys-vault-role with credentials-only inline policy	shipped in this PR	scripts/provision-vault-role.sh
Bucket policy gated on agentkeys_actor_omni PrincipalTag	shipped in this PR	scripts/apply-vault-bucket-policy.sh

Today the CLI does the encrypt + direct S3 PUT (acting as the worker). This is functionally correct and produces the same envelope bytes the worker will produce; the architecture violation is purely topological — the encrypt should happen inside the worker, not on the operator's laptop.

What this issue ships

1. Lambda function (agentkeys-worker-creds per arch.md §28 layout):
   • Endpoint POST /v1/cred/store — takes {cap, service, plaintext}, verifies cap on-chain, derives KEK via signer mTLS, encrypts, PUTs to $VAULT_BUCKET/bots/<actor>/credentials/<service>.enc
   • Endpoint POST /v1/cred/fetch — takes {cap, service}, verifies cap, GETs blob, decrypts via signer-derived KEK, returns plaintext over the TLS-terminated channel
   • Endpoint POST /v1/cred/teardown — takes {cap, target_actor}, verifies cap, wipes bots/<target_actor>/credentials/* prefix
2. mTLS plumbing: worker presents its attestation-issued cert; signer accepts only attested worker callers (arch.md §11)
3. CLI flag --credential-backend=sidecar activates the daemon → worker path (today returns "not yet implemented")
4. Sidecar daemon hop: agentkeys-daemon proxies CLI requests to the worker (cap-token cache + SSE drop event push)

Acceptance

• agentkeys store openrouter sk-or-... with --credential-backend=sidecar succeeds and the blob lands at s3://$VAULT_BUCKET/bots/<actor>/credentials/openrouter.enc — same path the v2 envelope-v2 CLI writes today
• agentkeys read openrouter with --credential-backend=sidecar returns the plaintext
• Cap-mint without the right scope is rejected by the worker (independent re-verify against chain)
• KEK never leaves the signer enclave; the operator's laptop never sees it

Why this is stage 2

The worker is architecturally required but operationally optional for stage-1 smoke testing. The CLI's direct path produces correct envelope bytes; future migration to the worker is invisible to anyone reading them later (same KEK, same AAD, same nonce shape). Deferring lets stage 1 ship the on-chain registry + sidecar daemon + WebAuthn binding first; stage 2 adds the worker hop.

See arch.md §15.1 + §28 (agentkeys-worker-creds/ layout) for the full design.

🤖 Generated with Claude Code

[hanwencheng]

Partial coverage landing in PR #87

PR #87 (v2 stage 1 — Foundation, closes #89) ships the Rust microservice variant of the credentials worker described here. Tracking the delta so the Lambda + mTLS work this issue describes is well-scoped.

Already shipped in PR #87

• ✅ agentkeys-worker-creds Rust microservice (crates/agentkeys-worker-creds/): axum-based HTTP server with POST /v1/cred/store, POST /v1/cred/fetch, POST /v1/cred/teardown.
• ✅ Cap signature verification: P-256 ECDSA verify against broker pubkey loaded from env (BROKER_CAP_PUBKEY_PEM).
• ✅ Independent on-chain re-verify of cap before any S3 op (arch.md §15.1's "independent re-verify" requirement):
  1. SidecarRegistry.isActive(deviceKeyHash) — reject if device revoked
  2. AgentKeysScope.isServiceInScope(operator, actor, service) — reject if service not in agent's scope
  3. K3EpochCounter.currentEpoch == cap.k3_epoch — reject on epoch drift
• ✅ AES-256-GCM envelope at s3://$VAULT_BUCKET/bots/<actor_omni_hex>/credentials/<service>.enc — same v2 envelope shape (1B version || 12B nonce || ciphertext || 16B tag) the CLI writes today, so worker-vs-CLI envelope bytes are interchangeable.
• ✅ $VAULT_BUCKET separation from $MAIL_BUCKET per arch.md §17 (PR v2 stage 1 — sovereign sidecar + on-chain identity + credentials-service worker (#89) #87 also ships provision-vault-bucket.sh + provision-vault-role.sh + apply-vault-bucket-policy.sh).
• ✅ --credential-backend=sidecar daemon → worker path: agentkeys-daemon proxy listens on $XDG_RUNTIME_DIR/agentkeys-proxy.sock (or TCP 127.0.0.1:9090); CLI's --credential-backend=sidecar routes credential CRUD through the daemon → broker cap-mint → worker chain.
• ✅ Cap-token cache + SSE drop event push: 5-min TTL in-memory cache in the daemon, fail-closed on stale broker (>60s last-contact), SSE drop events from broker invalidate cached caps.

Residual #91 work

These remain after PR #87 merges:

• Lambda variant (agentkeys-worker-creds deployed as AWS Lambda + API Gateway per arch.md §28). The microservice variant is functionally equivalent and sufficient for stage-1 closure; Lambda variant adds easier ops + S3 trigger composability + no idle compute cost. Should share most of the request-handling code with the microservice via a lib.rs.
• mTLS plumbing for KEK derivation from signer (arch.md §11 + §15.1). Today KEK comes from AGENTKEYS_WORKER_KEK_HEX env (stage-1 simplification per arch.md §22b.2). Stage-2 hardening:
  • Worker presents attestation-issued cert to signer
  • Signer accepts only attested worker callers
  • Signer derives cred_kek = HKDF(K3, operator_omni || k3_epoch || "cred-kek-v1") per-operator + per-data-class
  • Returns KEK over the mTLS channel; never leaves the signer enclave's process memory
  • Worker caches per-actor KEK with TTL + invalidates on K3 epoch bump
• Cap-mint without right scope → independent rejection by worker — PR v2 stage 1 — sovereign sidecar + on-chain identity + credentials-service worker (#89) #87's verify.rs does the chain re-check, but we should add an integration test that constructs a cap with service = "x" but on-chain scope ["y", "z"] and asserts the worker rejects it (broker would reject too but independent worker rejection is the arch.md §15.1 guarantee).
• POST /v1/cred/teardown integration test — handler exists, no e2e test on chain.

Why "microservice instead of Lambda" is fine for stage-1 closure

arch.md §28 lists both variants as supported deployment shapes. The arch.md decision was "Lambda variant default for AWS deployments, microservice variant for self-hosted ops" — and stage-1 explicitly targets self-hosted (each operator runs their own broker on EC2 today). The microservice form is the right primary deliverable; the Lambda is the next-deployment-target-up.

Cross-references

• arch.md §15.1 — credentials-service worker contract
• arch.md §17 — per-data-class bucket + role separation
• arch.md §22b.2 — stage-1 KEK-from-env simplification + this issue as the hardening pointer
• arch.md §28 — worker deployment shapes (Lambda vs microservice)