E3: Credential and key custody

[gnanirahulnutakki]

Goal. hold as few secrets as possible, and protect the ones that remain with envelope encryption and per-tenant keys so no single leak decrypts every tenant — and with no god credential in the center.

Phase / depends. the principles are day-one (P1); the concrete custody surfaces land when the first held secret arrives with the write path (P2 — Git credentials and the intent signing key). Depends on: E1 (workspaces are the key-scoping unit). This epic makes the predecessor's single-env-key blast radius structurally impossible (ADR-0006).

Features

• F3.1 — no-central-admin-credential posture
• F3.2 — KMS envelope encryption with per-tenant data keys
• F3.3 — intent signing-key custody
• F3.4 — key rotation and key-ring
• F3.5 — boot-time custody checks
• F3.6 — secret leak prevention

Exit criteria

• The hub stores no cluster-admin kubeconfigs; reach and action use scoped/brokered identity.
• Every hub-held secret is envelope-encrypted with a per-tenant DEK wrapped by a KMS master key.
• The intent signing key lives in KMS/HSM and is rotatable; rotation works via a key-ring with no data loss.
• Boot-time custody checks fail closed; secrets never reach logs, errors, or git.

---

Source: docs/EPICS.md, docs/SITH-NOTION.md · part of the Sith implementation backlog (see the master roadmap issue).

[gnanirahulnutakki]

E3 design escalation — reference KMS and first Git credential topology

Confirmed repo blocker. ADR-0006 is still Proposed, and docs/EPICS.md deliberately leaves three owner decisions open: the cloud/on-prem KMS reference, the gitops.open-pr host + credential model, and signer public-key distribution. There is no custody implementation on dev. Those choices cannot be hidden behind a neutral interface because they determine the actual cross-workspace blast radius, boot checks, rotation semantics, audit source, and air-gap story.

Current primary-source constraints

• GitHub App installation tokens can be narrowed at mint time to selected repositories and a subset of the App permissions, and expire after one hour: https://docs.github.com/en/enterprise-cloud@latest/apps/creating-github-apps/authenticating-with-a-github-app/generating-an-installation-access-token-for-a-github-app
• The long-lived App private key remains the credential that mints those tokens. One Sith-wide App key would therefore reach every installation of that App if compromised, even though each minted token is narrow. That conflicts with ADR-0006s per-workspace blast-radius goal unless GR explicitly accepts the cross-workspace exception.
• AWS KMS encryption context is cryptographically bound AAD, can be required by key policy/grants, and is logged in CloudTrail; it must contain only non-secret workspace/purpose identifiers: https://docs.aws.amazon.com/kms/latest/developerguide/encrypt_context.html
• AWS KMS supports imported asymmetric key material and KMS-held asymmetric signing, but the provider and on-prem equivalent still define rotation, portability, and operational cost: https://docs.aws.amazon.com/kms/latest/developerguide/importing-keys.html and https://docs.aws.amazon.com/kms/latest/developerguide/symmetric-asymmetric.html

Decision required from GR

1. Recommended first-write topology: GitHub only; one customer/workspace-owned GitHub App credential boundary; mint one-hour installation tokens just in time for the exact allowlisted repository and exact required permissions; never persist installation tokens. Store the App private key only as a named per-workspace envelope-encrypted exception. This preserves the ADR blast-radius boundary, at the cost of per-workspace App onboarding and rotation.
2. Shared Sith GitHub App: one HSM/KMS-protected App key with per-installation, per-repository token narrowing. Lower onboarding cost, but a signing-key compromise spans all installed workspaces. I do not recommend this without an explicit ADR exception and compensating controls.
3. User-to-server GitHub App tokens: binds access to both user and App permissions and improves human attribution, but adds browser authorization/refresh-token custody and makes headless governed execution depend on a user session. Better as a later delegated mode than the P2 service path.

Separately choose the reference custody pair: AWS KMS + Vault Transit/on-prem HSM is the existing research recommendation, but selecting it commits Sith to an AWS reference adapter plus a portability contract and non-trivial KMS/HSM operational cost. Signer public-key distribution to spokes remains a separate E3/E4 decision even after the Git credential choice.

No custody code or child issue is being opened until these security/product choices are confirmed. A provider-neutral fake-KMS seam would create an attractive but unproven security boundary and would not unblock real gitops.open-pr execution. Per the blocker protocol, this run is pivoting to the independent E6 existing-chain export seam.