PRD: Remote Secret Loading via Azure Key Vault (Staging First)

[cgrpa]

PRD: Remote Secret Loading via Azure Key Vault (Staging First)

Problem Statement

Remote runs of the bot currently depend on configuration mechanisms that are not explicitly governed as a single platform contract. The team wants staging (stg) deployments to load runtime secrets from Azure Key Vault through Azure Container Apps, while keeping local development ergonomics intact.

The current risk profile includes configuration drift, unclear ownership boundaries between deployment and infrastructure, and weak guarantees that required secrets exist and are usable before deployment.

Solution

Implement a platform-layer secret contract where Azure Container App environment variables are sourced from Azure Key Vault references, with Terraform as the source of truth for wiring and validation.

Secret values will be created and rotated outside Terraform using a scripted, documented CLI flow. Terraform will manage references, identity access, validation gates, and environment-variable mapping only.

Rollout is staged in two phases:

1. Infrastructure and operations phase:
   • Terraform wiring for Key Vault references.
   • Required-secret validation with fail-fast behavior.
   • RBAC for Container App managed identity.
   • Bootstrap/rotation script and runbook.
2. Application cleanup phase:
   • Development-only user-secrets loading.
   • Environment mode normalization to remove ambiguous development toggles.

User Stories

1. As a platform engineer, I want remote runtime secrets to come from Key Vault, so that sensitive values are centrally managed.
2. As a maintainer, I want Terraform to own secret wiring, so that secret mapping is deterministic and reviewable.
3. As a maintainer, I want secret values excluded from Terraform state, so that sensitive material is not persisted in plan/state artifacts.
4. As a release engineer, I want deploys to fail fast when required secrets are missing, so that broken revisions are blocked before rollout.
5. As a release engineer, I want deploys to fail when required secrets are disabled, so that unusable configurations are caught early.
6. As an operator, I want versionless Key Vault references, so that secret rotation does not require Terraform edits per version.
7. As an operator, I want a scripted bootstrap flow, so that staging setup is repeatable and less error-prone.
8. As an operator, I want secure prompt fallback for missing script inputs, so that setup remains safe and usable interactively.
9. As an operator, I want all-or-nothing secret updates by default, so that environments cannot become partially configured.
10. As a developer, I want local development to keep using user-secrets, so that local workflows stay fast and familiar.
11. As a developer, I want remote environments to avoid user-secrets loading, so that remote behavior is explicit and predictable.
12. As a developer, I want configuration keys to remain unchanged (DiscordToken, GeminiKey, GrokKey, PerplexityApiKey), so that app code churn is minimized.
13. As a platform engineer, I want Key Vault secret names to match existing app keys, so that mapping is straightforward.
14. As a platform engineer, I want Container App secret aliases managed explicitly, so that naming remains clear and controlled.
15. As a platform engineer, I want required secret names to be an input variable, so that additional environments can adopt the pattern with minimal changes.
16. As a platform engineer, I want alias-map and required-list parity checks, so that wiring drift cannot silently occur.
17. As a security reviewer, I want managed identity access to be least-privileged and explicit, so that Key Vault read access is auditable.
18. As a security reviewer, I want RBAC mode preserved for the Key Vault, so that one authorization model is used consistently.
19. As a delivery team member, I want staging-first rollout, so that production adoption can happen after validation.
20. As a maintainer, I want an enforcement toggle defaulted to strict, so that policy is strong by default with a controlled local escape hatch.
21. As a CI owner, I want CI enforcement kept strict, so that pull requests cannot bypass secret requirements remotely.
22. As a maintainer, I want deployment workflow changes deferred and tracked, so that current releases remain stable while ownership boundaries are improved later.
23. As an operator, I want explicit runbook steps for post-rotation app refresh, so that secret updates can be applied on a deterministic timeline.
24. As a new team member, I want clear documentation of the end-to-end secret flow, so that onboarding and troubleshooting are faster.
25. As a reviewer, I want phased implementation boundaries, so that risk and rollback are simpler to manage.

Implementation Decisions

• Platform boundary:
  • Secret retrieval in remote environments is handled by Container Apps + Key Vault references, not by adding Key Vault provider logic in app startup.
• Secret value ownership:
  • Secret values are created and rotated outside Terraform.
  • Terraform manages only contracts and references (identity, mapping, validation).
• Canonical runtime keys:
  • Runtime configuration keys remain unchanged: DiscordToken, GeminiKey, GrokKey, PerplexityApiKey.
  • Key Vault secret names use these exact names.
• Container App secret aliasing:
  • Container App secret identifiers use explicit kebab-case aliases via an explicit key-to-alias map.
  • Alias map is treated as a contract and must stay in sync with required secret names.
• Required-secret contract:
  • Introduce required_secret_names as a Terraform input variable with defaults for the four current required keys.
  • Implement strict parity validation between required_secret_names and alias map keys.
• Fail-fast policy:
  • Use hard-stop resource preconditions (not non-blocking checks) to enforce readiness.
  • Enforce that required secrets exist in the target Key Vault.
  • Enforce that required secrets are enabled.
  • Include enforce_required_secret_presence toggle defaulted true.
  • CI remains strict; local-only override is allowed for bootstrap/recovery use.
• Identity and access:
  • Keep Key Vault in RBAC authorization mode.
  • Use existing system-assigned identity on the Container App for Key Vault access.
  • Add/maintain Key Vault role assignment so the app identity can read secrets.
• Reference semantics:
  • Use versionless Key Vault secret IDs for runtime references to support independent rotation.
• Deployment ownership boundary:
  • For now, keep current deployment action behavior unchanged.
  • Track follow-up work to move to explicit image-only deployment commands so Terraform remains sole owner of secret/env wiring.
• Secret operations UX:
  • Add a scripted bootstrap/rotation workflow with env-var inputs and secure prompts for missing values.
  • Default to all-or-nothing updates.
  • Keep restart/revision refresh separate from secret-write operation, with explicit runbook steps.
• Environment-mode cleanup:
  • Consolidate development environment detection on standard host environment semantics.
  • Restrict user-secrets loading to Development only.

Testing Decisions

• Good tests principle:
  • Validate external behavior and contracts (what is enforced), not implementation mechanics (how expressions are written).
  • Prefer tests and checks that fail on broken runtime outcomes (missing mappings, missing secrets, incorrect environment behavior).
• Terraform contract validation:
  • Verify plan/apply fails when any required secret is missing.
  • Verify plan/apply fails when any required secret is disabled.
  • Verify plan/apply fails when alias map keys and required secret keys diverge.
  • Verify successful plan when required secrets exist, are enabled, and mapping is complete.
  • Verify enforce_required_secret_presence=true behavior in CI paths.
  • Verify local override path can disable enforcement for bootstrap scenarios.
• App configuration behavior validation:
  • Verify Development continues to load local user-secrets.
  • Verify non-Development environments do not depend on user-secrets.
  • Verify existing runtime key names continue to resolve correctly from environment variables.
  • Verify development command-prefix/environment behavior is consistent after normalization.
• Script/runbook validation:
  • Verify script prompts for missing values and refuses partial updates by default.
  • Verify script updates all required secrets when complete input is present.
  • Verify documented post-rotation refresh procedure works end-to-end.
• Prior art for style and scope:
  • Existing configuration-focused unit tests and integration tests in the .NET test project provide precedent for contract-level behavior checks.
  • Existing CI jobs already run restore/build/test and container smoke paths, which should remain compatible with the new secret contract boundaries.

Out of Scope

• Switching the deployment pipeline to explicit image-only CLI updates in this change set.
• Introducing application-level Key Vault SDK/provider integration in startup.
• Production (prd) rollout in this change set.
• Automatic secret rotation scheduling or secret lifecycle governance beyond bootstrap/runbook guidance.
• Reworking unrelated bot features, command semantics, or model integration behavior.

Further Notes

• Rollout target is staging first, with reusable Terraform inputs enabling later production adoption.
• The PRD should be mirrored to the issue tracker after repo access is available from the active GitHub integration context.
• A follow-up work item is required for deployment workflow hardening so secret/env wiring cannot drift outside Terraform ownership.