Secrets Management

Secrets Management

Every EPIC secret lives in an Azure Key Vault. Azure apps read it directly through a managed identity. OpenShift workloads get a copy written by a sync job; nobody edits an OpenShift secret by hand.

Where secrets live

Vault	Resource group	Serves
demi-kv-test	c4b0a8-test-rg	OpenShift 6cdc9e-dev and 6cdc9e-test, and the Azure test apps
demi-kv-prod	rg-demi-prod	OpenShift 6cdc9e-prod, and the Azure prod apps

There is no dev vault. demi-kv-test is the nonprod vault and covers both 6cdc9e-dev and 6cdc9e-test; entries meant for dev carry a dev- prefix in the sync mapping (see below).

demi-kv-<env> is owned and deployed by eagle-demi (azure/modules/key-vault.bicep). Other repos reference it by name; they do not deploy their own vault.

Why the vaults are private

Landing-zone policy Deny-PublicPaaSEndpoints forces publicNetworkAccess: Disabled on every vault. A call from outside the vault's private endpoint gets ForbiddenByConnection, even from the vault's own owner. A GitHub-hosted runner cannot read it.

The one place that can reach a vault is a devbox VM inside the private endpoint subnet (demi-devbox-<env>), running as the environment's managed identity. Setting or reading a secret value by hand always happens from there:

# on demi-devbox-<env>
az keyvault secret set --vault-name demi-kv-<env> --name <name> --value '<value>'
az keyvault secret list --vault-name demi-kv-<env> -o table

How Azure apps read secrets

An app setting is a versionless Key Vault reference: @Microsoft.KeyVault(SecretUri=<vaultUri>secrets/<name>). The app's managed identity holds Key Vault Secrets User on the vault, and that is the only grant it needs — no client secret, no value in Bicep, a param file, or a deployment record. Because the reference has no /<version> suffix, App Service re-reads it on its own after a rotation, no redeploy required.

How OpenShift namespaces get secrets

OpenShift pods cannot read the vault themselves: the cluster's issuer does not federate to Entra, and the vault answers only inside the VNet. A Function app, demi-secret-sync-<env>, bridges the gap. It runs inside the vault's VNet, reads the secrets it is told to sync, writes each one into the OpenShift Secret that consumes it, and restarts the Deployments and CronJobs that read it so the pods pick up the new value.

• demi-secret-sync-test owns 6cdc9e-dev and 6cdc9e-test. demi-secret-sync-prod owns 6cdc9e-prod.
• Triggers: an Event Grid subscription on the vault (SecretNewVersionCreated, filtered to the mapped names), plus a daily timer for drift. Both run the same reconcile logic.
• The mapping file lists names only, never values: which vault secret feeds which key of which OpenShift Secret, and which Deployments or CronJobs to restart when it changes. One OpenShift Secret with several keys is several mapping entries that write the same object.
• The job writes each Secret with oc apply and stamps it with the vault secret's version and a timestamp, so a later run can tell whether the live copy matches the vault.
• A missing or empty vault secret stops that entry's sync and raises an alert. It never writes an empty value over a real one.
• The job authenticates to the Kubernetes API as a dedicated secret-sync ServiceAccount, one per namespace, with a Role limited to secrets (create, get, patch) and deployments (get, patch, for the restart). The ServiceAccount's own token is itself a vault secret (openshift-token-<env>, or dev-openshift-token for 6cdc9e-dev on the shared test vault) — it is never stored in GitHub.

Adding a secret

1. Set the value in the vault, from the environment's devbox: az keyvault secret set --vault-name demi-kv-<env> --name <name> --value '<value>'.
2. For an Azure app: add the name to the vault's secret list and reference it as @Microsoft.KeyVault(SecretUri=...) in that app's Bicep. For an OpenShift consumer: add one entry per key to the sync mapping file, naming the OpenShift Secret, the key, and the Deployments or CronJobs to restart.
3. Merge. The next deploy (Azure) or the next sync event or daily run (OpenShift) picks it up.

Rotating a secret

Rotation is always the same first step: set a new version of the secret in the vault, using the same az keyvault secret set command as above.

• OpenShift consumers pick up the new version on the next sync event or the next daily run; the job restarts the Deployments and CronJobs listed against that secret.
• Azure apps on Flex Consumption must be stopped, then started — not restarted. A Key Vault reference is cached for up to 24 hours, and a plain restart does not force it to re-read.

What stays outside the vault

eagle-edge's EDGE_SECRET stays a GitHub environment secret. The GitHub-hosted runner that deploys eagle-edge cannot reach the vault, and Front Door rule sets have no managed-identity path to read one. This is a documented exception, not an oversight.

Dead OpenShift secrets

A number of OpenShift secrets in dev and test are no longer read by anything and are kept only until they are deleted: eagle-api-minio-keys, getok-credentials, eagle-cron-keycloak, eagle-cron-secret, forms-flow-*, met-fider-*, pn-details-card-*, mongo-migration, prod-minio-readonly, certbot, test-vanity-certificates, demi-keycloak-admin, notify-push-secret, eagle-analytics-secrets. demi-app-secrets in the test namespace is likewise unused once its values live in the vault. DEMI test does not read eagle-api-minio-keys either: it reaches the same test object-store account through minio-access-key / minio-secret-key in demi-kv-test. Deleting a secret in 6cdc9e-prod needs Daniel's own login, not the automation service account.