Skip to content

Secret Sync

Daniel Truong edited this page Sep 11, 2026 · 4 revisions

Key Vault and secret sync

demi-kv-<env> is the source of truth for every EPIC secret. demi-api-fc-<env> and the other Azure apps read it directly through a managed identity. OpenShift namespaces cannot reach it, so a second Function app, demi-secret-sync-<env>, copies mapped secrets out and keeps the OpenShift Secret objects in sync. Team-wide behaviour and rotation steps are in Secrets Management in the dev guides wiki; this page is the DEMI-specific wiring.

Scope

The sync Function runs in test only, covering namespaces 6cdc9e-dev and 6cdc9e-test. Prod OpenShift secrets stay in OpenShift and are managed by hand; prod demi-kv-prod serves only its Azure-native consumers (demi-api-fc-prod Key Vault references, analytics headers). Reason: the prod spoke c4b0a8-prod-vwan-spoke has no route table, landing-zone policy blocks creating one, and hub BGP routes swallow 142.34.0.0/16, so nothing on that subnet can reach the OpenShift API — see the hub BGP entry in Environment-Reality-and-Operational-Gotchas. Bicep parameter deploySecretSync (default true, false for prod) skips the sync module in prod.

Resources per environment

Resource demi-kv-test env demi-kv-prod env
Vault demi-kv-test, c4b0a8-test-rg demi-kv-prod, rg-demi-prod
Private endpoint pe-demi-kv-test pe-demi-kv-prod, c4b0a8-prod-cond-ext-pe-subnet
Sync Function app demi-secret-sync-test none
Sync identity demi-identity-test n/a
Namespaces synced 6cdc9e-dev, 6cdc9e-test none

There is no dev vault or dev Function app: demi-kv-test is the nonprod vault and serves 6cdc9e-dev as well as 6cdc9e-test. Entries meant for 6cdc9e-dev carry a dev- prefix in the mapping file so one vault can hold both namespaces' values without a name clash.

The Function app runs Flex Consumption, scale to zero, on the same VNet integration subnet as demi-api-fc-test, under demi-identity-test. The identity holds Key Vault Secrets User on the vault; nothing else reads secret values. Reaching the OpenShift API from that subnet needs a route table attach, not just outboundVnetRouting; see the hub BGP gotcha in Environment-Reality-and-Operational-Gotchas.

Triggers

  • An Event Grid subscription on the vault, Microsoft.KeyVault.SecretNewVersionCreated, filtered to the names in the mapping file.
  • A daily timer at 06:00 UTC that reconciles the same mapping, to catch drift a missed event would leave behind.

Both triggers run the same reconcile: read each mapped vault secret, compare it against the live OpenShift Secret, write and restart only what changed. A run that finds nothing changed writes nothing and restarts nothing. A write overwrites the mapped keys of a Secret and leaves every other key as it is, so a key set by hand or by a Helm release is neither deleted nor treated as drift.

Mapping file

eagle-demi/src/secret-sync/mapping.json lists names only, never values. Each entry names a vault secret, the target namespace, the OpenShift Secret name and key it fills, and the Deployments or CronJobs to restart when that key changes. An OpenShift Secret with several keys — eagle-api-mongodb has five in dev — is several entries writing the same object.

Required secret names

The names demi-kv-<env> must hold for the API to start, from requiredSecretNames in azure/modules/key-vault.bicep:

Secret What it is
admin-api-key break-glass sysadmin credential
track-client-secret Keycloak client secret of the Track service account
role-sync-client-secret Keycloak client secret the team sync grants roles with
docling-api-key outbound key DEMI presents to docling-serve as X-Api-Key
minio-access-key object-store access key for the NRS store the corpus lives in
minio-secret-key object-store secret key paired with it
analytics-shared-header header APIM stamps on every analytics call
analytics-audit-header second credential APIM stamps on POST /analytics-machine/audit

Optional per environment, named in that environment's optionalSecretNames parameter when set:

Secret What it is Set where
notify-api-key function key eagle-notify accepts on POST /api/events where notifyApiBase is configured
edge-secret value eagle-edge's Front Door rule set stamps as X-Edge-Secret where Front Door fronts the app
openshift-token-<env> ServiceAccount token the sync writes OpenShift Secrets with every environment that syncs
dev-openshift-token the same, for 6cdc9e-dev demi-kv-test only

OpenShift side

Each synced namespace carries a ServiceAccount secret-sync with a Role scoped to secrets (create, get, patch) and deployments (get, patch, for the rollout restart). Its token is read once after the Role is applied and stored in the vault as openshift-token-<env> (or dev-openshift-token for 6cdc9e-dev) — never in GitHub. Revoking it is deleting the token Secret; OpenShift issues a fresh one on request, and the vault entry is updated to match.

See also

DEMI Wiki

Home

Current system

Azure

Domain

ADRs

History

Related

Clone this wiki locally