ADR 007 Service to Service Credentials

ADR-007: Service-to-Service Credentials

Status

Accepted 2026-08-06. On main and deployed (verified 2026-08-26): the apikeys container exists in demi-cosmos-test and demi-cosmos-prod, requireWrite guards every mutating route in src/http/routes.js, and DEMI_ALLOWED_CLIENTS is read in src/config.js. Supersedes the single-ADMIN_API_KEY model.

Context

DEMI was built as one service with one consumer. It is becoming a shared EAO service: eagle-admin already reads it, and Track and Submit are expected to follow. The credential model did not move with it.

Before this change there was exactly one inbound credential:

// src/helpers/auth.js
const validKeys = [process.env.ADMIN_API_KEY].filter(Boolean);
if (apiKey && matchesConfiguredKey(apiKey, validKeys)) {
  return onSuccess({
    preferred_username: 'internal-service',
    realm_access: { roles: ['sysadmin', 'staff', 'demi-admin'] }
  });
}

Every consequence of that is structural, not a bug to patch:

Property	State before
Per-caller identity	None. Every key holder is literally the same principal, internal-service
Least privilege	Impossible. The one key grants full sysadmin
Revocation	Rotating the single key breaks every consumer at once
Expiry	None
Audit	None. A successful authentication was never logged, and http-logger recorded no identity field
Registry	None. The set of live credentials was whoever happens to hold the string

With one consumer that is merely untidy. With five it means a leak from any consumer is indistinguishable from any other and cannot be revoked without an outage for all of them.

A second, quieter problem: authMiddleware asked one question — is this caller privileged? — and used the answer to guard GET /db/stats and DELETE /projects/:id identically. A consumer that only needed to read could not be expressed, so every integration would have received the ability to delete every project.

Decision

Keycloak service accounts are the standard for applications. A key registry serves callers that cannot hold a Keycloak client. Consumers get least privilege, not blanket sysadmin.

1. Applications use Keycloak service accounts

A CSS integration request creates a client in realm eao-epic with its service account enabled and only the DEMI roles it needs. The consumer performs client_credentials, caches the token, and calls DEMI with a normal Authorization: Bearer.

This required no change to the authentication path. DEMI already verified RS256 against the realm JWKS with the issuer pinned, and resolveAccess already turned realm_access.roles into a visibility predicate. A service-account token was always going to be accepted; what was missing was any way to say which clients may act privileged, which DEMI_ALLOWED_CLIENTS supplies.

The allowlist defaults to empty, meaning permissive, and an unlisted client is demoted to the public tier rather than rejected. Both choices are deliberate: DEMI's own frontend and eagle-admin's staff users authenticate against this same realm, so an allowlist defaulting to on would have logged real users out the moment it shipped, and a 401 would break a page that only ever needed public reads.

2. Non-Keycloak callers use registry keys

Some callers cannot hold a Keycloak client — the extraction host is one. They get a key from a registry in the apikeys container:

{ id: keyId, name, hash, roles, projectScope, createdAt, createdBy, expiresAt, revokedAt, lastUsedAt }

Format is demi_<env>_<keyId>_<secret>. The keyId is public and carried in the key on purpose: without it, verification means scanning and hash-comparing every stored key; with it, it is a single-partition point read. It also lets a leaked key be traced to its consumer from a log line without the secret ever being logged.

Issuance is POST /admin/api-keys, which returns the plaintext exactly once. There is no update endpoint — rotation is issue-new-then-revoke-old, which is what allows rotation with no window where the consumer has no working credential, and means a key's roles can never be widened after issuance.

3. Read privilege and write privilege are separated

demi-service-read joins SECURE_ROLES, so it reads everything the ACL allows and no row's read[] array had to change. The requireWrite middleware guards every mutating route, keyed on WRITE_ROLES — exactly the older privileged set, so no caller that could write lost the ability. The only thing added is a tier that cannot write.

4. Every request logs its caller

http-logger records the principal, plus keyId or azp. ADMIN_API_KEY remains as a documented break-glass credential: the registry cannot mint its first key without something to authenticate the minting call.

Alternatives rejected

Keep one shared key, hand it to everyone. What we had. Fails on revocation before anything else — the first consumer to leak it takes the rest down with it.

A key registry alone, no service accounts. Would make DEMI an identity provider: its own issuance, rotation, revocation and audit, all reimplementing what CSS already operates. The landing zone also expects client secrets to live in Key Vault with rotation, which a homegrown store sits awkwardly against. Registry keys are kept for the callers that genuinely cannot use Keycloak, and no further.

Password grant against a public client, which is how ENGAGE reaches eagle-api (met-api/src/met_api/services/rest_service.py:143-165 — a human-shaped account driven through eagle-admin-console). It works, and it is why the estate has no clean precedent to copy. It means a service identity that is indistinguishable from a staff user, cannot be scoped separately, and whose credential is a username and password. Not repeated here.

Azure API Management or Front Door in front of DEMI. Rejected on cost. Authentication is not the problem an API gateway is bought for.

Consequences

• Each new consumer needs a CSS integration request. That is process, and it is the point: the list of things that can reach DEMI becomes something someone approves rather than something that accumulates.
• Consumers must cache tokens. met-api caches for 55s against 60s token lifetimes; copy that.
• Revocation is eventually consistent, bounded by a 60s lookup cache. Immediate on the instance that processed the revocation, up to 60s elsewhere. Acceptable for a service credential; it would not be for a human session.
• The apikeys container must be created out of band — container creation is control-plane and the app's managed identity cannot do it. See Azure-Environments.
• Key expiry defaults to 90 days, matching the landing zone's maximum secret lifetime. Keys expire whether or not anyone is watching, so lastUsedAt exists to find the ones nobody is using before they lapse in production.
• The service-account half rests on an unproven assumption. Whether a CSS service-account token carries realm_access.roles in the shape DEMI reads is unverified: CSS manages client roles and the roles scope, and eagle-admin requests scope: 'openid roles'. Prove it with one throwaway client before onboarding Track or Submit. The registry-key path is the only one exercised end to end.