remote: support source-referenced secrets in services.json (env:/file:) instead of inline-only values

[mushrafmim]

Problem

The remote package loads service definitions from services.json via Manager.LoadServices (remote/manager.go). Today, authentication secrets are embedded as literal values inside that file — auth.options.value (API key), token (bearer), and client_secret (OAuth2) are all plain strings.

This forces an all-or-nothing posture: because the secrets live inline, the entire config file becomes sensitive. That has real downsides for our deployment model:

• No clean ConfigMap/Secret split. Non-sensitive operational config (service IDs, URLs, timeouts, scopes, header names) can't live in a ConfigMap / be committed for GitOps, because it's interleaved with credentials.
• Coarse rotation. Rotating one credential means rewriting the whole blob.
• Poor fit for our secrets pipeline. We use HashiCorp Vault as the source of truth, synced into the cluster by the External Secrets Operator (ESO) via the nsw-vault-backend ClusterSecretStore, which generates native Kubernetes Secrets consumed through secretKeyRef / mounted volumes. The current "whole file is the secret" model can't consume those native Secrets by reference — it only accepts literal values.

In short: the config layer has no way to say "this field's value comes from somewhere else." Secrets and structure are fused.

Proposed Solution

Introduce a scheme-prefixed value reference for any secret-bearing field in services.json. A field value may be a literal, or a reference whose prefix names its source:

{
  "auth": {
    "type": "oauth2",
    "options": {
      "token_url": "https://idp.example.gov/token",
      "client_id": "my-client",
      "client_secret": "env:NPQS_CLIENT_SECRET",
      "scopes": ["npqs:submit"]
    }
  }
}

{
  "auth": {
    "type": "bearer",
    "options": { "token": "file:/var/run/secrets/npqs/token" }
  }
}

Supported sources initially:

• env:NAME — read from environment variable NAME (populated by ESO-generated Secret via secretKeyRef/envFrom).
• file:/path — read from a file (populated by an ESO-generated Secret mounted as a volume). Trailing newline trimmed.

Key design points:

• Per-field, not whole-file. Only fields explicitly marked with a known scheme are resolved; secret and non-secret fields coexist in the same options block (fixes OAuth2, which mixes both).
• Backward compatible. A value with no recognized scheme prefix is treated as a literal, exactly as today. Existing services.json files keep working unchanged — this is purely additive. An explicit literal: prefix is available as an escape hatch for the rare value that genuinely begins with a scheme name.
• Lazy resolution for rotation. file: references resolve inside the authenticator's Apply() (with a short TTL cache) rather than at client-construction time, so when ESO refreshes a mounted Secret the new value is picked up without a pod restart. env: resolves at startup (its natural lifecycle).
• Loud failures. An unset env var or unreadable/empty file errors at use (or at an eager validation pass on load) — it must never silently resolve to an empty string and send an unauthenticated request.

This lets non-sensitive config live in a ConfigMap (committable, auditable, GitOps-friendly) while only references point at ESO-generated native Secrets — aligning the SDK with the Vault → ESO → native Secret pipeline.

Scope / non-goals

• Initial sources are env: and file: only. A direct vault: source is intentionally out of scope — ESO already lands secrets as native Kubernetes Secrets, and a runtime Vault dependency in the SDK would defeat that. The scheme is extensible, so vault: (or others) can be added later without a breaking change.

Alternatives

• Whole-file ${VAR} interpolation (os.ExpandEnv over the file at load). Simpler (one line), but env-only, resolves once at load (no rotation without restart), and blindly rewrites the entire file — any literal containing $ is a footgun. Rejected in favor of the per-field, multi-source scheme.
• Flat top-level authSecretRef + fixed /secrets/<ref>/<key> mount convention. Clean for bearer/api_key but doesn't map onto OAuth2's mix of secret and non-secret fields, and couples the config to a single hard-coded mount layout. The per-field source:ref form is more flexible.
• Keep the whole file as a Kubernetes Secret. Makes all structure opaque, breaks GitOps/auditability, and forces a full-Secret rewrite to change a timeout. This is the status quo limitation we're trying to remove.
• Direct Vault integration in the SDK. Reintroduces the runtime coupling ESO exists to eliminate. Out of scope.

[sthanikan2000]

I think the main idea here is right — pointing each field at its own source (env: / file:) is much better than the whole-file options you listed. I just want to suggest two small changes to how it works, since both change the config format and are easier to get right now than later.

1. Allow both a short string and an object, not just the string prefix

The proposal puts the source inside the string ("env:NAME"). The downside is that you then need the literal: escape hatch for any real value that happens to start with a scheme name.

If we use a small value type with a custom JSON parser, it can accept both styles, and the escape hatch is no longer needed:

"client_secret": "env:NPQS_SECRET"             // short form (your proposal)
"client_secret": { "env": "NPQS_SECRET" }      // object form — no guessing
"client_secret": { "file": "/var/run/secrets/npqs/token" }
"value":         "plain text still works"       // plain string = literal (unchanged)

	String prefix only (proposal)	Allow both
Old files keep working	Yes	Yes
Risk of confusing a value with a source	Needs literal: escape hatch	None — the object form is clear
Length	Short	Short or clear, your choice
Easy to read	Lower	Higher (the source is a named key)
Amount of parsing code	A bit less	A bit more (one parser handles both)

2. Read the value on every request, instead of caching it with a TTL

The proposal caches a resolved file: value for a short time (TTL). But ESO secret files sit in memory (tmpfs), and the kubelet swaps them in one atomic step. So reading the file again is safe and takes only a few microseconds. The cache mostly just saves one file read per request — and only for bearer / api_key, since OAuth2 reads client_secret only when it refreshes the token.

	Cache with TTL (proposal)	Read on every request
Cost per request	Saves one file read when cached	One small file read (~µs)
How fast a rotated secret is picked up	kubelet delay + TTL	kubelet delay only
Extra code/state	Cached value + expiry time + lock + clearing	None
Chance of using an old value	Up to the TTL after rotation	None

A TTL cache only helps if you have very high traffic and the secret is on slow/network storage — which is not the case for ESO mounts. So reading every time is simpler, picks up new secrets faster, and is correct here. We can still keep your "fail loud" idea by checking all references once when the file loads (so a missing env var or unreadable file fails right away, not on the first request).

How it could look in code

Both changes come from one small shared type (say auth.Secret) with:

• a custom JSON parser that accepts a plain string, a prefixed string, or an object,
• a Resolve() method called inside each authenticator's Apply(),
• a Validate() method run when the file loads.

This keeps all the parsing and reading logic in one place that is easy to test, keeps the list of sources (env: / file:) in one spot so vault: is easy to add later, and replaces the three inline secret string fields in the same way.

[sanjiva]

Changing config values for containers is an antipattern AFAIK.

If correct we should read at startup time. Off the value needs to be changed you need to restart the container.

[mushrafmim]

We can inject the keys dynamically to the files, not as environment variables. I don't feel like, we should worry about key rotation at all, at this stage, since these are from ThunderID, the vault is not gonna rotate these at all.

[mushrafmim]

Thanks @sthanikan2000 — the per-field reference direction is solid, but I want to push back on both of the specific changes.

Re: suggestion 1 — accept both a string and an object form

I'd prefer to keep it to the prefixed-string form only.

• One way to do something. Supporting both "env:NAME" and { "env": "NAME" } means every config reader, every test, every doc example, and every code path has to handle two shapes for the same concept. I'd rather there be a single, obvious way to write a reference than carry two equivalent syntaxes forever to save very little.
• It avoids nesting that tends to grow. Once an object form exists, it's a natural home for "just one more key," and the config gets deeper over time. A flat "source:ref" string keeps each field a single scalar — easier to read, diff, and review in a ConfigMap.
• The ambiguity it removes is essentially theoretical. The object form's main justification is dropping the literal: escape hatch. But these fields hold secret references — tokens, API keys, client secrets — and none of those values start with env: or file:. Even in the vanishingly rare case one did, this isn't user-facing input: it's a small config written by an operator who understands the scheme, and literal: is right there as an explicit out. Not worth a permanent second syntax.

So: prefixed string only, with literal: as the documented escape hatch.

Re: suggestion 2 — read on every request

I'd actually go the other way and resolve once at startup (and Validate() there to fail loud), rather than re-reading per request.

The key point is what these secrets actually are. Every field in scope — the API key value, the bearer token, the OAuth2 client_secret — is a credential for an external IDP / government API. Vault can only rotate credentials it has privileged access to manage (databases, cloud IAM, PKI). It has no hook into an external IDP, so to Vault these are static stored secrets — it holds and serves them, but it never changes them.

That means there's no automated, mid-operation rotation to chase here. A change to one of these is a rare, deliberate, operator-driven event (policy or compromise). When that happens, restarting the pod is a perfectly acceptable way to pick up the new value — it's a planned action, not a hot path. Re-reading on every request adds machinery to handle a rotation pattern these secrets don't have.

So: resolve at startup, validate loudly there, restart to rotate. Simpler, and it matches how the values actually behave. If we ever point a field at a genuinely Vault-rotated credential, we can revisit lazy re-reads then — but I don't want to design for it now.

[lokewate]

Agree with the TLDR — flat scheme-prefixed strings, resolved and validated once at startup, fail loud on a missing credential.

One change: if we're shipping a single source scheme to start, make it file:, not env:. Env-var secrets leak through well-known paths (/proc//environ, crash reporters snapshotting the environment, printenv on a compromised container); ESO can land these as mounted files just as easily, so file: gives us better security at zero operational cost. env: can always be added later if something genuinely needs it.

[sthanikan2000]

Aligning with where this is landing — flat scheme-prefixed strings, resolved and validated once at startup, fail loud on a missing credential. Happy to drop both of my earlier suggestions:

• Resolve at startup, not per request — agreed. And since resolution timing is an internal detail rather than part of the config format, we can revisit lazy re-reads with no schema change if we ever point a field at a credential something actually rotates. So nothing's lost by starting simple.
• Prefixed-string only, not a string-or-object form — agreed. One obvious way to write a reference is worth more than the edge-case ambiguity it removes.

On the one still-open question, +1 to file:-only to start. Beyond @lokewate's leakage argument (/proc/<pid>/environ, crash dumps, printenv), there's a consistency point: once we resolve at startup, the one thing file: does better than env: — picking up a changed secret without a restart — no longer applies anyway. So the choice comes down purely to security, and there file: wins cleanly. env: can always be added later if something genuinely needs it.