remote: support source-referenced secrets in services.json (env:/file:) instead of inline-only values

[mushrafmim]

Problem

The remote package loads service definitions from services.json via Manager.LoadServices (remote/manager.go). Today, authentication secrets are embedded as literal values inside that file — auth.options.value (API key), token (bearer), and client_secret (OAuth2) are all plain strings.

This forces an all-or-nothing posture: because the secrets live inline, the entire config file becomes sensitive. That has real downsides for our deployment model:

• No clean ConfigMap/Secret split. Non-sensitive operational config (service IDs, URLs, timeouts, scopes, header names) can't live in a ConfigMap / be committed for GitOps, because it's interleaved with credentials.
• Coarse rotation. Rotating one credential means rewriting the whole blob.
• Poor fit for our secrets pipeline. We use HashiCorp Vault as the source of truth, synced into the cluster by the External Secrets Operator (ESO) via the nsw-vault-backend ClusterSecretStore, which generates native Kubernetes Secrets consumed through secretKeyRef / mounted volumes. The current "whole file is the secret" model can't consume those native Secrets by reference — it only accepts literal values.

In short: the config layer has no way to say "this field's value comes from somewhere else." Secrets and structure are fused.

Proposed Solution

Introduce a scheme-prefixed value reference for any secret-bearing field in services.json. A field value may be a literal, or a reference whose prefix names its source:

{
  "auth": {
    "type": "oauth2",
    "options": {
      "token_url": "https://idp.example.gov/token",
      "client_id": "my-client",
      "client_secret": "env:NPQS_CLIENT_SECRET",
      "scopes": ["npqs:submit"]
    }
  }
}

{
  "auth": {
    "type": "bearer",
    "options": { "token": "file:/var/run/secrets/npqs/token" }
  }
}

Supported sources initially:

• env:NAME — read from environment variable NAME (populated by ESO-generated Secret via secretKeyRef/envFrom).
• file:/path — read from a file (populated by an ESO-generated Secret mounted as a volume). Trailing newline trimmed.

Key design points:

• Per-field, not whole-file. Only fields explicitly marked with a known scheme are resolved; secret and non-secret fields coexist in the same options block (fixes OAuth2, which mixes both).
• Backward compatible. A value with no recognized scheme prefix is treated as a literal, exactly as today. Existing services.json files keep working unchanged — this is purely additive. An explicit literal: prefix is available as an escape hatch for the rare value that genuinely begins with a scheme name.
• Lazy resolution for rotation. file: references resolve inside the authenticator's Apply() (with a short TTL cache) rather than at client-construction time, so when ESO refreshes a mounted Secret the new value is picked up without a pod restart. env: resolves at startup (its natural lifecycle).
• Loud failures. An unset env var or unreadable/empty file errors at use (or at an eager validation pass on load) — it must never silently resolve to an empty string and send an unauthenticated request.

This lets non-sensitive config live in a ConfigMap (committable, auditable, GitOps-friendly) while only references point at ESO-generated native Secrets — aligning the SDK with the Vault → ESO → native Secret pipeline.

Scope / non-goals

• Initial sources are env: and file: only. A direct vault: source is intentionally out of scope — ESO already lands secrets as native Kubernetes Secrets, and a runtime Vault dependency in the SDK would defeat that. The scheme is extensible, so vault: (or others) can be added later without a breaking change.

Alternatives

• Whole-file ${VAR} interpolation (os.ExpandEnv over the file at load). Simpler (one line), but env-only, resolves once at load (no rotation without restart), and blindly rewrites the entire file — any literal containing $ is a footgun. Rejected in favor of the per-field, multi-source scheme.
• Flat top-level authSecretRef + fixed /secrets/<ref>/<key> mount convention. Clean for bearer/api_key but doesn't map onto OAuth2's mix of secret and non-secret fields, and couples the config to a single hard-coded mount layout. The per-field source:ref form is more flexible.
• Keep the whole file as a Kubernetes Secret. Makes all structure opaque, breaks GitOps/auditability, and forces a full-Secret rewrite to change a timeout. This is the status quo limitation we're trying to remove.
• Direct Vault integration in the SDK. Reintroduces the runtime coupling ESO exists to eliminate. Out of scope.

[sthanikan2000]

I think the main idea here is right — pointing each field at its own source (env: / file:) is much better than the whole-file options you listed. I just want to suggest two small changes to how it works, since both change the config format and are easier to get right now than later.

1. Allow both a short string and an object, not just the string prefix

The proposal puts the source inside the string ("env:NAME"). The downside is that you then need the literal: escape hatch for any real value that happens to start with a scheme name.

If we use a small value type with a custom JSON parser, it can accept both styles, and the escape hatch is no longer needed:

"client_secret": "env:NPQS_SECRET"             // short form (your proposal)
"client_secret": { "env": "NPQS_SECRET" }      // object form — no guessing
"client_secret": { "file": "/var/run/secrets/npqs/token" }
"value":         "plain text still works"       // plain string = literal (unchanged)

	String prefix only (proposal)	Allow both
Old files keep working	Yes	Yes
Risk of confusing a value with a source	Needs literal: escape hatch	None — the object form is clear
Length	Short	Short or clear, your choice
Easy to read	Lower	Higher (the source is a named key)
Amount of parsing code	A bit less	A bit more (one parser handles both)

2. Read the value on every request, instead of caching it with a TTL

The proposal caches a resolved file: value for a short time (TTL). But ESO secret files sit in memory (tmpfs), and the kubelet swaps them in one atomic step. So reading the file again is safe and takes only a few microseconds. The cache mostly just saves one file read per request — and only for bearer / api_key, since OAuth2 reads client_secret only when it refreshes the token.

	Cache with TTL (proposal)	Read on every request
Cost per request	Saves one file read when cached	One small file read (~µs)
How fast a rotated secret is picked up	kubelet delay + TTL	kubelet delay only
Extra code/state	Cached value + expiry time + lock + clearing	None
Chance of using an old value	Up to the TTL after rotation	None

A TTL cache only helps if you have very high traffic and the secret is on slow/network storage — which is not the case for ESO mounts. So reading every time is simpler, picks up new secrets faster, and is correct here. We can still keep your "fail loud" idea by checking all references once when the file loads (so a missing env var or unreadable file fails right away, not on the first request).

How it could look in code

Both changes come from one small shared type (say auth.Secret) with:

• a custom JSON parser that accepts a plain string, a prefixed string, or an object,
• a Resolve() method called inside each authenticator's Apply(),
• a Validate() method run when the file loads.

This keeps all the parsing and reading logic in one place that is easy to test, keeps the list of sources (env: / file:) in one spot so vault: is easy to add later, and replaces the three inline secret string fields in the same way.