Skip to content

feat: add external credential resolvers for config values#143

Merged
JoshMock merged 5 commits into
mainfrom
feat/external-credential-support
Apr 14, 2026
Merged

feat: add external credential resolvers for config values#143
JoshMock merged 5 commits into
mainfrom
feat/external-credential-support

Conversation

@MattDevy
Copy link
Copy Markdown
Contributor

@MattDevy MattDevy commented Apr 14, 2026

Summary

Adds support for fetching credentials from external sources via $(resolver:params) expression syntax in config files, as proposed in #128.

  • file: read from a file, e.g. $(file:/run/secrets/elastic_api_key)
  • env: read from environment variables, e.g. $(env:ELASTIC_API_KEY)
  • cmd: execute a shell command, e.g. $(cmd:pass show elastic/api-key)
  • keychain: read from macOS Keychain, e.g. $(keychain:elastic-cli/api-key)

Expressions are resolved after YAML parsing but before Zod validation, so downstream code (schemas, handlers, transport) requires zero changes. Any string field in the config supports expressions.

The resolver registry is extensible via registerResolver() for future sources.

Note: This also updates the README Configuration section to reflect the home-directory-only discovery from #142.

I've taken a stab at this based on the discussion in #128. Happy to tweak the approach or close this if the team prefers a different direction.

Future resolvers

The following could be added as future resolvers. Unlike the four included in this PR (which have zero external dependencies), these would require their respective CLIs or SDKs to be installed:

  • 1password - $(1password:op://vault/item/field) via the op CLI
  • vault - $(vault:secret/data/elastic#api_key) via HashiCorp Vault CLI
  • aws_sm - $(aws_sm:my-secret-name) via AWS CLI (aws secretsmanager get-secret-value)
  • gcp_secret - $(gcp_secret:my-secret/versions/latest) via gcloud secrets versions access

Test plan

  • npm run build && npm run test:unit passes (670 tests, 0 failures)
  • npx tsc --noEmit passes
  • npx eslint src passes
  • 38 new tests in test/config/resolvers.test.ts covering:
    • Expression parsing (single, embedded, multiple, duplicate, malformed)
    • Deep object walk (nested objects, arrays, primitives, error field paths)
    • file resolver (read, trimming, nonexistent file)
    • env resolver (set, unset, empty)
    • cmd resolver (success, failure, error messages)
    • keychain resolver (macOS, non-macOS, format validation, shell escaping)
    • Integration through loadConfig pipeline

Manual testing

Installed the CLI from this branch (npm run build && npm link) and verified each resolver end-to-end using --dry-run:

env resolver

$ ELASTIC_TEST_API_KEY=my-secret-from-env elastic es info --dry-run --config-file env-test.yml
dry run: inputs valid, no action performed

$ elastic es info --dry-run --config-file env-test.yml  # without the env var set
Error: Failed to resolve config expressions: Environment variable "ELASTIC_TEST_API_KEY" is not set or is empty

cmd resolver

$ elastic es info --dry-run --config-file cmd-test.yml  # api_key: $(cmd:echo my-secret-from-cmd)
dry run: inputs valid, no action performed

$ elastic es info --dry-run --config-file cmd-fail.yml  # api_key: $(cmd:nonexistent-command-12345)
Error: Failed to resolve config expressions: Command failed: nonexistent-command-12345
/bin/sh: nonexistent-command-12345: command not found

keychain resolver (macOS)

$ security add-generic-password -s elastic-cli-test -a test-api-key -w "my-secret-from-keychain" -U
$ elastic es info --dry-run --config-file keychain-test.yml  # api_key: $(keychain:elastic-cli-test/test-api-key)
dry run: inputs valid, no action performed

$ elastic es info --dry-run --config-file keychain-fail.yml  # api_key: $(keychain:nonexistent-service/nonexistent-account)
Error: Failed to resolve config expressions: Keychain lookup failed for service="nonexistent-service", account="nonexistent-account": ...
security: SecKeychainSearchCopyNext: The specified item could not be found in the keychain.

Adds $(resolver:params) expression syntax for config file values,
allowing secrets to be fetched from external sources instead of
stored in plaintext. Three built-in resolvers:

- env: read from environment variables
- cmd: execute a shell command and use stdout
- keychain: read from macOS Keychain (macOS only)

Closes #128
- Remove empty ResolverContext interface (and context param)
- Attach cause to re-thrown errors in cmd and keychain resolvers
@MattDevy MattDevy requested a review from JoshMock April 14, 2026 11:53
- Skip __proto__ and constructor keys in deep object walk
- Reject non-printable characters in keychain service/account params
@MattDevy
Copy link
Copy Markdown
Contributor Author

@JoshMock one thing to consider: expression resolution currently runs eagerly across all contexts, not just the active one. This means if you have:

current_context: local
contexts:
  local:
    elasticsearch:
      auth:
        api_key: $(env:LOCAL_KEY)
  staging:
    elasticsearch:
      auth:
        api_key: $(env:STAGING_KEY)

Running with current_context: local would still fail if STAGING_KEY is unset, because all expressions are resolved before Zod validation.

Ideally, resolution should be lazy: only resolve expressions for the active context. This requires restructuring the validation pipeline (currently Zod validates the entire config including all contexts, and it would reject unresolved expressions like $(env:X) as invalid URLs, etc). The fix would be splitting validation into two stages: structural validation of the outer shape, then full validation of just the active context after resolution.

Worth doing now, or fine to ship eager and follow up?

Supports $(file:/path/to/secret) syntax, useful for Docker/Kubernetes
secrets mounted at /run/secrets/.
- Reject non-regular files (directories, device files, sockets)
- Reject files larger than 64 KB
@JoshMock
Copy link
Copy Markdown
Member

Worth doing now, or fine to ship eager and follow up?

Let's ship the eager version for now and open an issue to track that as a future perf improvement.

Copy link
Copy Markdown
Member

@JoshMock JoshMock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fantastic implementation and impressive turnaround after our discussion yesterday. 👏

honestly, we could consider eventually extracting the resolver out and publishing it as a standalone package just for the OSS karma.

@JoshMock JoshMock merged commit e8d98a0 into main Apr 14, 2026
16 checks passed
@JoshMock JoshMock deleted the feat/external-credential-support branch April 14, 2026 16:30
@MattDevy
Copy link
Copy Markdown
Contributor Author

fantastic implementation and impressive turnaround after our discussion yesterday. 👏

honestly, we could consider eventually extracting the resolver out and publishing it as a standalone package just for the OSS karma.

That would be such a cool idea! I'll raise issues to track both

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants