[Discussion] External secrets integration

[gramidt]

Is your feature request related to a problem? Please describe.
The Collector does not provide a mechanism to retrieve secrets from external systems or processes. The current mechanisms are to either hardcode the secret in the corresponding config or use environment variable expansion. While the latter is better than the former, it still has significant room for improvement, since environment variables are prone to being accidentally exposed.

Key problems to think about:

• Expiration and auto-rotation of secrets/credentials
• The secrets management ecosystem is vast ( Hashicorp Vault, Volterra Vault, AWS Secrets Manager, AWS Cloud HSM, GCP Secrets Manager, Azure KeyVault, ...)
• Not all users will have an HSM or secret manager, but would still like to store their secrets more securely
• While not directly related, there should be a way to mark a config setting as "sensitive" and mask "sensitive" fields out if printing the config (debugging, etc.)

This problem is currently affecting multiple components that rely on secrets. If any component would like to adopt a more secure mechanism for secret management, they would have to implement them independently. The below list is just to reference a type of issue that currently exists, and is not to be considered a comprehensive list:

• newrelicexporter: Add warning to user for insecure storage of user credentials at rest opentelemetry-collector-contrib#2233
• sapmexporter: Add warning to user for insecure storage of user credentials at rest opentelemetry-collector-contrib#2232
• dynatraceexporter: Add warning to user for insecure storage of user credentials at rest  opentelemetry-collector-contrib#2231
• datadogexporter: Add warning to user for insecure storage of user credentials at rest opentelemetry-collector-contrib#2230

Describe the solution you'd like
I would love a mechanism that's similar to the current environment variable expansion but allows for integration into secret managers, HSMs, and other options such as files. This approach would enable all components to support secure secrets without having to do anything initially. The tricky part is determining how to support secret rotations without restarting the collector.

Describe alternatives you've considered

• Use a sidecar proxy approach when deploying the collector and try to instrument authentication there when applicable; however, this solution does not cover all bases and requires yet another process to run and manage.
• Use a sidecar approach when deploying the collector and use a custom templating to template the config prior to starting the collector. Still not a secure solution and requires yet another process to run and manage.

[gramidt]

One idea could be to use something like gomplate (MIT License), since it has many of the needed integrations already built in ( see https://github.com/hairyhenderson/gomplate/tree/master/data ). It could be used within config/config.go and would follow a similar pattern to the current environment variable expansion (

opentelemetry-collector/config/config.go

Line 683 in 9de6dd7

// expandEnvLoadedConfig is a utility function that goes recursively through a config object

), and be additive, so that the current method of variable expansion will continue to work. If implemented, we'll just need to ensure that the templating does not clash with Helm.

[gramidt]

@alolita @jpkrohling - While not critical at this moment, could you provide your thoughts on this when you have some spare cycles?

[jpkrohling]

I like the proposal and I could have made use of that for the PR that introduced the PerRPCCredentials option, where we did touch the topic of a generic mechanism to get conf value from external files.

I would make this opt-in with the possibility for components to register a callback, so that they can reload whatever needs to be reloaded when the config value changes.

[tigrannajaryan]

@pjanotti is looking into this.

[gramidt]

@pjanotti - Let me know if you would like assistance in your research and initial PRs.

[pjanotti]

Hi @gramidt, @jpkrohling, @alolita and @tigrannajaryan here is a doc with a design based on some prototyping. I would love to get some quick feedback so we can make this available soon.

[jpkrohling]

Thanks for the proposal, @pjanotti! I left a couple of comments there.

[gramidt]

Thank you for the proposal, @pjanotti! I have added it to my queue to review this afternoon.

[gramidt]

@pjanotti - Passing this document around to get further feedback from our teams. I'll be sure to share the feedback in the near future.

[gramidt]

@pjanotti - I apologize for the delay. Our team overall loved the proposal. It went above what our expectation were and couldn't be more excited to see this take form. Let me know if you need anything at all.