-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Print configuration used by OTEL Collector at runtime #5223
Comments
To expand on the problem here: it's hard to know what options are an API token, and we probably don't want to have an opt-in system for this where people declare "this field is an API token", since the probability that we or downstream distros forget to mark something as an API token is too high, and it conflicts with the "secure by default" philosophy of the OpenTelemetry project. As a start, we can assume some types are safe to show by default (we can always show bools, ints, I think this is the first problem that needs to be solved for this feature to eventually happen. |
@mx-psi I updated my original post with examples who secrets are handled by prometheus and promtail. I have no idea how they implemented it interally, but as per documentation:
So maybe OTEL can look into how the folks of prometheus have done it.
But I guess this will always be a possiblity that is hard or impossible to protect against? |
@Mario-Hofstaetter @mx-psi first of all, thank you for this issue. I have some confusions/questions here:
PS: @Mario-Hofstaetter metrics are also "pushed" and not exposed :) This is a bit of a biased coming from prometheus world. |
Neither, I guess I should not have mixed it in here. The two flags can be combined, but seperately they mean:
To keep this issue compact, I could move the dry run feature to another issue if you'd like?
exporters:
logging:
loglevel: debug to print infos about receiver, processor, export behavior. For OTEL collector, dry-running would IMHO mean:
However I am not entirely sure what is logged by telemetry logs in |
@bogdandrutu I was thinking about a similar feature available in the Datadog Agent, where the API/application keys are censored, but I didn't have a security model in mind, so it may be possible that we can get away without doing this if the approach is different. IIRC in the Datadog Agent we do this by exposing it on an endpoint, and the output is also something that gets sent to Datadog for technical support, so I assume these are the main reasons why we want edto remove the keys from the output in the Datadog case.
I guess if we support changing the configuration at runtime, we want to expose it in an endpoint, right? Also, with 'a tag', do you mean an opt-in or opt-out tag? And lastly, do you have thoughts about edge cases like how to censor HTTP headers on, say, the OTLP HTTP exporter? |
For starters a Sure it doesn't let you get the config from the running collector. But at least it provides a way to see the merged config after file inclusions and processing of multiple input files. Right now I'm struggling with an issue where |
@ringerc Adding functionality to the validate subcommand seems independent from this issue (solving this issue would not help with your problem since you don't have a running Collector). Could you file a separate issue for the 'validate' ask? |
@mx-psi The validate issue was handled separately. I was just using it as an example for why it's important to be able to see the config as actually resolved after merging of sections etc. |
Suggestion: --print-config-insecure, which prints everything without masking. Per Bogdan's comment, the user running such command already has runtime access to all secrets. For people who want an output with masking, let's keep another issue if someone wants to come up with a solution. But right now that extra requirement is blocking simple implementation of a much needed feature. |
Hi, any updated on this issue? This is a very helpful feature, especially when we have multiple source of configs |
The OpAMP folks have been making some progress on this, see #6833 and more recently #10282. I think the work OpAMP is doing could be leveraged here if someone is willing to pick this up. cc @evan-bradley, do you have any thoughts on this? |
This will be possible with sensitive values redacted once #10139 is complete. I plan to work on displaying this in the zPages extension afterward. Would we like a command/flag to display this information as well, or would the zPages extension be sufficient? |
zPages/something similar would be required to fully support this if configuration changes at runtime. I am not opposed to a command/flag as well if users ask for it |
+1 for the command/flag. Here is our use case: we are using the splunk-otel-collector, and plan to use the include config source feature which combines multiple template or plain files into one config file. Printing configuration via command/flag can surely help users understand what the complete config looks like before running the collector. |
I landed up dropping all use of DataDog so as the original requestor I won't be able to contribute here. |
+1 for command flag. The reason is that this would allow us to inspect the resolved configuration (again, similar use case to that mentioned above where we are either including additional configs from remote sources that get merged in to the final config) prior to collector startup. |
@evantorrie that's a great point. I see two different modes for this, one is "show exactly what I configured", which takes care of any resolutions and/or config merging, but before any of the collector logic kicks in. This can never fail. And the 2nd mode is to print the final configuration after parsing and defaults are applied - this might fail due to unmarshaling errors. |
Is your feature request related to a problem? Please describe.
otelcol can be configured by one or multiple
yaml
configuration files, which also support environment variable expension.Therefore, especially when using ENV vars, it is useful to be able to view the config otelcol is actually running with.
AFAIK currently there is no such feature.
Below I will suggest solutions I am using on a daily basis with other observability tools.
To cite @mx-psi from slack:
Somewhat remotely related to that concern: #2466
Describe the solution you'd like
All solutions should consider secrets like api tokens, any maybe censor them ******
Print configuration in command line
Probably the easiest way. For example Grafana promtail has a CLI flag
-print-config-stderr
which just dumps the config into the commandline, including expanded ENV vars. If the seperate-config.expand-env
flag is omitted, it is clearly visible where ENV vars are used in the configuration. This could be done forotelcol
in exactly the same way (however ENV vars are expanded by default in otelcol)Regarding secrets, I did a test using promtail
basic_auth
, and the password is actually masked with<secret>
in the console output:They also support a
-dry-run
flag, so nothing is actually pushed (otelcol
equivalent for dry-running would be: Don't push logs & traces, don't expose metrics), but I guess that could be a feature request on its own.Show configuration in HTTP endpoint
Otelcol has several HTTP endpoints (none active by default?), which could enable showing the active configuration in plain text.
For example, using
extensions: [health_check , zpages]
adds two seperate http endpoints.Maybe an optional extension "
config
" (lack of a better name) could be configured in a similar fashion to expose the configuration via http.prometheus
, its config (html) is available in http://localhost:9090/configClick to show prometheus web config <<<
Regarding secrets, prometheus does seem to hide secrets in the config:
promtail
, the config (html) is available in its http server: http://localhost:9080/configClick to show promtail web config <<<
Regarding secrets, I did a test using promtail
basic_auth
, and the password is actually masked with<secret>
in the web endpoint too:You get the idea.
Describe alternatives you've considered
?
The text was updated successfully, but these errors were encountered: