Support for environment variable substitution in configuration file

[dopuskh3]

I think that would be a good idea to substitute environment variables in the configuration file.

That could be done really easily using os.ExpandEnv on configuration string when loading configuration string.

That would be much easier to substitute environment variables only on configuration values. go -ini provides a valueMapper but yaml.v2 doesn't have such mechanism.

[brian-brazil]

We've previously discussed several times and decided against. For simplicity we keep to one way to configure a given thing, otherwise we'd end up with 4-5 ways.

[alexellis]

This could be a particularly powerful way to configure Prometheus in a cluster without the burdensome task of a separate config file for every configuration you can ever deploy.

[nutchalum]

It's a nice to have.
Now I've encountered a problem with external_label.
I've deploy prometheus Deployment and ConfigMap via kubernetes federation.
All of them have the same configuration, except external_label.
I can't think a way to injected this external_label (like source: clusterA)
So I have to deploy ConfigMap in the old way.
(I have around 7 clusters, that means 7 times instead of 1 time via Federation)

[Nick-Triller]

Environment variable substitution would allow to keep sensitive data like passwords used for remote read/write out of the config file, too.

InfluxData recently mentioned environment variable substitution in config files would be a useful feature in their eyes.

[ababushkin]

I don't think it would cause usability issues guys, its pretty common to use environment variable substitution in config files like these. Any chance to revise that decision?

Here are some benefits:

• Keeps sensitive configuration out of config files
• Simplifies build / deployment process, especially in containerised environments
• Inline with 12factor principles (https://12factor.net/)

Am I missing something here?

[boeboe]

Any update on this?

Related to #2664

Eventually we just ended up having separate configuration files per cluster, which was annoying since it was an exception among other services deployed within our clusters.

PS: another option is to provide small wrapper script doing this cluster specific substitions (sed -i prometheus.yml like) before actually starting prometheus. You will need a proper signal handling mechanism like dump-init however (https://github.com/Yelp/dumb-init), since you're prometheus will not be running as PID 1 anymore in this case.

# Install dumb-init
RUN curl -L https://github.com/Yelp/dumb-init/releases/download/v${DUMB_INIT_VERSION}/dumb-init_${DUMB_INIT_VERSION}_amd64 >/usr/bin/dumb-init &&  chmod +x /usr/bin/dumb-init 
...
ENTRYPOINT ["/usr/bin/dumb-init", "/entrypoint"]
CMD ["/bin/prometheus", "-config.file=/etc/prometheus/prometheus.yml", \
         "-storage.local.path=/prometheus", \
         "-web.console.libraries=/etc/prometheus/console_libraries", \
         "-web.console.templates=/etc/prometheus/consoles"]

... and within your entrypoint script:

#!/usr/bin/env bash
export CLUSTER_NAME=`grep search /etc/resolv.conf | awk '{print $NF}'`
# assuming your ConfigMap with prometheus.yml has a jinja style {{CLUSTER_NAME}} string for substitution...
sed -i "s/{{CLUSTER_NAME}}/${CLUSTER_NAME}/g" /etc/prometheus/prometheus.yml
exec "$@"

The disadvantage of this approach is you're introducing an external component dumb-init, breaking single process per container best practices, which some might not be willing to do.

[bwplotka]

We have similar issues when we run Prometheus in 2-replica statefulset which reuses single configmap. To support HA we have external_label called replica that suppose to have unique replica name (e.g hostname). We end up doing this:

<stateful-set>
...
spec:
      containers:
      - args:
        - -c
        - sed s/'@@HOSTNAME@@'/${HOSTNAME}/g /etc/prometheus/prometheus.yaml.tmpl
          > /etc/prometheus/prometheus.yaml && /prometheus/prometheus --config.file=/etc/prometheus/prometheus.yaml
          --log.level=info --query.max-concurrency=100 --query.timeout=2m --storage.tsdb.path=/prometheus-data
          --storage.tsdb.min-block-duration=2h --storage.tsdb.max-block-duration=2h
          --storage.tsdb.retention=12h --storage.tsdb.no-lockfile --alertmanager.timeout=1ms
        command:
        - /bin/sh
...

This is not fun and this solution screams for better approach. But I feel like this might be rather Kubernetes issue not enabling that, not necessarily Prometheus one.

[bwplotka]

Maybe moving external_labels to flags would at least fix that case?

[brian-brazil]

&& /prometheus/prometheus

A small tip: When you are using the shell like this where application is the last thing you are running do && exec /prometheus/prometheus so that the application replaces the shell process. This tends to make signal handling easier.

I'm unsure of the wisdom of using a hostname as an external label, that's going to make any remote storage endpoint have labels vary as Prometheus moves around over time. Two separate statefulsets with different label values like mydc1 and mydc2 might be better.

[bwplotka]

Cool, thanks for the advice!

Our host name is actually pod name so I think this is fine (: Won't change easily.

[fabxc]

Yea, the name for stateful sets is deterministic in contrast to deployments, where pods get a random suffix. So it will always be prometheus-infra-0, prometheus-infra-1 etc.

[liyaka]

I need to configure azure_sd_configs with sensitive info like client_secret and etc.
Since I keep my configuration in git, there is no option this info can appear in configuration file as is...
Naturally I would think of using environment variables and set them while deploying with Ansible.
What would be the best way to do it?..

[brian-brazil]

Naturally I would think of using environment variables and set them while deploying with Ansible.

Environment variables are a bad idea for secrets. Ansible has its own secret and templating support, which is the standard way to approach this.