Support for environment variable substitution in configuration file

[dopuskh3]

I think that would be a good idea to substitute environment variables in the configuration file.

That could be done really easily using os.ExpandEnv on configuration string when loading configuration string.

That would be much easier to substitute environment variables only on configuration values. go -ini provides a valueMapper but yaml.v2 doesn't have such mechanism.

[brian-brazil]

We've previously discussed several times and decided against. For simplicity we keep to one way to configure a given thing, otherwise we'd end up with 4-5 ways.

[alexellis]

This could be a particularly powerful way to configure Prometheus in a cluster without the burdensome task of a separate config file for every configuration you can ever deploy.

[nutchalum]

It's a nice to have.
Now I've encountered a problem with external_label.
I've deploy prometheus Deployment and ConfigMap via kubernetes federation.
All of them have the same configuration, except external_label.
I can't think a way to injected this external_label (like source: clusterA)
So I have to deploy ConfigMap in the old way.
(I have around 7 clusters, that means 7 times instead of 1 time via Federation)

[Nick-Triller]

Environment variable substitution would allow to keep sensitive data like passwords used for remote read/write out of the config file, too.

InfluxData recently mentioned environment variable substitution in config files would be a useful feature in their eyes.

[ababushkin]

I don't think it would cause usability issues guys, its pretty common to use environment variable substitution in config files like these. Any chance to revise that decision?

Here are some benefits:

• Keeps sensitive configuration out of config files
• Simplifies build / deployment process, especially in containerised environments
• Inline with 12factor principles (https://12factor.net/)

Am I missing something here?

[boeboe]

Any update on this?

Related to #2664

Eventually we just ended up having separate configuration files per cluster, which was annoying since it was an exception among other services deployed within our clusters.

PS: another option is to provide small wrapper script doing this cluster specific substitions (sed -i prometheus.yml like) before actually starting prometheus. You will need a proper signal handling mechanism like dump-init however (https://github.com/Yelp/dumb-init), since you're prometheus will not be running as PID 1 anymore in this case.

# Install dumb-init
RUN curl -L https://github.com/Yelp/dumb-init/releases/download/v${DUMB_INIT_VERSION}/dumb-init_${DUMB_INIT_VERSION}_amd64 >/usr/bin/dumb-init &&  chmod +x /usr/bin/dumb-init 
...
ENTRYPOINT ["/usr/bin/dumb-init", "/entrypoint"]
CMD ["/bin/prometheus", "-config.file=/etc/prometheus/prometheus.yml", \
         "-storage.local.path=/prometheus", \
         "-web.console.libraries=/etc/prometheus/console_libraries", \
         "-web.console.templates=/etc/prometheus/consoles"]

... and within your entrypoint script:

#!/usr/bin/env bash
export CLUSTER_NAME=`grep search /etc/resolv.conf | awk '{print $NF}'`
# assuming your ConfigMap with prometheus.yml has a jinja style {{CLUSTER_NAME}} string for substitution...
sed -i "s/{{CLUSTER_NAME}}/${CLUSTER_NAME}/g" /etc/prometheus/prometheus.yml
exec "$@"

The disadvantage of this approach is you're introducing an external component dumb-init, breaking single process per container best practices, which some might not be willing to do.

[bwplotka]

We have similar issues when we run Prometheus in 2-replica statefulset which reuses single configmap. To support HA we have external_label called replica that suppose to have unique replica name (e.g hostname). We end up doing this:

<stateful-set>
...
spec:
      containers:
      - args:
        - -c
        - sed s/'@@HOSTNAME@@'/${HOSTNAME}/g /etc/prometheus/prometheus.yaml.tmpl
          > /etc/prometheus/prometheus.yaml && /prometheus/prometheus --config.file=/etc/prometheus/prometheus.yaml
          --log.level=info --query.max-concurrency=100 --query.timeout=2m --storage.tsdb.path=/prometheus-data
          --storage.tsdb.min-block-duration=2h --storage.tsdb.max-block-duration=2h
          --storage.tsdb.retention=12h --storage.tsdb.no-lockfile --alertmanager.timeout=1ms
        command:
        - /bin/sh
...

This is not fun and this solution screams for better approach. But I feel like this might be rather Kubernetes issue not enabling that, not necessarily Prometheus one.

[bwplotka]

Maybe moving external_labels to flags would at least fix that case?

[brian-brazil]

&& /prometheus/prometheus

A small tip: When you are using the shell like this where application is the last thing you are running do && exec /prometheus/prometheus so that the application replaces the shell process. This tends to make signal handling easier.

I'm unsure of the wisdom of using a hostname as an external label, that's going to make any remote storage endpoint have labels vary as Prometheus moves around over time. Two separate statefulsets with different label values like mydc1 and mydc2 might be better.

[bwplotka]

Cool, thanks for the advice!

Our host name is actually pod name so I think this is fine (: Won't change easily.

[fabxc]

Yea, the name for stateful sets is deterministic in contrast to deployments, where pods get a random suffix. So it will always be prometheus-infra-0, prometheus-infra-1 etc.

[liyaka]

I need to configure azure_sd_configs with sensitive info like client_secret and etc.
Since I keep my configuration in git, there is no option this info can appear in configuration file as is...
Naturally I would think of using environment variables and set them while deploying with Ansible.
What would be the best way to do it?..

[brian-brazil]

Naturally I would think of using environment variables and set them while deploying with Ansible.

Environment variables are a bad idea for secrets. Ansible has its own secret and templating support, which is the standard way to approach this.

[abh]

FWIW, the prometheus-config-reloader for prometheus-operator looks to have implemented a solution for this for the statefulset use case recently:

prometheus-operator/prometheus-operator#1132

prometheus-operator/prometheus-operator@608e714#diff-a2dba0afa9edde401541fc9054415bba

[bwplotka]

If you use Thanos you have that out-of-the-box as well: https://github.com/improbable-eng/thanos/blob/master/cmd/thanos/sidecar.go#L73

[orubel]

One reason to have this is so that you can have better devops for SWARMS

If you are doing lots of builds, you want to be able to template your SWARMS and this does NOT make that possible. I build 4-50 node swarms constantly based on customer demands and being able to dynamically configure them would be IDEAL and save me hours.

[dbendelman]

B"H

We solved this by creating an entrypoint script that preprocesses environment variables before calling the binary. Example showing how to set --log.level:

# Prometheus
docker run -d -p 9090:9090 -e PROM_LOG_LEVEL=error hiveco/prometheus:v2.2.1

# Alertmanager
docker run -d -p 9093:9093 -e ALERTMGR_LOG_LEVEL=error hiveco/alertmanager:v0.15.0-rc.1

Here are the Prometheus and Alertmanager images, with sources linked there. Multiple tags are available depending on the version you need. These are super bare-bones wrappers around the official images, simply working around this particular issue and then getting out of the way.

The Prometheus authors made a great product and are making sound arguments for not including this functionality natively. The above images, however, might help people whose infrastructures or policies allow for taking a more relaxed approach.

[djbingham]

Environment variables are a bad idea for secrets.

What about using Docker secrets? My understanding is the expected way of controlling secret values in containers is to add a Docker secret, which generates a file inside your container, then within the container read that file into environment variables at runtime.

[brian-brazil]

@djbingham That is still putting secrets in environment variables.

[djbingham]

@brian-brazil Yes it is, but I'm led to believe (from the official documentation for Docker) that this is the expected way to handle such things. If that is actually a bad idea, please could you elaborate on why?

[brian-brazil]

See http://movingfast.io/articles/environment-variables-considered-harmful/ and https://diogomonica.com/2017/03/27/why-you-shouldnt-use-env-variables-for-secret-data/ for the reasoning. Environment variables were never designed with secrets in mind, nor are they traditionally a place to put secrets.

[djbingham]

Good explanations, thanks.

[dbsanfte]

Secrets are being used as an excuse to avoid adding some pretty basic and expected functionality, methinks.

http://docs.grafana.org/installation/configuration/#using-environment-variables

Justify what's wrong with that, as not just Grafana but NPM and others have the exact same regime of overrides.

[fljmayer]

So we have to put environment-specific proxy settings into a configuration file that can otherwise be kept environment-independent? That doesn't seem like good solution.

[sgeisbacher]

Environment variables are a bad idea for secrets.

@brian-brazil but storing them statically in config-files in your git-repo is better? 👎 👎 👎

We've previously discussed several times and decided against. For simplicity we keep to one way to configure a given thing, otherwise we'd end up with 4-5 ways.

@brian-brazil but running "config-pre-render-scripts" to set aws-secret-key in different docker-deployments increases simplicity? 👎 👎 👎

imho sometimes you have no clue what you are talking about, sorry.

[Jake-Smullin]

Env variables make so much sense here... Can't check in my AWS creds for an ec2_sd scrape job, and mounting them in as secrets to ENV vars and then leaving in the scrape config $ACCESS_KEY for example would be perfect.

[civik]

Also, in case feature parity matters to anyone - TICK stack components are adding this functionality in recent commits - its not in a release yet, but it will be soon. influxdata/telegraf#5648

[ShadowOfLies]

@brian-brazil I don't quite understand why your personal objection should make life difficult for everyone else. Environment variables are not used only for secrets.

Refusing an industry standard method of configuration seems quite unnatural in this industry.

Even closing this thread after 2 years won't make people stop asking for it.

[gannonex]

I'd like to leave a note here that might help others mired by the ignorance of some foolish crusade against environment variables. Our systems utilize stack and we were able to get some of our configuration of this project accomplished by adding an "entrypoint" subsection to the prometheus section of the configuration. Everything else we could do with a "configs" subsection using version 3.6 of compose that inserts the prometheus.yml file into place.

In discussions with my colleagues I found that we are unanimous in our belief that the stubbornness exhibited here doesn't bode well for the future of the Prometheus project. When someone offers up their work for others to save them time it is a truly wonderful and generous thing. In doing so you want your efforts to have mass appeal, be easy to use and versatile enough to wedge in just about anywhere.

Good folks of Prometheus please reconsider your desire to dictate the infrastructure of those who might make use of your project. You will save yourselves countless future threads full of angry developers who after hours of attempting to force a square peg in a round hole, unfortunately, decide to vent their frustrations upon you. This thread is a messed up situation that never had to happen, and I'm sure everyone involved would much rather be hailing you as heroes than reviling you as villains.

[Serrvosky]

I'm trying to implement Prometheus on several kubernetes clusters, and I just want to change remote write URL in each of them. Because Prometheus developers thinks that env variables are dangerous or only for sensitive data (all @brian-brazil replies are only about dangerous of putting sensitive data on them), I have to rethink about my implementation.
It doesn't take any sense, having several config maps to have almost the same information.
And it will be much more difficult to implement a automatic and simpler solution.

Please rethink about this feature. I read all the comment of this topic, and a lot of people ask for this feature. I think it will be good for Prometheus to have it.

The best part of this topic:

brian-brazil closed this on Jan 21, 2017

It's still in discuss for a lot of people.

[Qix-]

I don't see a point in even responding/using prometheus if @brian-brazil is just going to let this issue rot and refuse to discuss anything with users.

[BarakBD-Globality]

Seems that there are plenty of users with this requirement. Why not just put it up for a poll? A good product is built by requirements, not individual opinions.

[Qix-]

I appreciate the staunch requirement of security, actually. I think a poll could end disatrously if the wrong things were polled for. However in this case, there is little to no evidence environment variables in sandboxed environments pose much threat, if any. It's just a case of the BDFL not caring about what people say.

[kron4eg]

There was a solution posted in this issue already: #2357 (comment)

[BarakBD-Globality]

@kron4eg - Thanks, but that is only for K8s. Not sure it will work for ECS.

[BarakBD-Globality]

envsubst (part of gettext) could be a solution, however I could not find a way easily install it in Prometheus Docker. This is also much less secure than simply having the templating for config files.

[dbsanfte]

Tbh the only solution to obstinate stubborness like this is a fork.

[proskehy]

Wow. This thread is, baffling.

[Markbnj]

I'll leave a vote in favor of allowing configuration from environment variables. It's a standard approach in cloud deployments, is well-supported by existing deployment tooling, and like others I am not very convinced by the arguments against using the environment to pass secrets. The ability to read secrets from mounted files would make it possible for people who are convinced by those arguments to avoid that vector. As it stands now the various proposed work-arounds add complexity and are worse than the alternatives imo.

I'd also like to put in a strong vote for civility toward the project maintainers. Prometheus is an amazing tool that we get to use for free, thanks entirely to the volunteer efforts of the maintainers. Ad hominem attacks on them are childish and unproductive imo.

[iordaniordanov]

+1 for env vars

[kinghuang]

I'm working on a federated deployment, and the inability to easily fill values on things like external_labels using environment variables is just maddening.

[rbkaspr]

@kinghuang Having read the response to your issue, the idea that configuring Prometheus is something that needs to be handled outside Prometheus is something I find utterly baffling.

On a personally relevant note, I've actually been migrating my organization away from Prometheus and over to Telegraf. The fact the environment variable substitution is supported out of the box is something I consider critical to a distributed monitoring solution. It's a shame that the current CNCF graduated monitoring solution considers something like environment variable substitution for configs outside the scope of the tool itself, when so many other tools both within and without CNCF consider such a thing a standard feature. Hopefully at some point this attitude will change.

[kron4eg]

https://www.robustperception.io/environment-substitution-with-docker

[slifty]

How has Grafana managed to provide support for environment config without becoming a full blown configuration management service?

I'll add my use case to the pile!

I'm using Docker compose + Amazon ECS to set up a Prometheus / Grafana / Alertsmanger.

For the Grafana portion of our setup we are able to deploy semi-configured images stored on dockerhub (the images and config files are fully are public), and configure secrets using environment variables passed into the container.

I'd like to add alertmanager to our stack but without support for env configuration I'm left unsure how I might configure alertmanager without baking the secret into the image (which is a bad practice and would mean I have to either use Amazon's proprietary container registry or paying for private dockerhub). I imagine I'll face the same problem when I get around to configuring the prometheus image to be more secure.

It sounds like I'm going to have to write a script to generate a config file within the container based on passed-in environment variables. I'm left unsure why the Prometheus team demands this of me.

Update: Good to know that Telegraf exists @rbkaspr, I will definitely check that out for future projects.

[roidelapluie]

For ayone following this issue: https://groups.google.com/forum/#!topic/prometheus-developers/tSCa4ukhtUw

Please stay polite and respectful in the discussions.

[beorn7]

Thanks @roidelapluie for beating me to it. It was my intention to backlink from here.

For context: The linked mail thread is a formal voting thread to get an explicit statement from the 22 members of the Prometheus team. (See the governance document to see who is a team member and what that means.)

[Qix-]

Doesn't matter, they're all voting no without giving any information.

Time to fork.

[rbkaspr]

I get the frustration surrounding this issue, but let's wait until we have a majority of nay votes before we start talking forks. So far we only have 2 nays and one who seems to be leaning hard into a nay vote. Even folks who have previously closed issues relating to this feature request are at least having a reasoned discussion about implications, and not just slapping it down.

That said, I do feel that some of the arguments against environment variable expansions are head-scratching, to say the least. The main argument I really take issue with is the idea that somehow adding environment variable expansion as a config option for Prometheus will somehow turn it into a CM tool. As someone who's day-to-day job involves working with actual CM tools, this feature is not magically going to turn Prometheus into Salt, Chef, Ansible, or Puppet. If someone were to request Prometheus suddenly start pushing configurations out to other tools and systems, I'm confident that no one in the user community would bat an eye at such a request being rejected. When you have developers closing issues about how to configure Prometheus as out of scope for the Prometheus team, that's where people start having issues. Especially when (as has been mentioned before), so many other projects both in and out of CNCF have managed to solve this issue, and Kubernetes makes it so simple to take advantage of.

With that out of the way, I appreciate @beorn7 and @roidelapluie linking the developer discussion here, it's good to see that the team is actually having a serious discussion about this request internally rather than just silently rejecting it without explanation. Hopefully we'll land on a positive outcome once January 6th rolls around.

[brian-brazil]

The vote is complete, with results tallied at https://groups.google.com/forum/#!msg/prometheus-developers/tSCa4ukhtUw/J-j0bSEYCQAJ

You can read the whole thread for full details and discussion, but the brief summary is that the vote was 11:2 in favour of keeping the status quo of not supporting environment variable substitution.

Our governance means that this outcome is final, barring a future vote overturning this. I'll now be locking this issue. You are always free to bring things up on the -users/-developers list.

[roidelapluie]

A second vote happened this week and the special use-case of external labels was considered for env variable substitution.

Starting Prometheus 2.27, we will be able to substitute env variables in external labels if you use --enable-feature=expand-external-labels

This can be tested with the main docker image.