Support for environment variable substitution in configuration file

[dopuskh3]

I think that would be a good idea to substitute environment variables in the configuration file.

That could be done really easily using os.ExpandEnv on configuration string when loading configuration string.

That would be much easier to substitute environment variables only on configuration values. go -ini provides a valueMapper but yaml.v2 doesn't have such mechanism.

[brian-brazil]

We've previously discussed several times and decided against. For simplicity we keep to one way to configure a given thing, otherwise we'd end up with 4-5 ways.

[alexellis]

This could be a particularly powerful way to configure Prometheus in a cluster without the burdensome task of a separate config file for every configuration you can ever deploy.

[nutchalum]

It's a nice to have.
Now I've encountered a problem with external_label.
I've deploy prometheus Deployment and ConfigMap via kubernetes federation.
All of them have the same configuration, except external_label.
I can't think a way to injected this external_label (like source: clusterA)
So I have to deploy ConfigMap in the old way.
(I have around 7 clusters, that means 7 times instead of 1 time via Federation)

[Nick-Triller]

Environment variable substitution would allow to keep sensitive data like passwords used for remote read/write out of the config file, too.

InfluxData recently mentioned environment variable substitution in config files would be a useful feature in their eyes.

[ababushkin]

I don't think it would cause usability issues guys, its pretty common to use environment variable substitution in config files like these. Any chance to revise that decision?

Here are some benefits:

• Keeps sensitive configuration out of config files
• Simplifies build / deployment process, especially in containerised environments
• Inline with 12factor principles (https://12factor.net/)

Am I missing something here?

[boeboe]

Any update on this?

Related to #2664

Eventually we just ended up having separate configuration files per cluster, which was annoying since it was an exception among other services deployed within our clusters.

PS: another option is to provide small wrapper script doing this cluster specific substitions (sed -i prometheus.yml like) before actually starting prometheus. You will need a proper signal handling mechanism like dump-init however (https://github.com/Yelp/dumb-init), since you're prometheus will not be running as PID 1 anymore in this case.

# Install dumb-init
RUN curl -L https://github.com/Yelp/dumb-init/releases/download/v${DUMB_INIT_VERSION}/dumb-init_${DUMB_INIT_VERSION}_amd64 >/usr/bin/dumb-init &&  chmod +x /usr/bin/dumb-init 
...
ENTRYPOINT ["/usr/bin/dumb-init", "/entrypoint"]
CMD ["/bin/prometheus", "-config.file=/etc/prometheus/prometheus.yml", \
         "-storage.local.path=/prometheus", \
         "-web.console.libraries=/etc/prometheus/console_libraries", \
         "-web.console.templates=/etc/prometheus/consoles"]

... and within your entrypoint script:

#!/usr/bin/env bash
export CLUSTER_NAME=`grep search /etc/resolv.conf | awk '{print $NF}'`
# assuming your ConfigMap with prometheus.yml has a jinja style {{CLUSTER_NAME}} string for substitution...
sed -i "s/{{CLUSTER_NAME}}/${CLUSTER_NAME}/g" /etc/prometheus/prometheus.yml
exec "$@"

The disadvantage of this approach is you're introducing an external component dumb-init, breaking single process per container best practices, which some might not be willing to do.

[bwplotka]

We have similar issues when we run Prometheus in 2-replica statefulset which reuses single configmap. To support HA we have external_label called replica that suppose to have unique replica name (e.g hostname). We end up doing this:

<stateful-set>
...
spec:
      containers:
      - args:
        - -c
        - sed s/'@@HOSTNAME@@'/${HOSTNAME}/g /etc/prometheus/prometheus.yaml.tmpl
          > /etc/prometheus/prometheus.yaml && /prometheus/prometheus --config.file=/etc/prometheus/prometheus.yaml
          --log.level=info --query.max-concurrency=100 --query.timeout=2m --storage.tsdb.path=/prometheus-data
          --storage.tsdb.min-block-duration=2h --storage.tsdb.max-block-duration=2h
          --storage.tsdb.retention=12h --storage.tsdb.no-lockfile --alertmanager.timeout=1ms
        command:
        - /bin/sh
...

This is not fun and this solution screams for better approach. But I feel like this might be rather Kubernetes issue not enabling that, not necessarily Prometheus one.

[bwplotka]

Maybe moving external_labels to flags would at least fix that case?

[brian-brazil]

&& /prometheus/prometheus

A small tip: When you are using the shell like this where application is the last thing you are running do && exec /prometheus/prometheus so that the application replaces the shell process. This tends to make signal handling easier.

I'm unsure of the wisdom of using a hostname as an external label, that's going to make any remote storage endpoint have labels vary as Prometheus moves around over time. Two separate statefulsets with different label values like mydc1 and mydc2 might be better.

[bwplotka]

Cool, thanks for the advice!

Our host name is actually pod name so I think this is fine (: Won't change easily.

[fabxc]

Yea, the name for stateful sets is deterministic in contrast to deployments, where pods get a random suffix. So it will always be prometheus-infra-0, prometheus-infra-1 etc.

[liyaka]

I need to configure azure_sd_configs with sensitive info like client_secret and etc.
Since I keep my configuration in git, there is no option this info can appear in configuration file as is...
Naturally I would think of using environment variables and set them while deploying with Ansible.
What would be the best way to do it?..

[brian-brazil]

Naturally I would think of using environment variables and set them while deploying with Ansible.

Environment variables are a bad idea for secrets. Ansible has its own secret and templating support, which is the standard way to approach this.

[abh]

FWIW, the prometheus-config-reloader for prometheus-operator looks to have implemented a solution for this for the statefulset use case recently:

coreos/prometheus-operator#1132

coreos/prometheus-operator@608e714#diff-a2dba0afa9edde401541fc9054415bba

[bwplotka]

If you use Thanos you have that out-of-the-box as well: https://github.com/improbable-eng/thanos/blob/master/cmd/thanos/sidecar.go#L73

[orubel]

One reason to have this is so that you can have better devops for SWARMS

If you are doing lots of builds, you want to be able to template your SWARMS and this does NOT make that possible. I build 4-50 node swarms constantly based on customer demands and being able to dynamically configure them would be IDEAL and save me hours.

[dbendelman]

B"H

We solved this by creating an entrypoint script that preprocesses environment variables before calling the binary. Example showing how to set --log.level:

# Prometheus
docker run -d -p 9090:9090 -e PROM_LOG_LEVEL=error hiveco/prometheus:v2.2.1

# Alertmanager
docker run -d -p 9093:9093 -e ALERTMGR_LOG_LEVEL=error hiveco/alertmanager:v0.15.0-rc.1

Here are the Prometheus and Alertmanager images, with sources linked there. Multiple tags are available depending on the version you need. These are super bare-bones wrappers around the official images, simply working around this particular issue and then getting out of the way.

The Prometheus authors made a great product and are making sound arguments for not including this functionality natively. The above images, however, might help people whose infrastructures or policies allow for taking a more relaxed approach.

[djbingham]

Environment variables are a bad idea for secrets.

What about using Docker secrets? My understanding is the expected way of controlling secret values in containers is to add a Docker secret, which generates a file inside your container, then within the container read that file into environment variables at runtime.

[brian-brazil]

@djbingham That is still putting secrets in environment variables.

[djbingham]

@brian-brazil Yes it is, but I'm led to believe (from the official documentation for Docker) that this is the expected way to handle such things. If that is actually a bad idea, please could you elaborate on why?

[brian-brazil]

See http://movingfast.io/articles/environment-variables-considered-harmful/ and https://diogomonica.com/2017/03/27/why-you-shouldnt-use-env-variables-for-secret-data/ for the reasoning. Environment variables were never designed with secrets in mind, nor are they traditionally a place to put secrets.

[djbingham]

Good explanations, thanks.

[dbsanfte]

Secrets are being used as an excuse to avoid adding some pretty basic and expected functionality, methinks.

http://docs.grafana.org/installation/configuration/#using-environment-variables

Justify what's wrong with that, as not just Grafana but NPM and others have the exact same regime of overrides.

[fljmayer]

So we have to put environment-specific proxy settings into a configuration file that can otherwise be kept environment-independent? That doesn't seem like good solution.

[sgeisbacher]

Environment variables are a bad idea for secrets.

@brian-brazil but storing them statically in config-files in your git-repo is better? 👎 👎 👎

We've previously discussed several times and decided against. For simplicity we keep to one way to configure a given thing, otherwise we'd end up with 4-5 ways.

@brian-brazil but running "config-pre-render-scripts" to set aws-secret-key in different docker-deployments increases simplicity? 👎 👎 👎

imho sometimes you have no clue what you are talking about, sorry.

[Jake-Smullin]

Env variables make so much sense here... Can't check in my AWS creds for an ec2_sd scrape job, and mounting them in as secrets to ENV vars and then leaving in the scrape config $ACCESS_KEY for example would be perfect.

[speedooo]

Is it possible use the same way as file_sd_configs reading details fromm external file for external_labels?just read key-value pair

[mohr-michael-adam]

Is it safe to launch Prometheus (or AlertManager) then erase the config file after it is running? This way the sensitive data only exists on disk in plaintext for a brief moment. Not claiming this to be secure but it helps.

[logemann]

I think its obvious for several reasons to have that feature. For me, i want to have a more dynamic way of definiting ports of targets depending on the docker-compose.yml file. But i could think of thousands more.

[zakkg3]

This should be a must-have, not a nice to have.
@brian-brazil what's the problem on having 2 different ways of configuring things? Like ELK, grafana, and.. probably most of the things :D

[logemann]

@zakkg3 you wont get any positive feedback from the author regarding that. As you see in my closed reminder case. Sad but true. He dont mind that any other framework/product has such a mechanism.

[zakkg3]

I see @logemann I ve made a custom image with confd to workaround, feel free to fork:
https://github.com/zakkg3/Prometheus-confd

[nmaupu]

I am also surprised not to be capable of doing that. I would rather prefer using http_proxy environment variable instead of having to put this in global/http_config/proxy_url while it's already available and configured elsewhere. Those configurations are environment dependent and should be easily replaced in the same base code...

config:
  global:
    http_config:
      proxy_url: ${http_proxy}

would be much much better.

By the way, alertmanager doesn't seem to use http_proxy and no_proxy standard environment variables, am I missing something here ?

[Qix-]

@brian-brazil

We've previously discussed several times and decided against. For simplicity we keep to one way to configure a given thing, otherwise we'd end up with 4-5 ways.

You're encouraging poor security practices by keeping credentials on the filesystem during e.g. Dockerfile-based builds. Not being able to configure secrets via environment variables is going to cause vulnerabilities in production systems.

It's quite sad it's not available.

[brian-brazil]

See #2357 (comment) above, putting secrets in environment variables is not a good security practice.

[Qix-]

@brian-brazil You're basing that off one article that has a bunch of holes in it.

You can easily see environment variables using procfs, sure. They are also inherited to child processes, sure. But this is assuming you're running untrusted code in production in the first place. Most people that know what they're doing, or have proper QA processes in place, are not doing this.

Assuming a container has been compromised means that any form of secret storage is vulnerable. The only alternative is to somehow detect that the production application has consumed the credentials and to dispose of them - which means you have to trust the application to also dispose of them in memory.

That's a tall order for most software today, and highly unrealistic in terms of threat modeling - to require that degree of secrets handling isn't necessary in what I'd consider the "common case" for most production cases. The threat model generally takes into account the method of secret handling.

Storing credentials in a repository is a non-starter. That violates federal regulations under which we operate. I assume the same applies to many people.

Storing credentials in k8s secrets doesn't work because we're stuck with volume mounts or environment variables, both of which you're staunchly against.

As you've mentioned elsewhere, you are now expecting us to write glue code that initializes the configuration at runtime as an added step using a substitution method (e.g. templates) to pull in the secrets from volume mounts or environment variables anyway in order to build up prometheus configuration.

That's insane.

It isn't your job to dictate my system's threat model. It's my job. That's what I was hired to do.

You have an army of users asking you for this feature. You're declining for a dogmatic reason that contradicts many of the points made against both your reasoning as well as against the articles you yourself linked to back up your viewpoint.

Not having this feature is getting in the way of my work, and causing me more security overhead than not having it as I have to write configuration scripts which implies error surface area and liability. Plus, it's another piece of code I have to have audited.

Please reconsider. Even if you put a warning label in big, red capital letters next to the documentation for environment configuration, whatever it takes.

---

EDIT: some more points:

But sometimes software dumps out its environment in times of debugging or failure.

No it doesn't. I've yet to see a production-ready piece of software recklessly dump the entirety of its environment map to stdout/err/file/etc. when it dies. Not only is it needless, it doesn't do anything to help developers anyway. Rarely are problems in production related to the environment, especially with containers.

But child processes can maliciously access the parent process's secrets if they're exposed via environment variables.

This is assuming you're running software that runs child processes, and those child processes aren't also trusted. Compartmentalized security, credential auditing, rotation, codebase auditing, firewall rules - all of these things that should be happening will combat this.

But writing to the filesystem prevents all of that for free.

Again, this is only a net benefit if you are worried about the process somehow transmitting the credentials - which, in most cases, is not something you would worry about. Plus, if it's running in the container, you should be modeling the security as inherently compromised by the child process anyway - therefore, the assumption is that the child process can still read the filesystem just fine.

But what about UNIX permissions?

Sure, I guess. It's still a weak argument against proper security modeling, and only increases the chance for mis-configuration from within the container itself.

There's still the problem of secrets storage and transportation into the container or machine itself, too. That's fixed by exactly none of this.

Also, it's better if the application disposes of the credentials from memory too, which means you can unlink the credentials from the filesystem.

Sounds great - except not even prometheus can realistically do this. It runs based on HTTP + REST, which means it has to keep any URIs or bearer tokens in memory, which also means that it can't feasibly dispose of them if it wants to make more than one read/write.

[nmaupu]

Using environment variables for configuration is not necessary to pass critical secrets to a container. My use case (using HTTP_PROXY env vars) is not really secret, it is pushed via a podpreset and it would be neat to be reused instead of repeating the code elsewhere ...

[alexellis]

On the OpenFaaS project we are also advising users not to store secrets in environmental variables, but to use secrets in files instead.

I saw that the configuration for HTTP alert endpoints now supports reading credentials from a file at runtime. This would be ideal for other secrets too.

#3639

[Qix-]

On the OpenFaaS project we are also advising users not to store secrets in environmental variables, but to use secrets in files instead.

Why? How is that any better?

[fvclaus]

Wow. This is pretty ignorant. I am running prometheus in a docker container. To force my environment variables in, I modified the entrypoint of the official image and run this script instead:

#!/bin/ash

set -e

cp /etc/prometheus/prometheus-template.yml /etc/prometheus/prometheus.yml

env |
while IFS='=' read -r name value
do
  # Convert to lowercase: FOO -> foo
  name=$(echo $name | tr '[:upper:]' '[:lower:]')
  # Replace sed seperator character / -> \/
  value=$(echo $value | sed 's;/;\\/;g')
  # Replace occurrences in file.
  sed --in-place=.bak "s/((${name}))/${value}/g" /etc/prometheus/prometheus.yml
done

/bin/prometheus --config.file=/etc/prometheus/prometheus.yml \
  --log.level=debug \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.console.templates=/etc/prometheus/consoles

[dbsanfte]

Users are already routing around the brain damage of @brian-brazil. Very satisfying to watch.

[Qix-]

@alexellis Please respond. How does that help? I'm genuinely curious.

[abh]

Wow. This is pretty ignorant.

It’s hard to be motivated to help when you start out like that.

I am running prometheus in a docker container. To force my environment variables in, I modified the entrypoint of the official image and run this script instead:

Pay attention to the file paths, use a temporary file in /tmp if necessary. Remove the -i option from sed if you don’t need the backup file created (or again, use a different path). Those are basic file permission / shell scripting things, unrelated to Prometheus.

[Qix-]

/tmp is not magic. What benefits does it provide in this case?

I don't see how a backup file makes any difference.

Neither of those things have anything to do with file permissions.

Please explain @abh.

[abh]

The message I replied to originally mentioned how the proposed shell script got various permission errors. Having a “sidecar” to process configuration and reloads is a completely reasonable and easily implementable setup. The important part is for the primary software to have good facilities for managing configuration reloads, implementing every possible configuration source or templating doesn’t make any sense. You want environment variables. I want consul. She wants a grpc api. He wants integration with an ansible system.