Using env_file with environment

[ferrouswheel]

The documentation is slightly ambiguous about what happens when you specify environment variables using both env_file and environment. It does say "Environment variables specified in environment override these values" but what about when environment doesn't specify the values like so:

  environment:
   POSTGRES_PASSWORD:
   POSTGRES_USER:
  env_file: .env

It'd be nice to have POSTGRES_PASSWORD and POSTGRES_USER in .env and have them in docker-compose.yml at the same time. Basically telling people, "hey, you need to provide these values" either in .env or in the shell environment you run docker-compose up in.

If that's not possible or would only add confusion, then it'd be cool to add that the override happens even when they are blank and read from the shell environment.

[cressie176]

@ferrouswheel Sorry for off topic, but have you actually got POSTGRES_PASSWORD and POSTGRES_USER to work? If I specify those variable with -e to docker run everything works fine, but using docker-compose cannot connect

[ferrouswheel]

@cressie176 I have. I use it for my open source docker-based CI tool. See the docker-compose.yml there and the README.md.

As I remember it, the postgres container sets the user/password the first time you run it, so if you run it once with no credentials, it'll use the defaults the next time your start the container. Try removing the container completely and recreating it with the credentials you want to use.

[cressie176]

Makes sense. Thanks

[thaJeztah]

@ferrouswheel so, this issue is resolved, or did I interpret your previous comment incorrectly?

[ferrouswheel]

@thaJeztah No the issue still stands. My comment was just a reply to the unrelated question.

[pikeas]

+1, just got bit by this (even the specific case of POSTGRES_PASSWORD!).

I provided POSTGRES_PASSWORD as an unset value in environment, and POSTGRES_PASSWORD=foo in .env. Compose used the empty value in my environment instead of the value in .env.

Building on this (can open another issue, but compose already has a few too many of those...), there should be a way to require strict checking of all (or a subset of) environment variables, so that containers will fail to start if those variables are not set.

The simplest thing that could work is adding a new key called environment_strict.

[aanm]

Doesn't the extends solve this?
https://docs.docker.com/compose/extends/#define-the-production-environment

[lupa18]

I'm not sure if here or I must open a new issue, but.. when I use environment values are parsing correct, but if I use env_file with exactly the same values, container doesn't works well.
The error can be reproduced for example with this:

db:
  image: mdillon/postgis
  env_file: env.conf

and this in env.conf:

POSTGRES_USER: user
POSTGRES_PASSWORD: secret
POSTGRES_DB: base

Then, no error appears, but user and database are not created. If I put them inside compose file with environment it works well.

docker-compose version 1.5.2

thanks in advance !

[dnephin]

I don't think that's a valid env_file. It should be POSTGRES_USER=user, etc. https://docs.docker.com/compose/compose-file/#env-file

[shin-]

Closing: docker/docs#4177

[damianobarbati]

@lupa18 how did you solve?

@shin- I'm not seeing the env_file respected.

.env.development file:

POSTGRES_USER=root
POSTGRES_PASSWORD=webapp
POSTGRES_DB=webapp

docker-compose.yml file:

    db:
        env_file: .env.${NODE_ENV}
        environment:
            - POSTGRES_USER=${POSTGRES_USER:?}
            - POSTGRES_PASSWORD=${POSTGRES_PASSWORD:?}
            - POSTGRES_DB=${POSTGRES_DB:?}
        image: postgres:12-alpine
        container_name: ${POSTGRES_DB:?}-db
        volumes:
            - ./docker-entrypoint-initdb.d:/docker-entrypoint-initdb.d
            - ./pg-data:/var/lib/postgresql/data
        ports:
            - 5432:5432
        restart: unless-stopped

$ NODE_ENV=development NAME=webapp VERSION=1.1.0 docker-compose up --force-recreate --always-recreate-deps --renew-anon-volumes --remove-orphans --detach --build db

ERROR: Missing mandatory value for "environment" option in service "db":

Weird thing is this happens only with postgres image, because the other image in the yml is built correctly using the envs.

[rajeshhazari]

I've this config that works, you have defined your environment variables as array is the issue:

compose-file

version: '3'
services:
  app-dev:
    env_file: ./scripts/env-${NODE_ENV}.conf
    environment:
        POSTGRES_USER: ${POSTGRES_USER}
        POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
        POSTGRES_DB: ${POSTGRES_DB}
    image: postgres:12-alpine
    container_name: "${POSTGRES_DB}-db"
    restart: always
    volumes:
        - ./pg-data:/var/lib/postgresql/data
    command: postgres -c config_file=/etc/postgresql/postgresql.conf
    ports:
        - 6432:5432
    networks:
        - dbnet
networks:
  dbnet:
    driver: bridge

scripts/env-dev.conf

POSTGRES_PASSWORD=user321
POSTGRES_USER=user
POSTGRES_DB=app

export NODE_ENV=dev; export NAME=app_db;export VERSION=1.1.0;docker-compose up -d

[lnksz]

I hit the same issue, and the above solution doesn't work.
I created a reproducer repo: https://github.com/lnksz/docker-compose-env-repro

The variable definition from the default .env file can be used in the environment: attribute, but the variable from the env_file: attribute is only appended to the list of environment values, and seems like it cannot be used to initialize another variable from that.

I just cannot see this behavior described in the docs.
https://docs.docker.com/compose/environment-variables/set-environment-variables/#use-the-env_file-attribute

The env_file attribute lets you use multiple .env files in your Compose application...

Sounds much like the env_file: attribute has the same properties like the default .env file.

[jrzerr]

The above recommended by @rajeshhazari does not work.

Slightly simplified to avoid needing to set any variables in your shell

compose.yaml

version: '3'
services:
  app-dev:
    env_file: ./env.conf
    environment:
        POSTGRES_USER: ${POSTGRES_USER}
        POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
        POSTGRES_DB: ${POSTGRES_DB}
    image: postgres:12-alpine
    container_name: "${POSTGRES_DB}-db"
    restart: always
    volumes:
        - ./pg-data:/var/lib/postgresql/data
    command: postgres -c config_file=/etc/postgresql/postgresql.conf
    ports:
        - 6432:5432
    networks:
        - dbnet
networks:
  dbnet:
    driver: bridge

env.conf

POSTGRES_PASSWORD=user321
POSTGRES_USER=user
POSTGRES_DB=app

output of docker compose config

 dockertest % docker compose config
WARN[0000] The "POSTGRES_PASSWORD" variable is not set. Defaulting to a blank string. 
WARN[0000] The "POSTGRES_DB" variable is not set. Defaulting to a blank string. 
WARN[0000] The "POSTGRES_USER" variable is not set. Defaulting to a blank string. 
WARN[0000] The "POSTGRES_DB" variable is not set. Defaulting to a blank string. 
WARN[0000] /Users/jeremyzerr/zerrtech/dockertest/compose.yaml: `version` is obsolete 
name: dockertest
services:
  app-dev:
    command:
      - postgres
      - -c
      - config_file=/etc/postgresql/postgresql.conf
    container_name: -db
    environment:
      POSTGRES_DB: ""
      POSTGRES_PASSWORD: ""
      POSTGRES_USER: ""
    image: postgres:12-alpine
    networks:
      dbnet: null
    ports:
      - mode: ingress
        target: 5432
        published: "6432"
        protocol: tcp
    restart: always
    volumes:
      - type: bind
        source: /Users/jeremyzerr/zerrtech/dockertest/pg-data
        target: /var/lib/postgresql/data
        bind:
          create_host_path: true
networks:
  dbnet:
    name: dockertest_dbnet
    driver: bridge

Result

Clearly shows that the POSTGRES env vars are blank underneath the environment: property

I don't see why .env is treated differently than setting the env_file: to a different file. I don't understand the reasoning there.

[rouba002]

I believe the answer is as always in documentation. But two things are happening here.

https://docs.docker.com/compose/environment-variables/envvars-precedence/

First, those variables, as they are not defined in your shell ENV are just replaced by blanks. Effectively overriding your env.conf, higher precedence. Try to set one of your VARIABLES in shell, eg. export POSTGRES_USER="aaa", and then docker compose config again.

The '.env' file is really treated in a special way with docker compose and is, AFAIK only source used to interpolate compose.yml file.
So if you copy your env.conf to .env yo should see variables replaced except for POSTGRES_USER, which is still taken from ENV of your shell.

https://docs.docker.com/compose/environment-variables/set-environment-variables/
There is this small warning notice

Important

Substitution from .env files is a Docker Compose CLI feature.

It is not supported by Swarm when running docker stack deploy.```

I do not claim this is 100% true, so feel free to correct me.

BR, rouba