Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

--env-file option #6170

Closed
nmaro opened this issue Sep 6, 2018 · 27 comments · Fixed by #6535

Comments

@nmaro
Copy link

@nmaro nmaro commented Sep 6, 2018

This option has been discussed in a couple of tickets but never had a ticket for itself.

One should be able to execute

docker-compose up --env-file=.env.test (perhaps multiple env files)

to override the usage of .env file.

@alancapc

This comment has been minimized.

Copy link

@alancapc alancapc commented Sep 6, 2018

Good idea, this one.

@shin-

This comment has been minimized.

Copy link
Contributor

@shin- shin- commented Sep 6, 2018

Current plans for environment files are spelled out here: #745 (comment)

While I'm not entirely excluding the possibility of an --env-file flag at some point in the future, I think we'd first want to see whether the mentioned changes alleviate the issues with the current model. But at this time, I don't think the added complexity is worth it.

@nmaro

This comment has been minimized.

Copy link
Author

@nmaro nmaro commented Sep 7, 2018

From what I understand that solution doesn't really solve an other requirement (which I assume many have): having many docker-compose file in the same folder and being able to launch them using different .env files.

To bring a concrete example, I have a docker-compose.yml and a docker-compose.test.yml. The second is based on the first one. When developing I just want to use docker-compose.yml with my local unchecked .env file (development environment), but docker-compose.test.yml is run during CI and should use a checked .env.test file.

How would you deal with this scenario?

For reference:

docker-compose.yml

version: '3'
services:
  mysql:
    image: mysql:5.7
    environment:
      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
      MYSQL_DATABASE: ${MYSQL_DATABASE}
    ports:
      - '3306:3306'
version: '3'
services:
  mysql:
    container_name: ch.migros.reactions-api-test-mysql
  mockserver:
    image: jamesdbloom/mockserver
    ports:
      - '1080:1080'
      - '1090:1090'
  api:
    build: ./api
    env_file: ./api/.env.test
    command: ['start:integration']
    ports:
      - '4000:4000'
    depends_on:
      - 'mockserver'
      - 'mysql'
  test:
    build: ./test
    env_file: ./test/.env.test
    depends_on:
      - 'api'
@janhn

This comment has been minimized.

Copy link

@janhn janhn commented Sep 13, 2018

On a related note, the '.env' filename is really way too generic to have as a default.
Should be something like .docker-compose.env instead.

Where .env is used for different things, and without the ability to override it makes the whole environment variable mechanism unusable.

@codextremist

This comment has been minimized.

Copy link

@codextremist codextremist commented Sep 13, 2018

Am I missing something ? Is the --env-file option already valid? If it is not, the documentation says otherwise
https://docs.docker.com/compose/environment-variables/#pass-environment-variables-to-containers

@nmaro

This comment has been minimized.

Copy link
Author

@nmaro nmaro commented Sep 13, 2018

Am I missing something ? Is the --env-file option already valid? If it is not, the documentation says otherwise
https://docs.docker.com/compose/environment-variables/#pass-environment-variables-to-containers

No that's different. With env_file you pass variables to containers, but not to the process that generates the containers. For example it wouldn't be enough to put MYSQL_ROOT_PASSWORD in the example above via env_file because somehow it is needed at the time the container is created.

At least according to my understanding. Maybe someone can add more details.

@pirate

This comment has been minimized.

Copy link

@pirate pirate commented Sep 13, 2018

To expand on what @nmaro is saying and provide a summary of the ticket status as I understand it, @codextremist what this issue is discussing is being able to do something like so:

project folder:

dev.env
beta.env
prod.env
docker-compose.yml
docker-compose up --env-file=dev.env

Which would pass some variables from dev.env into the context of the docker-compose.yml file, not directly to the containers like docker --env-file=dev.env would do.
docker-compose.yml then defines which variables get passed down to containers and how, allowing only the settings needed for that specific image to be passed down e.g.:

version: '3'
services:
  mysql:
    image: mysql:5.7
    environment:
      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}     (passed in from dev.env or prod.env, etc)
      MYSQL_DATABASE: ${MYSQL_DATABASE}
      ...

Right now only one .env file is allowed in the root of the project, which allows for defining settings separately from the docker-compose file, but it doesn't allow for having separate sets of settings for different environments (potentially risky if people put prod passwords by accident into dev environments). With --env-file, people could explicitly require passing the environment file they want to run with, instead of implicitly looking for a .env.

This proposal is distinct from the --env-file=<filename> option that already exists for the docker command, we're asking for a similar option to pass env variables to the docker-compose command, I think most users would find it intuitive and useful given the existing usage pattern with docker.

I strongly support this proposal since this would make many people's beta/prod/dev setup easier given the ability to pass env files by command line :) Current hacks involving symlinking .env to the env file you want to have active are error-prone and less clear and explicit than a CLI flag would be. To maintain backwards compatibility, .env could still be loaded, future users would be encouraged to use --env-file=<filename> primarily, and .env for machine-specific overrides not added to version control.

@nolazybits

This comment has been minimized.

Copy link

@nolazybits nolazybits commented Sep 26, 2018

@shin- just wondering what is the current status on the --env-file option.
I see it is part of the doc but I am wondering if one can specify multiple --env-file to have them aggregated. (as talk in other issues). Would you mind commenting please?

@shin-

This comment has been minimized.

Copy link
Contributor

@shin- shin- commented Sep 26, 2018

@nolazybits Assuming you're talking about the env_file field inside the Compose file, since we don't have an --env-file flag in docker-compose:

Yes, you can specify multiple env files for a service and have them be combined. See the documentation.

giorgiosironi added a commit to elifesciences/elife-xpub that referenced this issue Oct 26, 2018
Have to copy it in place since docker/compose#6170
@nolazybits

This comment has been minimized.

Copy link

@nolazybits nolazybits commented Oct 29, 2018

Ah yeah got mixed up. Thanks for your reply though :)

So to come back to this issue (no actual --env-file)
@shin-

But at this time, I don't think the added complexity is worth it.

docker-compose offers a -f to get the docker-compose.yaml and to me those yaml file are going in par with the env files, so it would be nice to be able to specify the env files those docker-compose file are needing.

Copied from another issue

Hello, I would really like to see a --env-file option that could be use the same way as -f (i.e multiple --env-file could be loaded). My use case is the following:

I have a monorepo
each packages has a docker-compose and a .env file
Now in my CI I would like to do something like
docker-compose -f ./Dockerfile.base.yml -f ./packages/<moduleA>/Dockerfile -f ./packages/<moduleB>/Dockerfile --env-file ./packages/<moduleA>/.env --env-file ./packages/<moduleB>/.env && docker run -it <main_module> lerna run test --scope moduleA,moduleB

This will spin the needed environments for my modules and test only those modules

Without the -env-file option I guess I can merge those .env myself with a CI script...

Thanks again @shin-

@meglio

This comment has been minimized.

Copy link

@meglio meglio commented Nov 10, 2018

I'm interested in using --env-file as well.

@vijayant123

This comment has been minimized.

Copy link

@vijayant123 vijayant123 commented Nov 20, 2018

Can we please have this essential feature? The --env-file CLI option would complement the docker compose's ability to replace ${ENV_VARIABLE} with the variable's value. This somehow is essential in our many use cases.
Regards,
Vijayant

@Wirone

This comment has been minimized.

Copy link

@Wirone Wirone commented Dec 2, 2018

@shin- is there possibility to load multiple env files ONLY for processing docker-compose.yml while starting project? In Symfony 4.2+ there is Ruby's behavior for common .env file with ability to override values with .env.local. The problem is, we want to make docker-compose.yml configurable per developer (Traefik host, database local port etc), so we can't use .env for it and if we do:

env_file:
  - .env
  - .env.local

the values will be populated into container and there won't be possibility to dynamically change application's behavior with environmental variables from .env/.env.local because they won't be loaded since APP_ENV will be already loaded.

So what I would like to do is loading env files only for creating environment, but without passing them to container.. Any suggestions?

@Wirone

This comment has been minimized.

Copy link

@Wirone Wirone commented Dec 3, 2018

Another case. I moved docker-related environment variables to docker/docker.env (ignored in Git) and changed docker-compose.yml:

version: '3.4'

services:
  # ...

  php:
    env_file: docker/docker.env
    user: "${RUN_AS_USER:?You must define user:group for permissions handling, look at README}"
    build:
      dockerfile: docker/Dockerfile
      target: php-dev
      context: .

Now, when running project, I get ERROR: Missing mandatory value for "user" option in service "php": You must define user:group for permissions handling, look at README even if RUN_AS_USER is set in docker/docker.env. When I remove :? part, I get WARNING: The RUN_AS_USER variable is not set. Defaulting to a blank string..

I know I can run export RUN_AS_USER=$UID or RUN_AS_USER=user docker-compose up -d --build but it would be soooooo good if docker-compose could pre-populate environment variables used within docker-compose.yml.

@deepaksood619

This comment has been minimized.

Copy link

@deepaksood619 deepaksood619 commented Dec 17, 2018

What if we can have a direct key value in docker-compose which specifies which file to use for substituting values in docker-compose file.

Example -

version: '3.7'
services:
  service-1:
    substitute_env_file:
      - ${MACHINE_ENV}.env

So for every different machine MACHINE_ENV can be defined in topmost environment. For example, developer can define MACHINE_ENV=dev for his machine, MACHINE_ENV=prod for production, MACHINE_ENV=staging for staging and so on, on respective machines. And according we can have dev.env, prod.env, staging.env, all checked into version control and using a single command of docker-compose up will load all environments according to the respective machine.

The practical use case that I face daily is while deploying kafka cluster, where environment variable KAFKA_ADVERTISED_LISTENERS must be passed for each kafka broker, and it cannot be put in .env because there are multiple kafka brokers in same machine, which need different values for each broker (but environment variable cannot be changed). Now this cannot be used in dev.env and prod.env because of the same name of every broker environment variable.

@landsman

This comment has been minimized.

Copy link

@landsman landsman commented Jan 6, 2019

Guys, please, let's create the way how load specific .env file for each docker-compose.yml file.

Now I have clean project with folder structure:

app
db
doc
docker
|--- /dev/
|--------- docker-compose.yml
|--- /prod/
|------ docker-compose.yml
|--- Dockerfile
prod.env
dev.env
start_dev.sh
build_prod.sh
start_prod.sh
README.md
.gitignore

And I need just pass variables to each environment.
It can be argument for docker-compose command or definition in yaml.

My temporary solution: https://gist.github.com/landsman/514731b1cd94d379589a533a6b2d663f

This functionality missing me here a lot! 🙏

@zignd

This comment has been minimized.

Copy link

@zignd zignd commented Jan 9, 2019

Please add the --env-file flag, for the sake of consistency with the docker command at least.

For now, for those looking for a workaround, I'm using the env command, like this:

env $(cat .env.test) docker-compose up
@smcardle

This comment has been minimized.

Copy link

@smcardle smcardle commented Jan 20, 2019

My preference would also be for the --env-file option.

I have a use case where this would be great i.e. deploying alfresco into containers.

Alfresco uses a properties file of its own to configure the services called alfresco-global.properties. This is just a PROP=VAL file the same format as the .env file.

It would be great to use this same global.properties file to also enable the configuration for docker-compose rather than having multiple files with repeated properties for both the compose setup and the service setup.

Steve

@coltenkrauter

This comment has been minimized.

Copy link

@coltenkrauter coltenkrauter commented Jan 29, 2019

My preference would also be for the --env-file option.

@ianfixes

This comment has been minimized.

Copy link

@ianfixes ianfixes commented Feb 8, 2019

Please add the --env-file flag, for the sake of consistency with the docker command at least.

I agree

@Toilal

This comment has been minimized.

Copy link

@Toilal Toilal commented Feb 19, 2019

Other more specific filenames should be also read by default to avoid collisions with other tools parsing .env file (symfony/dotenv, vlucas/phpdotenv), maybe docker-compose.env or .docker-compose.env. If this file exists, .env file should be skipped.

@Toilal

This comment has been minimized.

Copy link

@Toilal Toilal commented Feb 19, 2019

And why not adding COMPOSE_ENV_FILE environnement variable to this list : https://docs.docker.com/compose/reference/envvars/ ? It seems to be really related to COMPOSE_FILE environnement variable.

@Wirone

This comment has been minimized.

Copy link

@Wirone Wirone commented Feb 26, 2019

AFAIS #6535 does not cover all use cases mentioned here. It brings basic functionality of overriding default env file which is loaded during build. If I'm correct, it doesn't allow specifying multiple env files to be loaded, which was described above in the example where env_file option inside docker-compose.yml was used.

Am I correct?

@gtirloni

This comment has been minimized.

Copy link

@gtirloni gtirloni commented May 14, 2019

A generic way to solve this is to implement something like kustomize but for Docker, then users are free to patch their YAML files as they please.

I also came here looking for a way to specify an env file for different environments and I think the workaround seems to be duplicating my docker-compose.yml files in an ugly way.

@markusait

This comment has been minimized.

Copy link

@markusait markusait commented May 21, 2019

Is it possible to also load env variables from a .yaml file. Like this node library does?

@Elyytscha

This comment has been minimized.

Copy link

@Elyytscha Elyytscha commented Jul 22, 2019

just saying i'm doing this via systemd:

[Unit]
Description=docker-compose
Requires=docker.service
After=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
StandardError=null
StandardOutput=null

Environment=ENVIRONMENT=test.env
EnvironmentFile=/usr/local/src/sonarqube/test.env

# pull images
ExecStartPre=/usr/bin/docker-compose -f /usr/local/src/sonarqube/docker-compose.yml pull

# Compose up
ExecStart=/usr/bin/docker-compose -f  /usr/local/src/sonarqube/docker-compose.yml up -d

# Compose down, remove containers and volumes
ExecStop=/usr/bin/docker-compose -f /usr/local/src/sonarqube/docker-compose.yml stop

[Install]
WantedBy=multi-user.target

compose:

services:
  sonarqube:
    image: docker.registry.at/sonarqube:${SONARQUBE_VERSION:-latest}
    ports:
      - "${SONARQUBE_PORT:-9000}:9000"
    env_file: ${ENVIRONMENT:-local.env}

which was the only way i found to use the env vars from a file directly in the compose file while executing it (i dont like the .env file, i want multiple env files named by my convention for multiple environments)

@laertis

This comment has been minimized.

Copy link

@laertis laertis commented Aug 20, 2019

And why not adding COMPOSE_ENV_FILE environnement variable to this list : https://docs.docker.com/compose/reference/envvars/ ? It seems to be really related to COMPOSE_FILE environnement variable.

That's actually a great idea although although you'd have to somehow resolve the cyclic dependency when someone sets COMPOSE_ENV_FILE inside the .env file as it is commonly done for COMPOSE_FILE

eg.

$cat .env

# COMPOSE CLI VARS
COMPOSE_FILE=docker-compose.yml:docker-compose.dev.yml

# EXTRA VARS
DNS=8.8.8.8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.