Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

--env-file option #6170

Closed
nickredmark opened this issue Sep 6, 2018 · 43 comments · Fixed by #6535
Closed

--env-file option #6170

nickredmark opened this issue Sep 6, 2018 · 43 comments · Fixed by #6535

Comments

@nickredmark
Copy link

@nickredmark nickredmark commented Sep 6, 2018

This option has been discussed in a couple of tickets but never had a ticket for itself.

One should be able to execute

docker-compose up --env-file=.env.test (perhaps multiple env files)

to override the usage of .env file.

@alancapc
Copy link

@alancapc alancapc commented Sep 6, 2018

Good idea, this one.

@shin-
Copy link

@shin- shin- commented Sep 6, 2018

Current plans for environment files are spelled out here: #745 (comment)

While I'm not entirely excluding the possibility of an --env-file flag at some point in the future, I think we'd first want to see whether the mentioned changes alleviate the issues with the current model. But at this time, I don't think the added complexity is worth it.

@nickredmark
Copy link
Author

@nickredmark nickredmark commented Sep 7, 2018

From what I understand that solution doesn't really solve an other requirement (which I assume many have): having many docker-compose file in the same folder and being able to launch them using different .env files.

To bring a concrete example, I have a docker-compose.yml and a docker-compose.test.yml. The second is based on the first one. When developing I just want to use docker-compose.yml with my local unchecked .env file (development environment), but docker-compose.test.yml is run during CI and should use a checked .env.test file.

How would you deal with this scenario?

For reference:

docker-compose.yml

version: '3'
services:
  mysql:
    image: mysql:5.7
    environment:
      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
      MYSQL_DATABASE: ${MYSQL_DATABASE}
    ports:
      - '3306:3306'
version: '3'
services:
  mysql:
    container_name: ch.migros.reactions-api-test-mysql
  mockserver:
    image: jamesdbloom/mockserver
    ports:
      - '1080:1080'
      - '1090:1090'
  api:
    build: ./api
    env_file: ./api/.env.test
    command: ['start:integration']
    ports:
      - '4000:4000'
    depends_on:
      - 'mockserver'
      - 'mysql'
  test:
    build: ./test
    env_file: ./test/.env.test
    depends_on:
      - 'api'

@janhn
Copy link

@janhn janhn commented Sep 13, 2018

On a related note, the '.env' filename is really way too generic to have as a default.
Should be something like .docker-compose.env instead.

Where .env is used for different things, and without the ability to override it makes the whole environment variable mechanism unusable.

@codextremist
Copy link

@codextremist codextremist commented Sep 13, 2018

Am I missing something ? Is the --env-file option already valid? If it is not, the documentation says otherwise
https://docs.docker.com/compose/environment-variables/#pass-environment-variables-to-containers

@nickredmark
Copy link
Author

@nickredmark nickredmark commented Sep 13, 2018

Am I missing something ? Is the --env-file option already valid? If it is not, the documentation says otherwise
https://docs.docker.com/compose/environment-variables/#pass-environment-variables-to-containers

No that's different. With env_file you pass variables to containers, but not to the process that generates the containers. For example it wouldn't be enough to put MYSQL_ROOT_PASSWORD in the example above via env_file because somehow it is needed at the time the container is created.

At least according to my understanding. Maybe someone can add more details.

@pirate
Copy link

@pirate pirate commented Sep 13, 2018

To expand on what @nmaro is saying and provide a summary of the ticket status as I understand it, @codextremist what this issue is discussing is being able to do something like so:

project folder:

dev.env
beta.env
prod.env
docker-compose.yml
docker-compose up --env-file=dev.env

Which would pass some variables from dev.env into the context of the docker-compose.yml file, not directly to the containers like docker --env-file=dev.env would do.
docker-compose.yml then defines which variables get passed down to containers and how, allowing only the settings needed for that specific image to be passed down e.g.:

version: '3'
services:
  mysql:
    image: mysql:5.7
    environment:
      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}     (passed in from dev.env or prod.env, etc)
      MYSQL_DATABASE: ${MYSQL_DATABASE}
      ...

Right now only one .env file is allowed in the root of the project, which allows for defining settings separately from the docker-compose file, but it doesn't allow for having separate sets of settings for different environments (potentially risky if people put prod passwords by accident into dev environments). With --env-file, people could explicitly require passing the environment file they want to run with, instead of implicitly looking for a .env.

This proposal is distinct from the --env-file=<filename> option that already exists for the docker command, we're asking for a similar option to pass env variables to the docker-compose command, I think most users would find it intuitive and useful given the existing usage pattern with docker.

I strongly support this proposal since this would make many people's beta/prod/dev setup easier given the ability to pass env files by command line :) Current hacks involving symlinking .env to the env file you want to have active are error-prone and less clear and explicit than a CLI flag would be. To maintain backwards compatibility, .env could still be loaded, future users would be encouraged to use --env-file=<filename> primarily, and .env for machine-specific overrides not added to version control.

@nolazybits
Copy link

@nolazybits nolazybits commented Sep 26, 2018

@shin- just wondering what is the current status on the --env-file option.
I see it is part of the doc but I am wondering if one can specify multiple --env-file to have them aggregated. (as talk in other issues). Would you mind commenting please?

@shin-
Copy link

@shin- shin- commented Sep 26, 2018

@nolazybits Assuming you're talking about the env_file field inside the Compose file, since we don't have an --env-file flag in docker-compose:

Yes, you can specify multiple env files for a service and have them be combined. See the documentation.

giorgiosironi added a commit to elifesciences/elife-xpub that referenced this issue Oct 26, 2018
@nolazybits
Copy link

@nolazybits nolazybits commented Oct 29, 2018

Ah yeah got mixed up. Thanks for your reply though :)

So to come back to this issue (no actual --env-file)
@shin-

But at this time, I don't think the added complexity is worth it.

docker-compose offers a -f to get the docker-compose.yaml and to me those yaml file are going in par with the env files, so it would be nice to be able to specify the env files those docker-compose file are needing.

Copied from another issue

Hello, I would really like to see a --env-file option that could be use the same way as -f (i.e multiple --env-file could be loaded). My use case is the following:

I have a monorepo
each packages has a docker-compose and a .env file
Now in my CI I would like to do something like
docker-compose -f ./Dockerfile.base.yml -f ./packages/<moduleA>/Dockerfile -f ./packages/<moduleB>/Dockerfile --env-file ./packages/<moduleA>/.env --env-file ./packages/<moduleB>/.env && docker run -it <main_module> lerna run test --scope moduleA,moduleB

This will spin the needed environments for my modules and test only those modules

Without the -env-file option I guess I can merge those .env myself with a CI script...

Thanks again @shin-

@meglio
Copy link

@meglio meglio commented Nov 10, 2018

I'm interested in using --env-file as well.

@vijayant123
Copy link

@vijayant123 vijayant123 commented Nov 20, 2018

Can we please have this essential feature? The --env-file CLI option would complement the docker compose's ability to replace ${ENV_VARIABLE} with the variable's value. This somehow is essential in our many use cases.
Regards,
Vijayant

@Wirone
Copy link

@Wirone Wirone commented Dec 2, 2018

@shin- is there possibility to load multiple env files ONLY for processing docker-compose.yml while starting project? In Symfony 4.2+ there is Ruby's behavior for common .env file with ability to override values with .env.local. The problem is, we want to make docker-compose.yml configurable per developer (Traefik host, database local port etc), so we can't use .env for it and if we do:

env_file:
  - .env
  - .env.local

the values will be populated into container and there won't be possibility to dynamically change application's behavior with environmental variables from .env/.env.local because they won't be loaded since APP_ENV will be already loaded.

So what I would like to do is loading env files only for creating environment, but without passing them to container.. Any suggestions?

@Wirone
Copy link

@Wirone Wirone commented Dec 3, 2018

Another case. I moved docker-related environment variables to docker/docker.env (ignored in Git) and changed docker-compose.yml:

version: '3.4'

services:
  # ...

  php:
    env_file: docker/docker.env
    user: "${RUN_AS_USER:?You must define user:group for permissions handling, look at README}"
    build:
      dockerfile: docker/Dockerfile
      target: php-dev
      context: .

Now, when running project, I get ERROR: Missing mandatory value for "user" option in service "php": You must define user:group for permissions handling, look at README even if RUN_AS_USER is set in docker/docker.env. When I remove :? part, I get WARNING: The RUN_AS_USER variable is not set. Defaulting to a blank string..

I know I can run export RUN_AS_USER=$UID or RUN_AS_USER=user docker-compose up -d --build but it would be soooooo good if docker-compose could pre-populate environment variables used within docker-compose.yml.

@deepaksood619
Copy link

@deepaksood619 deepaksood619 commented Dec 17, 2018

What if we can have a direct key value in docker-compose which specifies which file to use for substituting values in docker-compose file.

Example -

version: '3.7'
services:
  service-1:
    substitute_env_file:
      - ${MACHINE_ENV}.env

So for every different machine MACHINE_ENV can be defined in topmost environment. For example, developer can define MACHINE_ENV=dev for his machine, MACHINE_ENV=prod for production, MACHINE_ENV=staging for staging and so on, on respective machines. And according we can have dev.env, prod.env, staging.env, all checked into version control and using a single command of docker-compose up will load all environments according to the respective machine.

The practical use case that I face daily is while deploying kafka cluster, where environment variable KAFKA_ADVERTISED_LISTENERS must be passed for each kafka broker, and it cannot be put in .env because there are multiple kafka brokers in same machine, which need different values for each broker (but environment variable cannot be changed). Now this cannot be used in dev.env and prod.env because of the same name of every broker environment variable.

@landsman
Copy link

@landsman landsman commented Jan 6, 2019

Guys, please, let's create the way how load specific .env file for each docker-compose.yml file.

Now I have clean project with folder structure:

app
db
doc
docker
|--- /dev/
|--------- docker-compose.yml
|--- /prod/
|------ docker-compose.yml
|--- Dockerfile
prod.env
dev.env
start_dev.sh
build_prod.sh
start_prod.sh
README.md
.gitignore

And I need just pass variables to each environment.
It can be argument for docker-compose command or definition in yaml.

My temporary solution: https://gist.github.com/landsman/514731b1cd94d379589a533a6b2d663f

This functionality missing me here a lot! 🙏

@zignd
Copy link

@zignd zignd commented Jan 9, 2019

Please add the --env-file flag, for the sake of consistency with the docker command at least.

For now, for those looking for a workaround, I'm using the env command, like this:

env $(cat .env.test) docker-compose up

@smcardle
Copy link

@smcardle smcardle commented Jan 20, 2019

My preference would also be for the --env-file option.

I have a use case where this would be great i.e. deploying alfresco into containers.

Alfresco uses a properties file of its own to configure the services called alfresco-global.properties. This is just a PROP=VAL file the same format as the .env file.

It would be great to use this same global.properties file to also enable the configuration for docker-compose rather than having multiple files with repeated properties for both the compose setup and the service setup.

Steve

@coltenkrauter
Copy link

@coltenkrauter coltenkrauter commented Jan 29, 2019

My preference would also be for the --env-file option.

@ianfixes
Copy link

@ianfixes ianfixes commented Feb 8, 2019

Please add the --env-file flag, for the sake of consistency with the docker command at least.

I agree

@Toilal
Copy link

@Toilal Toilal commented Feb 19, 2019

Other more specific filenames should be also read by default to avoid collisions with other tools parsing .env file (symfony/dotenv, vlucas/phpdotenv), maybe docker-compose.env or .docker-compose.env. If this file exists, .env file should be skipped.

@Toilal
Copy link

@Toilal Toilal commented Feb 19, 2019

And why not adding COMPOSE_ENV_FILE environnement variable to this list : https://docs.docker.com/compose/reference/envvars/ ? It seems to be really related to COMPOSE_FILE environnement variable.

@Wirone
Copy link

@Wirone Wirone commented Feb 26, 2019

AFAIS #6535 does not cover all use cases mentioned here. It brings basic functionality of overriding default env file which is loaded during build. If I'm correct, it doesn't allow specifying multiple env files to be loaded, which was described above in the example where env_file option inside docker-compose.yml was used.

Am I correct?

@gtirloni
Copy link

@gtirloni gtirloni commented May 14, 2019

A generic way to solve this is to implement something like kustomize but for Docker, then users are free to patch their YAML files as they please.

I also came here looking for a way to specify an env file for different environments and I think the workaround seems to be duplicating my docker-compose.yml files in an ugly way.

@markusait
Copy link

@markusait markusait commented May 21, 2019

Is it possible to also load env variables from a .yaml file. Like this node library does?

@Elyytscha
Copy link

@Elyytscha Elyytscha commented Jul 22, 2019

just saying i'm doing this via systemd:

[Unit]
Description=docker-compose
Requires=docker.service
After=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
StandardError=null
StandardOutput=null

Environment=ENVIRONMENT=test.env
EnvironmentFile=/usr/local/src/sonarqube/test.env

# pull images
ExecStartPre=/usr/bin/docker-compose -f /usr/local/src/sonarqube/docker-compose.yml pull

# Compose up
ExecStart=/usr/bin/docker-compose -f  /usr/local/src/sonarqube/docker-compose.yml up -d

# Compose down, remove containers and volumes
ExecStop=/usr/bin/docker-compose -f /usr/local/src/sonarqube/docker-compose.yml stop

[Install]
WantedBy=multi-user.target

compose:

services:
  sonarqube:
    image: docker.registry.at/sonarqube:${SONARQUBE_VERSION:-latest}
    ports:
      - "${SONARQUBE_PORT:-9000}:9000"
    env_file: ${ENVIRONMENT:-local.env}

which was the only way i found to use the env vars from a file directly in the compose file while executing it (i dont like the .env file, i want multiple env files named by my convention for multiple environments)

@laertis
Copy link

@laertis laertis commented Aug 20, 2019

And why not adding COMPOSE_ENV_FILE environnement variable to this list : https://docs.docker.com/compose/reference/envvars/ ? It seems to be really related to COMPOSE_FILE environnement variable.

That's actually a great idea although although you'd have to somehow resolve the cyclic dependency when someone sets COMPOSE_ENV_FILE inside the .env file as it is commonly done for COMPOSE_FILE

eg.

$cat .env

# COMPOSE CLI VARS
COMPOSE_FILE=docker-compose.yml:docker-compose.dev.yml

# EXTRA VARS
DNS=8.8.8.8

@namgivu
Copy link

@namgivu namgivu commented Dec 10, 2019

Please add the --env-file flag, for the sake of consistency with the docker command at least.

For now, for those looking for a workaround, I'm using the env command, like this:

env $(cat .env.test) docker-compose up

This is my charm and working. Thank you!

@yairmark
Copy link

@yairmark yairmark commented Jan 20, 2020

Please add the --env-file flag, for the sake of consistency with the docker command at least.

For now, for those looking for a workaround, I'm using the env command, like this:

env $(cat .env.test) docker-compose up

To resolve the error env: #The: No such file or directory which is caused by env files with comments in change the above to:

env $(cat .env.test | grep "#" -v) docker-compose up

@Elyytscha
Copy link

@Elyytscha Elyytscha commented Jan 22, 2020

Please add the --env-file flag, for the sake of consistency with the docker command at least.
For now, for those looking for a workaround, I'm using the env command, like this:

env $(cat .env.test) docker-compose up

To resolve the error env: #The: No such file or directory which is caused by env files with comments in change the above to:

env $(cat .env.test | grep "#" -v) docker-compose up

just saying that all of those solutions with env in front are breaking auto-completion of docker-compose which i dont wanna miss

@json2d
Copy link

@json2d json2d commented Feb 8, 2020

Please add the --env-file flag, for the sake of consistency with the docker command at least.
For now, for those looking for a workaround, I'm using the env command, like this:

env $(cat .env.test) docker-compose up

To resolve the error env: #The: No such file or directory which is caused by env files with comments in change the above to:

env $(cat .env.test | grep "#" -v) docker-compose up

just saying that all of those solutions with env in front are breaking auto-completion of docker-compose which i dont wanna miss

i think lets just make a VSCode extension to do this for us and call it a day

@JustinGrote
Copy link

@JustinGrote JustinGrote commented Feb 17, 2020

This is my workaround that I put in my .bashrc:

export COMPOSE_HOME=/home/jgrote/docker
alias dc="env PWD=$COMPOSE_HOME $(cat $COMPOSE_HOME/.env | grep -v '#' | tr '\n' ' ') docker-compose"
alias dcup="dc up -d"
complete -F _docker_compose dc

I can use dc for my special one, and then docker-compose if I need to use a non-special one.

@Elyytscha this doesn't break autocompletion :)

@zignd
Copy link

@zignd zignd commented Feb 17, 2020

The feature has been implemented already #6535

@adamerose
Copy link

@adamerose adamerose commented Jun 10, 2020

The feature has been implemented already #6535

Has it been included in a release yet or is there a roadmap for when it will be? This still doesn't work for me and I don't see it anywhere in the docs, it's not listed here: https://docs.docker.com/compose/reference/up/

@Loki-Afro
Copy link

@Loki-Afro Loki-Afro commented Jul 6, 2020

@adamerose
works for me but did not find it in the documentation either

pwirth@frenzy ~ % docker-compose --version
docker-compose version 1.25.4, build unknown

@iraklisg
Copy link

@iraklisg iraklisg commented Aug 14, 2020

Is --env-file option available only with docker-compose up command? Because the command docker-compose run --env-file .env.test ... is not executed 😕

Docker Compose version 1.26.2

--EDIT--

My mistake, sorry. I stupidity messed up the options order... 😞

It should be docker-compose --env-file .env.test run service_name and not docker-compose run --env-file .env.test service_name

@MexsonFernandes
Copy link

@MexsonFernandes MexsonFernandes commented Aug 14, 2020

@iraklisg Check out this link https://docs.docker.com/compose/environment-variables/.

@bmarkovic
Copy link

@bmarkovic bmarkovic commented Aug 25, 2020

@MexsonFernandes the link doesn't mention --env-file for the docker-compose binary, which is what the discussion is about, given the merged PR.

Furthermore, pointing people to documentation in issues is rude, especially without an in-page anchor, and especially since few posts above the matter of documentation being outdated was raised. It's rude for two reasons. First, documentation changes over time and there is no guarantee that the information you saw will be there when someone else follows the link, and second, it's implies on your part an assumption that someone was either stupid or lazy as to not check relevant documentation before seeking answers in a Github issue.

@MexsonFernandes
Copy link

@MexsonFernandes MexsonFernandes commented Aug 25, 2020

@MexsonFernandes the link doesn't mention --env-file for the docker-compose binary, which is what the discussion is about, given the merged PR.

Furthermore, pointing people to documentation in issues is rude, especially without an in-page anchor, and especially since few posts above the matter of documentation being outdated was raised. It's rude for two reasons. First, documentation changes over time and there is no guarantee that the information you saw will be there when someone else follows the link, and second, it's implies on your part an assumption that someone was either stupid or lazy as to not check relevant documentation before seeking answers in a Github issue.

@bmarkovic my bad. I will make sure to read off and mention proper links out here. 👍🏼

@dmitry-kulikov
Copy link

@dmitry-kulikov dmitry-kulikov commented Oct 16, 2020

--env-file was added with version 1.25.0, it is not mentioned in CHANGELOG.md.

@fbergen
Copy link

@fbergen fbergen commented Dec 1, 2020

What if we can have a direct key value in docker-compose which specifies which file to use for substituting values in docker-compose file.

Example -

version: '3.7'
services:
  service-1:
    substitute_env_file:
      - ${MACHINE_ENV}.env

So for every different machine MACHINE_ENV can be defined in topmost environment. For example, developer can define MACHINE_ENV=dev for his machine, MACHINE_ENV=prod for production, MACHINE_ENV=staging for staging and so on, on respective machines. And according we can have dev.env, prod.env, staging.env, all checked into version control and using a single command of docker-compose up will load all environments according to the respective machine.

The practical use case that I face daily is while deploying kafka cluster, where environment variable KAFKA_ADVERTISED_LISTENERS must be passed for each kafka broker, and it cannot be put in .env because there are multiple kafka brokers in same machine, which need different values for each broker (but environment variable cannot be changed). Now this cannot be used in dev.env and prod.env because of the same name of every broker environment variable.

Running a similar version below, thought I'd mention:

docker-compose.yml

services:
  web:
     env_file: ${ENV_FILE}

and then only add one ENV_FILE var when running

ENV_FILE=.env.test docker-compose run web

@Mugane
Copy link

@Mugane Mugane commented May 11, 2021

This looks like a popular request and a glaring omission from the docker-compose workflow, with no solution presented.

Can someone explain why is this issue has been closed???

@daghsentinel
Copy link

@daghsentinel daghsentinel commented Dec 9, 2021

This looks like a popular request and a glaring omission from the docker-compose workflow, with no solution presented.

Can someone explain why is this issue has been closed???

Looks like it has been resolved.
#6170 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.