Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ability to specify what .env file to use (instead of $CWD/.env) #4841

Closed
bafoed opened this issue May 18, 2017 · 27 comments

Comments

@bafoed
Copy link

@bafoed bafoed commented May 18, 2017

For example I have following file structure:

docker-configs/
|-- compose.base.yml

projects/
|-- project1/
|------ .env
|------ overrides.yml
|-- project2/
|------ .env
|------ overrides.yml

When my working directory is project1/, I can run compose with docker-compose -f ../../docker-configs/compose.base.yml -f overrides.yml to substitute variables in compose.base.yml using parameters from .env file. The situation changes when I want to automate some compose-related tasks, e.g. stop selected project. If I replace all relative paths with absolute ones, this will not be enough, because for some unknown reason docker-compose is able to read .env files only from the current working directory.

According to the documentation (https://docs.docker.com/compose/env-file/), this behavior can not be changed. Maybe it's worth to add some additional argument, such as --cli-env-file which could take full paths for purposes like mine? Or maybe there are other ways for this?

I wanted to send a pull request, but, unfortunately, there are a lot of mentions of various "environments" in the code and I could not figure out where cwd is hardcoded. I think this is a very simple feature that does not even cause any problems in backward compatibility - there is no need to change the default behavior, just add a new argument. Any hardcoded things get better when they can be changed.

This idea was already offered six months ago, but unfortunately it remained without attention: #3381 (comment).

And thank you very much for such a great work that has already been done.

@bafoed

This comment has been minimized.

Copy link
Author

@bafoed bafoed commented May 18, 2017

I just saw that in version 1.12 the argument --project-directory was added, which, theoretically, could solve this problem, but it does not do it. I do not know whether this can be considered a bug.

By the way, I can not use this argument in any way:
root@srv:/# docker-compose --project-directory /projects/project1 -f /docker-configs/compose.base.yml -f /projects/project1/overrides.yml config - undefined variables errors.

root@srv:/projects/project1# docker-compose --project-directory /projects/project1 -f /docker-configs/compose.base.yml -f overrides.yml config - ok (just as without --project-directory)

Created /projects/test/docker-compose.yml:

root@srv:/projects/test# cat docker-compose.yml
version: '2'

# do nothing

root@srv:/projects/test# docker-compose config
networks: {}
services: {}
version: '2.0'
volumes: {}

root@srv:/projects/test# cd /
root@srv:/# docker-compose --project-directory /projects/test config
ERROR: 
        Can't find a suitable configuration file in this directory or any
        parent. Are you in the right directory?

        Supported filenames: docker-compose.yml, docker-compose.yaml
@bafoed

This comment has been minimized.

Copy link
Author

@bafoed bafoed commented May 18, 2017

I tried to figure out what the --project-directory argument does at all, but did not fully understand it. But, at least, it became clear that it has nothing to do with the .env file.
So I want to say that the current description of this argument is a little misleading: on the documentation page there is the text "where the docker-compose command is executed (current working directory)", and in the description of the argument - "Specify an alternate working directory". However, they are not related.

It is very sad that the ability to change the working directory is made quite limited, so I decided to rebuild docker-compose for my needs. The patch that adds the functionality I need looks like this: https://gist.github.com/bafoed/b4fcd7acc8ab62f27120c65e1973c8cf/revisions#diff-5bc02cefb3ea9e27f1a6776eabd1935d
I'm not sure that I understood the meaning of the --project-directory argument correctly and it can be used for such purposes, so I send it as a gist, rather than pull request.
I hope someone more familiar with this code will give a comment.

@shin-

This comment has been minimized.

Copy link
Contributor

@shin- shin- commented May 22, 2017

Thank you for the report.

I agree that --project-directory should change where the .env file is retrieved. Feel free to submit a PR with your changes.

@loadx

This comment has been minimized.

Copy link

@loadx loadx commented Jul 24, 2017

Hi All,

So I've had a quick look at the code for making project-directory flag work the way we agree it should.. in my opinion it should just override project_dir completly.

use cases:

  1. command: docker-compose --project-directory /my_files
    Then .env is loaded from /my_files and so is docker-compose.yml.

  2. command: docker-compose -f /compose_files/docker-compose.yaml --project-directory /my_files
    Then .env is loaded from /my_files along with the docker-compose.yaml. However it is over-ridden by the docker-compose.yaml loaded from /compose_files.

Do we agree this is the correct behavior? If so then I believe all we need to do is modify the project_dir to be over-written by the one specified in the flag.

https://github.com/docker/compose/blob/master/compose/cli/command.py#L24
@shin- Would you agree with this approach?

Cheers

@shin-

This comment has been minimized.

Copy link
Contributor

@shin- shin- commented Jul 24, 2017

Yes, there has been a couple PRs for it in #4709 and #4933 - I just haven't had time to test either thoroughly to make sure they don't break things in unexpected ways. It is on my radar though.

@loadx

This comment has been minimized.

Copy link

@loadx loadx commented Jul 25, 2017

@shin- cheers for the heads up. I will have a look at the PR's and see If i can perhaps assist there. It looks like there hasn't been a comprehensive change to the tests to verify the various edge cases as mentioned above so I should be able to rectify that.

@schmunk42

This comment has been minimized.

Copy link
Contributor

@schmunk42 schmunk42 commented Nov 17, 2017

@estarter continued from #745 (comment)

It's very good that --project-directory is not affecting -f - otherwise imagine what a nightmare would be to use --project-directory and multiple docker-compose files all in different directories.

You would have to include the files with relative paths, that would be much more consistent than using project-directory only for the default files (docker-compose.yml, docker-compose.override.yml). When the option would use the .env from the specified directory you could specify your default compose files there file COMPOSE_FILE. If you want to include with absolute paths you can still do that.

@shin- I agree to this 100% #4841 (comment) - to my understanding --project-directory $DIR should be equivalent to:

pushd $DIR && docker-compose [...] && popd
@estarter

This comment has been minimized.

Copy link

@estarter estarter commented Nov 17, 2017

You would have to include the files with relative paths, that would be much more consistent than using project-directory only for the default files (docker-compose.yml, docker-compose.override.yml).

I didn't mean to use --project-directory for the default files (docker-compose.yml, docker-compose.override.yml) actually, my only concern was .env file.

When the option would use the .env from the specified directory you could specify your default compose files there file COMPOSE_FILE. If you want to include with absolute paths you can still do that.

This is not quite a linux way. CLI tool that defines its own path convention is in practice difficult to use.
You propose to introduce a special path convention - when --project-directory is set than relative path is based on the parameter not current directory. That would not work with shell's path auto-complete feature, that would make relative path unusable for other commands (i can't re-use relative path with ls ), that makes reading of the path parameters of docker-compose a special thing.

@schmunk42 you described a perfect solution for the use-case when it's needed to change project directory:

pushd $DIR && docker-compose [...] && popd

I'm using this approach often working in shell for other commands, for example check on maven and its -f option.


What is missing for me is the option to define location of .env file.
Not sure if it'd be accepted to have a dedicated --env-file option?
Don't know how does that correlates with docker-compose.env plans?

@shin- WDYT?

@schmunk42

This comment has been minimized.

Copy link
Contributor

@schmunk42 schmunk42 commented Nov 17, 2017

@estarter Thank you for your valuable feedback.

You propose to introduce a special path convention - when --project-directory is set than relative path is based on the parameter not current directory.

Actually I meant relative paths should be relative from the file which uses them, but I also see that this raises other issues, ie. when merging files.

Before digging deeper into this topic, I have question @shin- ... Could you describe a bit more in detail what --project-directory is intended for? We looked through the sources and found ie. this:

:param working_dir: the directory to use for relative paths in the config

https://github.com/docker/compose/blob/master/compose/config/config.py#L151

Which paths are meant here, extends, env_file, volumes, ...? When playing around with the parameter we couldn't really figure out any effect at all?!

@zored

This comment has been minimized.

Copy link

@zored zored commented Jan 4, 2018

Is there env-variable for --project-directory like $COMPOSE_FILE? Or pushd / popd is currently the solution? It would be great to change folders with fixed docker-compose behaviour. 😃

@schmunk42

This comment has been minimized.

Copy link
Contributor

@schmunk42 schmunk42 commented Jan 4, 2018

@zored The best I've found so far...

(cd project-directory && docker-compose up -d)

via #745 (comment)

@verglor

This comment has been minimized.

Copy link

@verglor verglor commented Jan 23, 2018

Hello, I also think that adding --env-file option would be the best solution for specifying custom .env file without unnecessary (or even undesirable) change of project directory.

It would be even better if it is possible to specify multiple overriding env files (just like -f option does). Probably .env file from project directory should always be used and files specified in --env-file should just override it.

@chrisinmtown

This comment has been minimized.

Copy link

@chrisinmtown chrisinmtown commented Jan 30, 2018

I would like to offer a use case in support of this request.

We have a project with many dockers in complex relationships with each other, using the same relationships in dev and test environments. When pulling and launching containers in dev, docker-compose must pull images from our snapshot registry using snapshot versions. When pulling and launching in test, docker-compose must pull images from our staging registry using release-candidate versions.

Today I can do this by writing shell scripts that define a slew of environment variables then run docker-compose.

I would prefer instead if docker-compose would accept the proposed --env-file option discussed above. I could specify a dev.env file when launching in dev, and a test.env file in test. This would eliminate the need for kludgy shell scripts.

To emphasize, I'm concerned about passing values only to docker-compose, NOT so much to the docker images/containers.

FWIW this SO issue is close, but seems more interested in passing env var values to the DOCKERS:
https://stackoverflow.com/questions/29062522/different-env-file-but-same-yml-with-docker-compose

Thanks for listening.

@lystor

This comment has been minimized.

Copy link

@lystor lystor commented Feb 23, 2018

Please add support of "--env-file".
This feature is very important for us too.

@sir-gon

This comment has been minimized.

Copy link

@sir-gon sir-gon commented Mar 2, 2018

What about this issue? really is a must in my local and CI environments.

At every restart, my containers lost ENV vars, and I need to restart each one from their own directory!

@aJetHorn

This comment has been minimized.

Copy link

@aJetHorn aJetHorn commented Mar 7, 2018

Tangential, but since Google led me here I might as well help less experienced people like myself and add that anyone looking to emulate the functionality of --env-file without needing the path behavior being suggested above should just use overrides as mentioned in the first post.

@cropse

This comment has been minimized.

Copy link

@cropse cropse commented Apr 16, 2018

It's seen not too difficult for me to add the option --env-file, If this enhancement feature is great idea , @shin- please let me know, I would submit PR after couple week.

@idontusenumbers

This comment has been minimized.

Copy link

@idontusenumbers idontusenumbers commented May 20, 2018

Please reconsider how docker handles environment variables throughout before thoughtlessly adding new environment variable handling; the current situation (before custom .env files) is a quite complex and confusing. Adding additional complexity will further increase the confusion.

@nolazybits

This comment has been minimized.

Copy link

@nolazybits nolazybits commented Jun 1, 2018

Hello,

I would really like to see a --env-file option that could be use the same way as -f (i.e multiple --env-file could be loaded).
My use case is the following:

  • I have a monorepo
  • each packages has a docker-compose and a .env file

Now in my CI I would like to do something like
docker-compose -f ./Dockerfile.base.yml -f ./packages/<moduleA>/Dockerfile -f ./packages/<moduleB>/Dockerfile --env-file ./packages/<moduleA>/.env --env-file ./packages/<moduleB>/.env && docker run -it <main_module> lerna run test --scope moduleA,moduleB

This will spin the needed environments for my modules and test only those modules

Without the -env-file option I guess I can merge those .env myself with a CI script...

@Tensho

This comment has been minimized.

Copy link

@Tensho Tensho commented Aug 6, 2018

I'm another one struggler of docker-compose --env-file option absence. @shin, are you going to add it or fix docker-compose --project-directory behaviour?

@chrisinmtown

This comment has been minimized.

Copy link

@chrisinmtown chrisinmtown commented Aug 14, 2018

So the solution is not to add a new flag to specify an environment file, but instead to search the argument of the --project-directory flag for the (still-hardcoded) file name .env ?

@nolazybits

This comment has been minimized.

Copy link

@nolazybits nolazybits commented Aug 21, 2018

yeah I'll be keen to know how the code in the merged PR solves the problem described here.
@shin- could you clarify please?

@shin-

This comment has been minimized.

Copy link
Contributor

@shin- shin- commented Aug 21, 2018

The fix to the behavior of --project-directory resolves the issue raised by the original poster, allowing to configure where the .env file is retrieved from.

Other .env changes we're considering are described in this thread but aren't directly connected to this particular issue.

@nolazybits

This comment has been minimized.

Copy link

@nolazybits nolazybits commented Aug 21, 2018

Thanks for the answer @shin- . I'll check the thread and post a link to my comment here for the use case.

@sebnyberg

This comment has been minimized.

Copy link

@sebnyberg sebnyberg commented Aug 28, 2018

It would be great to have an --env-file option.

I use docker to run a local temporary database for testing purposes. Having the variables stored in some other .env file such as .env.local is quite nifty since it can be checked into version control.

@nmaro

This comment has been minimized.

Copy link

@nmaro nmaro commented Sep 6, 2018

Go and like the new ticket I created if you want this feature :)

@shellus

This comment has been minimized.

Copy link

@shellus shellus commented Apr 19, 2019

@zored到目前为止我发现的最好的......

(cd project-directory && docker-compose up -d)

通过#745(评论)

alias nginx='nginx_alias(){ (cd /data/compose/www/ && docker-compose exec nginx nginx $@);};nginx_alias'

nginx -s reload
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.