Add .env file support to COMPOSE_ENV_FILES

[nickjj]

Description

This is in relation to the new feature introduced in this PR: #11061

When COMPOSE_ENV_FILES is defined in an .env file it has no effect. This is a potential oversight / bug.

Here's a reproduceable example:

# docker-compose.yml
services:
  debian:
    image: "debian:bookworm-slim"
    restart: "${DOCKER_RESTART_POLICY:-unless-stopped}"

# .env
COMPOSE_ENV_FILES=.env,.env.local
DOCKER_RESTART_POLICY=yes

# .env.local
DOCKER_RESTART_POLICY=no

If you run docker compose config the value of restart is yes but it would be expected that it's no through the logic of:

• docker compose was called without explicitly defining any --env-file flags which means it will look for .env by default
• The new COMPOSE_ENV_FILES env var will be used which says to interpolate the .env file and then .env.local
  • This is the part that is failing the use case expectation
• no should be set

If you run COMPOSE_ENV_FILES=.env,.env.local docker compose config then everything works as expected but this requires altering your shell or creating a custom alias to run docker compose with that set. The original use case was to allow for running docker compose commands without modification.

[ndeloof]

I'm :-1 on this.
COMPOSE_ENV_FILES let you define the env files you want to include, as an alternative to a (potentially long) list of .env files to be included. From this point of view, this is an improvement.
With your proposed change, we are just back to defaulting to .env. If COMPOSE_ENV_FILES in this file doesn't include .env then resulting environment would just be inconsistent. And another env file might override this variable and include other files, what would then be the expected result ?
Basically, I'd say "this is going too far"

[nickjj]

As it currently stands (with COMPOSE_ENV_FILES as it is), we can do:

• docker compose --env-file .env --env-file .env.custom build
• COMPOSE_ENV_FILES=.env,.env.custom docker compose build

In both cases if you want docker compose to interpolate .env.custom you must modify how you use Docker Compose. You can no longer just call docker compose. You can do 1 of the following:

• Run either of the 2 options in the first set of bullets above
• Manually export COMPOSE_ENV_FILES=.env,.env.custom in your current shell
• Create an alias for docker compose but build out a lot of logic because you probably don't want this to apply to projects
• Modify your shell's rc file to auto-export COMPOSE_ENV_FILES but like the alias approach, this needs a lot of logic to work

In my opinion all 4 of the above cases are not a great developer experience since it involves every person who runs your project to customize how they call docker compose in one of the above ways.

This issue (and the original discussion) allows COMPOSE_ENV_FILES to be defined in .env and Docker Compose will look for that automatically. If it exists then it will interpolate all files in that list. The only extra step / functionality is this behavior. Everything else would work the same once COMPOSE_ENV_FILES is identified.

This would be a great addition because now you have a per-project solution to interpolate whatever env files you need and no one needs to modify how they call docker compose since you can include that in .env.

[ndeloof]

Manually export COMPOSE_ENV_FILES=.env,.env.custom in your current shell

indeed, this is what environment variables are for: tweak your environment so the same command will behave in a distinct way.

[nickjj]

indeed, this is what environment variables are for: tweak your environment so the same command will behave in a distinct way.

Right but now imagine you have 30 developers who are working on an app.

Now to run the project all 30 developers need to run export COMPOSE_ENV_FILES=.env,.env.custom every time they want to start up the project. Also, assume 75% of the developers don't even know what that command does because they are application developers and not necessarily heavy terminal users.

Likewise, imagine you open source a project that's used by thousands of folks. Suddenly you can't just docker compose up --build the project anymore. You need to export an environment variable. Not just once, but every single time you open a terminal.

[ndeloof]

having multiple env file suggest you want to combine those to cover multiple scenarios. If the default .env one automagically include more, then .. why not have all those variables set in a single file ?

[nickjj]

It wouldn't be automagical. It would be dependent on you explicitly setting COMPOSE_ENV_FILES in the .env file. If you don't set that env var in .env then Docker Compose would only interpolate variables in .env.

There's a number of use cases where you couldn't or wouldn't want to set everything in a single .env. Maybe you have a separate .env.local file which isn't commit to version control and it's meant to be local env vars to 1 specific developer where as your .env file is shared across multiple developers and commit to version control. You may also have a separate .env.secrets file for anything not safe to commit which is populated however you see fit (external secret store, etc.).

In any of the above cases you could or would want Docker Compose to interpolate, .env, .env.local and env.secrets.

[nickjj]

I think one last thing to bring up are the existing COMPOSE_* env variables. By default Docker Compose will pick them up and reference them when running docker compose commands without needing to manually export or reference --env-file.

Besides covering the use case description that spawned this feature, from an intuitiveness perspective the expected outcome is COMPOSE_ENV_FILES works the same as the other COMPOSE_* variables.

[ndeloof]

what if .env defines COMPOSE_ENV_FILES=x.env and x.env also defines COMPOSE_ENV_FILES=y.env?
the resulting COMPOSE_ENV_FILES would then list files which where not actually used to load its own value 🤪

IMHO this really goes way beyond usability rules, and I'll veto such a feature

[nickjj]

Ok no problem.

You may want to consider reverting COMPOSE_ENV_FILES entirely because as it currently stands it's really misleading since every other COMPOSE_* env var gets picked up by .env by default. This will be the only exception. As is, it also doesn't help solve the use case of wanting to reference multiple env files without needing every developer / user to modify their local shell every time they run a Docker Compose command.

When you put both of those together, I'm not sure how much value there is to having COMPOSE_ENV_FILES when you could just pass in --env-file flags. In either case if you plan to use this feature as is you need to do something custom to make it work and adding an env var or flag are both custom things.

[ndeloof]

other COMPOSE_* variables can be set by .env file, and .env file can be set by COMPOSE_ENV_FILES, but as a chicken-egg problem COMPOSE_ENV_FILES can't be set by a variable in a non-default env file, this just seems reasonable

[rLoka]

Has there been any news to this? Official documentation is misleading, it does not work as described there: https://docs.docker.com/compose/environment-variables/envvars/#compose_env_files

I am getting this in case I want to load vars from other env specified (upon up -d).

WARN[0000] The "VAR1" variable is not set. Defaulting to a blank string. 
WARN[0000] The "VAR2" variable is not set. Defaulting to a blank string. 
...

This would be an example of additional .env:

COMPOSE_ENV_FILES=.env.additional

Content of .env.additional:

VAR1=foo
VAR2=bar

[RicardoM17]

It wouldn't be automagical. It would be dependent on you explicitly setting COMPOSE_ENV_FILES in the .env file. If you don't set that env var in .env then Docker Compose would only interpolate variables in .env.

There's a number of use cases where you couldn't or wouldn't want to set everything in a single .env. Maybe you have a separate .env.local file which isn't commit to version control and it's meant to be local env vars to 1 specific developer where as your .env file is shared across multiple developers and commit to version control. You may also have a separate .env.secrets file for anything not safe to commit which is populated however you see fit (external secret store, etc.).

In any of the above cases you could or would want Docker Compose to interpolate, .env, .env.local and env.secrets.

I am exactly in this use case and it's extremely sad to see such an issue that was raised and then it was vetoed. Unfortunately this is not the first time that this has happened with docker and compose projects. Honestly it seems that the maintainers are more worried about setting arbitrary boundaries that then completely handicap some of the features. Coincidentally another example of this is also regarding docker profiles.

Even the author of the feature (@glours ) admitted that not adding support via the .env was just an oversight that then @nickjj was kind enough to correct.

Plenty of people that use this tool have different use cases and requirements, that is fine and expected. Yes you can't support every single use case but many of them are downright obvious and more than relevant, like in this scenario. I cannot understand why this project goes so out of their way to reject help from other users while at the same time making their experience worse.

Just my 2 cts.

[RicardoM17]

@ndeloof then how is one supposed to set the .env files to be used by compose in a way that is scalable and reliable and most importantly is project in/dependent? I see your concerns as to how this can get out of hand if we are defining the .env files to be used by compose within the .env file itself. But while that is a valid concern it does not erase the issue.

Having to manually export a variable every single time or having to add lengthy options to a command is completely impractical.

More "permanent" solutions like aliasing the command or adding such a variable export to .bashrc do not properly work when one has more than 1 docker project.

So if we can't have this variable defined in .env file, even though this an oversight from the initial implementation (see my comment above), can we have it somewhere else? Like for example in the compose.yaml file?

Thanks