Interpolation of variables in .env ignores shell variables

[emersonf]

Description
The documentation says that:

Values present in the environment at runtime always override those defined inside the .env file. Similarly, values passed via command-line arguments take precedence as well.

This doesn't seem to be the case when interpolating a variable used inside the .env file.

Steps to reproduce the issue:
Assume a .env of:

VERSION=from_dot_env
IMAGE=foo:${VERSION}

and a compose.yml of:

services:
  echo:
    image: bash
    environment:
      IMAGE: ${IMAGE}
    command: echo "$IMAGE"

Describe the results you received:
I get:

% VERSION=from_shell docker compose run echo
foo:from_dot_env

Describe the results you expected:
I expected:

% VERSION=from_shell docker compose run echo
foo:from_shell

because VERSION=from_shell is present in the shell, and should "override [VERSION=from_dot_env] defined inside the .env file".

Additional information you deem important (e.g. issue happens only occasionally):
I get what I expect if I specify the entire IMAGE variable:

% IMAGE=bar:from_shell docker compose run echo
bar:from_shell

so it looks like the shell isn't being considered when interpolating variables defined in .env.

Output of docker compose version:
I see the same result on V1 and V2.

% docker compose version
Docker Compose version v2.2.1

% docker-compose version
docker-compose version 1.29.2, build 5becea4c
docker-py version: 5.0.0
CPython version: 3.9.0
OpenSSL version: OpenSSL 1.1.1h  22 Sep 2020

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.7.1)
  compose: Docker Compose (Docker Inc., v2.2.1)
  scan: Docker Scan (Docker Inc., v0.11.0)

Server:
 Containers: 72
  Running: 3
  Paused: 0
  Stopped: 69
 Images: 102
 Server Version: 20.10.11
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: local
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 11.7GiB
 Name: docker-desktop
 ID: PL5B:VYGD:V6Y2:T3NB:U6DU:IXBL:KQGG:G745:YCZF:BOY3:C575:3CC3
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details:

[ulyssessouza]

Actually the variables are evaluated inside the .env file. What the documentation says is if the variable is overwritten in the shell, this is what will be used. It doesn't say that the variables that were composed from an evaluation of others will be re-evaluated. Eg:

.env:

VERSION=from_dot_env
IMAGE=foo:${VERSION}

docker-compose.yml:

services:
  echo:
    image: bash
    environment:
      IMAGE: ${IMAGE}
      VERSION: ${VERSION}
    command: bash -c "echo ${IMAGE} && echo ${VERSION}"

$ docker compose config:

services:
  echo:
    command:
    - bash
    - -c
    - echo foo:from_dot_env && echo from_dot_env
    environment:
      IMAGE: foo:from_dot_env
      VERSION: from_dot_env
    image: bash
    networks:
      default: null
networks:
  default:
    name: test_default

$ VERSION=from_shell docker compose config:

services:
  echo:
    command:
    - bash
    - -c
    - echo foo:from_dot_env && echo from_shell
    environment:
      IMAGE: foo:from_dot_env
      VERSION: from_shell
    image: bash
    networks:
      default: null
networks:
  default:
    name: test_default

As you can see, the variable VERSION was properly overwritten with the shell value.

[emersonf]

@ulyssessouza thanks for looking into this. I think if the documentation can be read in different ways, it's worth clarifying.

It doesn't say that the variables that were composed from an evaluation of others will be re-evaluated.

It doesn't say they aren't, either. This sentence implies they are, at least to me:

Values present in the environment at runtime always override those defined inside the .env file. Similarly, values passed via command-line arguments take precedence as well.

So I think the documentation should be updated to avoid ambiguity.

But far more interestingly than the documentation itself, I think it would be very useful if the shell-defined variables would have precedence even during .env file interpolation. Besides that approach having lower conceptual weight, it gives more control to users. We'd be able to create variables composed of other variables in .env, and only have to override the narrowest variable we care about to customise the environment, e.g. just the image VERSION in my example, not the entire IMAGE repository.

Is that something that could be considered?

[blacksheep--]

This used to work in V1 and leads to breaking changes in our stacks.

This is the output using docker-compose V1.26.2:

/app/test # docker-compose config
services:
  echo:
    command: bash -c "echo foo:from_dot_env && echo from_dot_env"
    environment:
      IMAGE: foo:from_dot_env
      VERSION: from_dot_env
    image: bash
version: '3.7'

/app/test # VERSION=from_shell docker-compose config
services:
  echo:
    command: bash -c "echo foo:from_shell && echo from_shell"
    environment:
      IMAGE: foo:from_shell
      VERSION: from_shell
    image: bash
version: '3.7'

The way it worked, always respecting the proposed precedence from the documentation is clearly what one is expecting when using it.