Interpolation of variables in .env ignores variable precedence in V2 - was working in V1

[blacksheep--]

Description

The documentation is stating:

Values present in the environment at runtime always override those defined inside the .env file. Similarly, values passed via command-line arguments take precedence as well.

Interpolation in .env recently broke with that behaviour and is ignoring precendence of variables in the shell. The behaviour was introduced in V2 and was working in V1 leading to an unnecessary breaking change.

Reference: 9045

Steps to reproduce the issue:
Assume a .env of:

VERSION=from_dot_env
IMAGE=foo:${VERSION}

and a compose.yml of:

services:
  echo:
    image: bash
    environment:
      IMAGE: ${IMAGE}
    command: echo "$IMAGE"

Describe the results you received:

in V2 (using --compatibility does not make any difference):

% VERSION=from_shell docker compose config
name: test
services:
  echo:
    command:
    - bash
    - -c
    - echo foo:from_dot_env && echo from_shell
    environment:
      IMAGE: foo:from_dot_env
      VERSION: from_shell
    image: bash
    networks:
      default: null
networks:
  default:
    name: test_default

in V1:

% VERSION=from_shell docker-compose config
services:
  echo:
    command: bash -c "echo foo:from_shell && echo from_shell"
    environment:
      IMAGE: foo:from_shell
      VERSION: from_shell
    image: bash
version: '3.7'

Describe the results you expected:

I expect V2 to also return the same consistent output regarding environment variable precedence as V1 is producing:

% VERSION=from_shell docker-compose config
services:
  echo:
    command: bash -c "echo foo:from_shell && echo from_shell"
    environment:
      IMAGE: foo:from_shell
      VERSION: from_shell
    image: bash
version: '3.7'

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker compose version:

Docker Compose version v2.4.1

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.8.2)
  compose: Docker Compose (Docker Inc., v2.4.1)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 3
  Running: 2
  Paused: 0
  Stopped: 1
 Images: 52
 Server Version: 20.10.14
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc version: v1.0.3-0-gf46b6ba
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.102.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 9.72GiB
 Name: docker-desktop
 ID: D35Z:TDUA:SCK3:3SYM:BJL6:C3ME:ZB6G:BJ6X:O65Q:IRCD:QYOH:ZTAZ
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 60
  Goroutines: 56
  System Time: 2022-05-04T15:09:48.264070158Z
  EventsListeners: 4
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

Additional environment details:

[ndeloof]

fixed by #9475

[tboddyspargo]

@ndeloof - Based on my experience, #9475 introduced a regression in v2.5.1 with regard to my usage of this feature in V2... I could be misunderstanding the documentation, but I've been relying on shell variables to override ANY env file. However, when I reproduced the test and erroneous behavior described by @blacksheep-- above with one slight modification (the config file and env file are explicitly passed as an argument), then the shell value is still ignored.

Output of docker compose version

Docker Compose version v2.5.1

test.env

IMAGE=from_env

compose-config.yaml

services:
  echo:
    image: bash
    environment:
      - IMAGE
    command: echo "$IMAGE"

Result without shell override

$ docker-compose -f compose-config.yaml --env-file test.env config
name: test
services:
  echo:
    command:
    - echo
    - from_env
    environment:
      IMAGE: from_env
    image: bash
    networks:
      default: null
networks:
  default:
    name: test_default

Result with shell override

$ IMAGE=from_shell docker-compose -f compose-config.yaml --env-file test.env config
name: test
services:
  echo:
    command:
    - echo
    - from_env
    environment:
      IMAGE: from_env
    image: bash
    networks:
      default: null
networks:
  default:
    name: test_default

The above test correctly showed IMAGE: from_shell in Compose v2.5.0.

The new behavior in v.2.5.1 does not align with my reading of documentation:

When you set the same environment variable in multiple files, here’s the priority used by Compose to choose which value to use:

1. Compose file
2. Shell environment variables
3. Environment file
4. Dockerfile
5. Variable is not defined
   -- https://docs.docker.com/compose/environment-variables/#set-environment-variables-with-docker-compose-run

[OktarinTentakel]

+1

Currently .env overrides everything provided in shell, which is the opposite of what the documentation says.

EDIT: using compose v1 instead of v2 cirumvents the problem

[ndeloof]

@tboddyspargo indeed, my fix is incomplete, sorry for this

[vsimon]

Hi @ndeloof FYI I built from the latest master to try to get past this bug and I am seeing that the variables in an .env file still override any variable provided through the shell. The go.mod file has compose-go 1.2.6.

❯ docker-compose version
Docker Compose version v2.5.1-16-g5b6b674d

[blacksheep--]

works now 💯 👍 thank you