[BUG] env_file directive in include does not affect interpolation for service env_file

[ShadowLNC]

Description

Within an include directive, I can specify an env_file. This changes interpolation for the included YAML, but if that YAML file has a service with an env_file, that environment file doesn't have its interpolation changed, and I believe it should.

Steps To Reproduce

docker-compose.yml

include:
    - path: include.compose.yml
      env_file:
          - .env
          - extras.env

include.compose.yml

services:
    mine:
        image: alpine
        command: env
        env_file: fakefile.env
        environment:
            COMPOSE_EXPLICIT: "abc"
            COMPOSE_FROM_MAIN: "${COMPOSE_FROM_MAIN}"
            COMPOSE_FROM_EXTRAS: "${COMPOSE_FROM_EXTRAS}"

.env

ENV_FROM_MAIN="env-frommain"
COMPOSE_FROM_MAIN="compose-frommain"

extras.env

ENV_FROM_EXTRAS="env-fromextras"
COMPOSE_FROM_EXTRAS="compose-fromextras"

fakefile.env

INTERPOLATE_FROM_MAIN="${ENV_FROM_MAIN}"
INTERPOLATE_FROM_EXTRAS="${ENV_FROM_EXTRAS}"

Then, I run docker compose up to see the environment:

docker compose up (breaks extras.env)

The extras.env file is missing from the env file interpolation step, but present on the YAML interpolation.

WARN[0000] The "ENV_FROM_EXTRAS" variable is not set. Defaulting to a blank string. 
[+] Running 0/0
 ⠋ Container fake-mine-1  Recreated                                                                                                        0.0s 
Attaching to mine-1
mine-1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
mine-1  | HOSTNAME=4bbcdec26225
mine-1  | COMPOSE_FROM_MAIN=compose-frommain
mine-1  | COMPOSE_FROM_EXTRAS=compose-fromextras
mine-1  | COMPOSE_EXPLICIT=abc
mine-1  | INTERPOLATE_FROM_MAIN=env-frommain
mine-1  | INTERPOLATE_FROM_EXTRAS=
mine-1  | HOME=/root
mine-1 exited with code 0

docker compose --env-file=extras.env up (breaks .env)

The .env file is missing from the env file interpolation step, but present on the YAML interpolation.

WARN[0000] The "ENV_FROM_MAIN" variable is not set. Defaulting to a blank string. 
[+] Running 0/0
 ⠋ Container fake-mine-1  Recreated                                                                                                        0.1s 
Attaching to mine-1
mine-1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
mine-1  | HOSTNAME=fd388d4b94b9
mine-1  | INTERPOLATE_FROM_EXTRAS=env-fromextras
mine-1  | INTERPOLATE_FROM_MAIN=
mine-1  | COMPOSE_FROM_EXTRAS=compose-fromextras
mine-1  | COMPOSE_EXPLICIT=abc
mine-1  | COMPOSE_FROM_MAIN=compose-frommain
mine-1  | HOME=/root
mine-1 exited with code 0

In both cases I would expect both files (.env and extras.env) to be used because of the env_file directive under include, irrespective of the CLI --env-file argument. My expectation holds true for the YAML interpolation, but not when interpolating an environment file referenced by the YAML file.

Compose Version

$ docker compose version
Docker Compose version v2.26.1-desktop.1
$ docker-compose version
Docker Compose version v2.26.1-desktop.1

Docker Environment

docker info

Client:
 Version:    26.0.0
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.1-desktop.1
    Path:     /Users/scott/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.26.1-desktop.1
    Path:     /Users/scott/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.27
    Path:     /Users/scott/.docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/scott/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.23
    Path:     /Users/scott/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     /Users/scott/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.1.0
    Path:     /Users/scott/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/scott/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.6.3
    Path:     /Users/scott/.docker/cli-plugins/docker-scout
WARNING: Plugin "/Users/scott/.docker/cli-plugins/docker-scan" is not valid: failed to fetch metadata: fork/exec /Users/scott/.docker/cli-plugins/docker-scan: no such file or directory

Server:
 Containers: 2
  Running: 1
  Paused: 0
  Stopped: 1
 Images: 41
 Server Version: 26.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.6.22-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 8
 Total Memory: 7.755GiB
 Name: docker-desktop
 ID: 85b60fc6-eedd-4b8c-996a-89f62a45ab85
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/scott/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Anything else?

No response

[ndeloof]

Note: this require we re-implement WithServicesEnvironmentResolved within loadYamlModel so that env_file get loaded before we merge the included model with the including one. This is obviously feasible but need to handle with care as we already got a bunch of regressions in this area

[clementgoclock]

Got something similar here but not using include directive.
Any interpolation variables from something else than .env is not working.

Exemple :

    env_file: vpn.env
    environment:
      HOMEPAGE_VAR_USER_NAME: "${USER_NAME:-Téléporteur pas configuré !}"
      HOMEPAGE_VAR_COHORT: "${SCHOOL_COHORT:-Sans Promo}"
      HOMEPAGE_VAR_VPN_NAME: "${VPN_NAME:-sans-vpn}" 

Are using default values defined in environment. If i copy variables or rename vpn.env to .env it works.

Variables from vpn.env are correctly passed to the CT btw:

/app # printenv | grep COHORT
SCHOOL_COHORT=CacaHati
HOMEPAGE_VAR_COHORT=Sans Promo

@ShadowLNC could you try without include to check if interpolation is working with an env file different from .env ?

Compose version :

Docker Compose version v2.26.1-desktop.1

[ndeloof]

@clementgoclock this is a distinct issue, please fill another one so we can track it.
AFAICT this one would be hard to address, as you expect (higher-priority) environment to be resolved based on (lower-priority) env_file values. This would require some significant refactoring...