Regression in handling of environment variables

[camilledejoye]

---

BUG REPORT INFORMATION

Description

From the documentation:

Values in the shell take precedence over those specified in the .env file.

This is no longer true since the 2.5.1 release.

Steps to reproduce the issue:

1. Create a docker-compose.yaml file with:

services:
  alpine:
    image: alpine:latest
    environment:
      - MY_VAR

2. Create a .env file with:

MY_VAR=file

3. Run MY_VAR=shell docker compose run alpine printenv MY_VAR

Describe the results you received:

Current output: file

Describe the results you expected:

Expected output: shell

Additional information you deem important (e.g. issue happens only occasionally):

Works on Docker Compose 2.5.0.

Output of docker compose version:

Docker Compose version 2.5.1

Output of docker version:

Client:
 Version:           20.10.16
 API version:       1.41
 Go version:        go1.18.1
 Git commit:        aa7e414fdc
 Built:             Sat May 14 12:48:33 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.16
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.1
  Git commit:       f756502055
  Built:            Sat May 14 12:47:27 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.6.4
  GitCommit:        212e8b6fa2f44b9c21b2798135fc6fb7c53efc16.m
 runc:
  Version:          1.1.2
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.8.2-docker)
  compose: Docker Compose (Docker Inc., 2.5.0)

Server:
 Containers: 186
  Running: 12
  Paused: 0
  Stopped: 174
 Images: 22
 Server Version: 20.10.16
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16.m
 runc version:
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.17.9-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.57GiB
 Name: cdejoye-arch
 ID: YFDR:NZWH:MQ5W:LNXA:3F47:VZMQ:7MPC:MUKR:4LT6:SUFM:TRTW:BTFN
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details:

[maxcleme]

@camilledejoye

Are you sure it was "working"in 2.5.0 ?

I tried with 2.4.1, same results, however afaik it is intended,

When documentation is speaking about shell environment variable, it speaks about shell inside the container, not shell starting docker, otherwise many others env variable will be changed and you don't want that (eg: $HOME).

Settings container shell variable works fine by using env file (as you did), -e can also be used to override, as well as defining them in the inside shell.

docker compose run --rm -e MY_VAR=foo alpine printenv MY_VAR
docker compose run --rm alpine sh -c "MY_VAR=shell printenv MY_VAR"

EDIT: Please ignore that comment, I can confirm the regression, I step back up until 2.2.3 and shell variable are well handled there, I'll look at it.

[ssc-]

@maxcleme The replacement of a variable set in .env by the (outer) shell env as described here https://docs.docker.com/compose/environment-variables/#the-env-file stopped working.

Minimal setup as described by @camilledejoye

Call without overwriting .env

$ docker-compose config
docker-compose config
name: example
services:
  alpine:
    environment:
      MY_VAR: file
    image: alpine:latest
    networks:
      default: null
networks:
  default:
    name: example_default

Call with set variable (expected to be replaced)

$ docker-compose version
Docker Compose version 2.5.1

$ MY_VAR=test docker-compose config
name: example
services:
  alpine:
    environment:
      MY_VAR: file
    image: alpine:latest
    networks:
      default: null
networks:
  default:
    name: example_default

Old / exptected behavior

$ docker-compose version
Docker Compose version 2.5.0

$ MY_VAR=test docker-compose config
name: example
services:
  alpine:
    environment:
      MY_VAR: test
    image: alpine:latest
    networks:
      default: null
networks:
  default:
    name: example_default

[maxcleme]

@ssc-

Got it, I can confirm the regression, I step back up until 2.2.3 and shell variable are well handled there, I'll look at it.

[ssc-]

@maxcleme
Think it's this one here..

compose/cmd/compose/compose.go

Lines 214 to 221 in 7e3564b

	append(po,
	cli.WithWorkingDirectory(o.ProjectDir),
	cli.WithOsEnv,
	cli.WithEnvFile(o.EnvFile),
	cli.WithDotEnv,
	cli.WithConfigFileEnv,
	cli.WithDefaultConfigPath,
	cli.WithName(o.ProjectName))...)

the order changed with 65b827d

[maxcleme]

@ssc- Sure it is, I'm asking internally to better understand what was the motivation behind this, I highly doubt it was unintended.

[maxcleme]

@ssc- @camilledejoye We just bump compose-go to latest version in order to integrate one remaining fix that was missing, are you able to try it out by building compose plugin locally ?

There is still some edge cases that needs to be fixes, we are going to include new test cases for env variables in the next release, in the mean time I hope that using local build will fix your issues.

[ssc-]

@maxcleme yep, I can build/try locally, any particular branch or just HEAD (v2)?

[maxcleme]

@ssc- It has been integrated into v2

#9495

[ssc-]

Thanks a lot @maxcleme, can confirm, works as expected now 👍

$ ../compose/bin/docker-compose -v
Docker Compose version v2.5.1-16-g5b6b674d

$ ../compose/bin/docker-compose config
name: example
services:
  alpine:
    environment:
      MY_VAR: file
    image: alpine:latest
    networks:
      default: null
networks:
  default:
    name: example_default

$ MY_VAR=test ../compose/bin/docker-compose config
name: example
services:
  alpine:
    environment:
      MY_VAR: test
    image: alpine:latest
    networks:
      default: null
networks:
  default:
    name: example_default

[camilledejoye]

@maxcleme I confirm as well that it's working for the simple use cases I gave here and in the real project from which I discover this change in the first place.

[mloskot]

I can confirm this regression when using on Linux (w/ Bash) and Windows (w/ PowerShell 7):

docker version: 20.10.14
docker compose version: v2.5.1

$ cat .env
TAG=v1.5

$ cat docker-compose.yml
version: '2'
services:
  web:
    image: "webapp:${TAG}"

$ docker compose config
name: docker-env
services:
  web:
    image: webapp:v1.5
    networks:
      default: null
networks:
  default:
    name: docker-env_default

$ export TAG=XXX

$ docker compose config
name: docker-env
services:
  web:
    image: webapp:v1.5
    networks:
      default: null
networks:
  default:
    name: docker-env_default

whereas I expected image: webapp:XXX in the last output.

FWIW, https://docs.docker.com/compose/environment-variables/ explains that export TAG=XXX action is expected to override the .env definition:

Values set in the shell environment override those set in the .env file.

[camilledejoye]

Fixed on 2.6.0

[the-ge]

And the regression is live again in 2.8.0.