Add support to use --env-file for docker compose up

[im-kulikov]

It currently lacks support for --env-file flag, to load env vars form a file and automatically load env variables from .env like it does docker-compose.

$ docker compose up --help
Create and start containers

Usage:
  docker compose up [SERVICE...] [flags]

Flags:
      --build                     Build images before starting containers.
  -d, --detach                    Detached mode: Run containers in the background
  -e, --environment stringArray   Environment variables
  -f, --file stringArray          Compose configuration files
  -h, --help                      help for up
  -p, --project-name string       Project name
      --remove-orphans            Remove containers for services not defined in the Compose file.
      --workdir string            Work dir

Global Flags:
      --config DIRECTORY   Location of the client config files DIRECTORY (default "/Users/kulikov/.docker")
  -c, --context string     context
  -D, --debug              Enable debug output in the logs
  -H, --host string        Daemon socket(s) to connect to

[ndeloof]

Support for .env file [has been introduced](daacc87 and is available with compose-cli 1.0.6
not sure which version is deployed with Desktop, you can check with docker version => Cloud integration

but --env-file is indeed not implemented

[ndeloof]

related: compose-spec/compose-go#97

[im-kulikov]

@ndeloof it's strange, but .env doesn't applying with docker compose up (docker desktop for macOS v3.1.0)

PS: values doesn't provided into docker-compose file

[ndeloof]

Docker Desktop 3.1.0 indeed includes compose-cli "Cloud integration" 1.0.7

I have the same installed.
I ran a quick test using this minimal compose file:

services:
  test:
     image: alpine
     command: echo hello ${FOO}

and .env file:

FOO=1000

and get the expected result:

➜  docker compose up
The new 'docker compose' command is currently experimental. To provide feedback or request new features please open issues at https://github.com/docker/compose-cli
[+] Running 2/2
 ⠿ Network "test_default"  Created                                                                                         0.1s
 ⠿ test_test_1             Created                                                                                         0.0s
Attaching to test_test_1
test_test_1 | hello 1000

[im-kulikov]

pls reopen, because I think there's bug with dotenv:

for example:

→ tree
.
|____docker
| |____docker-compose.yml
|____.env

dotenv file:

→ cat .env
FOO=1000

docker-compose file:

→ cat docker/docker-compose.yml
services:
  test:
     image: alpine
     command: echo hello ${FOO}

Running your example:

→ docker compose up -f docker/docker-compose.yml
The new 'docker compose' command is currently experimental. To provide feedback or request new features please open issues at https://github.com/docker/compose-cli
[+] Running 2/2
 ⠿ Network "docker_default"  Created                                                                                                                                                                                        0.1s
 ⠿ docker_test_1             Created                                                                                                                                                                                        0.1s
Attaching to docker_test_1
docker_test_1 | hello

and see what I have with docker-compose (python)

→ docker-compose -f docker/docker-compose.yml up
Recreating docker_test_1 ... done
Attaching to docker_test_1
test_1  | hello 1000
docker_test_1 exited with code 0

[im-kulikov]

As I can see if copying dotenv into docker folder (from example)

→ tree
.
|____docker
| |____.env
| |____docker-compose.yml
|____.env

dotenv from docker folder

→ cat docker/.env
FOO=2000

• compose-cli

→ docker compose up -f docker/docker-compose.yml
The new 'docker compose' command is currently experimental. To provide feedback or request new features please open issues at https://github.com/docker/compose-cli
[+] Running 1/0
 ⠿ docker_test_1  Recreated                                                                                                                                                                                                 0.0s
Attaching to docker_test_1
docker_test_1 | hello 2000

• docker-compose (python)

→ docker-compose -f docker/docker-compose.yml up
Recreating docker_test_1 ... done
Attaching to docker_test_1
test_1  | hello 1000
docker_test_1 exited with code 0

it's not same behaviour 😕

[im-kulikov]

Problem is here

func (o ProjectOptions) GetWorkingDir() (string, error) {
	if o.WorkingDir != "" {
		return o.WorkingDir, nil
	}
	for _, path := range o.ConfigPaths {
		if path != "-" {
			absPath, err := filepath.Abs(path)
			if err != nil {
				return "", err
			}
			return filepath.Dir(absPath), nil
		}
	}
	return os.Getwd()
}

[ndeloof]

thanks for digging into this, will look further tomorrow

[ndeloof]

Doing some experiments I noticed this behaviour changed in docker-compose 1.28 as docker/compose#7965
.env file is loaded (Environment.from_env_file(...) relative to override_dir. This path used to only be set by --project-dir before 1.28 but is now set as (first) compose file parent folder - same as implemented by compose-cli

see docker/compose#7928

[im-kulikov]

docker/compose#7928 (comment)

The easiest and most common usage consists on calling load_dotenv when the application starts, which will load environment variables from a file named .env in the current directory or any of its parents or from the path specificied

We should verify if the current behavior accidentally changed, or always has been like this. If it's always been like this, we may want to keep the behavior, as I can see this being a breaking change (users may explicitly switch to a different working-directory to use a different .env file)

As I understand, they decided to save old behavior.

[im-kulikov]

Latest versions 1.24-1.27 had this behavior and load .env file from folder, where it (docker-compose) was started.

[ndeloof]

I tend to agree, but then this has to be documented and debated on docker/compose#7928.
I would be happy to get this implemented "the python-dotenv way" if this is the decision taken by docker-compose maintainers
cc @ulyssessouza

[ndeloof]

closing in the meantime, as "work as expected" regarding the current status of docker-compose 1.28
I will create a task to align with docker-compose changes if docker/compose#7965 is reverted

[laithrafid]

this bug should be reopened

docker compose -d -f docker-compose.yaml --env-file db.env  up

Works ok , however when use -d flag throws warning in docker compose logs , and when i digged into env variables is not being passed

docker compose -d -f docker-compose.yaml --env-file db.env  up -d

WARN[0000] The "MYSQL_VERSION" variable is not set. Defaulting to a blank string.
WARN[0000] The "MYSQL_USER" variable is not set. Defaulting to a blank string.
WARN[0000] The "MYSQL_PASSWORD" variable is not set. Defaulting to a blank string.
WARN[0000] The "MYSQL_DATABASE" variable is not set. Defaulting to a blank string.
WARN[0000] The "ELASTIC_PASSWORD" variable is not set. Defaulting to a blank string.
WARN[0000] The "ELASTIC_VERSION" variable is not set. Defaulting to a blank string.
WARN[0000] The "ELASTIC_VERSION" variable is not set. Defaulting to a blank string.
WARN[0000] The "ELASTIC_VERSION" variable is not set. Defaulting to a blank string.
WARN[0000] The "CASS_VERSION" variable is not set. Defaulting to a blank string.
WARN[0000] The "CASS_VERSION" variable is not set. Defaulting to a blank string.
WARN[0000] The "MYSQL_VERSION" variable is not set. Defaulting to a blank string.

workaround this issue is to run docker-compose instead of using docker compose command

docker-compose  -f docker-compose.yaml --env-file db.env  up -d

docker-compose -v

docker-compose version 1.29.2, build 5becea4c

docker info 

Client:
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc., v0.7.1)
compose: Docker Compose (Docker Inc., v2.2.3)
scan: Docker Scan (Docker Inc., v0.17.0)

Server Version: 20.10.12
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
runc version: v1.0.2-0-g52b36a2
init version: de40ad0
Security Options:
seccomp
Profile: default
cgroupns
Kernel Version: 5.10.76-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 3
Total Memory: 3.842GiB
Name: docker-desktop
ID: ZFAI:DZXQ:55OC:NRPY:AMZW:EQHR:TV5D:M4KK:JFX3:U7P2:LBAZ:THK4
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
hubproxy.docker.internal:5000
127.0.0.0/8
Live Restore Enabled: false

docker version

Client:
Cloud integration: v1.0.22
Version: 20.10.12
API version: 1.41
Go version: go1.16.12
Git commit: e91ed57
Built: Mon Dec 13 11:46:56 2021
OS/Arch: darwin/amd64
Context: default
Experimental: true

Server: Docker Desktop 4.5.0 (74594)
Engine:
Version: 20.10.12
API version: 1.41 (minimum version 1.12)
Go version: go1.16.12
Git commit: 459d0df
Built: Mon Dec 13 11:43:56 2021
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: 1.4.12
GitCommit: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
runc:
Version: 1.0.2
GitCommit: v1.0.2-0-g52b36a2
docker-init:
Version: 0.19.0
GitCommit: de40ad0

[ndeloof]

@laithrafid Compose v2 is now hosted on https://github.com/docker/compose, please use this repo if you want to report a bug.

There's no -d flag you can pass to docker compose, only up -d is valid.