`docker compose` variable substitution broken

[tim-timman]

Description

Docker compose is not properly substituting variables from .env files as strings. This worked before and still works for the docker-compose

Steps to reproduce the issue:

1. Create these files:

   # compose.yml
   
   services:
     my_service:
       image: hello-world
       environment:
         MY_PASSWORD: ${MY_PASSWORD}

   # .env
   
   MY_PASSWORD=[*s.,x4J[s&t%cHTD~|+h,G!KD<_0}-j2.

2. Run docker compose config

Describe the results you received:

yaml: line 4: did not find expected alphabetic or numeric character

Describe the results you expected:

services:
  my_service:
    environment:
      MY_PASSWORD: '[*s.,x4J[s&t%cHTD~|+h,G!KD<_0}-j'
    image: hello-world
version: '3.9'

Additional information you deem important (e.g. issue happens only occasionally):

• Single or double quoting doesn't make a difference.
• docker-compose config gives the correct output as did docker compose config with the previous version from Docker Desktop 3.5.2.
• It breaks if the MY_PASSWORD starts with any of these characters: {, }, [, ], ,, &, *, ?, |, >, !, %, @, `
• The char # effectively comments it out; not sure if it's intended behavior but I'm not sure it's useful on a key=value line
• A backslash \ at either the start of the env value or before the variable substitution allows it to parse without errors, but the character ends up in the string.

Output of docker version:

Client:
 Cloud integration: 1.0.17
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:55:20 2021
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Fri Jul 30 19:52:10 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., v0.6.1-docker)
  compose: Docker Compose (Docker Inc., v2.0.0-rc.1)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 12
  Running: 11
  Paused: 0
  Stopped: 1
 Images: 98
 Server Version: 20.10.8
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e25210fe30a0a703442421b0f60afac609f950a3
 runc version: v1.0.1-0-g4144b63
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.47-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 3
 Total Memory: 3.842GiB
 Name: docker-desktop
 ID: RI76:ZL4I:OAZE:WNKU:PZUB:REK6:QBKR:YKER:CNVV:DNXL:3NQJ:6FRT
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

[tim-timman]

The issue we're seeing is that the value of an environment variable seems to be participating in the yaml parsing of the document, which points to this PR likely being the culprit: compose-spec/compose-go#142

These issues seems to all be variations of the variable substitution regression.

• Variable substitution happens in .env files when using docker compose, but not when using docker-compose. #1988
• Empty string in env var for command not accepted in v2.0.0-rc.1, was always fine before #2013
• Environment variable contents cause YAML error in docker-compose pull after upgrading to Docker Desktop 3.6.0 with Docker Compose v2.0.0-rc.1 #2018

[Ilyes512]

@tim-timman
Yup, I was about to create an issue where ENV vars containing json strings are being parsed as yaml instead as string.

For example:

.env:

EXAMPLE={"foobar":"value"}

docker-compose.yml:

version: '3.7'
services:
  example:
    image: some/image
    build:
      context: .
      args:
        FOOBAR: $FOOBAR

When I run docker-compose config with v1 I get:

services:
  example:
    build:
      args:
        FOOBAR: '{"foobar":"example"}'
      context: /home/<the_rest>/test
    image: some/image
version: '3.7'

But when I run it with V2 I get:

services.example.build.args.FOOBAR must be a string, number, boolean or null

Because the json string is being parsed.

[ulyssessouza]

Looks like this is fixed with compose-spec/compose-go#167

Just notice that quotes are required.