Relative paths for local volumes not working on windows

[stadlerism]

Description

On Windows, using Docker Compose V2, volumes specified using a relative path are not correctly mounted.

Steps to reproduce the issue:

1. Create the following docker-compose.yml

services:
  demo-fail:
    image: busybox
    entrypoint: /bin/true
    volumes:
    - demo_volume:/tmp/demo-volume
volumes:
  demo_volume:
    driver: local
    driver_opts:
      type: none
      device: ./demo-volume
      o: bind

2. Create the directory demo-volume next to the docker-compose.yml
3. run docker compose up

Describe the results you received:

Mounting the volume fails with the error
Error response from daemon: error while mounting volume '/var/lib/docker/volumes/tinkerbox_demo_volume/_data': failed to mount local volume: mount demo-volume:/var/lib/docker/volumes/tinkerbox_demo_volume/_data, flags: 0x1000: no such file or directory

Describe the results you expected:
Volume is mounted correctly inside the docker container.

Additional information you deem important (e.g. issue happens only occasionally):
Using an absolute path, the volume is mounted as expected.
As opposed to Docker Compose V1, Docker Compose V2 on Windows does not resolve relative paths into absolute paths.
Using Docker Compose V1:

> docker volume inspect tinkerbox_demo_volume
[
    {
        "CreatedAt": "2021-11-12T10:36:51Z",
        "Driver": "local",
        "Labels": {
            "com.docker.compose.project": "tinkerbox",
            "com.docker.compose.version": "1.29.2",
            "com.docker.compose.volume": "demo_volume"
        },
        "Mountpoint": "/var/lib/docker/volumes/tinkerbox_demo_volume/_data",
        "Name": "tinkerbox_demo_volume",
        "Options": {
            "device": "C:\\ABSOLUTE\\PATH\\TO\\demo-volume",
            "o": "bind",
            "type": "none"
        },
        "Scope": "local"
    }
]

Using Docker Compose V2:

> docker volume inspect tinkerbox_demo_volume
[
    {
        "CreatedAt": "2021-11-12T10:37:59Z",
        "Driver": "local",
        "Labels": {
            "com.docker.compose.project": "tinkerbox",
            "com.docker.compose.version": "2.0.0",
            "com.docker.compose.volume": "demo_volume"
        },
        "Mountpoint": "/var/lib/docker/volumes/tinkerbox_demo_volume/_data",
        "Name": "tinkerbox_demo_volume",
        "Options": {
            "device": "./demo-volume",
            "o": "bind",
            "type": "none"
        },
        "Scope": "local"
    }
]

Output of docker compose version:

Docker Compose version v2.0.0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., v0.6.3)
  compose: Docker Compose (Docker Inc., v2.0.0)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 2
  Running: 1
  Paused: 0
  Stopped: 1
 Images: 142
 Server Version: 20.10.8
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e25210fe30a0a703442421b0f60afac609f950a3
 runc version: v1.0.1-0-g4144b63
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.4.72-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 15.63GiB
 Name: docker-desktop
 ID: 5PON:NFXZ:GKCC:FXEA:NDR2:HM42:GO4Q:6HES:D5MR:RIE7:6SKY:SUMV
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

Additional environment details:
Host OS: Windows 10, Version 20H2, Build 19042.1288

[ndeloof]

Any reason you use this complex syntax, vs the common bind mount syntax: - ./demo_volume:/tmp/demo-volume ?

[stadlerism]

In my YAML these volumes are mounted in several different containers, hence having a central place where they are configured makes sense.

Granted, this can also be achieved by using YAML anchors and aliases. Still, this is only a workaround and does not address the issue at its core.

[ndeloof]

right, just asking because this is a pretty uncommon use of volumes - I actually never considered one could use the local volume driver to setup a bind mount

[DStape]

I also observe this on linux so it's not unique to windows.

$ docker compose version
Docker Compose version v2.2.2

$ docker version                                                                               
Client: Docker Engine - Community
 Version:           20.10.3
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        48d30b5
 Built:             Fri Jan 29 14:33:25 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.9
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.8
  Git commit:       79ea9d3
  Built:            Mon Oct  4 16:06:50 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.11
  GitCommit:        5b46e404f6b9f661a205e28d59c982d3634148f8
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[Faboslav]

Same case for OSX on M1 with:
Docker Compose version v2.3.3

Client:
 Cloud integration: v1.0.22
 Version:           20.10.13
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 10 14:08:43 2022
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.6.1 (76265)
 Engine:
  Version:          20.10.13
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.15
  Git commit:       906f57f
  Built:            Thu Mar 10 14:05:37 2022
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.5.10
  GitCommit:        2a1d4dbdb2a1030dc5b01e96fb110a9d9f150ecc
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[spreadpando]

This caused quite a bit of confusion for me. The use of relative paths in docker-compose is outlined in the "getting started" portion of the docs. I've only experienced this on Windows so far, and didn't have a problem with MacOS or WSL, but this issue complicates things for a cross-platform dev team wanting to use docker-compose.

[IvanOVasiliev]

When will be or has been this issue fixed?

[ndeloof]

This has been fixed by compose-spec/compose-go#332
available in v2.17.0-rc.1

[arcanisgk]

It's not to be a joke but... it happens to me even worse... I don't even get an error in the Windows 11 terminal, this is my YML:

version: "3.9"

services:
  reverse-proxy:
    env_file:
      - .env
    container_name: Proxy-Server
    image: nginx
    restart: always
    depends_on:
      - webserver
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro
      - ./config/nginx/nginx.conf:/etc/nginx/nginx.conf
      - certs:/etc/nginx/certs:ro
    extra_hosts:
      - "lh-2.dock:127.0.0.1"
      - "pma.lh-2.dock:127.0.0.1"
    ports:
      - 80:80
      - 443:443
    networks:
      - lamp-network
    environment:
      - TRUST_DOWNSTREAM_PROXY=true
      - ENABLE_WEBSOCKETS=true

  webserver:
    env_file:
      - .env
    container_name: LH-2-Web-Server
    build:
      context: ./bin/php81
      args:
        VIRTUAL_HOST: lh-2.dock
    restart: always
    expose:
      - 80
      - 443
    networks:
      - lamp-network
    volumes:
      - ./../project/:/var/www/html:rw
      - ./../project/public:/var/www/html/public:rw
      - ./config/vhosts:/etc/apache2/sites-enabled
      - ./config/php/php.ini:/usr/local/etc/php/php.ini
      - ./log/apache2:/var/log/apache2
      - ./log/cron:/var/log/cron
      - certs:/etc/ssl/certs:rw
    environment:
      VIRTUAL_HOST: lh-2.dock
      LH_WEB_MASTER: icarosnet@email.com
      LH_APACHE_DOCUMENT_ROOT: /var/www/html/
      LH_DOCUMENT_ROOT: public
    extra_hosts:
      - "host.docker.internal:host-gateway"
    labels:
      - "lh2.setup.description=Web Server"
      - "lh2.setup.role=webserver"

volumes:
  certs:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: ./config/ssl

networks:
  lamp-network:
    name: lamp-network
    driver: bridge

my php 81 Dockerfile where I execute several actions:

FROM php:8.1-apache-bullseye

ARG DEBIAN_FRONTEND=noninteractive

ARG VIRTUAL_HOST

RUN apt-get update &&  \
    apt-get upgrade -y --no-install-recommends --fix-missing

RUN apt-get install -y --no-install-recommends --fix-missing tzdata sed build-essential dialog nano apt-utils cron wget git curl zip openssl gettext-base libnss3-tools

RUN curl -JLO "https://dl.filippo.io/mkcert/latest?for=linux/amd64" && \
    chmod +x mkcert-v*-linux-amd64 && \
    cp mkcert-v*-linux-amd64 /usr/local/bin/mkcert

RUN mkcert -install

RUN mkdir -p /etc/ssl/certs

RUN cd /etc/ssl/certs

RUN mkcert \
    -cert-file /etc/ssl/certs/${VIRTUAL_HOST}.crt \
    -key-file /etc/ssl/certs/${VIRTUAL_HOST}.key \
    "${VIRTUAL_HOST}" "*.${VIRTUAL_HOST}"

RUN chmod 0777 /etc/ssl/certs/${VIRTUAL_HOST}.crt

RUN chmod 0777 /etc/ssl/certs/${VIRTUAL_HOST}.key

RUN a2enmod rewrite 
RUN a2enmod ssl 
RUN a2enmod headers 
RUN a2enmod proxy_wstunnel

RUN service apache2 restart

RUN mkdir -p /var/log/cron && \
    chmod 755 /var/log/cron && \
    touch /var/log/cron/cron.log

RUN echo "* * * * * root /var/www/html/Asset/resource/cron/cron.sh >> /var/log/cron/cron.log 2>&1" > /etc/cron.d/cron-task && \
    chmod 0644 /etc/cron.d/cron-task

CMD cron && /usr/local/bin/apache2-foreground && chmod +x /var/www/html/Asset/resource/cron/cron.sh && tail -f /var/log/cron/cron.log

the problem: if I clear the docker desktop cache and run everything with a simple "up" from the terminal. the declared volume is not mounted.

Log:

2023-10-10 11:07:44 AH00526: Syntax error on line 26 of /etc/apache2/sites-enabled/vhost.conf:
2023-10-10 11:07:44 SSLCertificateFile: file '/etc/ssl/certs/lh-2.dock.crt' does not exist or is empty
2023-10-10 11:07:45 AH00526: Syntax error on line 26 of /etc/apache2/sites-enabled/vhost.conf:
2023-10-10 11:07:45 SSLCertificateFile: file '/etc/ssl/certs/lh-2.dock.crt' does not exist or is empty
2023-10-10 11:07:47 AH00526: Syntax error on line 26 of /etc/apache2/sites-enabled/vhost.conf:
2023-10-10 11:07:47 SSLCertificateFile: file '/etc/ssl/certs/lh-2.dock.crt' does not exist or is empty

when inspection for the ssl files:

Even removing "ro" and "rw" from the paths the problem persists... it is as if when mounting the volume, it ignores the existing files or those created by the Dockerfile

### What's going on here????

[ndeloof]

those created by the Dockerfile

as you use a bind mount, files set by the Dockerfile will be overridden. Bind mount replace the Docker image filesystem folder by a unix mount to your configured source, so any previously existing file is hidden.

[arcanisgk]

those created by the Dockerfile

as you use a bind mount, files set by the Dockerfile will be overridden. Bind mount replace the Docker image filesystem folder by a unix mount to your configured source, so any previously existing file is hidden.

some kind of mount point that keeps all the files: files created in the Dockerfile, file previously coming from the FS of the Host OS?

Otherwise I don't see a way to share the files in the same place for two containers and the OS Host.