Compose v2 `extends` doesn't merge `volumes` sequence using path as key

[thirdgen88]

Description

When using extends to pull in an external service definition, the expected behavior is to merge the volumes sequence with the key as the target path in the container. This design allows you to declare a named volume to be required in the source service, but then to customize the name of that volume in the destination service (the one with the extends key) as long as you target the same path.

It seems that docker compose is not doing this quite right where docker-compose is.

Steps to reproduce the issue:

1. Suppose you have a stack file, foo.yml:

# foo.yml
---
services:
  my-app:
    image: ubuntu
    volumes:
      - my-app-vol:/app

and a docker-compose.yml:

# docker-compose.yml
---
services:
  app1:
    extends:
      file: foo.yml
      service: my-app
    volumes:
      - app1-data:/app

volumes:
  app1-data:

2. Attempt to compile the config with docker compose config and observe:

$ docker compose config
service "app1" refers to undefined volume my-app-vol: invalid compose project

3. Note that docker-compose config produces the expected compiled config with the single volume, merged from the two volumes sequences key'ing off of the target path:

$ docker-compose config
services:
  app1:
    image: ubuntu
    volumes:
    - app1-data:/app:rw
version: '3.9'
volumes:
  app1-data: {}

Describe the results you received:
Using docker compose fails to properly merge the volumes sequence and complains about an invalid compose project.

Describe the results you expected:
Use of docker compose should be equivalent to docker-compose, which seems to produce the correct behavior.

Output of docker-compose --version:

docker-compose version 1.29.2, build 5becea4c

Output of docker version:

Client: Docker Engine - Community
 Cloud integration: 1.0.17
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        f0df350
 Built:             Wed Jun  2 11:56:47 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:54:58 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  compose: Docker Compose (Docker Inc., v2.0.0-beta.6)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 34
  Running: 4
  Paused: 0
  Stopped: 30
 Images: 82
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: lbeujprm4qt955n8rv1dff6rc
  Is Manager: true
  ClusterID: z9wresmnlyehnlxmnnew1uzub
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8  
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 192.168.65.3
  Manager Addresses:
   192.168.65.3:2377
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.4.72-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 24.91GiB
 Name: docker-desktop
 ID: NWF2:DOVN:QUXC:I3AH:2G2C:EN6M:VE44:IAB3:IZ3E:TMMY:VZJB:ZJRK
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

Additional environment details (AWS ECS, Azure ACI, local, etc.):
Docker Desktop 3.5.2 (66501) running in WSL2 engine.

[ndeloof]

compose-go support for extends uses mergo to apply overrides: https://github.com/compose-spec/compose-go/blob/e58fef05b0b00e98038926f1a9c4139876b37760/loader/loader.go#L500
But for volumes, this need to be revisited as this results into appending the override to the volume list, with the same target path (a map[target]config would have been a better structure, but ... ¯\(°_o)/¯).
Need some special management for this scenario

[ndeloof]

internally tracked as https://docker.atlassian.net/browse/IL-416

[thaJeztah]

(a map[target]config would have been a better structure

Maps have their own disadvantages as well; if "stability" is important (for docker stack / docker services this was important), maps would cause golang's "order randomisation"); also it would cause the destination path to be either duplicated (once as key, once as value), or more difficult to deal with (the struct itself would not contain all the information).

Before I forget (noticed this ticket); be sure to take (env)variables will have to be taken into account when checking for duplicates (e.g. $HOME:$HOME)

[fbartels]

This issue seems to not only concern cases where "extends" is used, but also appears when volume definitions are spread over multiple files.

$ docker-compose --version
Docker Compose version 2.0.1
$ docker version
Client: Docker Engine - Community
 Version:           20.10.9
 API version:       1.41
 Go version:        go1.16.8
 Git commit:        c2ea9bc
 Built:             Mon Oct  4 16:08:29 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.9
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.8
  Git commit:       79ea9d3
  Built:            Mon Oct  4 16:06:37 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.11
  GitCommit:        5b46e404f6b9f661a205e28d59c982d3634148f8
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.6.3-docker)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 131
 Server Version: 20.10.9
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 5b46e404f6b9f661a205e28d59c982d3634148f8
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-88-generic
 Operating System: Ubuntu 20.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 62.75GiB
 Name: clifford
 ID: FI2X:VOQH:AEUH:H6Z6:XH2B:CRDS:GAAX:JKE7:5QCO:SXY2:CB6P:JRYD
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: fbartels
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

I have two compose files. One named docker-compose.yml and one docker-compose.volume.yml and after upgrading docker-compose this now throws the following error:

$ docker-compose -f test/docker-compose.yml -f test/docker-compose.volume.yml ps
service "my-container" refers to undefined volume my-volume/: invalid compose project

These volumes are defined in the second compose file, e.g. like this:

version: "3.5"

services:
  my-container:
    volumes:
      - my-volume/:/storage

volumes:
  my-volume:

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it had not recent activity during the stale period.