volume-subpath does not work for docker service

[yy8402]

Description

with the latest v26.0.0, volume-subpath does not work for docker service.

When creating a docker service, volume-subpath can be properly set, but the corresponding docker container cannot make use of it.

Reproduce

1. create a docker volume
2. use docker run to start nginx container using the volume, get into the volume folder, make sub-folder and create a file
3. delete nginx container
4. use docker run to start nginx container using the volume with subpath, verify subpath setting is effective
5. delete nginx container
6. use docker service create to start nginx service using the volume with subpath
7. check the corresponding container for the nginx service, verify subpath setting is not effective

please check the attached stdout for detailed info.
docker_subpath.txt

supra@test140:~/test.subpath$ sudo docker volume create test123
test123
supra@test140:~/test.subpath$ sudo docker run -d   --name nginx-subpath   --mount type=volume,source=test123,destination=/supra/www  nginx
5828c2d578b3cd9ee6c84abc048333eaa066f47c6d554c98837ac171560c7c4f
supra@test140:~/test.subpath$ sudo docker exec -it nginx-subpath bash
root@5828c2d578b3:/# mkdir /supra/www/subpath
root@5828c2d578b3:/# echo "hello, world!" > /supra/www/subpath/hello
root@5828c2d578b3:/# exit
exit
supra@test140:~/test.subpath$ sudo docker exec -it nginx-subpath cat /supra/www/subpath/hello
hello, world!
supra@test140:~/test.subpath$ sudo docker rm nginx-subpath --force
nginx-subpath
supra@test140:~/test.subpath$ sudo docker run -d   --name nginx-subpath   --mount type=volume,source=test123,destination=/supra/www,volume-subpath=s
ubpath  nginx
fba1b8e309900211be8b3b8381fd81bdf6c13ccd5553aa0394e179e999ac47b8
supra@test140:~/test.subpath$ sudo docker exec -it nginx-subpath cat /supra/www/hello
hello, world!
supra@test140:~/test.subpath$ sudo docker rm nginx-subpath --force
nginx-subpath
supra@test140:~/test.subpath$ sudo docker service create   --name nginx-subpath   --mount type=volume,source=test123,destination=/supra/www,volume-o
pt=volume-subpath=subpath  nginx
re8616r9smrg65gc8y3wnibc5
overall progress: 1 out of 1 tasks 
1/1: running   [==================================================>] 
verify: Service re8616r9smrg65gc8y3wnibc5 converged 
supra@test140:~/test.subpath$ sudo docker ps | grep nginx-subpath 
2b364c693bd5   nginx:latest   "/docker-entrypoint.…"   51 seconds ago      Up 50 seconds      80/tcp                                          nginx-
subpath.1.qti4x2n1mm3f6dchkz45jou9z
supra@test140:~/test.subpath$ sudo docker exec 2b364c693bd5 ls /supra/www
subpath
supra@test140:~/test.subpath$ sudo docker inspect 2b364c693bd5 
...
        "HostConfig": {
            "Mounts": [
                {
                    "Type": "volume",
                    "Source": "test123",
                    "Target": "/supra/www",
                    "VolumeOptions": {
                        "DriverConfig": {
                            "Options": {
                                "volume-subpath": "subpath"
                            }
                        }
                    }
                }
            ],
...
        }
        ...
        "Mounts": [
            {
                "Type": "volume",
                "Name": "test123",
                "Source": "/var/lib/docker/volumes/test123/_data",
                "Destination": "/supra/www",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            }
        ],
...
supra@test140:~/test.subpath$ 

Expected behavior

volume-subpath works for docker service that subpath setting is effective for the corresponding container

docker version

Client: Docker Engine - Community
 Version:           26.0.0
 API version:       1.45
 Go version:        go1.21.8
 Git commit:        2ae903e
 Built:             Wed Mar 20 15:17:48 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.0.0
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.8
  Git commit:       8b79278
  Built:            Wed Mar 20 15:17:48 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    26.0.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.25.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 344
 Server Version: 26.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: active
  NodeID: sdhyrptnzk3m6bfx0204donns
  Is Manager: true
  ClusterID: cfch8g5foc82sviq50t73pp26
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8  
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 192.168.0.140
  Manager Addresses:
   192.168.0.140:2377
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-91-generic
 Operating System: Ubuntu 22.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.61GiB
 Name: test140
 ID: a74e44ab-98a1-410e-99d6-f7a8a9ccddef
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  SupraRegistry:5000
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

No response

[derchr]

I can confirm the problem. The new subpath option is somehow not picked up, even though docker compose properly recognizes the subpath option.