Docker swarm unable to mount Ceph volume in alternate pool

[jdeglopper]

Summary

We are using the rexray/rbd plugin to mount Ceph RBD volumes in a Docker swarm. This seems to work fine for volumes on the default rbd pool, but I'm getting Docker errors mounting volumes from an alternate pool.

Bug Reports

Version

ID                  NAME                DESCRIPTION            ENABLED
322b0a4a8add        rexray/rbd:latest   REX-Ray for Ceph RBD   true

docker plugin inspect 322b0a4a8add 
[...]
"Enabled": true,
        "Id": "322b0a4a8add785edce78fce10e8a37c3a912b24ec0c2584fac0b6bca446bf77",
        "Name": "rexray/rbd:latest",
        "PluginReference": "rexray/rbd:latest",
[...]

[root@nodeA2 web]# docker version
Client:
 Version:         1.13.1
 API version:     1.26
 Package version: docker-1.13.1-63.git94f4240.el7.centos.x86_64
 Go version:      go1.9.4
 Git commit:      94f4240/1.13.1
 Built:           Fri May 18 15:44:33 2018
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: docker-1.13.1-63.git94f4240.el7.centos.x86_64
 Go version:      go1.9.4
 Git commit:      94f4240/1.13.1
 Built:           Fri May 18 15:44:33 2018
 OS/Arch:         linux/amd64
 Experimental:    true

Ceph mimic in Docker on CentOS 7.5 with a ml-stable 4.18 kernel

[root@nodeA2 web]# uname -a
Linux nodeA2 4.18.3-1.el7.elrepo.x86_64 #1 SMP Sat Aug 18 09:30:18 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@nodeA2 web]# cat /etc/redhat-release 
CentOS Linux release 7.5.1804 (Core) 

Expected Behavior

Docker swarm mounts the existing RBD volume by name, defined as an external volume in the docker-compose file

Actual Behavior

Docker produces a mount error:

"Err": "VolumeDriver.Mount: docker-legacy: Mount: mysqltest: failed: resource not found",

Steps To Reproduce

Create (and format, not shown) a docker volume in an alternate pool:

[root@nodeC1 ~]# docker volume create --driver=rexray/rbd --name=nvme.mysqltest --opt=size=16
nvme.mysqltest
[root@nodeC1 ~]# docker volume inspect nvme.mysqltest
[
    {
        "Driver": "rexray/rbd:latest",
        "Labels": {},
        "Mountpoint": "/var/lib/docker/plugins/8a488638419276eaedb75a7c17eb74f07eebec3060a20667e2579687bfeb9d9f/rootfs",
        "Name": "mysqltest",
        "Options": {
            "size": "16"
        },
        "Scope": "global",
        "Status": {
            "availabilityZone": "",
            "fields": null,
            "iops": 0,
            "name": "mysqltest",
            "server": "rbd",
            "service": "rbd",
            "size": 16,
            "type": "nvme"
        }
    }
]

[root@nodeC1 ~]# rbd ls nvme
mysqlTB
mysqltest
[root@nodeC1 ~]# rbd info nvme/mysqltest
rbd image 'mysqltest':
	size 16 GiB in 4096 objects
	order 22 (4 MiB objects)
	id: 28eed42ae8944a
	block_name_prefix: rbd_data.28eed42ae8944a
	format: 2
	features: layering
	op_features: 
	flags: 
	create_timestamp: Fri Aug 24 14:39:42 2018

Define a Docker swarm service which uses this volume:

version: '3'

services:

  web:
    image: nginx:latest
    ports:
      - "90:80"
    volumes:
       - nvme.mysqltest:/code
       - /var/data/web/site.conf:/etc/nginx/conf.d/site.conf
    deploy:
      mode: replicated
      replicas: 1
    networks:
      - internal

networks:
  internal:
     driver: overlay
     ipam:
       driver: default
       config:
       - subnet: 10.1.3.0/24

volumes:
  nvme.mysqltest:
    external: true

Deploy the docker service:

[root@nodeC1 webtest]# docker stack deploy webtest2 -c test.yml 
Creating network webtest2_internal
Creating service webtest2_web
[root@nodeC1 webtest]# docker stack ps webtest2
ID            NAME                IMAGE         NODE    DESIRED STATE  CURRENT STATE         ERROR                             PORTS
qahs94a58rzi  webtest2_web.1      nginx:latest  nodeA2  Ready          Ready 3 seconds ago                                     
a062tcho5fyn   \_ webtest2_web.1  nginx:latest  nodeA2  Shutdown       Failed 3 seconds ago  "starting container failed: er…"  

Inspect a failed container and note the error message:

[root@nodeC1 webtest]# docker inspect qahs94a58rzi
[...]
"Mounts": [
                    {
                        "Type": "volume",
                        "Source": "nvme.mysqltest",
                        "Target": "/code"
                    },
[...]

"Status": {
            "Timestamp": "2018-08-24T19:08:51.7424901Z",
            "State": "failed",
            "Message": "starting",
            "Err": "starting container failed: error while mounting volume '': VolumeDriver.Mount: docker-legacy: Mount: mysqltest: failed: resource not found",
            "ContainerStatus": {
                "ContainerID": "84c3ab361f40db15c0b04fff5bac3215f893923eebdd38100666f16b1a053a02",
                "ExitCode": 128
            },
[...]

[jdeglopper]

Possibly relevant log entries:

Aug 24 16:46:15 nodeA2 dockerd-current: time="2018-08-24T16:46:15-04:00" level=info msg="time=\"2018-08-24T20:46:15Z\" level=error msg=\"resource not found\" host=\"unix:///var/run/rexray/947717413.sock\" instanceID=\"rbd=10.7.102.12\" resourceID=mysqlTB route=volumeInspect server=peppermint-hound-lk service=rbd storageDriver=rbd task=0 time=1535143575108 tls=false txCR=1535143575 txID=a3f2db1a-a950-40a0-57ba-3906ec71fd82 " plugin=322b0a4a8add785edce78fce10e8a37c3a912b24ec0c2584fac0b6bca446bf77
Aug 24 16:46:15 nodeA2 dockerd-current: time="2018-08-24T16:46:15-04:00" level=info msg="time=\"2018-08-24T20:46:15Z\" level=error msg=\"error: api call failed\" error.resourceID=mysqlTB host=\"unix:///var/run/rexray/947717413.sock\" route=volumeInspect server=peppermint-hound-lk time=1535143575108 tls=false txCR=1535143575 txID=a3f2db1a-a950-40a0-57ba-3906ec71fd82 " plugin=322b0a4a8add785edce78fce10e8a37c3a912b24ec0c2584fac0b6bca446bf77
Aug 24 16:46:15 nodeA2 dockerd-current: time="2018-08-24T16:46:15-04:00" level=info msg="time=\"2018-08-24T20:46:15Z\" level=error msg=\"docker-legacy: Mount: mysqlTB: failed: resource not found\" host=\"unix:///var/run/rexray/947717413.sock\" integrationDriver=linux osDriver=linux service=rbd storageDriver=libstorage time=1535143575108 " plugin=322b0a4a8add785edce78fce10e8a37c3a912b24ec0c2584fac0b6bca446bf77
Aug 24 16:46:15 nodeA2 dockerd-current: time="2018-08-24T16:46:15.121777485-04:00" level=error msg="Handler for POST /v1.22/containers/create returned error: VolumeDriver.Mount: docker-legacy: Mount: mysqlTB: failed: resource not found"
Aug 24 16:48:45 nodeA2 dockerd-current: time="2018-08-24T16:48:45-04:00" level=info msg="time=\"2018-08-24T20:48:45Z\" level=error msg=\"docker-legacy: Get: nvme/mysqlTB: failed: http error\" host=\"unix:///var/run/rexray/947717413.sock\" integrationDriver=linux osDriver=linux service=rbd storageDriver=libstorage time=1535143725182 " plugin=322b0a4a8add785edce78fce10e8a37c3a912b24ec0c2584fac0b6bca446bf77
Aug 24 16:48:45 nodeA2 dockerd-current: time="2018-08-24T16:48:45.183008440-04:00" level=error msg="Handler for GET /v1.26/volumes/nvme/mysqlTB returned error: get nvme/mysqlTB: no such volume"

[jdeglopper]

In addition, the volume mount fails from a command-line docker run, without involving swarm or compose. ```
[root@nodeA2 web]# docker run -d --name web_mysql -v nvme.mysqlTB:/var/lib/mysql --network webnet mysql:5.7
/usr/bin/docker-current: Error response from daemon: VolumeDriver.Mount: docker-legacy: Mount: mysqlTB: failed: resource not found.
See '/usr/bin/docker-current run --help'.
[root@nodeA2 web]# docker volume inspect nvme.mysqlTB
[
{
"Driver": "rexray/rbd:latest",
"Labels": null,
"Mountpoint": "/var/lib/docker/plugins/322b0a4a8add785edce78fce10e8a37c3a912b24ec0c2584fac0b6bca446bf77/rootfs",
"Name": "mysqlTB",
"Options": {},
"Scope": "global",
"Status": {
"availabilityZone": "",
"fields": null,
"iops": 0,
"name": "mysqlTB",
"server": "rbd",
"service": "rbd",
"size": 360,
"type": "nvme"
}
}
]

[rodneygp]

Any news on this?