Issue when install this plugin: fatal error: rados/librados.h Centos 7.2

[phamthanhnhan14]

Hi ,
I try to install this plugin with command:
go get github.com/yp-engineering/rbd-docker-plugin
# github.com/noahdesu/go-ceph/rados
src/github.com/noahdesu/go-ceph/rados/conn.go:5:29: fatal error: rados/librados.h: No such file or directory
// #include <rados/librados.h>
^
compilation terminated.

I have googled it but can't find a good resovle for this case.
Please take it easy.
Thanks,
Nhan Pham T.

[rmartinez3]

make sure to have librados2-devel and librbd1-devel

if not follow this link to see how to install it
http://docs.ceph.com/docs/hammer/rados/api/librados-intro/

this should fix the issue. It fix the issue for me

however now im getting a different error when doing a go get

# github.com/yp-engineering/rbd-docker-plugin 2 src/github.com/yp-engineering/rbd-docker-plugin/main.go:83: cannot use d (type cephRBDVolumeDriver) as type "github.com/docker/go-plugins-helpers/volume".Driver in argument to "github.com/docker/go-pl ugins-helpers/volume".NewHa: 3 cephRBDVolumeDriver does not implement "github.com/docker/go-plugins-helpers/volume".Driver (missing Get method)

[mrapczynski]

I'm getting the same error also on Centos 7.2, but this time using the release candidate for Ceph 10.x (Jewel).

[rmartinez3]

The problem here is more of a missing function dependency for the docker plugin. docker version +9 will not work with this plugin since need to implement the method that is missing . This issue has been noted by #23.

I think there are pulll requests waiting to merge to fix this problem

#24

[mrapczynski]

@rmartinez3 Great that explains a lot because I am on 1.10 as well. Thanks!

[hekaldama]

Try latest and see if that fixes you. If not, reopen with details. Thanks!

[mrapczynski]

@hekaldama OK build works as expected. I'm a first time user of the plugin so getting a strange error, and not sure where to go with it. When I checked Ceph afterwards, my RBD image test1 did get created.

I tried creating a test container as follows:

docker run -i -t --volume-driver=rbd -v test1:/mnt/test1 centos:6.7 bash

... and got the following back after 20 seconds or so ...

docker: Error response from daemon:
Post http://%2Frun%2Fdocker%2Fplugins%2Frbd.sock/VolumeDriver.Mount: http: ContentLength=17 with Body length 0.

The following was reported by the plugin in the logs:

rbd-volume-plugin: 2016/04/13 11:58:16 driver.go:440: INFO: Path request(test1) => /var/lib/docker-volumes/rbd/docker/test1
rbd-volume-plugin: 2016/04/13 11:58:16 driver.go:440: INFO: Path request(test1) => /var/lib/docker-volumes/rbd/docker/test1
rbd-volume-plugin: 2016/04/13 11:58:16 driver.go:440: INFO: Path request(test1) => /var/lib/docker-volumes/rbd/docker/test1
rbd-volume-plugin: 2016/04/13 11:58:16 driver.go:440: INFO: Path request(test1) => /var/lib/docker-volumes/rbd/docker/test1
rbd-volume-plugin: 2016/04/13 11:58:16 driver.go:440: INFO: Path request(test1) => /var/lib/docker-volumes/rbd/docker/test1
rbd-volume-plugin: 2016/04/13 11:58:16 driver.go:269: INFO: Mount(test1)
rbd-volume-plugin: 2016/04/13 11:58:16 driver.go:681: INFO: rbdImageExists(docker/test1)
rbd-volume-plugin: 2016/04/13 11:58:16 driver.go:289: WARN: Image does not exist: test1
rbd-volume-plugin: 2016/04/13 11:58:16 driver.go:135: INFO: createImage(test1)
rbd-volume-plugin: 2016/04/13 11:58:16 driver.go:681: INFO: rbdImageExists(docker/test1)
rbd-volume-plugin: 2016/04/13 11:58:16 driver.go:705: INFO: Attempting to create RBD Image: (%s/%s, %s, %s) docker test1 256000 xfs
rbd-volume-plugin: 2016/04/13 11:58:16 driver.go:818: INFO: lockImage(docker/test1)
rbd-volume-plugin: 2016/04/13 11:58:17 main.go:70: INFO: Setting up Ceph Driver for PluginID=rbd, cluster=, user=admin, pool=docker, mount=/var/lib/docker-volumes, config=
rbd-volume-plugin: 2016/04/13 11:58:17 driver.go:83: INFO: newCephRBDVolumeDriver: setting base mount dir=/var/lib/docker-volumes/rbd
rbd-volume-plugin: 2016/04/13 11:58:17 driver.go:554: INFO: connecting to Ceph and default pool context
rbd-volume-plugin: 2016/04/13 11:58:17 main.go:82: INFO: Creating Docker VolumeDriver Handler
rbd-volume-plugin: 2016/04/13 11:58:17 main.go:86: INFO: Opening Socket for Docker to connect: /run/docker/plugins/rbd.sock
rbd-volume-plugin: 2016/04/13 11:58:32 driver.go:459: INFO: Unmount(test1)
rbd-volume-plugin: 2016/04/13 11:58:32 driver.go:477: WARN: Volume is not in known mounts: will attempt limited Unmount: test1
rbd-volume-plugin: 2016/04/13 11:58:32 driver.go:492: ERROR: unmounting device(/dev/rbd/docker/test1): exit status 32
rbd-volume-plugin: 2016/04/13 11:58:32 driver.go:500: ERROR: unmapping image device(/dev/rbd/docker/test1): exit status 22
rbd-volume-plugin: 2016/04/13 11:58:32 driver.go:861: INFO: unlockImage(docker/test1, dockertest.ad.fhda.edu)
rbd-volume-plugin: 2016/04/13 11:58:32 driver.go:508: ERROR: unlocking RBD image(test1): RBD image not found
rbd-volume-plugin: 2016/04/13 11:58:32 driver.go:440: INFO: Path request(test1) => /var/lib/docker-volumes/rbd/docker/test1
rbd-volume-plugin: 2016/04/13 11:58:32 driver.go:383: INFO: List request => []
rbd-volume-plugin: 2016/04/13 11:58:33 driver.go:440: INFO: Path request(test1) => /var/lib/docker-volumes/rbd/docker/test1
rbd-volume-plugin: 2016/04/13 11:58:33 driver.go:440: INFO: Path request(test1) => /var/lib/docker-volumes/rbd/docker/test1
rbd-volume-plugin: 2016/04/13 11:58:33 driver.go:440: INFO: Path request(test1) => /var/lib/docker-volumes/rbd/docker/test1

[hekaldama]

I am unsure what is going on at this time. I will continue to look at this, but I recommend using the supported versions of ceph + kernel / distro described:

http://docs.ceph.com/docs/master/start/os-recommendations/

and pairing that with docker 1.10 or newer. That is what I am going to be testing now.

[hekaldama]

Tested to be working with:

$ cat /etc/redhat-release && uname -r && docker --version && go version && ceph --version
CentOS Linux release 7.2.1511 (Core) 
3.10.0-327.3.1.el7.x86_64
Docker version 1.10.2, build c3959b1
go version go1.6.1 linux/amd64
ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3)

running micro-osd.sh like:

$ micro-osd.sh tmp

and starting rbd-docker-plugin with:

# rbd-docker-plugin -create=true -config=tmp/ceph.conf

and then ran your example of:

$ docker run -i -t --volume-driver=rbd -v test1:/mnt/test1 centos:6.7 bash

which ran without error.

[hekaldama]

Will try with different kernel tomorrow.

[mrapczynski]

@hekaldama OK thanks. My OS and Docker version are the same except that I'm running a release candidate of Jewel for Ceph, 10.1.1 - so that could be the issue. Thus far, this new version of Ceph has been flawless.

I can mount and use the same RBD images right on the host without issues - so whatever is breaking seems to be limited to the automation provided by the plugin. Also, it may be useful to know that the plugin leaves an image locked after it gives up and quits. I've had to use rbd to manually unlock it each time.