How to use rexray/csi-nfs with docker plugin ?

[keyolk]

I'm tyring to use rexray for my docker and NFS server.
I found this page, https://rexray.readthedocs.io/en/stable/user-guide/schedulers/docker/plug-ins/csi-nfs/ .
But it is not clear to me : (
Anyone can inform me how to use it by docker plugin ?

[clintkitson]

Hello @keyolk,

You're right I think we're definitely missing some examples. This is the plugin page https://github.com/thecodeteam/csi-nfs

@codenrhoden Do you have any examples that you can drop in here for now?

[briantopping]

I'm pretty sure that the following commands might work on the client:

docker plugin install rexray/csi-nfs
docker volume create --driver rexray/csi-nfs --name test
docker volume ls
docker run -v test:/data busybox mount | grep "/data"

But I can't figure out how to set the address of the server and directory? The last command fails with the following:

docker: Error response from daemon: error while mounting volume '/var/lib/docker/plugins/1a4940d2e0ab04a9363bd36b69c4018c555ad88c25be7d2525eb3896dfa900f1/rootfs': VolumeDriver.Mount: NodePublishVolume failed: 6: mount failed: exit status 1
mounting arguments: -t nfs : /dev/csi-nfs-mounts/:
output: mount: bad address ''.
ERRO[0000] error getting events from daemon: net/http: request canceled 

After that, something at https://rexray.readthedocs.io/en/stable/user-guide/schedulers/ should help?

[briantopping]

Thanks, I saw those, but it seems they are only used for parameterizing the call to the imported Mount in the test, no?

https://github.com/thecodeteam/csi-nfs/blob/master/nfs/nfs_test.go#L28

I think it has something to do with the li parameter passed in here: https://github.com/thecodeteam/csi-nfs/blob/master/provider/provider.go#L162-L163. If I understand well enough, this method is implementing the SPI for CSI.

[clintkitson]

Sorry, deleted that comment. I believe it is just looking for a host and export separated by a dash. For example, nfshost-volume1 would equal nfshhost://volume1

[clintkitson]

Can you also try this? docker volume create -d rexray test2 -o host=192.168.75.2 -o export=/data

[briantopping]

That does seem to cause the mount to work when the command is of the form docker volume create -d rexray/csi-nfs test2 -o host=192.168.75.2 -o export=/data (note the additional discrimination on the driver name). Now I need to get it wired into the container...

[briantopping]

Ok, it's close. Here's what I am seeing:

1. In my case, the command is docker volume create -d rexray/csi-nfs test5 -o host=10.10.0.10 -o export=/mnt/tank/dcos/grafana
2. When I run a container with docker run -v test5:/data busybox ls "/data" the output of the docker run is the following:

[root@public01 ~]# docker run -v test5:/data busybox ls "/data"
docker: Error response from daemon: error while mounting volume '/var/lib/docker/plugins/1a4940d2e0ab04a9363bd36b69c4018c555ad88c25be7d2525eb3896dfa900f1/rootfs': VolumeDriver.Mount: NodePublishVolume failed: 6: mount failed: exit status 255
mounting arguments: -o bind /dev/csi-nfs-mounts/10.10.0.10:#mnt#tank#dcos#grafana /var/lib/rexray/docker/volumes/test5
output: mount: mounting /dev/csi-nfs-mounts/10.10.0.10: on /var/lib/rexray/docker/volumes/test5 failed: No such file or directory.
ERRO[0000] error getting events from daemon: net/http: request canceled 

My belief is that the test5 token keys the volume defined in the docker volume create and the :/data after it mounts the test5 volume at /data on the docker container.

I think what's happening is Docker is parsing the mounting arguments and dropping the #mnt#tank#dcos#grafana when in fact the directory that the share is mounted on is literally 10.10.0.10:#mnt#tank#dcos#grafana:

[root@public01 ~]# mount | grep /dcos
10.10.0.10:/mnt/tank/dcos/grafana on /dev/csi-nfs-mounts/10.10.0.10:#mnt#tank#dcos#grafana type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.10,mountvers=3,mountproto=tcp,local_lock=none,addr=10.10.0.10)
[root@public01 ~]# ls -l /dev/csi-nfs-mounts/
total 1
drwxr-xr-x 2 root       root       40 Nov 19 12:43 :
drwxrwxrwx 2 prometheus prometheus  3 Nov 19 14:05 10.10.0.10:#mnt#tank#dcos#grafana

My versions:

[root@public01 ~]# docker version
Client:
 Version:      17.05.0-ce
 API version:  1.29
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:06:25 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.05.0-ce
 API version:  1.29 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:06:25 2017
 OS/Arch:      linux/amd64
 Experimental: false
[root@public01 ~]# rexray version
REX-Ray
-------
Binary: /usr/bin/rexray
Flavor: client+agent+controller
SemVer: 0.11.0
OsArch: Linux-x86_64
Commit: 508023f4b0e4974f1844c26b39af7c4219f4c7a9
Formed: Mon, 16 Oct 2017 01:07:06 MDT

Not sure if any of this word jumble is making sense, I may have to RTFM more. This is really super helpful though, thanks @clintkitson!

[clintkitson]

@briantopping Can you try an export path that is just one level deep? The example I had comes from testing that occurred so I'm wondering if its related to the path structure submitted when you create the volume.

[briantopping]

Thanks @clintkitson I can do that but need to create a new server for it. My NAS doesn't allow shares that are not in a data volume mount point, which is at least two directories deep.

Is there any way you could try the reverse, to quickly mount a deep directory? Then we could converge with results.

[briantopping]

Indeed, same problem:

[root@public01 ~]# mount -v dev.coglative.com:/data /mnt
[root@public01 ~]# ls /mnt
test
[root@public01 ~]# umount /mnt
[root@public01 ~]# docker volume create -d rexray/csi-nfs data -o host=dev.example.com -o export=/data
data
[root@public01 ~]# docker run -v data:/data busybox
docker: Error response from daemon: error while mounting volume '/var/lib/docker/plugins/1a4940d2e0ab04a9363bd36b69c4018c555ad88c25be7d2525eb3896dfa900f1/rootfs': VolumeDriver.Mount: NodePublishVolume failed: 6: mount failed: exit status 255
mounting arguments: -o bind /dev/csi-nfs-mounts/dev.example.com:#data /var/lib/rexray/docker/volumes/data
output: mount: mounting /dev/csi-nfs-mounts/dev.example.com: on /var/lib/rexray/docker/volumes/data failed: No such file or directory.
ERRO[0000] error getting events from daemon: net/http: request canceled 
[root@public01 ~]# mount
<snip/>
dev.example.com:/data on /dev/csi-nfs-mounts/dev.example.com:#data type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.10.1.39,local_lock=none,addr=x.x.x.15)

[clintkitson]

Does /var/lib/rexray/docker/volumes exist?

…

On Mon, Nov 20, 2017 at 7:58 PM Brian Topping ***@***.***> wrote: Indeed, same problem: ***@***.*** ~]# mount -v dev.coglative.com:/data /mnt ***@***.*** ~]# ls /mnt test ***@***.*** ~]# umount /mnt ***@***.*** ~]# docker volume create -d rexray/csi-nfs data -o host=dev.example.com -o export=/data data ***@***.*** ~]# docker run -v data:/data busybox docker: Error response from daemon: error while mounting volume '/var/lib/docker/plugins/1a4940d2e0ab04a9363bd36b69c4018c555ad88c25be7d2525eb3896dfa900f1/rootfs': VolumeDriver.Mount: NodePublishVolume failed: 6: mount failed: exit status 255 mounting arguments: -o bind /dev/csi-nfs-mounts/dev.example.com:#data /var/lib/rexray/docker/volumes/data output: mount: mounting /dev/csi-nfs-mounts/dev.example.com: on /var/lib/rexray/docker/volumes/data failed: No such file or directory. ERRO[0000] error getting events from daemon: net/http: request canceled ***@***.*** ~]# mount ... dev.example.com:/data on /dev/csi-nfs-mounts/dev.example.com:#data type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.10.1.39,local_lock=none,addr=x.x.x.15) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1114 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABVMMdMYnjbstIoT5RFZKnzKaJUbuetqks5s4kqAgaJpZM4Qhbbs> .

[briantopping]

[root@public01 ~]# ls -alt /var/lib/rexray/docker/
total 0
drwxr-xr-x. 5 root root 46 Nov  6 14:31 ..
drwxr-xr-x  3 root root 21 Nov  6 11:56 .
drwxr-xr-x  2 root root  6 Nov  6 11:56 volumes
[root@public01 ~]# df /var/lib/rexray/docker/volumes
Filesystem              1K-blocks    Used Available Use% Mounted on
/dev/mapper/centos-root  17811456 3949020  13862436  23% /

[clintkitson]

And the data directory? It is trying do a bind mount to that path if you reference the error.

…

On Mon, Nov 20, 2017 at 8:02 PM Brian Topping ***@***.***> wrote: ***@***.*** ~]# ls -alf /var/lib/rexray/docker/volumes . .. ***@***.*** ~]# df !$ df /var/lib/rexray/docker/volumes Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/centos-root 17811456 3949020 13862436 23% / — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1114 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABVMMfJGT9oLHYCss0gm_u-og58x3hTcks5s4ktjgaJpZM4Qhbbs> .

[briantopping]

And the data directory? It is trying do a bind mount to that path if you reference the error.

Ah good point. /var/lib/rexray/docker/volumes/data does not exist, the directory is mounted at /dev/csi-nfs-mounts/dev.example.com:#data, which is what I guess I meant by "same problem".

This is some of the first bind mounting I am doing, do I actually have to create directories for this stuff? I would have thought Docker does that automatically?

[clintkitson]

RR and the plugin should be doing this for you so very surprised you are hitting this error. Trying to troubleshoot based on what you're error is saying. What is the current RR config?

…

On Mon, Nov 20, 2017 at 8:09 PM Brian Topping ***@***.***> wrote: And the data directory? It is trying do a bind mount to that path if you reference the error. Ah good point. /var/lib/rexray/docker/volumes/data does not exist, the directory is mounted at /dev/csi-nfs-mounts/dev.example.com:#data, which is what I guess I meant by "same problem". This is some of the first bind mounting I am doing, do I actually have to create directories for this stuff? I would have thought Docker does that automatically? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1114 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABVMMdoBIskdnuurWlZsLTI5j5MTyFhXks5s4kzjgaJpZM4Qhbbs> .

[briantopping]

Thanks! Is this easier on Slack?

[root@public01 ~]# cat /etc/rexray/config.yml 
rexray:
  loglevel: info
  modules:
    default-docker:
      disabled: true
  service: vfs

Behavior is the same regardless of whether default-docker.disabled is true or false, commented or uncommented.

[clintkitson]

Want to capture this for others as well. What command are you using and what relevant env variables are set?

…

On Mon, Nov 20, 2017 at 8:14 PM Brian Topping ***@***.***> wrote: Thanks! Is this easier on Slack? ***@***.*** ~]# cat /etc/rexray/config.yml rexray: # loglevel: debug modules: default-docker: # disabled: true Behavior is the same regardless of whether default-docker.disabled: true is commented or uncommented. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1114 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABVMMaVn4ihqS0a8L4NEUuGUVHCTZDNOks5s4k47gaJpZM4Qhbbs> .

[briantopping]

Sure thing. No environment variables visible in export that are beyond standard profile:

[root@public01 ~]# export
declare -x HISTCONTROL="ignoredups"
declare -x HISTSIZE="1000"
declare -x HOME="/root"
declare -x HOSTNAME="public01.mc1.example.com"
declare -x LANG="en_US.UTF-8"
declare -x LESSOPEN="||/usr/bin/lesspipe.sh %s"
declare -x LOGNAME="root"
declare -x LS_COLORS="rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:"
declare -x MAIL="/var/spool/mail/root"
declare -x OLDPWD
declare -x PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/mesosphere/bin/dcos-path:/root/bin"
declare -x PWD="/root"
declare -x SHELL="/bin/bash"
declare -x SHLVL="1"
declare -x SSH_CLIENT="10.9.254.130 58822 22"
declare -x SSH_CONNECTION="10.9.254.130 58822 10.10.1.39 22"
declare -x SSH_TTY="/dev/pts/0"
declare -x TERM="xterm-256color"
declare -x USER="root"
declare -x XDG_RUNTIME_DIR="/run/user/0"
declare -x XDG_SESSION_ID="49"

Command to create volume and start docker container is:

docker volume create -d rexray/csi-nfs data -o host=dev.example.com -o export=/data
docker run -v data:/data busybox`

First command is fine, second as in output above.

[clintkitson]

What was the command to start RR?

…

On Mon, Nov 20, 2017 at 8:23 PM Brian Topping ***@***.***> wrote: Sure thing. No environment variables visible in export that are beyond standard profile: ***@***.*** ~]# export declare -x HISTCONTROL="ignoredups" declare -x HISTSIZE="1000" declare -x HOME="/root" declare -x HOSTNAME="public01.mc1.example.com" declare -x LANG="en_US.UTF-8" declare -x LESSOPEN="||/usr/bin/lesspipe.sh %s" declare -x LOGNAME="root" declare -x LS_COLORS="rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:" declare -x MAIL="/var/spool/mail/root" declare -x OLDPWD declare -x PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/mesosphere/bin/dcos-path:/root/bin" declare -x PWD="/root" declare -x SHELL="/bin/bash" declare -x SHLVL="1" declare -x SSH_CLIENT="10.9.254.130 58822 22" declare -x SSH_CONNECTION="10.9.254.130 58822 10.10.1.39 22" declare -x SSH_TTY="/dev/pts/0" declare -x TERM="xterm-256color" declare -x USER="root" declare -x XDG_RUNTIME_DIR="/run/user/0" declare -x XDG_SESSION_ID="49" Command to create volume and start docker container is: docker volume create -d rexray/csi-nfs data -o host=dev.example.com -o export=/data docker run -v data:/data busybox` First command is fine, second as in output above. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1114 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABVMMcR6R-xOnUiMVZdlemQ7n3BGqTQcks5s4lBdgaJpZM4Qhbbs> .

[briantopping]

[root@public01 ~]# cat /etc/systemd/system/rexray.service
[Unit]
Description=rexray
Wants=scini.service
Before=docker.service
After=scini.service

[Service]
EnvironmentFile=/etc/rexray/rexray.env
ExecStart=/usr/bin/rexray start
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process

[Install]
WantedBy=docker.service

I just noticed the /etc/rexray/rexray.env file, but it is empty.

[root@public01 ~]# systemctl status -l rexray
● rexray.service - rexray
   Loaded: loaded (/etc/systemd/system/rexray.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2017-11-20 21:31:50 MST; 6min ago
 Main PID: 1693 (rexray)
   Memory: 11.9M
   CGroup: /system.slice/rexray.service
           └─1693 /usr/bin/rexray start

Nov 20 21:31:53 public01.mc1.example.com rexray[1693]: REX-Ray
Nov 20 21:31:53 public01.mc1.example.com rexray[1693]: -------
Nov 20 21:31:53 public01.mc1.example.com rexray[1693]: Binary: /usr/bin/rexray
Nov 20 21:31:53 public01.mc1.example.com rexray[1693]: Flavor: client+agent+controller
Nov 20 21:31:53 public01.mc1.example.com rexray[1693]: SemVer: 0.11.0
Nov 20 21:31:53 public01.mc1.example.com rexray[1693]: OsArch: Linux-x86_64
Nov 20 21:31:53 public01.mc1.example.com rexray[1693]: Commit: 508023f4b0e4974f1844c26b39af7c4219f4c7a9
Nov 20 21:31:53 public01.mc1.example.com rexray[1693]: Formed: Mon, 16 Oct 2017 01:07:06 MDT
Nov 20 21:31:53 public01.mc1.example.com rexray[1693]: time="2017-11-20T21:31:53-07:00" level=info msg="http request" host="unix:///var/run/rexray/691450720.sock" route=volumesForService server=quill-crystal-si time=1511238713608 tls=false
Nov 20 21:31:53 public01.mc1.example.com rexray[1693]: time="2017-11-20T21:31:53-07:00" level=info msg="executing task" host="unix:///var/run/rexray/691450720.sock" instanceID="vfs:vfs=public01.mc1.example.com" route=volumesForService server=quill-crystal-si service=vfs storageDriver=vfs task=0 time=1511238713609 tls=false txCR=1511238713 txID=2aff048d-06cb-4a7d-4b2c-97be2f3d8034

[clintkitson]

Yes, but I am seeing you reference it as rexray/csi-nfs preivously? This indicates you also have the managed plugin running simultaneously?

…

On Mon, Nov 20, 2017 at 8:36 PM, Brian Topping ***@***.***> wrote: ***@***.*** ~]# cat /etc/systemd/system/rexray.service [Unit] Description=rexray Wants=scini.service Before=docker.service After=scini.service [Service] EnvironmentFile=/etc/rexray/rexray.env ExecStart=/usr/bin/rexray start ExecReload=/bin/kill -HUP $MAINPID KillMode=process [Install] WantedBy=docker.service I just noticed the /etc/rexray/rexray.env file, but it is empty. ***@***.*** ~]# systemctl status rexray ● rexray.service - rexray Loaded: loaded (/etc/systemd/system/rexray.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2017-11-20 21:31:50 MST; 2s ago Main PID: 1693 (rexray) Memory: 7.3M CGroup: /system.slice/rexray.service └─1693 /usr/bin/rexray start Nov 20 21:31:53 public01.mc1.coglative.com rexray[1693]: time="2017-11-20T21:31:53-07:00" level=info msg="registered custom context key" externalID=Libstorage-Localdevices internalID=4 keyBitmask=2 Nov 20 21:31:53 public01.mc1.coglative.com rexray[1693]: time="2017-11-20T21:31:53-07:00" level=info msg="registered custom context key" externalID=Authorization internalID=5 keyBitmask=2 Nov 20 21:31:53 public01.mc1.coglative.com rexray[1693]: time="2017-11-20T21:31:53-07:00" level=info msg="http request" host="unix:///var/run/rexray/691450720.sock" route=services server=quill-crystal-si time=1511238713065 tls=false Nov 20 21:31:53 public01.mc1.coglative.com rexray[1693]: time="2017-11-20T21:31:53-07:00" level=info msg="initializing supported cache" host="unix:///var/run/rexray/691450720.sock" server=quill-crystal-si service=vfs storage...me=1511238713066 Nov 20 21:31:53 public01.mc1.coglative.com rexray[1693]: time="2017-11-20T21:31:53-07:00" level=info msg="initializing instance ID cache" host="unix:///var/run/rexray/691450720.sock" server=quill-crystal-si service=vfs stora...me=1511238713101 Nov 20 21:31:53 public01.mc1.coglative.com rexray[1693]: time="2017-11-20T21:31:53-07:00" level=info msg="http request" host="unix:///var/run/rexray/691450720.sock" route=serviceInspect server=quill-crystal-si time=1511238713137 tls=false Nov 20 21:31:53 public01.mc1.coglative.com rexray[1693]: time="2017-11-20T21:31:53-07:00" level=info msg="successefully dialed libStorage server" host="unix:///var/run/rexray/691450720.sock" server=quill-crystal-si service=v...me=1511238713139 Nov 20 21:31:53 public01.mc1.coglative.com rexray[1693]: time="2017-11-20T21:31:53-07:00" level=info msg="storage driver initialized" host="unix:///var/run/rexray/691450720.sock" service=vfs storageDriver=libstorage time=1511238713140 Nov 20 21:31:53 public01.mc1.coglative.com rexray[1693]: time="2017-11-20T21:31:53-07:00" level=info msg="os driver initialized" host="unix:///var/run/rexray/691450720.sock" osDriver=linux service=vfs storageDriver=libstorag...me=1511238713156 Nov 20 21:31:53 public01.mc1.coglative.com rexray[1693]: time="2017-11-20T21:31:53-07:00" level=info msg="linux integration driver successfully initialized" host="unix:///var/run/rexray/691450720.sock" integrationDriver=linu...te.default.avail Hint: Some lines were ellipsized, use -l to show in full. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1114 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABVMMZv0GPyiTbRGm0nP3vvMZhE1Y55sks5s4lNWgaJpZM4Qhbbs> .

[briantopping]

Good catch, I hadn't considered that nuance, thanks!

The eventual goal is the docker containers run under dc/os, but for now just testing them at the command line. So I guess I don't need to worry whether rexray.service is running if I am just launching with docker run, correct?

Outside of that, I am not actually launching RR. If I should be, it's not clear how.

[clintkitson]

I just performed a test of this locally on a single host.

docker run --net=host -d --name nfs --privileged -v /data1:/nfsshare -e SHARED_DIRECTORY=/nfsshare itsthenetwork/nfs-server-alpine:latest
X_CSI_NATIVE=true X_CSI_DRIVER=csi-nfs rexray start &
docker volume create -o host=127.0.0.1 -o export=/ -d rexray test2
docker run -ti -v test2:/test2 alpine sh

Filesystem           1K-blocks      Used Available Use% Mounted on
overlay                8065444   1629688   6419372  20% /
tmpfs                    65536         0     65536   0% /dev
tmpfs                  1023420         0   1023420   0% /sys/fs/cgroup
127.0.0.1:/            8065536   1629696   6419456  20% /test2
/dev/xvda1             8065444   1629688   6419372  20% /etc/resolv.conf
/dev/xvda1             8065444   1629688   6419372  20% /etc/hostname
/dev/xvda1             8065444   1629688   6419372  20% /etc/hosts
shm                      65536         0     65536   0% /dev/shm

It is important to mention however that I am experiencing a problem with
the managed plugin, rexray/csi-nfs. As standalone managed via systemctl or
on the cmd line does seem to be working as expected. The error is below
which references a problem with the nfs utilities in the alpine image.

mounting arguments: -t nfs 127.0.0.1:/test1 /dev/csi-nfs-mounts/127.0.0.1:
#test1
output: flock: unrecognized option: e
BusyBox v1.26.2 (2017-06-11 06:38:32 GMT) multi-call binary.

Usage: flock [-sxun] FD|{FILE [-c] PROG ARGS}

[Un]lock file descriptor, or lock FILE, run PROG

-s Shared lock
-x Exclusive lock (default)
-u Unlock FD
-n Fail rather than wait
mount.nfs: requested NFS version or transport protocol is not supported
mount: mounting 127.0.0.1:/test1 on /dev/csi-nfs-mounts/127.0.0.1:#test1
failed: Not supported.

[clintkitson]

If you can, I suggest starting over re Docker. Stopping RR, stopping Docker, rm -rf /var/lib/docker, start Docker. Then start RR and I give you an example of above.

[briantopping]

I shall, thanks, that use case explains quite a bit. Appreciate you staying up late! Will report back with results.

[briantopping]

Sorry for the delay on this on my end, life interruptions. It's top of my stack tho... Happy holidays!

[benfab]

Hi all,

I am getting the "no such file or directory" error when I am installing rexray with the docker plugin command:

docker plugin install rexray/csi-nfs

error message

mounting /dev/csi-nfs-mounts/dev.example.com: on /var/lib/rexray/docker/volumes/data failed: No such file or directory.

Nevertheless, when I am running the rexray driver directly from the shell as below, it works fine.

X_CSI_NATIVE=true X_CSI_DRIVER=csi-nfs rexray start &
docker volume create -o host=<ip> -o export=/vol/data/ -d rexray test5
docker run -ti -v test5:/test5 alpine sh

Is there any extra configuration required when installing Rexray with the docker plugin command?

Thanks!

[clintkitson]

Hello @benfab,

We did identify a problem in this thread re the rexray/csi-nfs managed plugin missing some dependencies which is why you are getting this error. We will work to get this fixed in the upcoming release.

[IAPOLINARIO]

@briantopping @clintkitson Did you guys managed to solve the issue? I'm getting the same errors here.

[clintkitson]

@IAPOLINARIO Looking into it for this next release. There are really two things in this issue 1) lack of example docs 2) the plugin itself doesn't have the right dependencies. Outside of the docker managed plugin it does work with examples listed here.

[clintkitson]

I am assuming the NFS error that is occurring is related to what Alpine supports. This could be remedied with building the csi-nfs managed plugin using Ubuntu as the base instead to accommodate.

For example https://hub.docker.com/r/cpuguy83/nfs-client/~/dockerfile/

[IAPOLINARIO]

@clintkitson Thanks for the quick answer.

Let me see if I understood how to configure RR as a service (using CSI-NFS):

Step 1: install RR curl -sSL https://dl.bintray.com/emccode/rexray/install | sh
Step 2: Create a configuration file at /etc/rexray/config.yml
Step 3: Start RR service with sudo X_CSI_NATIVE=true X_CSI_DRIVER=csi-nfs rexray start
Step 4: Create a volume rexray volume create pg_data --size=20
Step 5: Run the container using the pre-created volume: docker run -dit --name pg -e POSTGRES_PASSWORD=mysecretpassword --volume-driver=rexray -v pg_data:/var/lib/postgresql/data postgres

Am I missing something? Are there any other steps before those?

And I still have some questions:

1 -How the file /etc/rexray/config.yml should be configured to work with CSI-NFS?
2 - How can I change the mount point from /dev to /another-volume?
3 - What the option -o host=127.0.0.1 mean?
4 - What the option -o export=/ mean?

Thanks in advance for your help.

[briantopping]

@IAPOLINARIO: A few things:

• I am running RR on a machine installed with DC/OS. I haven't looked at the origin of the package, but it has an env vars file in /etc/rexray. I put the env vars in there, no additional configuration is necessary, although the default came with a vfs service that I removed as it was giving parse errors.
• Your step 4 needs to have the -o host=127.0.0.1 -o export=/. The host is the NFS server, the export is the directory on that server that is being exported. These are the two fields that a typical mount command defines as host:export at the end of the command line.
• Turn on maximum debugging to help you see what's going on. There are three areas, for RR itself, for libstorage, and for the plugin (documented in the plugin README, added to the env variables).

The errors in RR are good if you know the internal sequence of events, but it will take some sleuthing in the source if you see an error. Best thing you can do is look at the most recently emitted debug statement, search the source, then figure out from there.

With all that, I was able to get connections, what I am having problems with at this point are more permission related.

[IAPOLINARIO]

@briantopping Worked! Tks for your help.

Taking the chance: @clintkitson is there any plan to release a version that supports DELL SC8000?

[sinahamedheidari]

@IAPOLINARIO How you found the correct configuration for the /etc/rexray/config.yml to work with CSI-NFS?

[IAPOLINARIO]

@IAPOLINARIO How you found the correct configuration for the /etc/rexray/config.yml to work with CSI-NFS?

Hey @sinahamedheidari, unfortunately, it was a long time ago and I don't remember exactly what I did back then. But as far as I can remember, the file doesn't exist and you need to create it manually.

Hope it can help you.