New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

/run/docker/plugins/.../rexray.sock: connect: no such file or directory #820

Open
Daniel-ltw opened this Issue Apr 27, 2017 · 52 comments

Comments

Projects
None yet
@Daniel-ltw
Copy link

Daniel-ltw commented Apr 27, 2017

Summary

I tried installing the rexray s3fs plugin and got the following error when trying to list volumes.

ls the directory and it does seem that rexray.sock does not exist at that location.

Thought it was a case of a single faulty instance, so remove the plugin and re-install it.
Still the same error.

If anyone has a solution for this, it would be greatly appreciated.

@clintkitson

This comment has been minimized.

Copy link
Member

clintkitson commented Apr 27, 2017

Thanks for filing the issue @Daniel-ltw. We had another conversation in the slack community yesterday with @ajeetraina about linuxkit and it sounds like you are having a similar problem.

I believe it relates to the Docker Engine in Linux Kit running from a container. When it looks to try and find the REX-Ray socket as defined in the error it can't find it. To fix this the docker container likely needs the /run/docker/plugins directory introduced so it can see sockets that the REX-Ray managed plugin advertises.

@Daniel-ltw

This comment has been minimized.

Copy link

Daniel-ltw commented Apr 27, 2017

👍

Hope there will be good news soon regarding a fix for this issue.

Really keen to use the plugin.

@ajeetraina

This comment has been minimized.

Copy link

ajeetraina commented Apr 28, 2017

I raised this issue with LinuxKit linuxkit/linuxkit#1710

@clintkitson

This comment has been minimized.

Copy link
Member

clintkitson commented Apr 28, 2017

@ajeetraina Is it possible that you just need to add - /run/docker/plugins:/run/docker/plugins to make it work under https://github.com/linuxkit/linuxkit/blob/master/examples/docker.yml#L68-L78?

@ajeetraina

This comment has been minimized.

Copy link

ajeetraina commented Apr 28, 2017

@clintkitson

This comment has been minimized.

Copy link
Member

clintkitson commented Apr 28, 2017

Try adding this as an onboot

  - name: plugins
    image: "library/alpine:3.5"
    binds:
     - /run:/run
    capabilities:
     - CAP_SYS_ADMIN
    command: ["mkdir", "-p", "/run/docker/plugins"]

and update the docker server as follows:

    binds:
     - /var/lib/docker:/var/lib/docker
     - /lib/modules:/lib/modules
     - /run/docker/plugins:/run/docker/plugins
@clintkitson

This comment has been minimized.

Copy link
Member

clintkitson commented Apr 28, 2017

I think the last thing here has to do with the types of Go binaries that can be run on the distro. It looks like based on the alpine image things might need to be statically compiled, or proper deps need to be added.

@ajeetraina

This comment has been minimized.

Copy link

ajeetraina commented Apr 29, 2017

@clintonskitson Tried making entry onto onboot and binds section, but still docker service is not coming up once LinuxKit OS comes up.

@clintkitson

This comment has been minimized.

Copy link
Member

clintkitson commented May 1, 2017

@ajeetraina You should check the docker.log to see if it says anything.

@ajeetraina

This comment has been minimized.

Copy link

ajeetraina commented May 7, 2017

Here is the complete information:

Details about Linuxkit OS:

/ # docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 17.04.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary:
containerd version: 422e31ce907fd9c3833a38d7b8fdd023e5a76e73
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.25-linuxkit
Operating System: Alpine Linux v3.5 (containerized)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.654GiB
Name: moby-42010a840002
ID: W7QF:T5WD:ZPYM:QWCC:LT4M:OXTT:VT6H:UR34:7RCX:2GLS:642P:S6JB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

My YAML file from where I built up plugin:

ajeetsinghraina81@my-swarm-worker-fgft:~/linuxkit/examples$ cat gcpwithdocker.yml
kernel:
image: "linuxkit/kernel:4.9.x"
cmdline: "console=ttyS0 console=tty0 page_poison=1"
init:

  • linuxkit/init:63eed9ca7a09d2ce4c0c5e7238ac005fa44f564b
  • linuxkit/runc:b0fb122e10dbb7e4e45115177a61a3f8d68c19a9
  • linuxkit/containerd:18eaf72f3f4f9a9f29ca1951f66df701f873060b
  • linuxkit/ca-certificates:eabc5a6e59f05aa91529d80e9a595b85b046f935
    onboot:
  • name: sysctl
    image: "linuxkit/sysctl:1f5ec5d5e6f7a7a1b3d2ff9dd9e36fd6fb14756a"
    net: host
    pid: host
    ipc: host
    capabilities:
    • CAP_SYS_ADMIN
      readonly: true
  • name: dhcpcd
    image: "linuxkit/dhcpcd:48e249ebef6a521eed886b3bce032db69fbb4afa"
    binds:
    • /var:/var
    • /tmp/etc:/etc
      capabilities:
    • CAP_NET_ADMIN
    • CAP_NET_BIND_SERVICE
    • CAP_NET_RAW
      net: host
      command: ["/sbin/dhcpcd", "--nobackground", "-f", "/dhcpcd.conf", "-1"]
  • name: sysfs
    image: linuxkit/sysfs:6c1d06f28ddd9681799d3950cddf044b930b221c
  • name: binfmt
    image: "linuxkit/binfmt:8881283ac627be1542811bd25c85e7782aebc692"
    binds:
    • /proc/sys/fs/binfmt_misc:/binfmt_misc
      readonly: true
  • name: format
    image: "linuxkit/format:53748000acf515549d398e6ae68545c26c0f3a2e"
    binds:
    • /dev:/dev
      capabilities:
    • CAP_SYS_ADMIN
    • CAP_MKNOD
  • name: metadata
    image: "linuxkit/metadata:c5567e65e9125f0a4c4b8cb9d56a86377be62652"
    binds:
    • /dev:/dev
    • /var:/var
    • /tmp/etc/resolv.conf:/etc/resolv.conf
      net: host
      capabilities:
    • CAP_SYS_ADMIN
  • name: mount
    image: "linuxkit/mount:d2669e7c8ddda99fa0618a414d44261eba6e299a"
    binds:
    • /dev:/dev
    • /var:/var:rshared,rbind
      capabilities:
    • CAP_SYS_ADMIN
      rootfsPropagation: shared
      command: ["/mount.sh", "/var/lib/docker"]
  • name: plugins
    image: "library/alpine:3.5"
    binds:
    • /run:/run
      capabilities:
    • CAP_SYS_ADMIN
      command: ["mkdir", "-p", "/run/docker/plugins"]

services:

  • name: rngd
    image: "linuxkit/rngd:c42fd499690b2cb6e4e6cb99e41dfafca1cf5b14"
    capabilities:
    • CAP_SYS_ADMIN
      oomScoreAdj: -800
      readonly: true
  • name: sshd
    image: "linuxkit/sshd:e108d208adf692c8a0954f602743e0eec445364e"
    capabilities:
    • all
      net: host
      pid: host
      binds:
    • /var/config/ssh/authorized_keys:/root/.ssh/authorized_keys
    • /tmp/etc/resolv.conf:/etc/resolv.conf
  • name: dhcpcd
    image: "linuxkit/dhcpcd:57a8ef29d3a910645b2b24c124f9ce9ef53ce703"
    binds:
    • /var:/var
    • /tmp/etc:/etc
      capabilities:
    • CAP_NET_ADMIN
    • CAP_NET_BIND_SERVICE
    • CAP_NET_RAW
      net: host
      oomScoreAdj: -800
  • name: nginx
    image: "nginx:alpine"
    capabilities:
    • CAP_NET_BIND_SERVICE
    • CAP_CHOWN
    • CAP_SETUID
    • CAP_SETGID
    • CAP_DAC_OVERRIDE
      net: host
  • name: ntpd
    image: "linuxkit/openntpd:a570316d7fc49ca1daa29bd945499f4963d227af"
    capabilities:
    • CAP_SYS_TIME
    • CAP_SYS_NICE
    • CAP_SYS_CHROOT
    • CAP_SETUID
    • CAP_SETGID
      net: host
  • name: docker
    image: "linuxkit/docker-ce:741bf21513328f674e0cdcaa55492b0b75974e08"
    capabilities:
    • all
      net: host
      mounts:
    • type: cgroup
      options: ["rw","nosuid","noexec","nodev","relatime"]
      binds:
    • /var/lib/docker:/var/lib/docker
    • /lib/modules:/lib/modules
    • /run/docker/plugins:/run/docker/plugins

files:
- path: etc/docker/daemon.json
contents: '{"debug": true}'

trust:
image:
- linuxkit/kernel
outputs:

  • format: kernel+initrd
  • format: gcp-img

This built up the following services:

/ # runc list
ID PID STATUS BUNDLE CREATED OWNER
dhcpcd 1056 running /containers/services/dhcpcd 2017-05-07T03:30:55.991655374Z root
docker 1055 running /containers/services/docker 2017-05-07T03:30:56.004875912Z root
nginx 1080 running /containers/services/nginx 2017-05-07T03:30:56.080045836Z root
ntpd 1100 running /containers/services/ntpd 2017-05-07T03:30:56.093808269Z root
rngd 1234 running /containers/services/rngd 2017-05-07T03:30:56.423514327Z root
sshd 1140 running /containers/services/sshd 2017-05-07T03:30:56.09789584Z root

I entered into docker service:

/ # runc exec -t docker sh
/ # docker version
Client:
Version: 17.04.0-ce
API version: 1.28
Go version: go1.7.5
Git commit: 4845c56
Built: Wed Apr 5 23:33:17 2017
OS/Arch: linux/amd64
Server:
Version: 17.04.0-ce
API version: 1.28 (minimum version 1.12)
Go version: go1.7.5
Git commit: 4845c56
Built: Wed Apr 5 23:33:17 2017
OS/Arch: linux/amd64
Experimental: false
/ #

I installed RexRay Volume Plugin:

/ # docker plugin install --grant-all-permissions rexray/gcepd GCEPD_TAG=rexray
latest: Pulling from rexray/gcepd
5a5bf882331c: Download complete
Digest: sha256:c27e5d1af1777743612e41ea70d7e7dbc82e1eccace81a007220ab068f28f896
Status: Downloaded newer image for rexray/gcepd:latest
Installed plugin rexray/gcepd
/ # docker plugin ls
ID NAME DESCRIPTION ENABLED
a55d51ab0ab6 rexray/gcepd:latest REX-Ray for GCE Persistent Disks false

Tried enabling it:
/ # docker plugin enable rexray/gcepd:latest
rexray/gcepd:latest
/ # docker plugin ls
ID NAME DESCRIPTION ENABLED
a55d51ab0ab6 rexray/gcepd:latest REX-Ray for GCE Persistent Disks false

@ajeetraina

This comment has been minimized.

Copy link

ajeetraina commented May 7, 2017

Just want to verify with you if the mount point looks good:

 # docker plugin install  rexray/gcepd GCEPD_TAG=rexray
Plugin "rexray/gcepd" is requesting the following privileges:
 - network: [host]
 - mount: [/dev]
 - allow-all-devices: [true]
 - capabilities: [CAP_SYS_ADMIN]
Do you grant the above permissions? [y/N] y
latest: Pulling from rexray/gcepd
5a5bf882331c: Download complete 
Digest: sha256:c27e5d1af1777743612e41ea70d7e7dbc82e1eccace81a007220ab068f28f896
Status: Downloaded newer image for rexray/gcepd:latest
Installed plugin rexray/gcepd
/ # 
@ajeetraina

This comment has been minimized.

Copy link

ajeetraina commented May 7, 2017

I am running this on top of GCE instance. Just wonder if it requires GCP credentials.

@clintkitson

This comment has been minimized.

Copy link
Member

clintkitson commented May 7, 2017

@ajeetraina Check my gist that I used without problems getting it docker started with my mods. But at this point it is hitting seccomp issues when trying to load the plugin.

https://gist.github.com/clintkitson/25233b7902adbe382e4cfd16f2e1d83d

@sebglon

This comment has been minimized.

Copy link
Contributor

sebglon commented May 17, 2017

I have the same on docker plugin install --grant-all-permissions=true rexray/s3fs whith docker 17.5

sudo docker plugin install --grant-all-permissions=true rexray/s3fs latest: Pulling from rexray/s3fs 26e03f1a0d2e: Download complete Digest: sha256:8dc11fd18494111b5fea862541a40983419648bf27cbd4c7039744d706de3256 Status: Downloaded newer image for rexray/s3fs:latest Error response from daemon: dial unix /run/docker/plugins/dd212c27f10f859148bed2d3ba8226e8c1ba5cf4196ab1d9266f866b4972c116/rexray.sock: connect: no such file or directory

@irdara

This comment has been minimized.

Copy link

irdara commented May 17, 2017

Any update on this? Running into the same issue

@plrunner

This comment has been minimized.

Copy link

plrunner commented May 29, 2017

+1

@fabio-barile

This comment has been minimized.

Copy link

fabio-barile commented Jun 29, 2017

Same issue...
Installing a plugin is not working

@kelu27

This comment has been minimized.

Copy link

kelu27 commented Jun 29, 2017

same issue with gcepd

@clintkitson

This comment has been minimized.

Copy link
Member

clintkitson commented Jun 29, 2017

@fabio-barile @kelu27 Are you reporting this issue in regards to using LinuxKit?

@Clcanny

This comment has been minimized.

Copy link

Clcanny commented Jul 15, 2017

same problem on weave plugin(v2, not depre...) +1

@savagecm

This comment has been minimized.

Copy link

savagecm commented Aug 2, 2017

same issue....

@clintkitson

This comment has been minimized.

Copy link
Member

clintkitson commented Aug 2, 2017

@savagecm Is this being reported in regards to LinuxKit?

@ajeetraina

This comment has been minimized.

Copy link

ajeetraina commented Aug 6, 2017

I am able to make RexRay Volume plugin enabled on Google Cloud Platform and this is how YAML file look like:

Ajeets-MacBook-Air:examples ajeetraina$ cat docker.yml 
kernel:
  image: linuxkit/kernel:4.9.40
  cmdline: "console=tty0 console=ttyS0"
init:
  - linuxkit/init:4dcee04c04c900a5796dc719f8d16fea7e771059
  - linuxkit/runc:f5960b83a8766ae083efc744fa63dbf877450e4f
  - linuxkit/containerd:8fc87b7f465bde9ece781899a007f47b6d3c096b
  - linuxkit/ca-certificates:67acf038c44bb191ebb704ec7bb39a1524052cdf
onboot:
  - name: sysctl
    image: linuxkit/sysctl:3f7a3f6f9e7e1d3f245c766fcf5c2b9e97382cfb
  - name: sysfs
    image: linuxkit/sysfs:006a65b30cfdd9d751d7ab042fde7eca2c3bc9dc
  - name: binfmt
    image: linuxkit/binfmt:257b5174a8e33bc62d5448cc026d72cae3713628
  - name: format
    image: linuxkit/format:efafddf9bc6165b5efaf09c532c15a1100a10e61
  - name: mount
    image: linuxkit/mount:54990a6a69cb3ead4da8a9c1f0b651e27aea8d3f
    command: ["/usr/bin/mountie", "/var/lib/docker"]
  - name: plugins
    image: "library/alpine:3.5"
    binds:
     - /run:/run
    capabilities:
     - all
    command: ["mkdir", "-p", "/run/docker/plugins"]
services:
  - name: getty
    image: linuxkit/getty:58620cff1b0bf8b5d144d087602115e996f18a02
    env:
     - INSECURE=true
  - name: rngd
    image: linuxkit/rngd:1516d5d70683a5d925fe475eb1b6164a2f67ac3b
  - name: dhcpcd
    image: linuxkit/dhcpcd:17423c1ccced74e3c005fd80486e8177841fe02b
  - name: ntpd
    image: linuxkit/openntpd:2874b66c9fa51fa5b4d11c8b50441eb94ee22a5a
  - name: docker
    image: docker:17.06.0-ce-dind
    capabilities:
     - all
    net: host
    mounts:
     - type: cgroup
       options: ["rw","nosuid","noexec","nodev","relatime"]
    binds:
     - /etc/resolv.conf:/etc/resolv.conf
     - /var/lib/docker:/var/lib/docker
     - /lib/modules:/lib/modules
     - /etc/docker/daemon.json:/etc/docker/daemon.json
     - /run/docker/plugins:/run/docker/plugins
    command: ["/usr/local/bin/docker-init", "/usr/local/bin/dockerd"]
files:
  - path: var/lib/docker
    directory: true
  - path: etc/docker/daemon.json
    contents: '{"debug": true}'
trust:
  org:
    - linuxkit
    - library

Listing the Plugin:

/ # docker plugin ls
ID                  NAME                  DESCRIPTION                        ENABLED
1cefad62ce9d        rexray/gcepd:latest   REX-Ray for GCE Persistent Disks   true
/ # docker plugin inspect rexray/gcepd
[
    {
        "Config": {
            "Args": {
                "Description": "",
                "Name": "",
                "Settable": null,
                "Value": null
            },
            "Description": "REX-Ray for GCE Persistent Disks",
            "DockerVersion": "17.06.0-ce",
            "Documentation": "https://github.com/codedellemc/rexray/.docker/plugin/gcepd",
            "Entrypoint": [
                "/rexray.sh",
                "rexray",
                "start",
                "-f",
                "--nopid"
            ],
            "Env": [
                {
                    "Description": "",
                    "Name": "REXRAY_FSTYPE",
                    "Settable": [
                        "value"
                    ],
                    "Value": "ext4"
                },
                {
                    "Description": "",
                    "Name": "REXRAY_LOGLEVEL",
                    "Settable": [
                        "value"
                    ],
                    "Value": "warn"
                },
                {
                    "Description": "",
                    "Name": "REXRAY_PREEMPT",
                    "Settable": [
                        "value"
                    ],
                    "Value": "false"
                },
                {
                    "Description": "",
                    "Name": "GCEPD_DEFAULTDISKTYPE",
                    "Settable": [
                        "value"
                    ],
                    "Value": ""
                },
                {
                    "Description": "",
                    "Name": "GCEPD_TAG",
                    "Settable": [
                        "value"
                    ],
                    "Value": ""
                },
                {
                    "Description": "",
                    "Name": "GCEPD_ZONE",
                    "Settable": [
                        "value"
                    ],
                    "Value": ""
                }
            ],
            "Interface": {
                "Socket": "rexray.sock",
                "Types": [
                    "docker.volumedriver/1.0"
                ]
            },
            "IpcHost": false,
            "Linux": {
                "AllowAllDevices": true,
                "Capabilities": [
                    "CAP_SYS_ADMIN"
                ],
                "Devices": null
            },
            "Mounts": [
                {
                    "Description": "",
                    "Destination": "/dev",
                    "Name": "",
                    "Options": [
                        "rbind"
                    ],
                    "Settable": null,
                    "Source": "/dev",
                    "Type": "bind"
                }
            ],
            "Network": {
                "Type": "host"
            },
            "PidHost": false,
            "PropagatedMount": "/var/lib/libstorage/volumes",
            "User": {},
            "WorkDir": "",
            "rootfs": {
                "diff_ids": [
                    "sha256:9f14605df71aca8c6dd617709a458c4aeeec94ce0698cddb8ccb795b88405b0d"
                ],
                "type": "layers"
            }
        },
        "Enabled": true,
        "Id": "1cefad62ce9d4a7d65e2ef43a419118374fb17744c76104e51f1d77333b159df",
        "Name": "rexray/gcepd:latest",
        "PluginReference": "docker.io/rexray/gcepd:latest",
        "Settings": {
            "Args": [],
            "Devices": [],
            "Env": [
                "REXRAY_FSTYPE=ext4",
                "REXRAY_LOGLEVEL=warn",
                "REXRAY_PREEMPT=false",
                "GCEPD_DEFAULTDISKTYPE=",
                "GCEPD_TAG=rexray",
                "GCEPD_ZONE="
            ],
            "Mounts": [
                {
                    "Description": "",
                    "Destination": "/dev",
                    "Name": "",
                    "Options": [
                        "rbind"
                    ],
                    "Settable": null,
                    "Source": "/dev",
                    "Type": "bind"
                }
            ]
        }
    }
]
/ # 

Though I am sure this should work now, but still I am not able to create volume on top of it. Is there any change in the command:

/ # docker volume create --driver rexray/gcepd --opt size=10 --name hello
Error response from daemon: create hello: VolumeDriver.Create: {"Error":"error executing xcli"}

Wonder if anything further is required to get it working.

@clintkitson Can you suggest here?

@ajeetraina

This comment has been minimized.

Copy link

ajeetraina commented Aug 7, 2017

Checked the logs:

 # docker-runc exec -t 1cefad62ce9d4a7d65e2ef43a419118374fb17744c76104e51f1d773
33b159df cat /var/log/rexray/rexray.log
                          ⌐▄Q▓▄Ç▓▄,▄_
                         Σ▄▓▓▓▓▓▓▓▓▓▓▄π
                       ╒▓▓▌▓▓▓▓▓▓▓▓▓▓▀▓▄▄.
                    ,_▄▀▓▓ ▓▓ ▓▓▓▓▓▓▓▓▓▓▓█
                   │▄▓▓ _▓▓▓▓▓▓▓▓▓┌▓▓▓▓▓█
                  _J┤▓▓▓▓▓▓▓▓▓▓▓▓▓├█▓█▓▀Γ
            ,▄▓▓▓▓▓▓^██▓▓▓▓▓▓▓▓▓▓▓▓▄▀▄▄▓▓Ω▄
            F▌▓▌█ⁿⁿⁿ  ⁿ└▀ⁿ██▓▀▀▀▀▀▀▀▀▀▀▌▓▓▓▌
             'ⁿ_  ,▄▄▄▄▄▄▄▄▄█_▄▄▄▄▄▄▄▄▄ⁿ▀~██
               Γ  ├▓▓▓▓▓█▀ⁿ█▌▓Ω]█▓▓▓▓▓▓ ├▓
               │  ├▓▓▓▓▓▌≡,__▄▓▓▓█▓▓▓▓▓ ╞█~   Y,┐
               ╞  ├▓▓▓▓▓▄▄__^^▓▓▓▌▓▓▓▓▓  ▓   /▓▓▓
                  ├▓▓▓▓▓▓▓▄▄═▄▓▓▓▓▓▓▓▓▓  π ⌐▄▓▓█║n
                _ ├▓▓▓▓▓▓▓▓▓~▓▓▓▓▓▓▓▓▓▓  ▄4▄▓▓▓██
                µ ├▓▓▓▓█▀█▓▓_▓▓███▓▓▓▓▓  ▓▓▓▓▓Ω4
                µ ├▓▀▀L   └ⁿ  ▀   ▀ ▓▓█w ▓▓▓▀ìⁿ
                ⌐ ├_                τ▀▓  Σ⌐└
                ~ ├▓▓  ▄  _     ╒  ┌▄▓▓  Γ
                  ├▓▓▓▌█═┴▓▄╒▀▄_▄▌═¢▓▓▓  ╚
               ⌠  ├▓▓▓▓▓ⁿ▄▓▓▓▓▓▓▓┐▄▓▓▓▓  └
               Ω_.└██▓▀ⁿÇⁿ▀▀▀▀▀▀█≡▀▀▀▀▀   µ
               ⁿ  .▄▄▓▓▓▓▄▄┌ ╖__▓_▄▄▄▄▄*Oⁿ
                 û▌├▓█▓▓▓██ⁿ ¡▓▓▓▓▓▓▓▓█▓╪
                 ╙Ω▀█ ▓██ⁿ    └█▀██▀▓█├█Å
                     ⁿⁿ             ⁿ ⁿ^
:::::::..  .,::::::    .,::      .::::::::..    :::.  .-:.     ::-.
;;;;'';;;; ;;;;''''    ';;;,  .,;; ;;;;'';;;;   ;;';;  ';;.   ;;;;'
 [[[,/[[['  [[cccc       '[[,,[['   [[[,/[[['  ,[[ '[[,  '[[,[[['
 $$$$$$c    $$""""        Y$$$Pcccc $$$$$$c   c$$$cc$$$c   c$$"
 888b "88bo,888oo,__    oP"''"Yo,   888b "88bo,888   888,,8P"'
 MMMM   "W" """"YUMMM,m"       "Mm, MMMM   "W" YMM   ""'mM"

REX-Ray
-------
Binary: /usr/bin/rexray
Flavor: client+agent+controller
SemVer: 0.9.2
OsArch: Linux-x86_64
Branch: v0.9.2
Commit: 9c7b7329884bf2feeffe101fc5dc3a239730427d
Formed: Wed, 28 Jun 2017 20:19:06 UTC

libStorage
----------
SemVer: 0.6.2
OsArch: Linux-x86_64
Branch: v0.9.2
Commit: 7368564cc71be9a0dbe03ddfbaeda281232eee72
Formed: Wed, 28 Jun 2017 20:18:44 UTC
@ajeetraina

This comment has been minimized.

Copy link

ajeetraina commented Aug 7, 2017

Disabling the RexRay Plugin to alter REXRAY_LOGLEVEL from warn to debug:

 # docker plugin disable -f rexray/gcepd
rexray/gcepd
/ # docker plugin ls
ID                  NAME                  DESCRIPTION                        ENABLED
05ed6c2fbe82        rexray/gcepd:latest   REX-Ray for GCE Persistent Disks   false

# docker plugin set 05 REXRAY_LOGLEVEL=debug
 # docker plugin enable rexray/gcepd
rexray/gcepd
 # docker plugin ls
ID                  NAME                  DESCRIPTION                        ENABLED
05ed6c2fbe82        rexray/gcepd:latest   REX-Ray for GCE Persistent Disks   true
/ # 


/ # docker-runc exec -t 05ed6c2fbe825bf6a87bb329e28a3b144c0b8497ea79d812087ce43c
b1d70cde cat /var/log/rexray/rexray.log
                          ⌐▄Q▓▄Ç▓▄,▄_
                         Σ▄▓▓▓▓▓▓▓▓▓▓▄π
                       ╒▓▓▌▓▓▓▓▓▓▓▓▓▓▀▓▄▄.
                    ,_▄▀▓▓ ▓▓ ▓▓▓▓▓▓▓▓▓▓▓█
                   │▄▓▓ _▓▓▓▓▓▓▓▓▓┌▓▓▓▓▓█
                  _J┤▓▓▓▓▓▓▓▓▓▓▓▓▓├█▓█▓▀Γ
            ,▄▓▓▓▓▓▓^██▓▓▓▓▓▓▓▓▓▓▓▓▄▀▄▄▓▓Ω▄
            F▌▓▌█ⁿⁿⁿ  ⁿ└▀ⁿ██▓▀▀▀▀▀▀▀▀▀▀▌▓▓▓▌
             'ⁿ_  ,▄▄▄▄▄▄▄▄▄█_▄▄▄▄▄▄▄▄▄ⁿ▀~██
               Γ  ├▓▓▓▓▓█▀ⁿ█▌▓Ω]█▓▓▓▓▓▓ ├▓
               │  ├▓▓▓▓▓▌≡,__▄▓▓▓█▓▓▓▓▓ ╞█~   Y,┐
               ╞  ├▓▓▓▓▓▄▄__^^▓▓▓▌▓▓▓▓▓  ▓   /▓▓▓
                  ├▓▓▓▓▓▓▓▄▄═▄▓▓▓▓▓▓▓▓▓  π ⌐▄▓▓█║n
                _ ├▓▓▓▓▓▓▓▓▓~▓▓▓▓▓▓▓▓▓▓  ▄4▄▓▓▓██
                µ ├▓▓▓▓█▀█▓▓_▓▓███▓▓▓▓▓  ▓▓▓▓▓Ω4
                µ ├▓▀▀L   └ⁿ  ▀   ▀ ▓▓█w ▓▓▓▀ìⁿ
                ⌐ ├_                τ▀▓  Σ⌐└
                ~ ├▓▓  ▄  _     ╒  ┌▄▓▓  Γ
                  ├▓▓▓▌█═┴▓▄╒▀▄_▄▌═¢▓▓▓  ╚
               ⌠  ├▓▓▓▓▓ⁿ▄▓▓▓▓▓▓▓┐▄▓▓▓▓  └
               Ω_.└██▓▀ⁿÇⁿ▀▀▀▀▀▀█≡▀▀▀▀▀   µ
               ⁿ  .▄▄▓▓▓▓▄▄┌ ╖__▓_▄▄▄▄▄*Oⁿ
                 û▌├▓█▓▓▓██ⁿ ¡▓▓▓▓▓▓▓▓█▓╪
                 ╙Ω▀█ ▓██ⁿ    └█▀██▀▓█├█Å
                     ⁿⁿ             ⁿ ⁿ^
:::::::..  .,::::::    .,::      .::::::::..    :::.  .-:.     ::-.
;;;;'';;;; ;;;;''''    ';;;,  .,;; ;;;;'';;;;   ;;';;  ';;.   ;;;;'
 [[[,/[[['  [[cccc       '[[,,[['   [[[,/[[['  ,[[ '[[,  '[[,[[['
 $$$$$$c    $$""""        Y$$$Pcccc $$$$$$c   c$$$cc$$$c   c$$"
 888b "88bo,888oo,__    oP"''"Yo,   888b "88bo,888   888,,8P"'
 MMMM   "W" """"YUMMM,m"       "Mm, MMMM   "W" YMM   ""'mM"
REX-Ray
-------
Binary: /usr/bin/rexray
Flavor: client+agent+controller
SemVer: 0.9.2
OsArch: Linux-x86_64
Branch: v0.9.2
Commit: 9c7b7329884bf2feeffe101fc5dc3a239730427d
Formed: Wed, 28 Jun 2017 20:19:06 UTC

libStorage
----------
SemVer: 0.6.2
OsArch: Linux-x86_64
Branch: v0.9.2
Commit: 7368564cc71be9a0dbe03ddfbaeda281232eee72
Formed: Wed, 28 Jun 2017 20:18:44 UTC


                          ⌐▄Q▓▄Ç▓▄,▄_
                         Σ▄▓▓▓▓▓▓▓▓▓▓▄π
                       ╒▓▓▌▓▓▓▓▓▓▓▓▓▓▀▓▄▄.
                    ,_▄▀▓▓ ▓▓ ▓▓▓▓▓▓▓▓▓▓▓█
                   │▄▓▓ _▓▓▓▓▓▓▓▓▓┌▓▓▓▓▓█
                  _J┤▓▓▓▓▓▓▓▓▓▓▓▓▓├█▓█▓▀Γ
            ,▄▓▓▓▓▓▓^██▓▓▓▓▓▓▓▓▓▓▓▓▄▀▄▄▓▓Ω▄
            F▌▓▌█ⁿⁿⁿ  ⁿ└▀ⁿ██▓▀▀▀▀▀▀▀▀▀▀▌▓▓▓▌
             'ⁿ_  ,▄▄▄▄▄▄▄▄▄█_▄▄▄▄▄▄▄▄▄ⁿ▀~██
               Γ  ├▓▓▓▓▓█▀ⁿ█▌▓Ω]█▓▓▓▓▓▓ ├▓
               │  ├▓▓▓▓▓▌≡,__▄▓▓▓█▓▓▓▓▓ ╞█~   Y,┐
               ╞  ├▓▓▓▓▓▄▄__^^▓▓▓▌▓▓▓▓▓  ▓   /▓▓▓
                  ├▓▓▓▓▓▓▓▄▄═▄▓▓▓▓▓▓▓▓▓  π ⌐▄▓▓█║n
                _ ├▓▓▓▓▓▓▓▓▓~▓▓▓▓▓▓▓▓▓▓  ▄4▄▓▓▓██
                µ ├▓▓▓▓█▀█▓▓_▓▓███▓▓▓▓▓  ▓▓▓▓▓Ω4
                µ ├▓▀▀L   └ⁿ  ▀   ▀ ▓▓█w ▓▓▓▀ìⁿ
                ⌐ ├_                τ▀▓  Σ⌐└
                ~ ├▓▓  ▄  _     ╒  ┌▄▓▓  Γ
                  ├▓▓▓▌█═┴▓▄╒▀▄_▄▌═¢▓▓▓  ╚
               ⌠  ├▓▓▓▓▓ⁿ▄▓▓▓▓▓▓▓┐▄▓▓▓▓  └
               Ω_.└██▓▀ⁿÇⁿ▀▀▀▀▀▀█≡▀▀▀▀▀   µ
               ⁿ  .▄▄▓▓▓▓▄▄┌ ╖__▓_▄▄▄▄▄*Oⁿ
                 û▌├▓█▓▓▓██ⁿ ¡▓▓▓▓▓▓▓▓█▓╪
                 ╙Ω▀█ ▓██ⁿ    └█▀██▀▓█├█Å
                     ⁿⁿ             ⁿ ⁿ^
:::::::..  .,::::::    .,::      .::::::::..    :::.  .-:.     ::-.
;;;;'';;;; ;;;;''''    ';;;,  .,;; ;;;;'';;;;   ;;';;  ';;.   ;;;;'
 [[[,/[[['  [[cccc       '[[,,[['   [[[,/[[['  ,[[ '[[,  '[[,[[['
 $$$$$$c    $$""""        Y$$$Pcccc $$$$$$c   c$$$cc$$$c   c$$"
 888b "88bo,888oo,__    oP"''"Yo,   888b "88bo,888   888,,8P"'
 MMMM   "W" """"YUMMM,m"       "Mm, MMMM   "W" YMM   ""'mM"

REX-Ray
-------
Binary: /usr/bin/rexray
Flavor: client+agent+controller
SemVer: 0.9.2
OsArch: Linux-x86_64
Branch: v0.9.2
Commit: 9c7b7329884bf2feeffe101fc5dc3a239730427d
Formed: Wed, 28 Jun 2017 20:19:06 UTC

libStorage
----------
SemVer: 0.6.2
OsArch: Linux-x86_64
Branch: v0.9.2
Commit: 7368564cc71be9a0dbe03ddfbaeda281232eee72
Formed: Wed, 28 Jun 2017 20:18:44 UTC

time="2017-08-07T17:01:54Z" level=debug msg="created scoped scope" new=libstorage.server parentScopes="rexray," 
time="2017-08-07T17:01:54Z" level=debug msg="created scoped scope" new=libstorage.server.services.gcepd parentScopes="libstorage.server,rexray," 
time="2017-08-07T17:01:55Z" level=debug msg="created scoped scope" new=rexray.modules.default-docker parentScopes="rexray," 
time="2017-08-07T17:01:55Z" level=debug msg="created scoped scope" new=libstorage.client parentScopes="rexray.modules.default-docker,rexray," 
@clintkitson

This comment has been minimized.

Copy link
Member

clintkitson commented Aug 7, 2017

@ajeetraina I am not seeing any failures in the log?

@ajeetraina

This comment has been minimized.

Copy link

ajeetraina commented Aug 7, 2017

@Daniel-ltw

This comment has been minimized.

Copy link

Daniel-ltw commented Aug 7, 2017

Sounds like good news 👍

@clintkitson

This comment has been minimized.

Copy link
Member

clintkitson commented Aug 7, 2017

@ajeetraina Not sure I understand. You should be trying to execute an executor command via this runc command you showed earlier (updated to latest plugin id). docker-runc exec -t 05ed6c2fbe825bf6a87bb329e28a3b144c0b8497ea79d812087ce43c b1d70cde /var/lib/libstorage/lsx-linux gce instanceID.

I believe this would be the right location and command.

@ajeetraina

This comment has been minimized.

Copy link

ajeetraina commented Aug 8, 2017

# docker-runc exec -t 05ed6c2fbe825bf6a87bb329e28a3b144c0b8497ea79d812087ce43c
b1d70cde /var/lib/libstorage/lsx-linux gcepd instanceID

/ # docker-runc exec -t 05ed6c2fbe825bf6a87bb329e28a3b144c0b8497ea79d812087ce43c
b1d70cde cat /var/log/rexray/rexray.log

                          ⌐▄Q▓▄Ç▓▄,▄_
                         Σ▄▓▓▓▓▓▓▓▓▓▓▄π
                       ╒▓▓▌▓▓▓▓▓▓▓▓▓▓▀▓▄▄.
                    ,_▄▀▓▓ ▓▓ ▓▓▓▓▓▓▓▓▓▓▓█
                   │▄▓▓ _▓▓▓▓▓▓▓▓▓┌▓▓▓▓▓█
                  _J┤▓▓▓▓▓▓▓▓▓▓▓▓▓├█▓█▓▀Γ
            ,▄▓▓▓▓▓▓^██▓▓▓▓▓▓▓▓▓▓▓▓▄▀▄▄▓▓Ω▄
            F▌▓▌█ⁿⁿⁿ  ⁿ└▀ⁿ██▓▀▀▀▀▀▀▀▀▀▀▌▓▓▓▌
             'ⁿ_  ,▄▄▄▄▄▄▄▄▄█_▄▄▄▄▄▄▄▄▄ⁿ▀~██
               Γ  ├▓▓▓▓▓█▀ⁿ█▌▓Ω]█▓▓▓▓▓▓ ├▓
               │  ├▓▓▓▓▓▌≡,__▄▓▓▓█▓▓▓▓▓ ╞█~   Y,┐
               ╞  ├▓▓▓▓▓▄▄__^^▓▓▓▌▓▓▓▓▓  ▓   /▓▓▓
                  ├▓▓▓▓▓▓▓▄▄═▄▓▓▓▓▓▓▓▓▓  π ⌐▄▓▓█║n
                _ ├▓▓▓▓▓▓▓▓▓~▓▓▓▓▓▓▓▓▓▓  ▄4▄▓▓▓██
                µ ├▓▓▓▓█▀█▓▓_▓▓███▓▓▓▓▓  ▓▓▓▓▓Ω4
                µ ├▓▀▀L   └ⁿ  ▀   ▀ ▓▓█w ▓▓▓▀ìⁿ
                ⌐ ├_                τ▀▓  Σ⌐└
                ~ ├▓▓  ▄  _     ╒  ┌▄▓▓  Γ
                  ├▓▓▓▌█═┴▓▄╒▀▄_▄▌═¢▓▓▓  ╚
               ⌠  ├▓▓▓▓▓ⁿ▄▓▓▓▓▓▓▓┐▄▓▓▓▓  └
               Ω_.└██▓▀ⁿÇⁿ▀▀▀▀▀▀█≡▀▀▀▀▀   µ
               ⁿ  .▄▄▓▓▓▓▄▄┌ ╖__▓_▄▄▄▄▄*Oⁿ
                 û▌├▓█▓▓▓██ⁿ ¡▓▓▓▓▓▓▓▓█▓╪
                 ╙Ω▀█ ▓██ⁿ    └█▀██▀▓█├█Å
                     ⁿⁿ             ⁿ ⁿ^
:::::::..  .,::::::    .,::      .::::::::..    :::.  .-:.     ::-.
;;;;'';;;; ;;;;''''    ';;;,  .,;; ;;;;'';;;;   ;;';;  ';;.   ;;;;'
 [[[,/[[['  [[cccc       '[[,,[['   [[[,/[[['  ,[[ '[[,  '[[,[[['
 $$$$$$c    $$""""        Y$$$Pcccc $$$$$$c   c$$$cc$$$c   c$$"
 888b "88bo,888oo,__    oP"''"Yo,   888b "88bo,888   888,,8P"'
 MMMM   "W" """"YUMMM,m"       "Mm, MMMM   "W" YMM   ""'mM"

REX-Ray
-------
Binary: /usr/bin/rexray
Flavor: client+agent+controller
SemVer: 0.9.2
OsArch: Linux-x86_64
Branch: v0.9.2
Commit: 9c7b7329884bf2feeffe101fc5dc3a239730427d
Formed: Wed, 28 Jun 2017 20:19:06 UTC

libStorage
----------
SemVer: 0.6.2
OsArch: Linux-x86_64
Branch: v0.9.2
Commit: 7368564cc71be9a0dbe03ddfbaeda281232eee72
Formed: Wed, 28 Jun 2017 20:18:44 UTC


                          ⌐▄Q▓▄Ç▓▄,▄_
                         Σ▄▓▓▓▓▓▓▓▓▓▓▄π
                       ╒▓▓▌▓▓▓▓▓▓▓▓▓▓▀▓▄▄.
                    ,_▄▀▓▓ ▓▓ ▓▓▓▓▓▓▓▓▓▓▓█
                   │▄▓▓ _▓▓▓▓▓▓▓▓▓┌▓▓▓▓▓█
                  _J┤▓▓▓▓▓▓▓▓▓▓▓▓▓├█▓█▓▀Γ
            ,▄▓▓▓▓▓▓^██▓▓▓▓▓▓▓▓▓▓▓▓▄▀▄▄▓▓Ω▄
            F▌▓▌█ⁿⁿⁿ  ⁿ└▀ⁿ██▓▀▀▀▀▀▀▀▀▀▀▌▓▓▓▌
             'ⁿ_  ,▄▄▄▄▄▄▄▄▄█_▄▄▄▄▄▄▄▄▄ⁿ▀~██
               Γ  ├▓▓▓▓▓█▀ⁿ█▌▓Ω]█▓▓▓▓▓▓ ├▓
               │  ├▓▓▓▓▓▌≡,__▄▓▓▓█▓▓▓▓▓ ╞█~   Y,┐
               ╞  ├▓▓▓▓▓▄▄__^^▓▓▓▌▓▓▓▓▓  ▓   /▓▓▓
                  ├▓▓▓▓▓▓▓▄▄═▄▓▓▓▓▓▓▓▓▓  π ⌐▄▓▓█║n
                _ ├▓▓▓▓▓▓▓▓▓~▓▓▓▓▓▓▓▓▓▓  ▄4▄▓▓▓██
                µ ├▓▓▓▓█▀█▓▓_▓▓███▓▓▓▓▓  ▓▓▓▓▓Ω4
                µ ├▓▀▀L   └ⁿ  ▀   ▀ ▓▓█w ▓▓▓▀ìⁿ
                ⌐ ├_                τ▀▓  Σ⌐└
                ~ ├▓▓  ▄  _     ╒  ┌▄▓▓  Γ
                  ├▓▓▓▌█═┴▓▄╒▀▄_▄▌═¢▓▓▓  ╚
               ⌠  ├▓▓▓▓▓ⁿ▄▓▓▓▓▓▓▓┐▄▓▓▓▓  └
               Ω_.└██▓▀ⁿÇⁿ▀▀▀▀▀▀█≡▀▀▀▀▀   µ
               ⁿ  .▄▄▓▓▓▓▄▄┌ ╖__▓_▄▄▄▄▄*Oⁿ
                 û▌├▓█▓▓▓██ⁿ ¡▓▓▓▓▓▓▓▓█▓╪
                 ╙Ω▀█ ▓██ⁿ    └█▀██▀▓█├█Å
                     ⁿⁿ             ⁿ ⁿ^
:::::::..  .,::::::    .,::      .::::::::..    :::.  .-:.     ::-.
;;;;'';;;; ;;;;''''    ';;;,  .,;; ;;;;'';;;;   ;;';;  ';;.   ;;;;'
 [[[,/[[['  [[cccc       '[[,,[['   [[[,/[[['  ,[[ '[[,  '[[,[[['
 $$$$$$c    $$""""        Y$$$Pcccc $$$$$$c   c$$$cc$$$c   c$$"
 888b "88bo,888oo,__    oP"''"Yo,   888b "88bo,888   888,,8P"'
 MMMM   "W" """"YUMMM,m"       "Mm, MMMM   "W" YMM   ""'mM"
REX-Ray
-------
Binary: /usr/bin/rexray
Flavor: client+agent+controller
SemVer: 0.9.2
OsArch: Linux-x86_64
Branch: v0.9.2
Commit: 9c7b7329884bf2feeffe101fc5dc3a239730427d
Formed: Wed, 28 Jun 2017 20:19:06 UTC
libStorage
----------
SemVer: 0.6.2
OsArch: Linux-x86_64
Branch: v0.9.2
Commit: 7368564cc71be9a0dbe03ddfbaeda281232eee72
Formed: Wed, 28 Jun 2017 20:18:44 UTC
time="2017-08-07T17:01:54Z" level=debug msg="created scoped scope" new=libstorage.server pare
ntScopes="rexray," 
time="2017-08-07T17:01:54Z" level=debug msg="created scoped scope" new=libstorage.server.serv
ices.gcepd parentScopes="libstorage.server,rexray," 
time="2017-08-07T17:01:55Z" level=debug msg="created scoped scope" new=rexray.modules.default
-docker parentScopes="rexray," 
time="2017-08-07T17:01:55Z" level=debug msg="created scoped scope" new=libstorage.client pare
ntScopes="rexray.modules.default-docker,rexray," 
/ # 

Facing the same error while I try to create volume.
Can you educate what is this executor all about? I never used this command earlier.
Also, I wonder if this limitation is coming from LInuxKit architecture.

@clintkitson

This comment has been minimized.

Copy link
Member

clintkitson commented Aug 8, 2017

@robertgartman

This comment has been minimized.

Copy link

robertgartman commented Aug 10, 2017

Encountering the

dial unix /run/docker/plugins/617c70e...dec28cf180c9ad/rexray.sock: connect: no such file or directory

Environment:
AWS EC2 AMI:

ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20170619.1 (ami-6d48500b),
Description: Canonical, Ubuntu, 16.04 LTS, amd64 xenial image build on 2017-06-19

A Docker Swarm worker is running on the instance:

root@ip-192-168-12-134:/home/ubuntu# docker info
Containers: 12
 Running: 3
 Paused: 0
 Stopped: 9
Images: 16
Server Version: 17.06.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 131
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: rsy0l12trm5h5840j928djktx
 Is Manager: false
 Node Address: 192.168.12.134
 Manager Addresses:
  192.168.11.10:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a
runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-1026-aws
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.858GiB
Name: ip-192-168-12-134
ID: AVAV:AQRO:BF7B:SJGC:TOB6:LZQE:WZ4I:T73F:XPHG:FCUS:NDYX:6UYN
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

The Docker engine and swarm is running fine. When trying to install s3fs:

root@ip-192-168-12-134:docker plugin install rexray/s3fs:0.9.2 S3FS_ACCESSKEY=B.........K S3FS_SECRETKEY=X.......rC
Plugin "rexray/s3fs:0.9.2" is requesting the following privileges:
 - network: [host]
 - mount: [/dev]
 - allow-all-devices: [true]
 - capabilities: [CAP_SYS_ADMIN]
Do you grant the above permissions? [y/N] y
0.9.2: Pulling from rexray/s3fs
5a6437c91cc6: Download complete
Digest: sha256:6ba105974d5298ff49def3c83e5547d41e573e0a3cb8604f1dcb1a0f90ee070b
Status: Downloaded newer image for rexray/s3fs:0.9.2
Error response from daemon: dial unix /run/docker/plugins/617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad/rexray.sock: connect: no such file or directory

Suspecting a general plug-in issue I tried installing...
docker plugin install tiborvass/sample-volume-plugin
...which went fine. Status ends up being:

root@ip-192-168-12-134:/home/ubuntu# docker plugin ls
ID                  NAME                                    DESCRIPTION                                     ENABLED
617c70e554c6        rexray/s3fs:0.9.2                       REX-Ray FUSE Driver for Amazon Simple Stor...   false
26334f9f6756        tiborvass/sample-volume-plugin:latest   A sample volume plugin for Docker               true

It's not possible to enable s3fs. The no such file or directory error is triggered.
I have have successfully installed rexray/s3fs:0.9.2 on a local Ubuntu 16.04 Xenial.

@clintkitson

This comment has been minimized.

Copy link
Member

clintkitson commented Aug 10, 2017

@robertgartman

This comment has been minimized.

Copy link

robertgartman commented Aug 10, 2017

@clintkitson
I have installed Docker engine via apt-get...

root@ip-192-168-12-134:/home/ubuntu# service docker status
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2017-08-04 08:49:23 UTC; 6 days ago
     Docs: https://docs.docker.com
 Main PID: 21325 (dockerd)
    Tasks: 270
   Memory: 624.3M
      CPU: 48min 35.456s
   CGroup: /system.slice/docker.service
           ├─ 2638 docker-containerd-shim d8b78e650ab7023fd8f476b0d154312eeb730c80fe25955310ed0f83e2cf562a /var/run/docker/libcontainerd/d8b78e650ab7023fd8f476b0d154312eeb730c80fe25955310ed0f83e2cf562a docker-runc
           ├─ 3357 docker-containerd-shim e977762bdeedb94bb15590f3ad39921498415a4ff350e2732c2ffbc17780161b /var/run/docker/libcontainerd/e977762bdeedb94bb15590f3ad39921498415a4ff350e2732c2ffbc17780161b docker-runc
           ├─ 7218 docker-containerd-shim 26334f9f6756fe8efe5fe3d1f389be7d972ce14d7b355ef6f61710914b966689 /var/run/docker/libcontainerd/26334f9f6756fe8efe5fe3d1f389be7d972ce14d7b355ef6f61710914b966689 docker-runc
           ├─13307 docker-containerd-shim c3706552a2f9bf36e5fb514b8ee1ee5bae1b9c9a9e1b1074bc2ae1e66fba7007 /var/run/docker/libcontainerd/c3706552a2f9bf36e5fb514b8ee1ee5bae1b9c9a9e1b1074bc2ae1e66fba7007 docker-runc
           ├─21325 /usr/bin/dockerd
           ├─21333 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc
           └─26334f9f6756fe8efe5fe3d1f389be7d972ce14d7b355ef6f61710914b966689
             └─7239 /usr/bin/sample-volume-plugin /data
@clintkitson

This comment has been minimized.

Copy link
Member

clintkitson commented Aug 10, 2017

@robertgartman

This comment has been minimized.

Copy link

robertgartman commented Aug 10, 2017

@clintkitson

More input:

Checking Docker engine logs via

journalctl -u docker.service

Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="                          ⌐▄Q▓▄Ç▓▄,▄_" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="                         Σ▄▓▓▓▓▓▓▓▓▓▓▄π" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="                       ╒▓▓▌▓▓▓▓▓▓▓▓▓▓▀▓▄▄." plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="                    ,_▄▀▓▓ ▓▓ ▓▓▓▓▓▓▓▓▓▓▓█" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="                   │▄▓▓ _▓▓▓▓▓▓▓▓▓┌▓▓▓▓▓█" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="                  _J┤▓▓▓▓▓▓▓▓▓▓▓▓▓├█▓█▓▀Γ" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="            ,▄▓▓▓▓▓▓^██▓▓▓▓▓▓▓▓▓▓▓▓▄▀▄▄▓▓Ω▄" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="            F▌▓▌█ⁿⁿⁿ  ⁿ└▀ⁿ██▓▀▀▀▀▀▀▀▀▀▀▌▓▓▓▌" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="             'ⁿ_  ,▄▄▄▄▄▄▄▄▄█_▄▄▄▄▄▄▄▄▄ⁿ▀~██" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="               Γ  ├▓▓▓▓▓█▀ⁿ█▌▓Ω]█▓▓▓▓▓▓ ├▓" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="               │  ├▓▓▓▓▓▌≡,__▄▓▓▓█▓▓▓▓▓ ╞█~   Y,┐" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="               ╞  ├▓▓▓▓▓▄▄__^^▓▓▓▌▓▓▓▓▓  ▓   /▓▓▓" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="                  ├▓▓▓▓▓▓▓▄▄═▄▓▓▓▓▓▓▓▓▓  π ⌐▄▓▓█║n" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="                _ ├▓▓▓▓▓▓▓▓▓~▓▓▓▓▓▓▓▓▓▓  ▄4▄▓▓▓██" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="                µ ├▓▓▓▓█▀█▓▓_▓▓███▓▓▓▓▓  ▓▓▓▓▓Ω4" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="                µ ├▓▀▀L   └ⁿ  ▀   ▀ ▓▓█w ▓▓▓▀ìⁿ" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="                ⌐ ├_                τ▀▓  Σ⌐└" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="                ~ ├▓▓  ▄  _     ╒  ┌▄▓▓  Γ" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="                  ├▓▓▓▌█═┴▓▄╒▀▄_▄▌═¢▓▓▓  ╚" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="               ⌠  ├▓▓▓▓▓ⁿ▄▓▓▓▓▓▓▓┐▄▓▓▓▓  └" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="               Ω_.└██▓▀ⁿÇⁿ▀▀▀▀▀▀█≡▀▀▀▀▀   µ" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="               ⁿ  .▄▄▓▓▓▓▄▄┌ ╖__▓_▄▄▄▄▄*Oⁿ" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="                 û▌├▓█▓▓▓██ⁿ ¡▓▓▓▓▓▓▓▓█▓╪" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="                 ╙Ω▀█ ▓██ⁿ    └█▀██▀▓█├█Å" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="                     ⁿⁿ             ⁿ ⁿ^" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg=":::::::..  .,::::::    .,::      .::::::::..    :::.  .-:.     ::-." plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg=";;;;'';;;; ;;;;''''    ';;;,  .,;; ;;;;'';;;;   ;;';;  ';;.   ;;;;'" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg=" [[[,/[[['  [[cccc       '[[,,[['   [[[,/[[['  ,[[ '[[,  '[[,[[['" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg=" $$$$$$c    $$\"\"\"\"        Y$$$Pcccc $$$$$$c   c$$$cc$$$c   c$$\"" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg=" 888b \"88bo,888oo,__    oP\"''\"Yo,   888b \"88bo,888   888,,8P\"'" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg=" MMMM   \"W\" \"\"\"\"YUMMM,m\"       \"Mm, MMMM   \"W\" YMM   \"\"'mM\"" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg=REX-Ray plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg=------- plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="Binary: /usr/bin/rexray" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="Flavor: client+agent+controller" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="SemVer: 0.9.2" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="OsArch: Linux-x86_64" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="Branch: v0.9.2" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="Commit: 9c7b7329884bf2feeffe101fc5dc3a239730427d" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="Formed: Wed, 28 Jun 2017 20:22:05 UTC" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg=libStorage plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg=---------- plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="SemVer: 0.6.2" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="OsArch: Linux-x86_64" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="Branch: v0.9.2" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="Commit: 7368564cc71be9a0dbe03ddfbaeda281232eee72" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info msg="Formed: Wed, 28 Jun 2017 20:21:45 UTC" plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:02 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:02Z" level=info plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:07 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:07Z" level=info msg="time=\"2017-08-10T18:46:07Z\" level=error msg=\"s3 connection failed\" error=\"AccessDenied: Access Denied\\n\\tstatus code: 403, request id: 8107014919EB7232\" region=us-east-1 server=maze-ferret-mw service=s3fs storageDriver=s3fs time=1502390767750 " plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:07 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:07Z" level=info msg="time=\"2017-08-10T18:46:07Z\" level=error msg=\"error starting libStorage server\" error=\"AccessDenied: Access Denied\\n\\tstatus code: 403, request id: 8107014919EB7232\" time=1502390767750 " plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:07 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:07Z" level=info msg="time=\"2017-08-10T18:46:07Z\" level=error msg=\"default module(s) failed to initialize\" error=\"AccessDenied: Access Denied\\n\\tstatus code: 403, request id: 8107014919EB7232\" time=1502390767750 " plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:07 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:07Z" level=info msg="time=\"2017-08-10T18:46:07Z\" level=error msg=\"daemon failed to initialize\" error=\"AccessDenied: Access Denied\\n\\tstatus code: 403, request id: 8107014919EB7232\" time=1502390767751 " plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:07 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:07Z" level=info msg="time=\"2017-08-10T18:46:07Z\" level=error msg=\"error starting rex-ray\" error=\"AccessDenied: Access Denied\\n\\tstatus code: 403, request id: 8107014919EB7232\" time=1502390767751 " plugin=617c70e554c6f151347fcf10d7000d60bac6271210a17d29f5dec28cf180c9ad
Aug 10 18:46:08 ip-192-168-12-134 dockerd[21325]: time="2017-08-10T18:46:08.858967337Z" level=error msg="Sending SIGTERM to plugin failed with error: rpc error: code = 2 desc = containerd: container not found"

AccessDenied.... hmmm. I realise there was a difference between installing locally and on AWS. When running locally I had "AmazonS3FullAccess" AMI policy assigned to the user. Then I locked down the user by removing "AmazonS3FullAccess" and instead adding my custom policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::*",
            "Condition": {
                "StringLike": {
                    "s3:prefix": "mydomain-rexray*"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": "s3:CreateBucket",
            "Resource": "arn:aws:s3:::*",
            "Condition": {
                "StringLike": {
                    "s3:LocationConstraint": "eu-west-1*"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::mydomain-rexray*",
                "arn:aws:s3:::mydomain-rexray*/*"
            ]
        }
    ]
}

Inspiration: http://blog.vizuri.com/practical-persistent-cloud-storage-for-docker-in-aws-using-rexray-pt-3

It turns out that my custom policy is the root cause of the installation issue. Switching back to "AmazonS3FullAccess" makes the plugin install just fine!

So probably not a bug after all! Thanks @clintkitson for extremely quick response.

Looking into...
http://libstorage.readthedocs.io/en/stable/user-guide/storage-providers/#aws-s3fs
.. there are no guidelines abut minimum S3 access rights required during plugin install. Do you know?

@clintkitson

This comment has been minimized.

Copy link
Member

clintkitson commented Aug 10, 2017

I personally haven't been able to figure out exactly what those are. Open to someone to lend a hand here and define those? I have always just been using the secret/access keys to make this work.

@marcellodesales

This comment has been minimized.

Copy link

marcellodesales commented Aug 21, 2017

Got EFS, EBS, S3 Plugins with AWS Policy Working

  • Just got the same errors descbried here... But got them solved.
  • @clintkitson you were asking for some debugging... I don't use secret/access key because of security. I prefer the Policies because it is more general.
  • I finally see the m all installed and working...
  • @robertgartman Definitely helped with the comment from above... The Policy file can be used to restrict which S3 buckets to read based on the name, etc...

EFS MOUNTS
screen shot 2017-08-20 at 6 05 09 pm

EBS MOUNTS
screen shot 2017-08-20 at 6 05 42 pm

Errors from S3 and EFS support

I just tried to install the S3 version in an EC2 AMI through Packer... as well and got the same error... I still can't do it over from Packer, maybe due to the missing security groups...

    amazon-ebs: Complete!
    amazon-ebs: latest: Pulling from rexray/efs
    amazon-ebs: 5fa4398d663c: Verifying Checksum
    amazon-ebs: 5fa4398d663c: Download complete
    amazon-ebs: Digest: sha256:9d507174acf741b233c827d54778d1e8e4ce40c386edda085448a593cec58dcb
    amazon-ebs: Status: Downloaded newer image for rexray/efs:latest
    amazon-ebs: Installed plugin rexray/efs
    amazon-ebs: latest: Pulling from rexray/ebs
    amazon-ebs: aba58f462f1b: Verifying Checksum
    amazon-ebs: aba58f462f1b: Download complete
    amazon-ebs: Digest: sha256:0563ef465fef628caa5fd40d432e4df32699a4e95f92810a30c76555b5b9c155
    amazon-ebs: Status: Downloaded newer image for rexray/ebs:latest
    amazon-ebs: Installed plugin rexray/ebs
    amazon-ebs: latest: Pulling from rexray/s3fs
    amazon-ebs: 5a6437c91cc6: Verifying Checksum
    amazon-ebs: 5a6437c91cc6: Download complete
    amazon-ebs: Digest: sha256:6ba105974d5298ff49def3c83e5547d41e573e0a3cb8604f1dcb1a0f90ee070b
    amazon-ebs: Status: Downloaded newer image for rexray/s3fs:latest
    amazon-ebs: Error response from daemon: dial unix /run/docker/plugins/05465763408c05ceac2ee011b631efeacd629151ec0e4913b9c28a30174f78be/rexray.sock: connect: no such file or directory

Policy

  • I had a partial policy file without S3 nor EFS...
  #description = "give REX-Ray and the ec2 instance the permissions it needs to perform necessary operations"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RexRayMin",
      "Effect": "Allow",
      "Action": [
                "ec2:AttachVolume",      <<<< ------   EBS Permissions
                "ec2:CreateVolume",
                "ec2:CreateSnapshot",
                "ec2:CreateTags",
                "ec2:DeleteVolume",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeInstances",
                "ec2:DescribeVolumes",
                "ec2:DescribeVolumeAttribute",
                "ec2:DescribeVolumeStatus",
                "ec2:DescribeSnapshots",
                "ec2:CopySnapshot",
                "ec2:DescribeSnapshotAttribute",
                "ec2:DetachVolume",
                "ec2:ModifySnapshotAttribute",
                "ec2:ModifyVolumeAttribute",
                "ec2:DescribeTags"
 ...
...
  • I did not have the Action for S3 Full access that @robertgartman described...

Fixed

  • Added the Action to the Policy s3:AmazonS3FullAccess and it worked for the S3 host.
  • In addition, I also added the permissions to get EFS to work as well...
....
....
                "ec2:ModifySnapshotAttribute",
                "ec2:ModifyVolumeAttribute",
                "ec2:DescribeTags",
                "s3:AmazonS3FullAccess",      <<<----- S3 Driver Support

                "ec2:CreateNetworkInterface",    <<<<------ From here down, EFS Support
                "ec2:DeleteNetworkInterface",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeNetworkInterfaceAttribute",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeVpcAttribute",
                "ec2:DescribeVpcs",
                "ec2:ModifyNetworkInterfaceAttribute",
                "elasticfilesystem:*",
                "kms:DescribeKey",
                "kms:ListAliases"
  • After that, I was able to install without errors
[root@ip-10-zz-yy-xx ec2-user]# sudo docker plugin install --grant-all-permissions rexray/s3fs
latest: Pulling from rexray/s3fs
5a6437c91cc6: Download complete
Digest: sha256:6ba105974d5298ff49def3c83e5547d41e573e0a3cb8604f1dcb1a0f90ee070b
Status: Downloaded newer image for rexray/s3fs:latest
Installed plugin rexray/s3fs

Docker Plugin ls

  • After setting up the UserData to install the 3 plugins , everything works properly!
[ec2-user@ip-10-82-48-59 ~]$ docker plugin ls
ID                  NAME                 DESCRIPTION                                     ENABLED
0eb15d4a9594        rexray/s3fs:latest   REX-Ray FUSE Driver for Amazon Simple Stor...   true
6bae636cb377        rexray/ebs:latest    REX-Ray for Amazon EBS                          true
9e231d9177c2        rexray/efs:latest    REX-Ray for Amazon EFS                          true

Running from Docker Containers

As a rule of thumb, I learned the following:

  • EFS = Concurrent Writes (x replicas scenarios)
  • EBS = Single Writes (1 replicate scenarios)

So, the example below shows the following:

  • Create 10 Docker Containers concurrently writing to the same file
Last login: Mon Aug 21 00:34:46 2017 from 10.82.49.230
[ec2-user@ip-10-zz-yy-xx ~]$ docker service create --replicas 10 
         --name echo-to-efs  
         --mount type=volume,source=echo_data,target=/test,volume-driver=rexray/efs 
        alpine /bin/sh -c 'echo "writing from $HOSTNAME" >> /test/echo.log ; cat /test/echo.log ; sleep 30'
iy4rzd4imq776abjlqdqf8l10
  • The docker ps shows some of them writing and failures
  • If someone can explain why some failures in the case please let me know...
  • I have REXRAY_PREEMPT=true so, not sure if that's not related
[ec2-user@ip-10-82-48-59 ~]$ docker service ps echo-to-efs
ID                  NAME                IMAGE               NODE                           DESIRED STATE       CURRENT STATE                 ERROR                              PORTS
8c8lxqcoc3n5        echo-to-efs.1       alpine:latest       ip-10-82-48-59.vpc.internal    Running             Running 8 seconds ago
23meciw5cvad        echo-to-efs.2       alpine:latest       ip-10-82-49-52.vpc.internal    Running             Running 2 seconds ago
si7lg30kb13b         \_ echo-to-efs.2   alpine:latest       ip-10-82-48-93.vpc.internal    Shutdown            Failed about a minute ago     "starting container failed: er…"
rmj163xu97ai        echo-to-efs.3       alpine:latest       ip-10-82-48-96.vpc.internal    Running             Running 7 seconds ago
5jtao2yu5udt        echo-to-efs.4       alpine:latest       ip-10-82-48-95.vpc.internal    Running             Running 8 seconds ago
wot9hd5nbjsz        echo-to-efs.5       alpine:latest       ip-10-82-49-71.vpc.internal    Running             Running 7 seconds ago
hcbbgfiuga9c        echo-to-efs.6       alpine:latest       ip-10-82-48-140.vpc.internal   Running             Starting 49 seconds ago
wrnnrsm16usr         \_ echo-to-efs.6   alpine:latest       ip-10-82-48-25.vpc.internal    Shutdown            Rejected 58 seconds ago       "create echo_data: VolumeDrive…"
w3v4x0pf9uri         \_ echo-to-efs.6   alpine:latest       ip-10-82-48-37.vpc.internal    Shutdown            Rejected about a minute ago   "create echo_data: VolumeDrive…"
okgda40hb4hy         \_ echo-to-efs.6   alpine:latest       ip-10-82-49-107.vpc.internal   Shutdown            Rejected about a minute ago   "create echo_data: VolumeDrive…"
bzoafno0gb4d        echo-to-efs.7       alpine:latest       ip-10-82-48-74.vpc.internal    Running             Running 7 seconds ago
rk7uebz4cdql        echo-to-efs.8       alpine:latest       ip-10-82-49-111.vpc.internal   Running             Running 6 seconds ago
hy1rfid4zitl        echo-to-efs.9       alpine:latest       ip-10-82-48-162.vpc.internal   Running             Running 7 seconds ago
k5lmmmu44yh3        echo-to-efs.10      alpine:latest       ip-10-82-48-187.vpc.internal   Running             Running 8 seconds ago
  • Now we can verify if the file is being written...
[ec2-user@ip-10-82-48-59 ~]$ docker service logs echo-to-efs
echo-to-efs.1.8c8lxqcoc3n5@ip-10-82-48-59.vpc.internal     | writing from 3da21bc313f7
echo-to-efs.3.rmj163xu97ai@ip-10-82-48-96.vpc.internal     | writing from 3da21bc313f7
echo-to-efs.1.8c8lxqcoc3n5@ip-10-82-48-59.vpc.internal     | writing from 334b26ce3747
echo-to-efs.7.bzoafno0gb4d@ip-10-82-48-74.vpc.internal     | writing from 3da21bc313f7
echo-to-efs.7.bzoafno0gb4d@ip-10-82-48-74.vpc.internal     | writing from 334b26ce3747
echo-to-efs.3.rmj163xu97ai@ip-10-82-48-96.vpc.internal     | writing from 334b26ce3747
echo-to-efs.1.8c8lxqcoc3n5@ip-10-82-48-59.vpc.internal     | writing from fcefd6eb5035
echo-to-efs.1.8c8lxqcoc3n5@ip-10-82-48-59.vpc.internal     | writing from 84ffd56fdcfa
echo-to-efs.1.8c8lxqcoc3n5@ip-10-82-48-59.vpc.internal     | writing from b7ecbd01debe
echo-to-efs.1.8c8lxqcoc3n5@ip-10-82-48-59.vpc.internal     | writing from e5ef2a2821a2
echo-to-efs.1.8c8lxqcoc3n5@ip-10-82-48-59.vpc.internal     | writing from 5179f81f04f9
echo-to-efs.1.8c8lxqcoc3n5@ip-10-82-48-59.vpc.internal     | writing from 5e859dce0c2f
echo-to-efs.7.bzoafno0gb4d@ip-10-82-48-74.vpc.internal     | writing from fcefd6eb5035
echo-to-efs.3.rmj163xu97ai@ip-10-82-48-96.vpc.internal     | writing from fcefd6eb5035
echo-to-efs.7.bzoafno0gb4d@ip-10-82-48-74.vpc.internal     | writing from 84ffd56fdcfa
echo-to-efs.7.bzoafno0gb4d@ip-10-82-48-74.vpc.internal     | writing from b7ecbd01debe
echo-to-efs.3.rmj163xu97ai@ip-10-82-48-96.vpc.internal     | writing from 84ffd56fdcfa
echo-to-efs.3.rmj163xu97ai@ip-10-82-48-96.vpc.internal     | writing from b7ecbd01debe
echo-to-efs.3.rmj163xu97ai@ip-10-82-48-96.vpc.internal     | writing from e5ef2a2821a2
echo-to-efs.4.5jtao2yu5udt@ip-10-82-48-95.vpc.internal     | writing from 3da21bc313f7
echo-to-efs.3.rmj163xu97ai@ip-10-82-48-96.vpc.internal     | writing from 5179f81f04f9
echo-to-efs.7.bzoafno0gb4d@ip-10-82-48-74.vpc.internal     | writing from e5ef2a2821a2
echo-to-efs.3.rmj163xu97ai@ip-10-82-48-96.vpc.internal     | writing from 5e859dce0c2f
...
...
  • Now see the current value in the file.
  • The command here will mount the volume and at the same and then read the file from EFS mount.
  • It does take a long while until the container prepares the mount... Like 2min... Not sure if that's expected.
 [ec2-user@ip-10-82-48-59 ~]$ docker run -ti --volume-driver=rexray/efs -v echo_data:/test alpine cat /test/echo.log
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
Digest: sha256:1072e499f3f655a032e88542330cf75b02e7bdf673278f701d7ba61629ee3ebe
Status: Downloaded newer image for alpine:latest
writing from 3da21bc313f7
writing from 334b26ce3747
writing from fcefd6eb5035
writing from 84ffd56fdcfa
writing from b7ecbd01debe
writing from e5ef2a2821a2
writing from 5179f81f04f9
writing from 5e859dce0c2f
writing from af94a5a29b4a
writing from a3f860caaf03
writing from b7b3d30d2f8e
writing from cf65a52760c0
writing from a551b529e599
writing from 0533784ce9e5
writing from d78f781baa4d
writing from 06b84229ec32
writing from 4e3974cfb8a9
writing from 007cd66062e7
writing from 665f76589ae2
writing from 68b04985559a
writing from 9a2e19e110fb
writing from 1ed7903abf86
writing from 763c7eea52fb
writing from 529c7fff9c71
writing from 7172d5fb56d9
writing from c33294ce8d9d
writing from b95a5b5f1bc1
writing from ff269d8a78ba
writing from b1299af12f30
writing from 4f970a6e234d
writing from a4a1f331b3c9
writing from bb3915d9d242
writing from 0fbafba12a0c
writing from 017f70a769da
writing from 8d657b23571f
writing from 570e2fa76e16
writing from 45a92f3d19b4
writing from 3b02cb3675e4
writing from e2d49358df6f

Docker Volumes Status

  • Docker volumes now maps all the Mounts with the Plugins
  • This is a major win for me!
[ec2-user@ip-10-82-48-59 ~]$ docker volume ls
DRIVER               VOLUME NAME
rexray/s3fs:latest   7820-3357-5600-us-west-2
rexray/s3fs:latest   cf-templates-8hmsxuf2muaa-us-west-2
rexray/efs:latest    echo_data
rexray/s3fs:latest   marcello-12345
rexray/s3fs:latest   marcello-testing-bucket-1
rexray/s3fs:latest   marcello-testing-bucket-2
rexray/s3fs:latest   new-config-publisher
rexray/s3fs:latest   publisher.intuit-caas
rexray/s3fs:latest   publisher.intuit-caas-new-update
rexray/ebs:latest    quay-registry-configs
rexray/ebs:latest    quay-registry-images-datastore
rexray/ebs:latest    quay-registry-mysql-datastore
rexray/ebs:latest    registry_configs
rexray/ebs:latest    registry_mysql-data
rexray/s3fs:latest   swarmbucket-vpc-155e5072
@joernott

This comment has been minimized.

Copy link

joernott commented Mar 13, 2018

I have a similar problem on CentOS 7.4 with rexray/rbd 0.11.2 and docker 17.12.1-ce in swarm mode.

rexray volume ls works as well as docker volume ls. However, docker plugin ls shows the plugin as disabled. I can find the socket as /run/docker/plugins/rexray.sock, but when I try to enable the plugin, docker complains, that it can't find the socket under /run/docker/plugins/2f6be6e7a907dfce1339211c1974987cadb14d0e66c32f92782f66fd2bfc89f.

When I create that folder, >I don't get the error message when I activate the plugin and it is shown as ENABLED "true". After a while it turns back to "false".

I fixed that by updating docker to 18.02.0-ce. The plugin ist still showing up as ENABLED=false but it can be used by the swarm.

@clintkitson

This comment has been minimized.

Copy link
Member

clintkitson commented Mar 13, 2018

Is it possible that you have REX-Ray running in two separate modes? You can launch REX-Ray in standalone mode by running rexray, and also as a managed plugin with docker plugin install. One of these modes should be chosen, otherwise Docker will be confused with which socket file to communicate with.

@joernott

This comment has been minimized.

Copy link

joernott commented Mar 20, 2018

This actually was the case as I used to run rexray as systemd service. However, updating docker to 17.12 broke the setup, 18.02 (which is not in the stable branch) works with the latest rexray, so this is a working combination. I will try to switch to the rexray-plugin on a new machine in the cluister without starting rexray as service, as soon as I find the time.

@dradux

This comment has been minimized.

Copy link

dradux commented Apr 2, 2018

@clintkitson , dont know if this helps but I ran into this same issue using a stock Ubuntu 16.04 LTS AMI using docker from the repos (v17.12.1-ce). I uninstalled docker and installed following docker.com instructions (note: I installed v17.03.2 due to other requirements) and the rexray plugin installed successfully after that.

@Lindenk

This comment has been minimized.

Copy link

Lindenk commented Jun 11, 2018

I'm also having this problem on every arch linux installation (x86 and arm) that I have (around 4+). Docker version on each is 18.05.0-ce, build f150324782. I also get the same error when attempting install on a stock vagrant image of ubuntu/bionic64 (17.12.1-ce, build 7390fc6).

@holms

This comment has been minimized.

Copy link

holms commented Aug 13, 2018

@clintkitson this should be mentioned in documentation! I've spent hours to investigate this issue, and this is an issue only on ONE node from whole cluster. Now I feel like an idiot.

@clintkitson

This comment has been minimized.

Copy link
Member

clintkitson commented Aug 13, 2018

@holms, can you describe what your issue more? There are a few things mixed together in this thread so its hard to understand exactly what you think is missing.

@holms

This comment has been minimized.

Copy link

holms commented Aug 13, 2018

@holms

This comment has been minimized.

Copy link

holms commented Aug 14, 2018

Ok, now only docker plugin rexray/dobs is installed and i've got same problem

TASK [rexray : Install Docker dobs driver] ************************************************************************************************************************************************************************************************************************************************************************************
changed: [178.62.xx]
changed: [159.65.xx]
changed: [167.99.xx]
changed: [206.189.xx]
changed: [142.93.xx]
changed: [142.93.xx]
fatal: [174.138.14.xx]: FAILED! => {"changed": true, "cmd": "docker plugin install --grant-all-permissions=true rexray/dobs", "delta": "0:00:46.361994", "end": "2018-08-14 12:08:41.139230", "msg": "non-zero return code", "rc": 1, "start": "2018-08-14 12:07:54.777236", "stderr": "Error response from daemon: dial unix /run/docker/plugins/f98f1c10ebec7d7a466bb443f4c860e8b52b5f7d36ef9bde5a43fafa15ef8206/rexray.sock: connect: no such file or directory", "stderr_lines": ["Error response from daemon: dial unix /run/docker/plugins/f98f1c10ebec7d7a466bb443f4c860e8b52b5f7d36ef9bde5a43fafa15ef8206/rexray.sock: connect: no such file or directory"], "stdout": "latest: Pulling from rexray/dobs\n89d5fc3414e7: Verifying Checksum\n89d5fc3414e7: Download complete\nDigest: sha256:eebf6abccef4486099446a62f994dd7099440d7b972cb16077008a4ab3656668\nStatus: Downloaded newer image for rexray/dobs:latest", "stdout_lines": ["latest: Pulling from rexray/dobs", "89d5fc3414e7: Verifying Checksum", "89d5fc3414e7: Download complete", "Digest: sha256:eebf6abccef4486099446a62f994dd7099440d7b972cb16077008a4ab3656668", "Status: Downloaded newer image for rexray/dobs:latest"]}
@holms

This comment has been minimized.

Copy link

holms commented Aug 14, 2018

Ok solved. Somehow one of the hosts had missing folders, and I have no idea why. Same OS, same build, same resources, same automation, same everything.. and yet 4 more dirs been created there, even after full recreation of that node:

sudo mkdir -p /var/lib/libstorage ;
sudo mkdir -p /var/lib/rexray ;
sudo mkdir -p /var/log/rexray ;
sudo mkdir -p /var/run/rexray ;
sudo mkdir -p /var/run/libstorage

Might save some time for someone

@RothAndrew

This comment has been minimized.

Copy link

RothAndrew commented Nov 15, 2018

Re: #820 (comment)

Using s3:AmazonS3FullAccess did not work for me. However, s3:* did work.

@mcgitty

This comment has been minimized.

Copy link

mcgitty commented Nov 29, 2018

I have this issue simply installing a rexray/efs in docker for mac:

docker --version
Docker version 18.06.1-ce, build e68fc7a

docker plugin install rexray/efs
Plugin "rexray/efs" is requesting the following privileges:
...
latest: Pulling from rexray/efs
369238b93d78: Download complete
Digest: sha256:309a4d0281addde8d94c6f5c08c940b106c4d163b7350c3ce226ce143fe897c6
Status: Downloaded newer image for rexray/efs:latest
Error response from daemon: dial unix /run/docker/plugins/38878189e2c7a16c162c4aaec2b785251ba5fbca766ae38229395f7218a9b75d/rexray.sock: connect: no such file or directory

@anthonyloukinas

This comment has been minimized.

Copy link

anthonyloukinas commented Dec 1, 2018

OS: Ubuntu 16.04.5 LTS
Kernel: 4.4.0-139-generic
Docker Version: 18.09.0, build 4d60db4

user@docker01:~$ docker plugin install rexray/s3fs
Plugin "rexray/s3fs" is requesting the following privileges:
 - network: [host]
 - mount: [/dev]
 - allow-all-devices: [true]
 - capabilities: [CAP_SYS_ADMIN]
Do you grant the above permissions? [y/N] y
latest: Pulling from rexray/s3fs
b5ce7aed9a2b: Download complete
Digest: sha256:f06a8d9e64e6d616298646082c33252703af807b274288643e9668789d06cd1e
Status: Downloaded newer image for rexray/s3fs:latest
Error response from daemon: dial unix /run/docker/plugins/5a940368177b92ef879f8530bfc32ea81b6398ceb11a08f27092fd6dda334d85/rexray.sock: connect: no such file or directory

Edit: I solved this problem by actualling installing the Rexray binaries from the frontpage of their website. I didn't realize this was required (initially found rexray as a volume driver where they didn't talk about the binaries being required).

source: https://rexray.readthedocs.io/en/latest/#getting-started

curl -sSL https://rexray.io/install | sh

Once you've installed the binary, in order for me to have gotten it to work i had to install s3fs on my linux distribution. For ubuntu/debian

apt install s3fs -y
@mcgitty

This comment has been minimized.

Copy link

mcgitty commented Dec 3, 2018

You have to set the access and secret key when you install the plugin.

docker plugin install rexray/s3fs S3FS_ACCESSKEY=xxx S3FS_SECRETKEY=yyy

I considered this a design flaw and reported here: #1285

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment