Skip to content
This repository has been archived by the owner on Nov 9, 2020. It is now read-only.

'Docker volume ls' command throws error after upgrading from 0.11.1 build to 0.12 build. #981

Closed
ashahi1 opened this issue Mar 1, 2017 · 5 comments
Closed

'Docker volume ls' command throws error after upgrading from 0.11.1 build to 0.12 build. #981

ashahi1 opened this issue Mar 1, 2017 · 5 comments
Labels
kind/bug P0

Comments

@ashahi1
Copy link
Contributor

ashahi1 commented Mar 1, 2017 •
edited

  1. Installed 0.11.1 build on vm and esx.
root@photon-xVmYMbyTn [ /tmp ]# ls
docker-volume-vsphere-0.11.1.f1792d6-1.x86_64.rpm
systemd-private-7e70dce8a837484da28dc3ef7e39a2d9-systemd-timesyncd.service-8Adao                                                                                                                     I
vgauthsvclog.txt.0
vmware-root
root@photon-xVmYMbyTn [ /tmp ]# rpm -ivh docker-volume-vsphere-0.11.1.f1792d6-1.                                                                                                                     x86_64.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:docker-volume-vsphere-0:0.11.1.f1################################# [100%]
  File: '/proc/1/exe' -> '/usr/lib/systemd/systemd'
Created symlink from /etc/systemd/system/multi-user.target.wants/docker-volume-v                                                                                                                     sphere.service to /usr/lib/systemd/system/docker-volume-vsphere.service.
  1. Created a volume - vol2511
root@photon-xVmYMbyTn [ ~ ]# docker volume create --driver=vmdk --name=vol2511 -o size=100mb
vol2511
root@photon-xVmYMbyTn [ ~ ]# docker volume inspect vol2511
[
    {
        "Name": "vol2511",
        "Driver": "vmdk",
        "Mountpoint": "/mnt/vmdk/vol2511",
        "Status": {
            "access": "read-write",
            "attach-as": "independent_persistent",
            "capacity": {
                "allocated": "13MB",
                "size": "100MB"
            },
            "clone-from": "None",
            "created": "Wed Mar  1 20:34:10 2017",
            "created by VM": "photon-VM1.3",
            "datastore": "sharedVmfs-0",
            "diskformat": "thin",
            "fstype": "ext4",
            "status": "detached"
        },
        "Labels": {},
        "Scope": "global"
    }
]
  1. Downloaded and upgraded to 0.12 build (latest)
root@photon-xVmYMbyTn [ /tmp ]# ls
docker-volume-vsphere-0.12.3f64805-1.x86_64.rpm  systemd-private-7e70dce8a837484da28dc3ef7e39a2d9-systemd-timesyncd.service-8AdaoI  vgauthsvclog.txt.0  vmware-root
root@photon-xVmYMbyTn [ /tmp ]# rpm -Uvh docker-volume-vsphere-0.12.3f64805-1.x86_64.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:docker-volume-vsphere-0:0.12.3f64################################# [ 50%]
  File: '/proc/1/exe' -> '/usr/lib/systemd/systemd'
  File: '/proc/1/exe' -> '/usr/lib/systemd/systemd'
Removed symlink /etc/systemd/system/multi-user.target.wants/docker-volume-vsphere.service.
Cleaning up / removing...
   2:docker-volume-vsphere-0:0.11.1.f1################################# [100%]
  File: '/proc/1/exe' -> '/usr/lib/systemd/systemd'

From ESX:

[root@sc2-rdops-vm05-dhcp-180-144:~] esxcli software vib install --no-sig-check -v /tmp/vmware-esx-vmdkops-0.12.3f64805.vib
Installation Result
   Message: Operation finished successfully.
   Reboot Required: false
   VIBs Installed: VMWare_bootbank_esx-vmdkops-service_0.12.3f64805-0.0.1
   VIBs Removed: VMWare_bootbank_esx-vmdkops-service_0.11.1.f1792d6-0.0.1
   VIBs Skipped:
  1. Tried running 'docker volume ls' but it throws error -
root@photon-xVmYMbyTn [ ~ ]# docker volume ls
list vmdk: Post http://%2Frun%2Fdocker%2Fplugins%2Fvmdk.sock/VolumeDriver.List: dial unix /run/docker/plugins/vmdk.sock: connect: no such file or directory
DRIVER              VOLUME NAME
vmdk                vol251
vmdk                vol2511

docker-volume-vsphere service was dead:

root@photon-xVmYMbyTn [ ~ ]# systemctl status docker-volume-vsphere
● docker-volume-vsphere.service - "Docker Volume Driver for vSphere"
   Loaded: loaded (/usr/lib/systemd/system/docker-volume-vsphere.service; disabled; vendor preset: enabled)
   Active: inactive (dead)

Mar 01 20:28:41 photon-xVmYMbyTn docker-volume-vsphere[716]: rflags 0x206
Mar 01 20:28:41 photon-xVmYMbyTn docker-volume-vsphere[716]: cs     0x33
Mar 01 20:28:41 photon-xVmYMbyTn docker-volume-vsphere[716]: fs     0x0
Mar 01 20:28:41 photon-xVmYMbyTn docker-volume-vsphere[716]: gs     0x0
Mar 01 20:28:41 photon-xVmYMbyTn systemd[1]: docker-volume-vsphere.service: Service hold-off time over, scheduling restart.
Mar 01 20:28:41 photon-xVmYMbyTn systemd[1]: Stopped "Docker Volume Driver for vSphere".
Mar 01 20:28:41 photon-xVmYMbyTn systemd[1]: Started "Docker Volume Driver for vSphere".
Mar 01 20:48:07 photon-xVmYMbyTn systemd[1]: Started "Docker Volume Driver for vSphere".
Mar 01 20:48:07 photon-xVmYMbyTn systemd[1]: Stopping "Docker Volume Driver for vSphere"...
Mar 01 20:48:07 photon-xVmYMbyTn systemd[1]: Stopped "Docker Volume Driver for vSphere".

Attached logs:
docker-volume-vsphere.txt
vmdk_ops.txt
@ashahi1
Copy link
Contributor Author

ashahi1 commented Mar 1, 2017

Docker Engine log:

root@photon-xVmYMbyTn [ ~ ]# journalctl -u docker.service
-- Logs begin at Wed 2017-03-01 19:54:24 UTC, end at Wed 2017-03-01 21:43:54 UTC. --
Mar 01 20:04:25 photon-xVmYMbyTn systemd[1]: Stopped Docker Daemon.
Mar 01 20:04:25 photon-xVmYMbyTn systemd[1]: Starting Docker Daemon...
Mar 01 20:04:25 photon-xVmYMbyTn docker[371]: time="2017-03-01T20:04:25.130424264Z" level=error msg="devmapper: Udev sync is not supported. This will lead to data loss and unexpected behavior. Inst
all a dynamic binary to use devicemapper or select a different storage driver. For more information, see https://docs.docker.com/engine/reference/commandline/daemon/#daemon-storage-driver-option"
Mar 01 20:04:25 photon-xVmYMbyTn docker[371]: time="2017-03-01T20:04:25.194567708Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Mar 01 20:04:25 photon-xVmYMbyTn docker[371]: time="2017-03-01T20:04:25.194964491Z" level=warning msg="Your kernel does not support kernel memory limit."
Mar 01 20:04:25 photon-xVmYMbyTn docker[371]: time="2017-03-01T20:04:25.195120145Z" level=warning msg="Your kernel does not support cgroup blkio weight"
Mar 01 20:04:25 photon-xVmYMbyTn docker[371]: time="2017-03-01T20:04:25.195247516Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
Mar 01 20:04:25 photon-xVmYMbyTn docker[371]: time="2017-03-01T20:04:25.195405438Z" level=warning msg="Your kernel does not support cgroup blkio throttle.read_bps_device"
Mar 01 20:04:25 photon-xVmYMbyTn docker[371]: time="2017-03-01T20:04:25.195586111Z" level=warning msg="Your kernel does not support cgroup blkio throttle.write_bps_device"
Mar 01 20:04:25 photon-xVmYMbyTn docker[371]: time="2017-03-01T20:04:25.195715699Z" level=warning msg="Your kernel does not support cgroup blkio throttle.read_iops_device"
Mar 01 20:04:25 photon-xVmYMbyTn docker[371]: time="2017-03-01T20:04:25.195850692Z" level=warning msg="Your kernel does not support cgroup blkio throttle.write_iops_device"
Mar 01 20:04:25 photon-xVmYMbyTn docker[371]: time="2017-03-01T20:04:25.196037800Z" level=warning msg="mountpoint for pids not found"
Mar 01 20:04:25 photon-xVmYMbyTn docker[371]: time="2017-03-01T20:04:25.196462576Z" level=info msg="Loading containers: start."
Mar 01 20:04:25 photon-xVmYMbyTn docker[371]: time="2017-03-01T20:04:25.221915645Z" level=info msg="Firewalld running: false"
Mar 01 20:04:25 photon-xVmYMbyTn docker[371]: time="2017-03-01T20:04:25.312406452Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be
 used to set a preferred IP address"
Mar 01 20:04:25 photon-xVmYMbyTn docker[371]: time="2017-03-01T20:04:25.368266235Z" level=info msg="Loading containers: done."
Mar 01 20:04:25 photon-xVmYMbyTn docker[371]: time="2017-03-01T20:04:25.368542542Z" level=info msg="Daemon has completed initialization"
Mar 01 20:04:25 photon-xVmYMbyTn docker[371]: time="2017-03-01T20:04:25.368678160Z" level=info msg="Docker daemon" commit=78d1802 graphdriver=overlay version=1.12.6
Mar 01 20:04:25 photon-xVmYMbyTn systemd[1]: Started Docker Daemon.
Mar 01 20:04:25 photon-xVmYMbyTn docker[371]: time="2017-03-01T20:04:25.374509380Z" level=info msg="API listen on /var/run/docker.sock"
Mar 01 20:04:25 photon-xVmYMbyTn systemd[1]: Stopping Docker Daemon...
Mar 01 20:04:25 photon-xVmYMbyTn docker[371]: time="2017-03-01T20:04:25.828474992Z" level=info msg="Processing signal 'terminated'"
Mar 01 20:04:25 photon-xVmYMbyTn systemd[1]: Stopped Docker Daemon.
Mar 01 20:04:25 photon-xVmYMbyTn systemd[1]: Starting Docker Daemon...
Mar 01 20:04:25 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:04:25.862375686Z" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting -tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOIN
G [!]"
Mar 01 20:04:25 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:04:25.865795928Z" level=info msg="[graphdriver] using prior storage driver \"overlay\""
Mar 01 20:04:25 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:04:25.870231131Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Mar 01 20:04:25 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:04:25.870395973Z" level=warning msg="Your kernel does not support kernel memory limit."
Mar 01 20:04:25 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:04:25.870434249Z" level=warning msg="Your kernel does not support cgroup blkio weight"
Mar 01 20:04:25 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:04:25.870448349Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
Mar 01 20:04:25 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:04:25.870456466Z" level=warning msg="Your kernel does not support cgroup blkio throttle.read_bps_device"
Mar 01 20:04:25 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:04:25.870463724Z" level=warning msg="Your kernel does not support cgroup blkio throttle.write_bps_device"
Mar 01 20:04:25 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:04:25.870471216Z" level=warning msg="Your kernel does not support cgroup blkio throttle.read_iops_device"
Mar 01 20:04:25 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:04:25.870483559Z" level=warning msg="Your kernel does not support cgroup blkio throttle.write_iops_device"
Mar 01 20:04:25 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:04:25.870534598Z" level=warning msg="mountpoint for pids not found"
Mar 01 20:04:25 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:04:25.870687377Z" level=info msg="Loading containers: start."
Mar 01 20:04:25 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:04:25.874682799Z" level=info msg="Firewalld running: false"
Mar 01 20:04:25 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:04:25.986921057Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be
 used to set a preferred IP address"
Mar 01 20:04:26 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:04:26.026079240Z" level=info msg="Loading containers: done."
Mar 01 20:04:26 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:04:26.026211990Z" level=info msg="Daemon has completed initialization"
Mar 01 20:04:26 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:04:26.026236402Z" level=info msg="Docker daemon" commit=78d1802 graphdriver=overlay version=1.12.6
Mar 01 20:04:26 photon-xVmYMbyTn systemd[1]: Started Docker Daemon.
Mar 01 20:04:26 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:04:26.038784593Z" level=info msg="API listen on /var/run/docker.sock"
Mar 01 20:04:26 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:04:26.038866928Z" level=info msg="API listen on [::]:2375"
Mar 01 20:28:41 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:28:41.782698740Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/vmdk.sock/VolumeDriver.Create: Post http://%2
Frun%2Fdocker%2Fplugins%2Fvmdk.sock/VolumeDriver.Create: EOF, retrying in 1s"
Mar 01 20:28:42 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:28:42.783896153Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/vmdk.sock/VolumeDriver.Create: Post http://%2
Frun%2Fdocker%2Fplugins%2Fvmdk.sock/VolumeDriver.Create: http: ContentLength=42 with Body length 0, retrying in 2s"
Mar 01 20:28:44 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:28:44.784568516Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/vmdk.sock/VolumeDriver.Create: Post http://%2
Frun%2Fdocker%2Fplugins%2Fvmdk.sock/VolumeDriver.Create: http: ContentLength=42 with Body length 0, retrying in 4s"
Mar 01 20:28:48 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:28:48.785189568Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/vmdk.sock/VolumeDriver.Create: Post http://%2
Frun%2Fdocker%2Fplugins%2Fvmdk.sock/VolumeDriver.Create: http: ContentLength=42 with Body length 0, retrying in 8s"
Mar 01 20:28:56 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:28:56.785730276Z" level=error msg="Handler for POST /v1.24/volumes/create returned error: create vol251: Post http://%2Frun%2Fdocke
r%2Fplugins%2Fvmdk.sock/VolumeDriver.Create: http: ContentLength=42 with Body length 0"
Mar 01 20:48:18 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:48:18.226813248Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/vmdk.sock/VolumeDriver.List: Post http://%2Fr
un%2Fdocker%2Fplugins%2Fvmdk.sock/VolumeDriver.List: dial unix /run/docker/plugins/vmdk.sock: connect: no such file or directory, retrying in 1s"
Mar 01 20:48:19 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:48:19.227425742Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/vmdk.sock/VolumeDriver.List: Post http://%2Fr
un%2Fdocker%2Fplugins%2Fvmdk.sock/VolumeDriver.List: dial unix /run/docker/plugins/vmdk.sock: connect: no such file or directory, retrying in 2s"
Mar 01 20:48:21 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:48:21.227884821Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/vmdk.sock/VolumeDriver.List: Post http://%2Fr
un%2Fdocker%2Fplugins%2Fvmdk.sock/VolumeDriver.List: dial unix /run/docker/plugins/vmdk.sock: connect: no such file or directory, retrying in 4s"
Mar 01 20:48:25 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:48:25.228268237Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/vmdk.sock/VolumeDriver.List: Post http://%2Fr
un%2Fdocker%2Fplugins%2Fvmdk.sock/VolumeDriver.List: dial unix /run/docker/plugins/vmdk.sock: connect: no such file or directory, retrying in 8s"
Mar 01 20:48:33 photon-xVmYMbyTn docker[550]: time="2017-03-01T20:48:33.228742857Z" level=warning msg="list vmdk: Post http://%2Frun%2Fdocker%2Fplugins%2Fvmdk.sock/VolumeDriver.List: dial unix /run
/docker/plugins/vmdk.sock: connect: no such file or directory"
Mar 01 21:42:21 photon-xVmYMbyTn docker[550]: time="2017-03-01T21:42:21.487659497Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/vmdk.sock/VolumeDriver.List: Post http://%2Fr
un%2Fdocker%2Fplugins%2Fvmdk.sock/VolumeDriver.List: dial unix /run/docker/plugins/vmdk.sock: connect: no such file or directory, retrying in 1s"
Mar 01 21:42:22 photon-xVmYMbyTn docker[550]: time="2017-03-01T21:42:22.488057624Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/vmdk.sock/VolumeDriver.List: Post http://%2Fr
un%2Fdocker%2Fplugins%2Fvmdk.sock/VolumeDriver.List: dial unix /run/docker/plugins/vmdk.sock: connect: no such file or directory, retrying in 2s"
Mar 01 21:42:24 photon-xVmYMbyTn docker[550]: time="2017-03-01T21:42:24.488501011Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/vmdk.sock/VolumeDriver.List: Post http://%2Fr
un%2Fdocker%2Fplugins%2Fvmdk.sock/VolumeDriver.List: dial unix /run/docker/plugins/vmdk.sock: connect: no such file or directory, retrying in 4s"
Mar 01 21:42:28 photon-xVmYMbyTn docker[550]: time="2017-03-01T21:42:28.489094166Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/vmdk.sock/VolumeDriver.List: Post http://%2Fr
un%2Fdocker%2Fplugins%2Fvmdk.sock/VolumeDriver.List: dial unix /run/docker/plugins/vmdk.sock: connect: no such file or directory, retrying in 8s"
Mar 01 21:42:36 photon-xVmYMbyTn docker[550]: time="2017-03-01T21:42:36.489721382Z" level=warning msg="list vmdk: Post http://%2Frun%2Fdocker%2Fplugins%2Fvmdk.sock/VolumeDriver.List: dial unix /run
/docker/plugins/vmdk.sock: connect: no such file or directory"
Mar 01 21:43:39 photon-xVmYMbyTn docker[550]: time="2017-03-01T21:43:39.836562373Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/vmdk.sock/VolumeDriver.List: Post http://%2Fr
un%2Fdocker%2Fplugins%2Fvmdk.sock/VolumeDriver.List: dial unix /run/docker/plugins/vmdk.sock: connect: no such file or directory, retrying in 1s"
Mar 01 21:43:40 photon-xVmYMbyTn docker[550]: time="2017-03-01T21:43:40.837072462Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/vmdk.sock/VolumeDriver.List: Post http://%2Fr
un%2Fdocker%2Fplugins%2Fvmdk.sock/VolumeDriver.List: dial unix /run/docker/plugins/vmdk.sock: connect: no such file or directory, retrying in 2s"
Mar 01 21:43:42 photon-xVmYMbyTn docker[550]: time="2017-03-01T21:43:42.837522953Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/vmdk.sock/VolumeDriver.List: Post http://%2Fr
un%2Fdocker%2Fplugins%2Fvmdk.sock/VolumeDriver.List: dial unix /run/docker/plugins/vmdk.sock: connect: no such file or directory, retrying in 4s"
Mar 01 21:43:46 photon-xVmYMbyTn docker[550]: time="2017-03-01T21:43:46.838109608Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/vmdk.sock/VolumeDriver.List: Post http://%2Fr
un%2Fdocker%2Fplugins%2Fvmdk.sock/VolumeDriver.List: dial unix /run/docker/plugins/vmdk.sock: connect: no such file or directory, retrying in 8s"
Mar 01 21:43:54 photon-xVmYMbyTn docker[550]: time="2017-03-01T21:43:54.838672174Z" level=warning msg="list vmdk: Post http://%2Frun%2Fdocker%2Fplugins%2Fvmdk.sock/VolumeDriver.List: dial unix /run
/docker/plugins/vmdk.sock: connect: no such file or directory"
root@photon-xVmYMbyTn [ ~ ]#

@pdhamdhere
Copy link
Contributor

Driver name in 0.11 is "vmdk".
Driver name in 0.12 is "vsphere". However, Docker still looking for "vmdk.sock". So, with 0.12 install, docker needs restart and it will pick up "vsphere.sock". Will update release note.

CC/ @govint

@pdhamdhere
Copy link
Contributor

Updated Release notes that 0.12 install requires docker service restart.

@govint
Copy link
Contributor

govint commented Mar 2, 2017

Whenever the plugin pkg is installed docker is restarted as well. Looks like upgrade doesn't restart docker?

@govint
Copy link
Contributor

govint commented Mar 2, 2017

We need to quickly move to the "managed plugin" format of Docker 1.13. Can't restart docker just because of a plugin install or upgrade - #886 may need to come in sooner.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
kind/bug P0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@govint @pdhamdhere @ashahi1