[Tenant] Inconsistent docker volume ls output in case of orphaned VMs

[ashahi1]

Do not see consistent output for 'docker volume ls' command across vms after _DEFAULT tenant is deleted.

Setup:
One ESX,
Two VMs (TestVM1-125 and TestVM2-125)
VSAN datastore.
Both VMs belong to same vsan datastore and same esx.

Steps:

1. Created a docker volume named testVolVsan from vm2 (TestVM2-125).

root@sc-rdops-vm02-dhcp-52-237:~# docker volume create --driver=vsphere --name=testVolVsan -o size=200mb
testVolVsan
root@sc-rdops-vm02-dhcp-52-237:~# docker volume ls
DRIVER              VOLUME NAME
vsphere             testVolVsan@vsanDatastore
vsphere             vol1@sharedVmfs-0
vsphere             vol2@sharedVmfs-0

Docker volume ls command from VM1

root@sc-rdops-vm02-dhcp-52-237:~# docker volume ls
DRIVER              VOLUME NAME
vsphere             testVolVsan@vsanDatastore
vsphere             vol1@sharedVmfs-0
vsphere             vol2@sharedVmfs-0

2. Admin cli command output listing volumes and default tenant .

[root@sc-rdops-vm18-dhcp-55-125:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume       Datastore      Created By VM  Created                   Attached To VM (name/uuid)  Policy          Capacity  Used   Disk Format  Filesystem Type  Access      Attach As
-----------  -------------  -------------  ------------------------  --------------------------  --------------  --------  -----  -----------  ---------------  ----------  ----------------------
vol1         sharedVmfs-0   ubuntu-VM0.0   Fri Mar  3 19:21:15 2017  detached                    N/A             10GB      145MB  thin         ext4             read-write  independent_persistent
vol2         sharedVmfs-0   ubuntu-VM1.0   Fri Mar  3 19:21:34 2017  detached                    N/A             10GB      145MB  thin         ext4             read-write  independent_persistent
testVolVsan  vsanDatastore  TestVM2-125    Fri Mar  3 19:55:00 2017  detached                    [VSAN default]  200MB     96MB   thin         ext4             read-write  independent_persistent

[root@sc-rdops-vm18-dhcp-55-125:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py tenant ls
Uuid                                  Name      Description               Default_datastore  VM_list
------------------------------------  --------  ------------------------  -----------------  -------
11111111-1111-1111-1111-111111111111  _DEFAULT  This is a default tenant

3. Then deleted the default tenant. Do not see any volumes or default tenant

[root@sc-rdops-vm18-dhcp-55-125:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py tenant rm --name=_DEFAULT
tenant rm succeeded
[root@sc-rdops-vm18-dhcp-55-125:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py tenant ls
Uuid  Name  Description  Default_datastore  VM_list
----  ----  -----------  -----------------  -------


[root@sc-rdops-vm18-dhcp-55-125:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume  Datastore  Created By VM  Created  Attached To VM (name/uuid)  Policy  Capacity  Used  Disk Format  Filesystem Type  Access  Attach As
------  ---------  -------------  -------  --------------------------  ------  --------  ----  -----------  ---------------  ------  ---------

4. Do not see same output for for docker volume ls command.

'docker volume ls' command from VM1

root@sc-rdops-vm02-dhcp-52-237:~# docker volume ls
list vsphere: VolumeDriver.List: VM TestVM1-125 does not belong to any tenant
DRIVER              VOLUME NAME

'docker volume ls' command from VM2

root@sc-rdops-vm02-dhcp-52-237:~# docker volume ls
list vsphere: VolumeDriver.List: VM TestVM2-125 does not belong to any tenant
DRIVER              VOLUME NAME
vsphere             testVolVsan
vsphere             testVolVsan@vsanDatastore
root@sc-rdops-vm02-dhcp-52-237:~#

Logs:
docker-volume-vsphere-VM2.txt
vmdk_ops.txt
docker-volume-vsphere-VM1.txt

[ashahi1]

CC / @pshahzeb

[ashahi1]

Also, after deleting the tenant, we see two volumes being listed on the vm if we do 'docker volume ls' - one short name and another long name.

'docker volume ls' command from VM2

root@sc-rdops-vm02-dhcp-52-237:~# docker volume ls
list vsphere: VolumeDriver.List: VM TestVM2-125 does not belong to any tenant
DRIVER              VOLUME NAME
vsphere             testVolVsan  <<<<<<<<<<<<<<<<<<<<<<< 1
vsphere             testVolVsan@vsanDatastore <<<<<<<<<<<<<<<< 2
root@sc-rdops-vm02-dhcp-52-237:~#

[msterin]

After deleting _DEFAULT tenant the following should happen by design:

• All Vms which do not belong to any tenant (i.e. the ones formerly in _DEFAULT) should be denied access to anything. 'docker volume ls' should say warning: VM XXX does not belong to any tenant and show empty list. docker volume create should fail
  //CC @lipingxue

[msterin]

@pshahzeb can you decipher the title and be clear on what's wrong compared to what was expected ? Currently it requires reading the lots of text just to figure it out.

it looks to me you are saying two different things:

1. After removing _DEFAULT tenant, volumes dissapear form 'vmdkops_admin ls' command. , This is a clear bug, admin command should show all volumes. If the volumes do not belong to ANY tenant. we still should show them , I'd suggest marking tenant name (i.e. folder name) with *. This code does not exist (to mark stuff with *) but all in all this is a bug
2. After removing _DEFAULT tenant. docker volume ls command still shows some volumes.
   You need to restart docker service , most likely it caches the list. If after docker restart all is as expected, this is most likely a docker bug. We do need to check what do we return to tdocker on List() when volumes are hidden by access control.

[pshahzeb]

@msterin Yes. The two things are apt as pointed.
This issue focuses on the second point.
The first part is being handled as part of #1043