Skip to content
This repository has been archived by the owner on Mar 26, 2020. It is now read-only.

failed to create volumes more than 600 #1364

Open
Madhu-1 opened this issue Dec 5, 2018 · 18 comments
Open

failed to create volumes more than 600 #1364

Madhu-1 opened this issue Dec 5, 2018 · 18 comments
Assignees
@aravindavk
Labels
priority: high

Comments

@Madhu-1
Copy link
Member

Madhu-1 commented Dec 5, 2018

Observed behavior

failed to create volumes more than 600 on my local setup

Expected/desired behavior

volume creation should be successful

Details on how to reproduce (minimal and precise)

create more PVC (around 1000)

logs:

I1205 07:53:41.594206       1 controller.go:1027] provision "default/gcs-app635" class "glusterfs-csi": volume "pvc-d8ea5cf4-f862-11e8-bb41-525400770e06" provisioned
I1205 07:53:41.594250       1 controller.go:1041] provision "default/gcs-app635" class "glusterfs-csi": trying to save persistentvolume "pvc-d8ea5cf4-f862-11e8-bb41-525400770e06"
I1205 07:53:41.601001       1 controller.go:1048] provision "default/gcs-app635" class "glusterfs-csi": persistentvolume "pvc-d8ea5cf4-f862-11e8-bb41-525400770e06" saved
I1205 07:53:41.601063       1 controller.go:1089] provision "default/gcs-app635" class "glusterfs-csi": succeeded
I1205 07:53:41.601432       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-app635", UID:"d8ea5cf4-f862-11e8-bb41-525400770e06", APIVersion:"v1", ResourceVersion:"53044", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-d8ea5cf4-f862-11e8-bb41-525400770e06
I1205 07:53:51.113151       1 controller.go:927] provision "default/gcs-app636" class "glusterfs-csi": started
I1205 07:53:51.128549       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-app636", UID:"df1f7786-f862-11e8-bb41-525400770e06", APIVersion:"v1", ResourceVersion:"53238", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/gcs-app636"
I1205 07:53:51.522971       1 controller.go:927] provision "default/gcs-app637" class "glusterfs-csi": started
I1205 07:53:51.530006       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-app637", UID:"e556c971-f862-11e8-bb41-525400770e06", APIVersion:"v1", ResourceVersion:"53244", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/gcs-app637"
W1205 07:53:57.951408       1 controller.go:686] Retrying syncing claim "default/gcs-app636" because failures 0 < threshold 15
E1205 07:53:57.951456       1 controller.go:701] error syncing claim "default/gcs-app636": failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: Transaction step vol-create.PrepareBricks failed on peer 74d48f37-812e-4459-b97b-10983c261312 with error: exit status 32; mount: mount /dev/mapper/vg--dev--vdd-brick_pvc--df1f7786--f862--11e8--bb41--525400770e06_s1_b2 on /run/glusterd2/bricks/pvc-df1f7786-f862-11e8-bb41-525400770e06/subvol1/brick2 failed: Cannot allocate memory
I1205 07:53:57.951826       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-app636", UID:"df1f7786-f862-11e8-bb41-525400770e06", APIVersion:"v1", ResourceVersion:"53238", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: Transaction step vol-create.PrepareBricks failed on peer 74d48f37-812e-4459-b97b-10983c261312 with error: exit status 32; mount: mount /dev/mapper/vg--dev--vdd-brick_pvc--df1f7786--f862--11e8--bb41--525400770e06_s1_b2 on /run/glusterd2/bricks/pvc-df1f7786-f862-11e8-bb41-525400770e06/subvol1/brick2 failed: Cannot allocate memory
W1205 07:53:58.113959       1 controller.go:686] Retrying syncing claim "default/gcs-app637" because failures 0 < threshold 15
E1205 07:53:58.114000       1 controller.go:701] error syncing claim "default/gcs-app637": failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: Transaction step vol-create.PrepareBricks failed on peer 74d48f37-812e-4459-b97b-10983c261312 with error: exit status 5;   fork failed: Cannot allocate memory
  Check of pool vg-dev-vdd/tp_pvc-e556c971-f862-11e8-bb41-525400770e06_s1_b2 failed (status:-1). Manual repair required!
  Failed to activate thin pool vg-dev-vdd/tp_pvc-e556c971-f862-11e8-bb41-525400770e06_s1_b2.
I1205 07:53:58.114323       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-app637", UID:"e556c971-f862-11e8-bb41-525400770e06", APIVersion:"v1", ResourceVersion:"53244", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: Transaction step vol-create.PrepareBricks failed on peer 74d48f37-812e-4459-b97b-10983c261312 with error: exit status 5;   fork failed: Cannot allocate memory
  Check of pool vg-dev-vdd/tp_pvc-e556c971-f862-11e8-bb41-525400770e06_s1_b2 failed (status:-1). Manual repair required!
  Failed to activate thin pool vg-dev-vdd/tp_pvc-e556c971-f862-11e8-bb41-525400770e06_s1_b2.

time="2018-12-05 07:55:52.396888" level=error msg="TxnSvc.Runstep failed on peer" error="fork/exec /usr/sbin/lvcreate: cannot allocate memory" remotepeer="8dbc62c9-9b1d-4ac9-b75f-f9f17385ff05(gluster-kube2-0)" reqid=5b1780d6-bffb-4537-ab3d-49e4beeadbd3 source="[rpc-client.go:77:transaction.runStepOn]" txnid=32245554-18c2-409b-8963-5336f9f76a2a
time="2018-12-05 07:55:52.397003" level=error msg="Step failed on node." error="fork/exec /usr/sbin/lvcreate: cannot allocate memory" node=8dbc62c9-9b1d-4ac9-b75f-f9f17385ff05 reqid=5b1780d6-bffb-4537-ab3d-49e4beeadbd3 source="[step.go:119:transaction.runStepFuncOnNodes]" step=vol-create.PrepareBricks txnid=32245554-18c2-409b-8963-5336f9f76a2a
time="2018-12-05 07:55:52.478842" level=info msg="client connected" address="10.233.66.6:666" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 07:55:52.481111" level=info msg="client connected" address="10.233.66.6:665" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 07:55:52.482492" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-4bce54a1-f854-11e8-bb41-525400770e06/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-4bce54a1-f854-11e8-bb41-525400770e06/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 07:55:52.483746" level=info msg="client disconnected" address="10.233.66.6:666" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 07:55:52.483964" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-4c55e734-f853-11e8-bb41-525400770e06/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-4c55e734-f853-11e8-bb41-525400770e06/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 07:55:52.485034" level=info msg="client disconnected" address="10.233.66.6:665" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 07:55:52.923351" level=info msg="client connected" address="10.233.65.7:669" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 07:55:52.925187" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-0feab6ff-f856-11e8-bb41-525400770e06/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-0feab6ff-f856-11e8-bb41-525400770e06/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 07:55:52.925491" level=info msg="client disconnected" address="10.233.65.7:669" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 07:55:52.926997" level=info msg="client connected" address="10.233.65.7:667" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 07:55:52.927513" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-d52e717c-f854-11e8-bb41-525400770e06/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-d52e717c-f854-11e8-bb41-525400770e06/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 07:55:52.927786" level=info msg="client connected" address="10.233.65.7:668" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 07:55:52.927932" level=info msg="client disconnected" address="10.233.65.7:667" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 07:55:52.929664" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-60cf8ca5-f857-11e8-bb41-525400770e06/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-60cf8ca5-f857-11e8-bb41-525400770e06/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 07:55:52.930000" level=info msg="client disconnected" address="10.233.65.7:668" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 07:55:52.970189" level=info msg="client connected" address="10.233.64.7:620" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 07:55:52.970747" level=info msg="client connected" address="10.233.64.7:619" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 07:55:52.971300" level=info msg="client connected" address="10.233.64.7:621" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 07:55:52.971750" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-a9bedc63-f853-11e8-bb41-525400770e06/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-a9bedc63-f853-11e8-bb41-525400770e06/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
@Madhu-1
Copy link
Member Author

Madhu-1 commented Dec 5, 2018

@aravindavk ^^

@Madhu-1
Copy link
Member Author

Madhu-1 commented Dec 5, 2018

dueto high memory utilization gd2 pods are not responding to pings properly

curl http://10.233.20.49:24007/ping -v
* About to connect() to 10.233.20.49 port 24007 (#0)
*   Trying 10.233.20.49...
* Connection refused
* Failed connect to 10.233.20.49:24007; Connection refused
* Closing connection 0
curl: (7) Failed connect to 10.233.20.49:24007; Connection refused
[vagrant@kube1 ~]$ 
[vagrant@kube1 ~]$ curl http://10.233.20.49:24007/ping -v
* About to connect() to 10.233.20.49 port 24007 (#0)
*   Trying 10.233.20.49...
* Connected to 10.233.20.49 (10.233.20.49) port 24007 (#0)
> GET /ping HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.233.20.49:24007
> Accept: */*
> 
< HTTP/1.1 200 OK
< X-Gluster-Cluster-Id: b4463b59-c41a-4e3b-9b7e-176a98de2d9a
< X-Gluster-Peer-Id: 74d48f37-812e-4459-b97b-10983c261312
< X-Request-Id: e8e945ba-a018-4cc3-8e3b-c79f7a870fd7
< Date: Wed, 05 Dec 2018 08:05:56 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host 10.233.20.49 left intact

@atinmu atinmu added bug priority: high GCS/1.0 Issue is blocker for Gluster for Container Storage labels Dec 5, 2018
@atinmu
Copy link
Contributor

atinmu commented Dec 5, 2018

@aravindavk / @rishubhjain - can one of you please check this?

@rmadaka
Copy link

rmadaka commented Dec 6, 2018 •
edited by Madhu-1

Create GCS setup with 16 vcpus and 32GB RAM for each kube node. Then try to create 1000 PVC using script. Each pvc size 1GB.

Observation:

-> I have tried to create 1000 PVC, 615 PVCs are bounded remaining pvcs are in pending state.

-> After 12 hours of idle time observed that GD2 pods rebooted more than 150 times

-> After 12 hour of idle time one of the etcd pod is in not ready state.

[vagrant@kube1 ~]$ kubectl -n gcs get pods
NAME                                   READY   STATUS             RESTARTS   AGE
csi-attacher-glusterfsplugin-0         2/2     Running            0          21h
csi-nodeplugin-glusterfsplugin-b7v2f   2/2     Running            0          21h
csi-nodeplugin-glusterfsplugin-g2vd9   2/2     Running            0          21h
csi-nodeplugin-glusterfsplugin-ppqr2   2/2     Running            0          21h
csi-provisioner-glusterfsplugin-0      3/3     Running            0          21h
etcd-95nxlqf2v7                        1/1     Running            0          21h
etcd-9q9h6mw86w                        1/1     Running            0          21h
etcd-operator-7cb5bd459b-gwdpd         1/1     Running            0          21h
etcd-sf5bntpkh4                        0/1     Running            0          21h
gluster-kube1-0                        1/1     Running            17         21h
gluster-kube2-0                        1/1     Running            194        21h
gluster-kube3-0                        0/1     CrashLoopBackOff   151        21h

glusterd2.log:

time="2018-12-05 15:20:17.981084" level=info msg="client connected" address="10.233.65.5:44539" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:17.984382" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-d77def0e-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-d77def0e-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:17.985775" level=info msg="client disconnected" address="10.233.65.5:44539" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:17.989872" level=info msg="client connected" address="10.233.64.6:45712" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:17.990696" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-0ad50608-f88f-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-0ad50608-f88f-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:17.990997" level=info msg="client disconnected" address="10.233.64.6:45712" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.011815" level=info msg="client connected" address=@ server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=unix
time="2018-12-05 15:20:18.076331" level=info msg="client connected" address="10.233.64.6:45710" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.077212" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-eb4f786e-f88e-11e8-af9b-525400090231/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-eb4f786e-f88e-11e8-af9b-525400090231/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.077569" level=info msg="client disconnected" address="10.233.64.6:45710" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.097058" level=info msg="client connected" address="10.233.64.6:45708" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.103134" level=info msg="client connected" address="10.233.64.6:45707" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.104073" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-3568e09f-f88f-11e8-af9b-525400090231/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-3568e09f-f88f-11e8-af9b-525400090231/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.104426" level=info msg="client disconnected" address="10.233.64.6:45707" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.113387" level=info msg="client connected" address="10.233.65.5:44545" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.114662" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-dcbbd0b8-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-dcbbd0b8-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.115276" level=info msg="client disconnected" address="10.233.65.5:44545" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.125024" level=info msg="client connected" address="10.233.64.6:45704" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.125940" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-c296ce61-f88e-11e8-af9b-525400090231/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-c296ce61-f88e-11e8-af9b-525400090231/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.126375" level=info msg="client disconnected" address="10.233.64.6:45704" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.137460" level=info msg="client connected" address="10.233.65.5:44534" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.138541" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-dd8a6870-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-dd8a6870-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.138997" level=info msg="client disconnected" address="10.233.65.5:44534" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.142198" level=info msg="client disconnected" address=@ server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.142413" level=info msg="client connected" address="10.233.64.6:45703" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.147982" level=info msg="client connected" address="10.233.64.6:45702" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.148924" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-0b0b3718-f88f-11e8-af9b-525400090231/subvol1/brick3/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-0b0b3718-f88f-11e8-af9b-525400090231/subvol1/brick3/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.149226" level=info msg="client disconnected" address="10.233.64.6:45702" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.180983" level=info msg="10.233.64.6 - - [05/Dec/2018:15:20:17 +0000] \"GET /v1/volumes/pvc-eb4f786e-f88e-11e8-af9b-525400090231/heal-info HTTP/1.1\" 200 619" reqid=f31aea33-fed5-4013-bd64-48d24efaba9e
time="2018-12-05 15:20:18.182514" level=info msg="client connected" address="10.233.64.6:45700" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.183553" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-c2c6a9d4-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-c2c6a9d4-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.184066" level=info msg="client disconnected" address="10.233.64.6:45700" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.203663" level=info msg="client connected" address="10.233.64.6:45698" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.212928" level=info msg="client connected" address="10.233.64.6:45697" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.213923" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-0b402fbc-f88f-11e8-af9b-525400090231/subvol1/brick3/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-0b402fbc-f88f-11e8-af9b-525400090231/subvol1/brick3/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.214304" level=info msg="client disconnected" address="10.233.64.6:45697" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.219886" level=info msg="client connected" address=@ server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=unix
time="2018-12-05 15:20:18.246074" level=info msg="client connected" address="10.233.64.6:45696" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.246989" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-35aac659-f88f-11e8-af9b-525400090231/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-35aac659-f88f-11e8-af9b-525400090231/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.247283" level=info msg="client disconnected" address="10.233.64.6:45696" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.254830" level=info msg="client connected" address="10.233.64.6:45694" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.262509" level=info msg="client connected" address="10.233.64.6:45693" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.263585" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-eb8ca6ac-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-eb8ca6ac-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.264275" level=info msg="client disconnected" address="10.233.64.6:45693" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.272470" level=info msg="client connected" address="10.233.64.6:45692" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.273479" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-c2f9bdb3-f88e-11e8-af9b-525400090231/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-c2f9bdb3-f88e-11e8-af9b-525400090231/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.273819" level=info msg="client disconnected" address="10.233.64.6:45692" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.279997" level=info msg="client connected" address="10.233.65.5:44537" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.284255" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-a7f594e1-f88e-11e8-af9b-525400090231/subvol1/brick3/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-a7f594e1-f88e-11e8-af9b-525400090231/subvol1/brick3/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.292147" level=info msg="client disconnected" address="10.233.65.5:44537" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.309320" level=info msg="client connected" address="10.233.64.6:45689" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.314686" level=info msg="client connected" address="10.233.64.6:45687" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.315512" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-0b76dd8f-f88f-11e8-af9b-525400090231/subvol1/brick3/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-0b76dd8f-f88f-11e8-af9b-525400090231/subvol1/brick3/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.315853" level=info msg="client disconnected" address="10.233.64.6:45687" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.318445" level=info msg="client connected" address="10.233.65.5:44533" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.319694" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-a8279e6a-f88e-11e8-af9b-525400090231/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-a8279e6a-f88e-11e8-af9b-525400090231/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.321219" level=info msg="client disconnected" address="10.233.65.5:44533" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.334817" level=info msg="client connected" address="10.233.64.6:45686" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.335976" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-c3301f36-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-c3301f36-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.336331" level=info msg="client disconnected" address="10.233.64.6:45686" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.340069" level=info msg="client connected" address="10.233.65.5:44531" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.342206" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-01bef564-f88f-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-01bef564-f88f-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.343999" level=info msg="client disconnected" address="10.233.65.5:44531" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.345090" level=info msg="client disconnected" address=@ server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.348236" level=info msg="client connected" address="10.233.64.6:45688" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.349062" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-35e90a04-f88f-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-35e90a04-f88f-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.349336" level=info msg="client disconnected" address="10.233.64.6:45688" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.362312" level=info msg="10.233.64.6 - - [05/Dec/2018:15:20:18 +0000] \"GET /v1/volumes/pvc-eb8ca6ac-f88e-11e8-af9b-525400090231/heal-info HTTP/1.1\" 200 619" reqid=b0875906-0116-4d8e-a480-4078a962dc2b
time="2018-12-05 15:20:18.366095" level=info msg="client connected" address="10.233.65.5:44530" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.369147" level=info msg="client disconnected" address="10.233.65.5:44530" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.371954" level=info msg="client connected" address="10.233.64.6:45685" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.413826" level=info msg="client connected" address="10.233.65.5:44525" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.415216" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-ca58b3e4-f88e-11e8-af9b-525400090231/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-ca58b3e4-f88e-11e8-af9b-525400090231/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.415861" level=info msg="client connected" address="10.233.64.6:45684" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.416440" level=info msg="client disconnected" address="10.233.65.5:44525" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.416806" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-c36a06cf-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-c36a06cf-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.417112" level=info msg="client disconnected" address="10.233.64.6:45684" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.417131" level=info msg="client connected" address=@ server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=unix
time="2018-12-05 15:20:18.422888" level=info msg="client connected" address="10.233.64.6:45683" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.438508" level=info msg="client connected" address="10.233.65.5:44524" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.438818" level=info msg="client connected" address="10.233.64.6:45682" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.440088" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-e87e652e-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-e87e652e-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.440583" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-0bb21ace-f88f-11e8-af9b-525400090231/subvol1/brick3/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-0bb21ace-f88f-11e8-af9b-525400090231/subvol1/brick3/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.440824" level=info msg="client disconnected" address="10.233.65.5:44524" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.440953" level=info msg="client disconnected" address="10.233.64.6:45682" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.453787" level=info msg="client connected" address="10.233.65.5:44522" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.454994" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-aaff0b1d-f88e-11e8-af9b-525400090231/subvol1/brick3/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-aaff0b1d-f88e-11e8-af9b-525400090231/subvol1/brick3/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.455504" level=info msg="client disconnected" address="10.233.65.5:44522" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.466104" level=info msg="client connected" address="10.233.64.6:45680" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.467268" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-ebc49846-f88e-11e8-af9b-525400090231/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-ebc49846-f88e-11e8-af9b-525400090231/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.467689" level=info msg="client disconnected" address="10.233.64.6:45680" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.473198" level=info msg="client connected" address="10.233.65.5:44519" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.474451" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-c4c6cfea-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-c4c6cfea-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.475070" level=info msg="client disconnected" address="10.233.65.5:44519" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.478415" level=info msg="client connected" address="10.233.64.6:45679" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.479260" level=info msg="client connected" address="10.233.64.6:45677" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.479632" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-c39efae7-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-c39efae7-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.479990" level=info msg="client disconnected" address="10.233.64.6:45679" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.490997" level=info msg="client connected" address="10.233.65.5:44518" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.492203" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-a85c2248-f88e-11e8-af9b-525400090231/subvol1/brick3/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-a85c2248-f88e-11e8-af9b-525400090231/subvol1/brick3/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.494487" level=info msg="client disconnected" address="10.233.65.5:44518" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.497854" level=info msg="client connected" address="10.233.64.6:45675" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.498604" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-3623c371-f88f-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-3623c371-f88f-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.498933" level=info msg="client disconnected" address="10.233.64.6:45675" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.514495" level=info msg="client connected" address="10.233.65.5:44516" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.515806" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-e726e2e9-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-e726e2e9-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.516437" level=info msg="client disconnected" address="10.233.65.5:44516" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.516854" level=info msg="client connected" address="10.233.64.6:45673" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.517574" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-0be9d973-f88f-11e8-af9b-525400090231/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-0be9d973-f88f-11e8-af9b-525400090231/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.517930" level=info msg="client disconnected" address="10.233.64.6:45673" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.524686" level=info msg="client disconnected" address=@ server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.535014" level=info msg="client connected" address="10.233.65.5:44514" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.536904" level=info msg="client connected" address="10.233.64.6:45672" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.537257" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-c66b5171-f88e-11e8-af9b-525400090231/subvol1/brick3/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-c66b5171-f88e-11e8-af9b-525400090231/subvol1/brick3/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.538370" level=info msg="client disconnected" address="10.233.65.5:44514" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.551488" level=info msg="10.233.64.6 - - [05/Dec/2018:15:20:18 +0000] \"GET /v1/volumes/pvc-ebc49846-f88e-11e8-af9b-525400090231/heal-info HTTP/1.1\" 200 619" reqid=cc4c34d8-14f5-4900-b68e-c20c5cc0c733
time="2018-12-05 15:20:18.555991" level=info msg="client connected" address="10.233.65.5:44512" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.557073" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-c52cd389-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-c52cd389-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.557650" level=info msg="client disconnected" address="10.233.65.5:44512" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.570378" level=info msg="client connected" address="10.233.65.5:44510" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.571407" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-e230d38c-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-e230d38c-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.571860" level=info msg="client disconnected" address="10.233.65.5:44510" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.588825" level=info msg="client connected" address="10.233.65.5:44508" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.589699" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-c4969cbc-f88e-11e8-af9b-525400090231/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-c4969cbc-f88e-11e8-af9b-525400090231/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.590179" level=info msg="client disconnected" address="10.233.65.5:44508" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.598297" level=info msg="client connected" address="10.233.64.6:45670" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.599167" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-c3e226fb-f88e-11e8-af9b-525400090231/subvol1/brick3/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-c3e226fb-f88e-11e8-af9b-525400090231/subvol1/brick3/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.599424" level=info msg="client disconnected" address="10.233.64.6:45670" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.606866" level=info msg="client connected" address=@ server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=unix
time="2018-12-05 15:20:18.607957" level=info msg="client connected" address="10.233.65.5:44506" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.608947" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-e8f64d7c-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-e8f64d7c-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.609528" level=info msg="client disconnected" address="10.233.65.5:44506" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.613480" level=info msg="client connected" address="10.233.64.6:45669" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.614427" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-0c239b4a-f88f-11e8-af9b-525400090231/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-0c239b4a-f88f-11e8-af9b-525400090231/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.614870" level=info msg="client disconnected" address="10.233.64.6:45669" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.619900" level=info msg="client connected" address="10.233.64.6:45668" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.625620" level=info msg="client connected" address="10.233.65.5:44503" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.626757" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-bdfe3b21-f88e-11e8-af9b-525400090231/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-bdfe3b21-f88e-11e8-af9b-525400090231/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.627365" level=info msg="client disconnected" address="10.233.65.5:44503" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.641772" level=info msg="client connected" address="10.233.64.6:45666" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.642625" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-ec05b364-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-ec05b364-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.644030" level=info msg="client disconnected" address="10.233.64.6:45666" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.644134" level=info msg="client connected" address="10.233.65.5:44502" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.644756" level=info msg="client connected" address="10.233.64.6:45665" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.645559" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-ddc1271e-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-ddc1271e-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.645838" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-c414ff33-f88e-11e8-af9b-525400090231/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-c414ff33-f88e-11e8-af9b-525400090231/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.646166" level=info msg="client disconnected" address="10.233.64.6:45665" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.647141" level=info msg="client disconnected" address="10.233.65.5:44502" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.658912" level=info msg="client connected" address="10.233.64.6:45662" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.660123" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-365b83b4-f88f-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-365b83b4-f88f-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.660864" level=info msg="client disconnected" address="10.233.64.6:45662" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.665872" level=info msg="client connected" address="10.233.65.5:44500" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.667175" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-edde13bf-f88e-11e8-af9b-525400090231/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-edde13bf-f88e-11e8-af9b-525400090231/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.667694" level=info msg="client disconnected" address="10.233.65.5:44500" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.672291" level=info msg="client connected" address="10.233.64.6:45661" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.692149" level=info msg="client connected" address="10.233.65.5:44497" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.693630" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-04bfdcb7-f88f-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-04bfdcb7-f88f-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.694295" level=info msg="client disconnected" address="10.233.65.5:44497" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.710864" level=info msg="client connected" address="10.233.65.5:44495" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.711955" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-e9724890-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-e9724890-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.712426" level=info msg="client disconnected" address="10.233.65.5:44495" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.723666" level=info msg="client connected" address="10.233.64.6:45658" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.724148" level=info msg="client disconnected" address=@ server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.728082" level=info msg="client connected" address="10.233.65.5:44493" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.728406" level=info msg="client connected" address="10.233.64.6:45657" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.729102" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-bc92e325-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-bc92e325-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.729198" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-c44c9d19-f88e-11e8-af9b-525400090231/subvol1/brick3/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-c44c9d19-f88e-11e8-af9b-525400090231/subvol1/brick3/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.729523" level=info msg="client disconnected" address="10.233.64.6:45657" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.729981" level=info msg="client disconnected" address="10.233.65.5:44493" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.743769" level=info msg="client connected" address="10.233.64.6:45656" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.744793" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-0c592a66-f88f-11e8-af9b-525400090231/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-0c592a66-f88f-11e8-af9b-525400090231/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.745153" level=info msg="client disconnected" address="10.233.64.6:45656" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.747892" level=info msg="client connected" address="10.233.65.5:44491" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.749035" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-ec05b364-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-ec05b364-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.749546" level=info msg="client disconnected" address="10.233.65.5:44491" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.750568" level=info msg="10.233.64.6 - - [05/Dec/2018:15:20:18 +0000] \"GET /v1/volumes/pvc-ec05b364-f88e-11e8-af9b-525400090231/heal-info HTTP/1.1\" 200 619" reqid=795ac44b-bcf6-455d-9dd2-2acd2b6e4108
time="2018-12-05 15:20:18.769347" level=info msg="client connected" address="10.233.65.5:44488" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.770664" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-c36a06cf-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-c36a06cf-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.771238" level=info msg="client disconnected" address="10.233.65.5:44488" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.773915" level=info msg="client connected" address="10.233.64.6:45654" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.774874" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-369c41a9-f88f-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-369c41a9-f88f-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.775165" level=info msg="client disconnected" address="10.233.64.6:45654" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.783131" level=info msg="client connected" address=@ server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=unix
time="2018-12-05 15:20:18.786799" level=info msg="client connected" address="10.233.64.6:45653" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.787317" level=info msg="client connected" address="10.233.64.6:45652" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.787578" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-c4969cbc-f88e-11e8-af9b-525400090231/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-c4969cbc-f88e-11e8-af9b-525400090231/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.787963" level=info msg="client disconnected" address="10.233.64.6:45653" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.788054" level=info msg="client connected" address="10.233.65.5:44487" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.788887" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-a88b9144-f88e-11e8-af9b-525400090231/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-a88b9144-f88e-11e8-af9b-525400090231/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.789516" level=info msg="client disconnected" address="10.233.65.5:44487" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.802786" level=info msg="client connected" address="10.233.64.6:45650" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.803577" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-0c91a35b-f88f-11e8-af9b-525400090231/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-0c91a35b-f88f-11e8-af9b-525400090231/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.803930" level=info msg="client disconnected" address="10.233.64.6:45650" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.806784" level=info msg="client connected" address="10.233.65.5:44485" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.807895" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-c4fa25d6-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-c4fa25d6-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.808391" level=info msg="client disconnected" address="10.233.65.5:44485" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.813550" level=info msg="client connected" address="10.233.64.6:45649" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.814477" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-ec3f0cea-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-ec3f0cea-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.816100" level=info msg="client disconnected" address="10.233.64.6:45649" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.824112" level=info msg="client connected" address="10.233.65.5:44484" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.825165" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-ea957875-f88e-11e8-af9b-525400090231/subvol1/brick3/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-ea957875-f88e-11e8-af9b-525400090231/subvol1/brick3/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.825621" level=info msg="client disconnected" address="10.233.65.5:44484" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.839654" level=info msg="client connected" address="10.233.64.6:45648" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.840761" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-c4c6cfea-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-c4c6cfea-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.841091" level=info msg="client disconnected" address="10.233.64.6:45648" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.842855" level=info msg="client connected" address="10.233.65.5:44482" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.843819" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-230545b7-f88f-11e8-af9b-525400090231/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-230545b7-f88f-11e8-af9b-525400090231/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.844242" level=info msg="client disconnected" address="10.233.65.5:44482" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.850342" level=info msg="client connected" address="10.233.64.6:45646" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.851689" level=info msg="client connected" address="10.233.64.6:45645" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.852474" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-0cccb30f-f88f-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-0cccb30f-f88f-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.852802" level=info msg="client disconnected" address="10.233.64.6:45645" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.859285" level=info msg="client connected" address="10.233.65.5:44478" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.861090" level=info msg="client disconnected" address="10.233.65.5:44478" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.879535" level=info msg="client connected" address="10.233.65.5:44475" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.880912" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-d0a7788d-f88e-11e8-af9b-525400090231/subvol1/brick3/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-d0a7788d-f88e-11e8-af9b-525400090231/subvol1/brick3/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.881630" level=info msg="client disconnected" address="10.233.65.5:44475" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.893336" level=info msg="client disconnected" address=@ server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.897059" level=info msg="client connected" address="10.233.65.5:44471" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.898267" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-f97bd77f-f88e-11e8-af9b-525400090231/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-f97bd77f-f88e-11e8-af9b-525400090231/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.898829" level=info msg="client disconnected" address="10.233.65.5:44471" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.899486" level=info msg="client connected" address="10.233.64.6:45642" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.906222" level=info msg="client connected" address="10.233.64.6:45641" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.907186" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-5a647dee-f88f-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-5a647dee-f88f-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.907601" level=info msg="client disconnected" address="10.233.64.6:45641" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.917239" level=info msg="client connected" address="10.233.65.5:44469" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.918292" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-fcbb5692-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-fcbb5692-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.918830" level=info msg="client disconnected" address="10.233.65.5:44469" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.919956" level=info msg="client connected" address="10.233.64.6:45640" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.920767" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-c4fa25d6-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-c4fa25d6-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.921072" level=info msg="client disconnected" address="10.233.64.6:45640" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.935094" level=info msg="client connected" address="10.233.64.6:45638" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.935287" level=info msg="client connected" address="10.233.65.5:44467" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.936215" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-0d0d48f5-f88f-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-0d0d48f5-f88f-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.936483" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-c2c6a9d4-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-c2c6a9d4-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.936568" level=info msg="client disconnected" address="10.233.64.6:45638" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.937109" level=info msg="client disconnected" address="10.233.65.5:44467" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.937521" level=info msg="10.233.64.6 - - [05/Dec/2018:15:20:18 +0000] \"GET /v1/volumes/pvc-ec3f0cea-f88e-11e8-af9b-525400090231/heal-info HTTP/1.1\" 200 619" reqid=ce188743-b14f-4a46-ab97-1d888c128f95
time="2018-12-05 15:20:18.954913" level=info msg="client connected" address="10.233.65.5:44464" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.957978" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-c296ce61-f88e-11e8-af9b-525400090231/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-c296ce61-f88e-11e8-af9b-525400090231/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:18.960503" level=info msg="client disconnected" address="10.233.65.5:44464" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:18.964445" level=info msg="client connected" address=@ server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=unix
time="2018-12-05 15:20:18.967165" level=info msg="client connected" address="10.233.64.6:45637" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.974591" level=info msg="client connected" address="10.233.65.5:44462" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:18.976569" level=info msg="client disconnected" address="10.233.65.5:44462" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:19.000148" level=info msg="client connected" address="10.233.65.5:44460" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:19.000988" level=info msg="client connected" address="10.233.64.6:45636" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:19.001376" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-1d4e1962-f88f-11e8-af9b-525400090231/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-1d4e1962-f88f-11e8-af9b-525400090231/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:19.001914" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-ec7c5831-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-ec7c5831-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:19.002022" level=info msg="client disconnected" address="10.233.65.5:44460" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:19.002495" level=info msg="client disconnected" address="10.233.64.6:45636" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:19.003948" level=info msg="client connected" address="10.233.64.6:45634" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:19.004759" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-c52cd389-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-c52cd389-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:19.005108" level=info msg="client disconnected" address="10.233.64.6:45634" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:19.019523" level=info msg="client connected" address="10.233.64.6:45632" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:19.020758" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-0d47d415-f88f-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-0d47d415-f88f-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:19.021091" level=info msg="client disconnected" address="10.233.64.6:45632" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:19.023771" level=info msg="client connected" address="10.233.65.5:44456" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:19.024933" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-d9408287-f88e-11e8-af9b-525400090231/subvol1/brick2/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-d9408287-f88e-11e8-af9b-525400090231/subvol1/brick2/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:19.025384" level=info msg="client disconnected" address="10.233.65.5:44456" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:19.032702" level=info msg="client connected" address="10.233.64.6:45633" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:19.033530" level=info msg="client connected" address="10.233.64.6:45629" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:19.034358" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-5aaaf95a-f88f-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-5aaaf95a-f88f-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:19.034757" level=info msg="client disconnected" address="10.233.64.6:45629" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:19.047077" level=info msg="client connected" address="10.233.65.5:44454" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:19.048492" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-e7d0105e-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-e7d0105e-f88e-11e8-af9b-525400090231/subvol1/brick1/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"
time="2018-12-05 15:20:19.049076" level=info msg="client disconnected" address="10.233.65.5:44454" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2018-12-05 15:20:19.064125" level=info msg="client connected" address="10.233.65.5:44453" server=sunrpc source="[server.go:155:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2018-12-05 15:20:19.065444" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-d9b665a5-f88e-11e8-af9b-525400090231/subvol1/brick1/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/briccommand terminated with exit code 137

csi-provisioner.log

@rmadaka
Copy link

rmadaka commented Dec 6, 2018

Attached CSI provisioner log above

@ksandha
Copy link

ksandha commented Dec 13, 2018

attacher.log
provisioner.log

hitting this issue at 758 pvc

@Madhu-1
Copy link
Member Author

Madhu-1 commented Dec 13, 2018

@aravindavk @rishubhjain PTAL

@Madhu-1
Copy link
Member Author

Madhu-1 commented Dec 13, 2018

@ksandha please attach gd2 logs

@ksandha
Copy link

ksandha commented Dec 13, 2018

glusterd2_758pvc.log

Madhu-1 added a commit to Madhu-1/glusterd2 that referenced this issue Dec 13, 2018
@Madhu-1
Volume Create: Add a check to validate if volume exists
3f22c26 
during volume create, before starting
of transaction,add a check to validate
volume already exists or not.

Partial Fixes: gluster#1364

Signed-off-by: Madhu Rajanna <mrajanna@redhat.com>
@Madhu-1 Madhu-1 mentioned this issue Dec 13, 2018
Merged
Madhu-1 added a commit to Madhu-1/glusterd2 that referenced this issue Dec 13, 2018
@Madhu-1
Volume Create: Add a check to validate if volume exists
628c137 
during volume create, before starting
of transaction,add a check to validate
volume already exists or not.

Partial Fixes: gluster#1364

Signed-off-by: Madhu Rajanna <mrajanna@redhat.com>
@ksandha
Copy link

ksandha commented Dec 13, 2018

  1. Started 1000 pvc with a script in sequential order
  2. At 758PVC were stuck and rest all in pending state.
    attched the logs in above comments.
  3. started to delete the pvc using script in sequential order.
    seeing no pvc in
[vagrant@kube1 ~]$ kubectl get pvc
No resources found.
[vagrant@kube1 ~]$
  1. Check the below command
[vagrant@kube1 ~]$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                STORAGECLASS    REASON   AGE
pvc-0018c46b-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc445   glusterfs-csi            158m
pvc-00556e31-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc446   glusterfs-csi            158m
pvc-0095de33-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc447   glusterfs-csi            158m
pvc-00cdd6b2-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc448   glusterfs-csi            158m
pvc-010475b3-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc449   glusterfs-csi            158m
pvc-0138ee5a-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc450   glusterfs-csi            158m
pvc-016e8c9e-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc451   glusterfs-csi            158m
pvc-01aba5c6-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc452   glusterfs-csi            158m
pvc-01e7277a-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc453   glusterfs-csi            158m
pvc-021e3d33-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc454   glusterfs-csi            158m
pvc-025a61a4-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc455   glusterfs-csi            158m
pvc-029ec7b5-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc456   glusterfs-csi            158m
pvc-02d1375c-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc457   glusterfs-csi            158m
pvc-030876a5-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc458   glusterfs-csi            158m
pvc-03419b4e-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc459   glusterfs-csi            158m
pvc-03757081-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc460   glusterfs-csi            158m
pvc-03b0a5e7-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc461   glusterfs-csi            158m
pvc-03ec1558-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc462   glusterfs-csi            157m
pvc-041f8596-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc463   glusterfs-csi            157m
pvc-0453ae22-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc464   glusterfs-csi            157m
pvc-0484da74-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc465   glusterfs-csi            157m
pvc-04b636a5-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc466   glusterfs-csi            157m
pvc-04efd2e7-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc467   glusterfs-csi            157m
pvc-052c739f-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc468   glusterfs-csi            157m
pvc-05690a80-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc469   glusterfs-csi            157m
pvc-05a0e36e-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc470   glusterfs-csi            157m
pvc-05fc045a-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc471   glusterfs-csi            157m
pvc-0633959c-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc472   glusterfs-csi            157m
pvc-066af0f1-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc473   glusterfs-csi            157m
pvc-069cd39b-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc474   glusterfs-csi            157m
pvc-06d5c33e-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc475   glusterfs-csi            157m
pvc-070c34a7-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc476   glusterfs-csi            157m
pvc-073e6650-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc477   glusterfs-csi            157m
pvc-0772dc2f-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc478   glusterfs-csi            157m
pvc-07ac3ff1-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc479   glusterfs-csi            157m
pvc-07e377bf-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc480   glusterfs-csi            157m
pvc-082c7cae-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc481   glusterfs-csi            157m
pvc-08685cc2-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc482   glusterfs-csi            157m
pvc-08a9f8fb-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc483   glusterfs-csi            157m
pvc-08ea5bfb-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc484   glusterfs-csi            157m
pvc-09272035-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc485   glusterfs-csi            157m
pvc-095fde85-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc486   glusterfs-csi            157m
pvc-099f0349-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc487   glusterfs-csi            156m
pvc-09e17087-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc488   glusterfs-csi            156m
pvc-0a216f16-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc489   glusterfs-csi            156m
pvc-0a5662fd-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc490   glusterfs-csi            156m
pvc-0a98f09d-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc491   glusterfs-csi            156m
pvc-0ad6b2f7-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc492   glusterfs-csi            156m
pvc-0b10b503-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc493   glusterfs-csi            156m
pvc-0b4f0cb5-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc494   glusterfs-csi            156m
pvc-0b90d199-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc495   glusterfs-csi            156m
pvc-0bcadedf-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc496   glusterfs-csi            156m
pvc-0c0028f8-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc497   glusterfs-csi            156m
pvc-0c374310-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc498   glusterfs-csi            156m
pvc-0c7132b2-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc499   glusterfs-csi            156m
pvc-0caccb1d-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc500   glusterfs-csi            156m
pvc-0ceab667-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc501   glusterfs-csi            156m
pvc-0d241fbe-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc502   glusterfs-csi            156m
pvc-0d5f8eaa-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc503   glusterfs-csi            156m
pvc-0d938d80-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc504   glusterfs-csi            156m
pvc-0dcc96a8-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc505   glusterfs-csi            156m
pvc-0e04505d-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc506   glusterfs-csi            156m
pvc-0e3a54b3-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc507   glusterfs-csi            156m
pvc-0e765e44-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc508   glusterfs-csi            156m
pvc-0ebe484a-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc509   glusterfs-csi            156m
pvc-0ef56e32-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc510   glusterfs-csi            156m
pvc-0f2ee6f2-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc511   glusterfs-csi            156m
pvc-0f604b9e-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc512   glusterfs-csi            155m
pvc-0fa06390-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc513   glusterfs-csi            155m
pvc-0fe51ed1-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc514   glusterfs-csi            155m
pvc-1019f8bf-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc515   glusterfs-csi            155m
pvc-104ec8be-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc516   glusterfs-csi            155m
pvc-1084bf99-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc517   glusterfs-csi            155m
pvc-10b86c0d-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc518   glusterfs-csi            155m
pvc-10fbd284-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc519   glusterfs-csi            155m
pvc-113f8453-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc520   glusterfs-csi            155m
pvc-116f86ab-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc521   glusterfs-csi            155m
pvc-11a962b8-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc522   glusterfs-csi            155m
pvc-11e53dc8-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc523   glusterfs-csi            155m
pvc-121e5ad0-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc524   glusterfs-csi            155m
pvc-12524244-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc525   glusterfs-csi            155m
pvc-12894cb9-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc526   glusterfs-csi            155m
pvc-12c2f9c9-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc527   glusterfs-csi            155m
pvc-12ffb38b-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc528   glusterfs-csi            155m
pvc-133360dd-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc529   glusterfs-csi            155m
pvc-136f8c25-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc530   glusterfs-csi            155m
pvc-13a7d206-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc531   glusterfs-csi            155m
pvc-13dce059-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc532   glusterfs-csi            155m
pvc-14172cbf-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc533   glusterfs-csi            155m
pvc-14516242-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc534   glusterfs-csi            155m
pvc-148e6fb4-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc535   glusterfs-csi            155m
pvc-14c5918b-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc536   glusterfs-csi            155m
pvc-14fac52c-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc537   glusterfs-csi            154m
pvc-15593bd7-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc538   glusterfs-csi            154m
pvc-15933439-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc539   glusterfs-csi            154m
pvc-15d56179-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc540   glusterfs-csi            154m
pvc-1610398c-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc541   glusterfs-csi            154m
pvc-164ab940-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc542   glusterfs-csi            154m
pvc-167ea045-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc543   glusterfs-csi            154m
pvc-16b11fe0-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc544   glusterfs-csi            154m
pvc-16e4a3a7-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc545   glusterfs-csi            154m
pvc-171d136a-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc546   glusterfs-csi            154m
pvc-175d3a5e-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc547   glusterfs-csi            154m
pvc-1794422a-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc548   glusterfs-csi            154m
pvc-17c6c4d6-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc549   glusterfs-csi            154m
pvc-17fbd759-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc550   glusterfs-csi            154m
pvc-18323ec9-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc551   glusterfs-csi            154m
pvc-186a85f8-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc552   glusterfs-csi            154m
pvc-18a27a64-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc553   glusterfs-csi            154m
pvc-18e57402-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc554   glusterfs-csi            154m
pvc-191cdf89-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc555   glusterfs-csi            154m
pvc-196e8f38-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc556   glusterfs-csi            154m
pvc-19a5f48d-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc557   glusterfs-csi            154m
pvc-19dda347-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc558   glusterfs-csi            153m
pvc-1a11f674-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc559   glusterfs-csi            153m
pvc-1a4cbabb-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc560   glusterfs-csi            153m
pvc-1a878179-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc561   glusterfs-csi            153m
pvc-1ac199f6-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc562   glusterfs-csi            153m
pvc-1af3b2e9-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc563   glusterfs-csi            153m
pvc-1b2d0485-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc564   glusterfs-csi            153m
pvc-1b6691fe-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc565   glusterfs-csi            153m
pvc-1b9b86c7-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc566   glusterfs-csi            153m
pvc-1bddece9-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc567   glusterfs-csi            153m
pvc-1c2642dc-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc568   glusterfs-csi            153m
pvc-1c5d41ab-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc569   glusterfs-csi            153m
pvc-1c9ba4e7-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc570   glusterfs-csi            153m
pvc-1cdaaa38-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc571   glusterfs-csi            153m
pvc-1d16bfcf-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc572   glusterfs-csi            153m
pvc-1d49a408-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc573   glusterfs-csi            153m
pvc-1d7eee2f-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc574   glusterfs-csi            153m
pvc-1dbd4f78-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc575   glusterfs-csi            153m
pvc-1df74419-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc576   glusterfs-csi            153m
pvc-1e33d270-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc577   glusterfs-csi            153m
pvc-1e6f6427-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc578   glusterfs-csi            153m
pvc-1eaef45e-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc579   glusterfs-csi            153m
pvc-1eed5004-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc580   glusterfs-csi            153m
pvc-1f296c9b-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc581   glusterfs-csi            153m
pvc-1f699f6d-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc582   glusterfs-csi            152m
pvc-1fa256d8-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc583   glusterfs-csi            152m
pvc-1fdc3f9a-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc584   glusterfs-csi            152m
pvc-2017f22a-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc585   glusterfs-csi            152m
pvc-204f062f-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc586   glusterfs-csi            152m
pvc-2082a375-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc587   glusterfs-csi            152m
pvc-20c2daed-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc588   glusterfs-csi            152m
pvc-21002306-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc589   glusterfs-csi            152m
pvc-2135b6fb-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc590   glusterfs-csi            152m
pvc-218a1850-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc591   glusterfs-csi            152m
pvc-21ca869f-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc592   glusterfs-csi            152m
pvc-22040f2d-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc593   glusterfs-csi            152m
pvc-224b05dc-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc594   glusterfs-csi            152m
pvc-228440ab-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc595   glusterfs-csi            152m
pvc-22bde4a0-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc596   glusterfs-csi            152m
pvc-23015a19-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc597   glusterfs-csi            152m
pvc-233e1828-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc598   glusterfs-csi            152m
pvc-23729c52-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc599   glusterfs-csi            151m
pvc-23b52083-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc600   glusterfs-csi            152m
pvc-23fb3612-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc601   glusterfs-csi            152m
pvc-24359ba4-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc602   glusterfs-csi            152m
pvc-24787499-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc603   glusterfs-csi            151m
pvc-24b63252-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc604   glusterfs-csi            151m
pvc-24f7e91d-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc605   glusterfs-csi            151m
pvc-253b963d-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc606   glusterfs-csi            151m
pvc-25786539-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc607   glusterfs-csi            151m
pvc-25b9b576-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc608   glusterfs-csi            151m
pvc-25f90460-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc609   glusterfs-csi            151m
pvc-2631e49f-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc610   glusterfs-csi            151m
pvc-266bdf81-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc611   glusterfs-csi            151m
pvc-26add129-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc612   glusterfs-csi            151m
pvc-26e73a23-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc613   glusterfs-csi            151m
pvc-2725af53-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc614   glusterfs-csi            151m
pvc-2761ee91-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc615   glusterfs-csi            151m
pvc-27a125cb-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc616   glusterfs-csi            151m
pvc-27dd89dd-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc617   glusterfs-csi            151m
pvc-281dd604-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc618   glusterfs-csi            151m
pvc-28626255-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc619   glusterfs-csi            151m
pvc-28ae7205-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc620   glusterfs-csi            151m
pvc-28e80f7a-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc621   glusterfs-csi            151m
pvc-2936cd5e-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc622   glusterfs-csi            151m
pvc-297f20cd-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc623   glusterfs-csi            151m
pvc-29b3a491-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc624   glusterfs-csi            150m
pvc-29e7dc66-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc625   glusterfs-csi            150m
pvc-2a221ddf-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc626   glusterfs-csi            150m
pvc-2a614118-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc627   glusterfs-csi            150m
pvc-2a9cc64f-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc628   glusterfs-csi            150m
pvc-2adf678d-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc629   glusterfs-csi            150m
pvc-2b1ca353-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc630   glusterfs-csi            150m
pvc-2b587c15-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc631   glusterfs-csi            150m
pvc-2b9b87cb-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc632   glusterfs-csi            150m
pvc-2bf11d7c-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc633   glusterfs-csi            150m
pvc-2c33c25f-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc634   glusterfs-csi            150m
pvc-2c6eb154-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc635   glusterfs-csi            150m
pvc-2caa9c38-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc636   glusterfs-csi            150m
pvc-2ce9ce2e-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc637   glusterfs-csi            150m
pvc-2d218f37-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc638   glusterfs-csi            150m
pvc-2d55c7de-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc639   glusterfs-csi            150m
pvc-2d8ee840-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc640   glusterfs-csi            150m
pvc-2dcd2610-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc641   glusterfs-csi            150m
pvc-2e09d325-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc642   glusterfs-csi            150m
pvc-2e460893-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc643   glusterfs-csi            150m
pvc-2e870664-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc644   glusterfs-csi            150m
pvc-2ec8e65e-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc645   glusterfs-csi            149m
pvc-2f10c8fb-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc646   glusterfs-csi            149m
pvc-2f4cabd2-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc647   glusterfs-csi            149m
pvc-2f89ce3f-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc648   glusterfs-csi            149m
pvc-2fc121de-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc649   glusterfs-csi            149m
pvc-2ffab8a7-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc650   glusterfs-csi            149m
pvc-302ddf05-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc651   glusterfs-csi            149m
pvc-3063b3e4-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc652   glusterfs-csi            149m
pvc-30a3ab40-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc653   glusterfs-csi            149m
pvc-30e9421a-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc654   glusterfs-csi            149m
pvc-312d5e24-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc655   glusterfs-csi            149m
pvc-316769de-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc656   glusterfs-csi            149m
pvc-31b007fd-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc657   glusterfs-csi            149m
pvc-31f80220-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc658   glusterfs-csi            149m
pvc-3232866e-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc659   glusterfs-csi            149m
pvc-326f5dfd-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc660   glusterfs-csi            149m
pvc-32a8539a-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc661   glusterfs-csi            149m
pvc-32e2a9c7-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc662   glusterfs-csi            149m
pvc-33193eda-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc663   glusterfs-csi            149m
pvc-334d516a-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc664   glusterfs-csi            149m
pvc-33911e0f-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc665   glusterfs-csi            148m
pvc-33dae9ac-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc666   glusterfs-csi            148m
pvc-341abb39-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc667   glusterfs-csi            148m
pvc-3454e6a2-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc668   glusterfs-csi            148m
pvc-349faf04-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc669   glusterfs-csi            148m
pvc-34ec51af-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc670   glusterfs-csi            148m
pvc-352a5dac-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc671   glusterfs-csi            148m
pvc-3560f70b-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc672   glusterfs-csi            148m
pvc-35b1da4f-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc673   glusterfs-csi            148m
pvc-360a88c1-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc674   glusterfs-csi            148m
pvc-3646e74d-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc675   glusterfs-csi            148m
pvc-367c8c72-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc676   glusterfs-csi            148m
pvc-36b372c2-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc677   glusterfs-csi            148m
pvc-36ec5ff2-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc678   glusterfs-csi            148m
pvc-3727f9f4-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc679   glusterfs-csi            148m
pvc-3766a24e-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc680   glusterfs-csi            148m
pvc-37a95698-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc681   glusterfs-csi            148m
pvc-37e6f56a-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc682   glusterfs-csi            148m
pvc-382a289b-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc683   glusterfs-csi            148m
pvc-386cebe8-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc684   glusterfs-csi            148m
pvc-38a58058-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc685   glusterfs-csi            147m
pvc-38d923d0-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc686   glusterfs-csi            147m
pvc-390e6e5d-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc687   glusterfs-csi            147m
pvc-394d62d5-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc688   glusterfs-csi            147m
pvc-3986b5ff-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc689   glusterfs-csi            147m
pvc-39c4d58c-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc690   glusterfs-csi            147m
pvc-39fc6664-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc691   glusterfs-csi            147m
pvc-3a34a3fb-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc692   glusterfs-csi            147m
pvc-3a6ba7a6-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc693   glusterfs-csi            147m
pvc-3aaadeb7-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc694   glusterfs-csi            147m
pvc-3af19694-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc695   glusterfs-csi            147m
pvc-3b2ebd51-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc696   glusterfs-csi            147m
pvc-3b6a4910-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc697   glusterfs-csi            147m
pvc-3ba08125-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc698   glusterfs-csi            147m
pvc-3bdce716-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc699   glusterfs-csi            147m
pvc-3c1de6dd-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc700   glusterfs-csi            147m
pvc-3c5bcdb3-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc701   glusterfs-csi            147m
pvc-3ca5ba38-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc702   glusterfs-csi            147m
pvc-3d045c2e-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc703   glusterfs-csi            147m
pvc-3d56dba8-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc704   glusterfs-csi            147m
pvc-3d8e4a3a-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc705   glusterfs-csi            146m
pvc-3dc7e177-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc706   glusterfs-csi            146m
pvc-3e066409-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc707   glusterfs-csi            146m
pvc-3e4464e5-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc708   glusterfs-csi            146m
pvc-3e7d5b36-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc709   glusterfs-csi            146m
pvc-3eb6417f-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc710   glusterfs-csi            146m
pvc-3ef36634-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc711   glusterfs-csi            146m
pvc-3f302f9e-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc712   glusterfs-csi            146m
pvc-3f6ab14c-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc713   glusterfs-csi            146m
pvc-3fa3a9b1-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc714   glusterfs-csi            146m
pvc-3ffb2c1e-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc715   glusterfs-csi            146m
pvc-40419019-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc716   glusterfs-csi            146m
pvc-407f6f47-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc717   glusterfs-csi            146m
pvc-40c5ca99-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc718   glusterfs-csi            146m
pvc-4107d51f-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc719   glusterfs-csi            146m
pvc-41400794-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc720   glusterfs-csi            146m
pvc-4177366d-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc721   glusterfs-csi            146m
pvc-41a9c3ac-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc722   glusterfs-csi            146m
pvc-41e02be6-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc723   glusterfs-csi            146m
pvc-42233412-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc724   glusterfs-csi            145m
pvc-425ed326-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc725   glusterfs-csi            145m
pvc-429ced15-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc726   glusterfs-csi            145m
pvc-42dca405-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc727   glusterfs-csi            145m
pvc-431a74ae-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc728   glusterfs-csi            145m
pvc-43519157-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc729   glusterfs-csi            145m
pvc-4396cda6-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc730   glusterfs-csi            145m
pvc-43d6e273-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc731   glusterfs-csi            145m
pvc-442cdd4e-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc732   glusterfs-csi            145m
pvc-446301ed-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc733   glusterfs-csi            145m
pvc-449b2f24-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc734   glusterfs-csi            145m
pvc-44d39286-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc735   glusterfs-csi            145m
pvc-45068abf-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc736   glusterfs-csi            145m
pvc-45449bb3-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc737   glusterfs-csi            145m
pvc-45841bb8-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc738   glusterfs-csi            145m
pvc-45c385cc-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc739   glusterfs-csi            145m
pvc-460c852a-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc740   glusterfs-csi            145m
pvc-464abebe-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc741   glusterfs-csi            145m
pvc-468998ed-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc742   glusterfs-csi            145m
pvc-46d841b1-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc743   glusterfs-csi            144m
pvc-47198f79-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc744   glusterfs-csi            144m
pvc-475a0fe5-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc745   glusterfs-csi            144m
pvc-478ff5a8-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc746   glusterfs-csi            144m
pvc-47e5694c-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc747   glusterfs-csi            144m
pvc-48219c85-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc748   glusterfs-csi            144m
pvc-48653356-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc749   glusterfs-csi            144m
pvc-489f1f88-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc750   glusterfs-csi            144m
pvc-48d95fbf-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc751   glusterfs-csi            144m
pvc-4910e44e-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc752   glusterfs-csi            144m
pvc-494fc11d-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc753   glusterfs-csi            144m
pvc-499618a4-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc754   glusterfs-csi            144m
pvc-49d72bb3-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc755   glusterfs-csi            144m
pvc-4a183420-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc756   glusterfs-csi            144m
pvc-4a518600-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc757   glusterfs-csi            144m
pvc-4a957183-feb6-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc758   glusterfs-csi            144m
pvc-9d7107d2-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc0     glusterfs-csi            168m
pvc-9da72c6c-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc1     glusterfs-csi            168m
pvc-9dd45026-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc2     glusterfs-csi            168m
pvc-9e04850b-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc3     glusterfs-csi            168m
pvc-9e3b216a-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc4     glusterfs-csi            168m
pvc-9e74ba16-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc5     glusterfs-csi            168m
pvc-9ea88ece-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc6     glusterfs-csi            168m
pvc-9edff6a9-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc7     glusterfs-csi            168m
pvc-9f172834-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc8     glusterfs-csi            168m
pvc-9f4f0121-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc9     glusterfs-csi            168m
pvc-9f8c0c90-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc10    glusterfs-csi            168m
pvc-9fc12499-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc11    glusterfs-csi            168m
pvc-9ffb4ed6-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc12    glusterfs-csi            168m
pvc-a038b8c0-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc13    glusterfs-csi            168m
pvc-a0703d6a-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc14    glusterfs-csi            168m
pvc-a0ac1023-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc15    glusterfs-csi            168m
pvc-a0e3bdb1-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc16    glusterfs-csi            168m
pvc-a119aa39-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc17    glusterfs-csi            168m
pvc-a153ab84-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc18    glusterfs-csi            168m
pvc-a18cda89-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc19    glusterfs-csi            168m
pvc-a1c6a2a0-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc20    glusterfs-csi            168m
pvc-a1fb9301-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc21    glusterfs-csi            168m
pvc-a231012a-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc22    glusterfs-csi            168m
pvc-a265ab14-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc23    glusterfs-csi            168m
pvc-a29b2b99-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc24    glusterfs-csi            168m
pvc-a2ceff29-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc25    glusterfs-csi            168m
pvc-a3034064-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc26    glusterfs-csi            168m
pvc-a33802f5-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc27    glusterfs-csi            168m
pvc-a3736b5d-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc28    glusterfs-csi            168m
pvc-a3a83abe-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc29    glusterfs-csi            168m
pvc-a3d8a2c5-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc30    glusterfs-csi            168m
pvc-a40a6b4f-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc31    glusterfs-csi            168m
pvc-a44337f0-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc32    glusterfs-csi            168m
pvc-a4774abd-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc33    glusterfs-csi            168m
pvc-a4af839a-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc34    glusterfs-csi            168m
pvc-a4e615ad-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc35    glusterfs-csi            168m
pvc-a51a92d5-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc36    glusterfs-csi            168m
pvc-a564474d-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc37    glusterfs-csi            168m
pvc-a5937a6b-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc38    glusterfs-csi            168m
pvc-a5d1c7e6-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc39    glusterfs-csi            168m
pvc-a604cad4-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc40    glusterfs-csi            168m
pvc-a63b797f-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc41    glusterfs-csi            168m
pvc-a66e086b-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc42    glusterfs-csi            168m
pvc-a6a581fc-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc43    glusterfs-csi            168m
pvc-a6d89f5d-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc44    glusterfs-csi            168m
pvc-a70c4a38-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc45    glusterfs-csi            168m
pvc-a740e8e5-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc46    glusterfs-csi            168m
pvc-a7761eb8-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc47    glusterfs-csi            168m
pvc-a7ab1d73-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc48    glusterfs-csi            168m
pvc-a7ecddb1-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc49    glusterfs-csi            168m
pvc-a8221d1f-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc50    glusterfs-csi            168m
pvc-a8563c3b-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc51    glusterfs-csi            168m
pvc-a890bd4b-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc52    glusterfs-csi            168m
pvc-a8c1e9b8-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc53    glusterfs-csi            168m
pvc-a8f31df9-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc54    glusterfs-csi            168m
pvc-a9243cd6-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc55    glusterfs-csi            168m
pvc-a961ca07-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc56    glusterfs-csi            168m
pvc-a999b48a-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc57    glusterfs-csi            168m
pvc-a9cd16e8-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc58    glusterfs-csi            168m
pvc-aa045dda-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc59    glusterfs-csi            168m
pvc-aa3d2428-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc60    glusterfs-csi            168m
pvc-aa7190bb-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc61    glusterfs-csi            168m
pvc-aaa75285-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc62    glusterfs-csi            168m
pvc-aad97dba-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc63    glusterfs-csi            168m
pvc-ab13073c-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc64    glusterfs-csi            168m
pvc-ab57f715-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc65    glusterfs-csi            168m
pvc-ab8a9432-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc66    glusterfs-csi            168m
pvc-abc4632c-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc67    glusterfs-csi            168m
pvc-ac00498e-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc68    glusterfs-csi            168m
pvc-ac37e94e-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc69    glusterfs-csi            168m
pvc-ac752910-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc70    glusterfs-csi            168m
pvc-aca7675a-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc71    glusterfs-csi            167m
pvc-acdb5792-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc72    glusterfs-csi            167m
pvc-ad1516dc-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc73    glusterfs-csi            167m
pvc-ad459233-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc74    glusterfs-csi            167m
pvc-ad7bfb4d-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc75    glusterfs-csi            167m
pvc-adb8f463-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc76    glusterfs-csi            167m
pvc-adee47bf-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc77    glusterfs-csi            167m
pvc-ae21c049-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc78    glusterfs-csi            167m
pvc-ae594745-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc79    glusterfs-csi            167m
pvc-ae8f3b13-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc80    glusterfs-csi            167m
pvc-aec6f859-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc81    glusterfs-csi            167m
pvc-af002ee2-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc82    glusterfs-csi            167m
pvc-af38b0d6-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc83    glusterfs-csi            167m
pvc-af6dfd55-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc84    glusterfs-csi            167m
pvc-af9f17d5-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc85    glusterfs-csi            167m
pvc-afd64199-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc86    glusterfs-csi            167m
pvc-b00bbff0-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc87    glusterfs-csi            167m
pvc-b041dcc9-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc88    glusterfs-csi            167m
pvc-b080ce07-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc89    glusterfs-csi            167m
pvc-b0b2a8ed-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc90    glusterfs-csi            167m
pvc-b0e87f5f-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc91    glusterfs-csi            167m
pvc-b11cae3c-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc92    glusterfs-csi            167m
pvc-b1652dec-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc93    glusterfs-csi            167m
pvc-b19a5308-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc94    glusterfs-csi            167m
pvc-b1cc7a4e-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc95    glusterfs-csi            167m
pvc-b20404ee-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc96    glusterfs-csi            167m
pvc-b2358a2a-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc97    glusterfs-csi            167m
pvc-b267e67f-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc98    glusterfs-csi            167m
pvc-b2a2bc33-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc99    glusterfs-csi            167m
pvc-b2d76d0a-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc100   glusterfs-csi            167m
pvc-b309cc5e-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc101   glusterfs-csi            167m
pvc-b33e5f22-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc102   glusterfs-csi            167m
pvc-b37763e9-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc103   glusterfs-csi            167m
pvc-b3b13d47-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc104   glusterfs-csi            167m
pvc-b3e79654-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc105   glusterfs-csi            167m
pvc-b41d2418-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc106   glusterfs-csi            167m
pvc-b4520324-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc107   glusterfs-csi            167m
pvc-b482ee96-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc108   glusterfs-csi            167m
pvc-b4d87cc8-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc109   glusterfs-csi            167m
pvc-b50f2a00-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc110   glusterfs-csi            167m
pvc-b54427cf-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc111   glusterfs-csi            167m
pvc-b578bb42-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc112   glusterfs-csi            167m
pvc-b5ae7125-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc113   glusterfs-csi            167m
pvc-b5e5b82b-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc114   glusterfs-csi            167m
pvc-b61cc344-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc115   glusterfs-csi            167m
pvc-b652a74a-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc116   glusterfs-csi            167m
pvc-b68c733f-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc117   glusterfs-csi            167m
pvc-b6bdd12b-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc118   glusterfs-csi            167m
pvc-b6f2dac5-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc119   glusterfs-csi            167m
pvc-b72a30b4-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc120   glusterfs-csi            167m
pvc-b76373f1-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc121   glusterfs-csi            167m
pvc-b79b5345-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc122   glusterfs-csi            167m
pvc-b7d11f06-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc123   glusterfs-csi            167m
pvc-b80e36ed-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc124   glusterfs-csi            167m
pvc-b84fffb4-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc125   glusterfs-csi            167m
pvc-b892f3e7-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc126   glusterfs-csi            167m
pvc-b8d01994-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc127   glusterfs-csi            167m
pvc-b9043b1f-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc128   glusterfs-csi            167m
pvc-b93b91ab-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc129   glusterfs-csi            166m
pvc-b970a701-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc130   glusterfs-csi            166m
pvc-b9a91b70-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc131   glusterfs-csi            166m
pvc-b9dc2d80-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc132   glusterfs-csi            166m
pvc-ba1015ee-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc133   glusterfs-csi            166m
pvc-ba49f753-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc134   glusterfs-csi            166m
pvc-ba8ef2c3-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc135   glusterfs-csi            166m
pvc-bac6502c-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc136   glusterfs-csi            166m
pvc-baf923d0-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc137   glusterfs-csi            166m
pvc-bb29a6ba-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc138   glusterfs-csi            166m
pvc-bb5f741a-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc139   glusterfs-csi            166m
pvc-bb96c656-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc140   glusterfs-csi            166m
pvc-bbcc7486-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc141   glusterfs-csi            166m
pvc-bc026672-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc142   glusterfs-csi            166m
pvc-bc5470e6-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc143   glusterfs-csi            166m
pvc-bc91d99d-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc144   glusterfs-csi            166m
pvc-bcc98cf7-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc145   glusterfs-csi            166m
pvc-bcfdfdd4-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc146   glusterfs-csi            166m
pvc-bd32174d-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc147   glusterfs-csi            166m
pvc-bd625d8d-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc148   glusterfs-csi            166m
pvc-bd97a2ca-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc149   glusterfs-csi            166m
pvc-bdc8a422-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc150   glusterfs-csi            166m
pvc-bdf9af66-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc151   glusterfs-csi            166m
pvc-be2dde70-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc152   glusterfs-csi            166m
pvc-be66448a-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc153   glusterfs-csi            166m
pvc-be9b92ae-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc154   glusterfs-csi            166m
pvc-bed0a316-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc155   glusterfs-csi            166m
pvc-bf0901c9-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc156   glusterfs-csi            166m
pvc-bf41890e-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc157   glusterfs-csi            166m
pvc-bf78b467-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc158   glusterfs-csi            166m
pvc-bfafa12d-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc159   glusterfs-csi            166m
pvc-bfe69eaf-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc160   glusterfs-csi            166m
pvc-c018bdf6-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc161   glusterfs-csi            166m
pvc-c051f696-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc162   glusterfs-csi            166m
pvc-c08c89fe-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc163   glusterfs-csi            166m
pvc-c0c63099-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc164   glusterfs-csi            166m
pvc-c0fff960-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc165   glusterfs-csi            166m
pvc-c13baa53-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc166   glusterfs-csi            166m
pvc-c175128f-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc167   glusterfs-csi            166m
pvc-c1b43ef2-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc168   glusterfs-csi            166m
pvc-c1edf4b2-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc169   glusterfs-csi            166m
pvc-c22513a0-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc170   glusterfs-csi            166m
pvc-c25844b4-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc171   glusterfs-csi            166m
pvc-c28c16ec-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc172   glusterfs-csi            166m
pvc-c2bb238c-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc173   glusterfs-csi            166m
pvc-c2f11bcc-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc174   glusterfs-csi            166m
pvc-c3297f04-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc175   glusterfs-csi            166m
pvc-c35e872f-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc176   glusterfs-csi            166m
pvc-c3a10999-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc177   glusterfs-csi            166m
pvc-c3d66414-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc178   glusterfs-csi            166m
pvc-c40c3ff0-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc179   glusterfs-csi            165m
pvc-c43bfaa3-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc180   glusterfs-csi            166m
pvc-c4717273-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc181   glusterfs-csi            165m
pvc-c4a65f10-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc182   glusterfs-csi            165m
pvc-c4d726cd-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc183   glusterfs-csi            165m
pvc-c50874c3-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc184   glusterfs-csi            165m
pvc-c542c50a-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc185   glusterfs-csi            165m
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                STORAGECLASS    REASON   AGE
pvc-c57ae57a-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc186   glusterfs-csi            165m
pvc-c5c4a866-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc187   glusterfs-csi            165m
pvc-c600c707-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc188   glusterfs-csi            165m
pvc-c64176d6-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc189   glusterfs-csi            165m
pvc-c6789753-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc190   glusterfs-csi            165m
pvc-c6b545bc-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc191   glusterfs-csi            165m
pvc-c6e7943c-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc192   glusterfs-csi            165m
pvc-c71d0f55-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc193   glusterfs-csi            165m
pvc-c7542c38-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc194   glusterfs-csi            165m
pvc-c78bbbd4-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc195   glusterfs-csi            165m
pvc-c7c0f414-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc196   glusterfs-csi            165m
pvc-c7f6d6b6-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc197   glusterfs-csi            165m
pvc-c84080aa-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc198   glusterfs-csi            165m
pvc-c87a13d4-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc199   glusterfs-csi            165m
pvc-c8b684e4-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc200   glusterfs-csi            165m
pvc-c8ec050c-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc201   glusterfs-csi            165m
pvc-c9206797-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc202   glusterfs-csi            165m
pvc-c954822a-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc203   glusterfs-csi            165m
pvc-c98a9d96-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc204   glusterfs-csi            165m
pvc-c9c8c9af-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc205   glusterfs-csi            165m
pvc-ca01928b-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc206   glusterfs-csi            165m
pvc-ca3d74da-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc207   glusterfs-csi            165m
pvc-ca75a18b-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc208   glusterfs-csi            165m
pvc-caab7224-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc209   glusterfs-csi            165m
pvc-cae67732-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc210   glusterfs-csi            165m
pvc-cb1ef6e6-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc211   glusterfs-csi            165m
pvc-cb545367-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc212   glusterfs-csi            165m
pvc-cb88ee98-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc213   glusterfs-csi            165m
pvc-cbcff1b7-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc214   glusterfs-csi            165m
pvc-cc023ad2-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc215   glusterfs-csi            165m
pvc-cc3b3045-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc216   glusterfs-csi            165m
pvc-cc7079ac-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc217   glusterfs-csi            165m
pvc-cca6a534-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc218   glusterfs-csi            165m
pvc-cce040e6-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc219   glusterfs-csi            165m
pvc-cd13d1df-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc220   glusterfs-csi            165m
pvc-cd45488e-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc221   glusterfs-csi            165m
pvc-cd7b8db3-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc222   glusterfs-csi            165m
pvc-cdacd048-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc223   glusterfs-csi            165m
pvc-cde1f6e8-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc224   glusterfs-csi            165m
pvc-ce17119e-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc225   glusterfs-csi            164m
pvc-ce5077a2-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc226   glusterfs-csi            164m
pvc-ce840437-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc227   glusterfs-csi            164m
pvc-ceb7de9d-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc228   glusterfs-csi            164m
pvc-ceed8c08-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc229   glusterfs-csi            164m
pvc-cf229f8b-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc230   glusterfs-csi            164m
pvc-cf69bb05-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc231   glusterfs-csi            164m
pvc-cfaaafa3-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc232   glusterfs-csi            164m
pvc-cfdc67fe-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc233   glusterfs-csi            164m
pvc-d0150a94-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc234   glusterfs-csi            164m
pvc-d04651db-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc235   glusterfs-csi            164m
pvc-d078fb74-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc236   glusterfs-csi            164m
pvc-d0ad84e0-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc237   glusterfs-csi            164m
pvc-d0e44cb0-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc238   glusterfs-csi            164m
pvc-d11b774f-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc239   glusterfs-csi            164m
pvc-d1569157-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc240   glusterfs-csi            164m
pvc-d18aad08-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc241   glusterfs-csi            164m
pvc-d1bece12-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc242   glusterfs-csi            164m
pvc-d1f8b655-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc243   glusterfs-csi            164m
pvc-d23a15a8-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc244   glusterfs-csi            164m
pvc-d273a6ca-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc245   glusterfs-csi            164m
pvc-d2aa38b8-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc246   glusterfs-csi            164m
pvc-d2db2ff0-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc247   glusterfs-csi            164m
pvc-d3105e66-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc248   glusterfs-csi            164m
pvc-d34a971b-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc249   glusterfs-csi            164m
pvc-d3864df1-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc250   glusterfs-csi            164m
pvc-d3bbf608-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc251   glusterfs-csi            164m
pvc-d3f96046-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc252   glusterfs-csi            164m
pvc-d43acb26-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc253   glusterfs-csi            164m
pvc-d470a124-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc254   glusterfs-csi            164m
pvc-d4a40724-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc255   glusterfs-csi            164m
pvc-d4dacf84-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc256   glusterfs-csi            164m
pvc-d5101518-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc257   glusterfs-csi            164m
pvc-d542fec3-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc258   glusterfs-csi            164m
pvc-d57abb70-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc259   glusterfs-csi            164m
pvc-d5af5884-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc260   glusterfs-csi            164m
pvc-d5e9ccd6-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc261   glusterfs-csi            164m
pvc-d619894a-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc262   glusterfs-csi            164m
pvc-d650cef7-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc263   glusterfs-csi            164m
pvc-d68ab115-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc264   glusterfs-csi            164m
pvc-d6cb31f6-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc265   glusterfs-csi            163m
pvc-d70963c7-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc266   glusterfs-csi            163m
pvc-d75b44cb-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc267   glusterfs-csi            163m
pvc-d796df4a-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc268   glusterfs-csi            163m
pvc-d7cd4b27-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc269   glusterfs-csi            163m
pvc-d7ff1029-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc270   glusterfs-csi            163m
pvc-d82f7360-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc271   glusterfs-csi            163m
pvc-d8681376-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc272   glusterfs-csi            163m
pvc-d8a26923-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc273   glusterfs-csi            163m
pvc-d8db8fe6-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc274   glusterfs-csi            163m
pvc-d919876a-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc275   glusterfs-csi            163m
pvc-d954ec69-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc276   glusterfs-csi            163m
pvc-d99029ea-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc277   glusterfs-csi            163m
pvc-d9cc55b6-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc278   glusterfs-csi            163m
pvc-da15e9ee-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc279   glusterfs-csi            163m
pvc-da4a054d-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc280   glusterfs-csi            163m
pvc-da7c922c-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc281   glusterfs-csi            163m
pvc-dab1a605-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc282   glusterfs-csi            163m
pvc-daeaf1de-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc283   glusterfs-csi            163m
pvc-db21e93b-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc284   glusterfs-csi            163m
pvc-db64c24e-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc285   glusterfs-csi            163m
pvc-dba412cf-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc286   glusterfs-csi            163m
pvc-dbd680bd-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc287   glusterfs-csi            163m
pvc-dc0ea717-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc288   glusterfs-csi            163m
pvc-dc5b8be1-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc289   glusterfs-csi            163m
pvc-dc956e63-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc290   glusterfs-csi            163m
pvc-dcc824cc-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc291   glusterfs-csi            163m
pvc-dd017fed-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc292   glusterfs-csi            163m
pvc-dd374aeb-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc293   glusterfs-csi            163m
pvc-dd6bef41-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc294   glusterfs-csi            163m
pvc-dda3b6a0-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc295   glusterfs-csi            163m
pvc-dde53272-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc296   glusterfs-csi            163m
pvc-de2adc32-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc297   glusterfs-csi            163m
pvc-de67cdd9-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc298   glusterfs-csi            163m
pvc-dea6c9f7-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc299   glusterfs-csi            163m
pvc-dee1c8bb-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc300   glusterfs-csi            163m
pvc-df1d1cb7-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc301   glusterfs-csi            162m
pvc-df552850-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc302   glusterfs-csi            162m
pvc-df86aba9-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc303   glusterfs-csi            162m
pvc-dfbc0a54-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc304   glusterfs-csi            162m
pvc-dfef51c6-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc305   glusterfs-csi            162m
pvc-e026f7c9-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc306   glusterfs-csi            162m
pvc-e05dca9f-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc307   glusterfs-csi            162m
pvc-e0918624-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc308   glusterfs-csi            162m
pvc-e0c775b6-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc309   glusterfs-csi            162m
pvc-e10c5df3-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc310   glusterfs-csi            162m
pvc-e143ccf4-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc311   glusterfs-csi            162m
pvc-e183ea76-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc312   glusterfs-csi            162m
pvc-e1ba3bce-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc313   glusterfs-csi            162m
pvc-e1e9984b-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc314   glusterfs-csi            162m
pvc-e2188cd7-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc315   glusterfs-csi            162m
pvc-e24b708b-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc316   glusterfs-csi            162m
pvc-e282e900-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc317   glusterfs-csi            162m
pvc-e2bbc786-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc318   glusterfs-csi            162m
pvc-e2f3ce10-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc319   glusterfs-csi            162m
pvc-e32fb424-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc320   glusterfs-csi            162m
pvc-e36ef66b-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc321   glusterfs-csi            162m
pvc-e3ab9e59-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc322   glusterfs-csi            162m
pvc-e3dd3065-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc323   glusterfs-csi            162m
pvc-e41a31b5-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc324   glusterfs-csi            162m
pvc-e451f95c-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc325   glusterfs-csi            162m
pvc-e49173f5-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc326   glusterfs-csi            162m
pvc-e4c465d0-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc327   glusterfs-csi            162m
pvc-e4f5f783-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc328   glusterfs-csi            162m
pvc-e52e3fe5-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc329   glusterfs-csi            162m
pvc-e56d315a-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc330   glusterfs-csi            162m
pvc-e5a5c32e-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc331   glusterfs-csi            162m
pvc-e5e64dd0-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc332   glusterfs-csi            162m
pvc-e61e43cf-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc333   glusterfs-csi            162m
pvc-e65fd349-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc334   glusterfs-csi            162m
pvc-e697d9d8-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc335   glusterfs-csi            162m
pvc-e6d09da0-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc336   glusterfs-csi            162m
pvc-e707d42b-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc337   glusterfs-csi            161m
pvc-e73c923f-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc338   glusterfs-csi            161m
pvc-e776c448-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc339   glusterfs-csi            161m
pvc-e7ab7c78-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc340   glusterfs-csi            161m
pvc-e7e81a98-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc341   glusterfs-csi            161m
pvc-e8203d55-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc342   glusterfs-csi            161m
pvc-e85bb19a-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc343   glusterfs-csi            161m
pvc-e8989e88-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc344   glusterfs-csi            161m
pvc-e8cfc077-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc345   glusterfs-csi            161m
pvc-e930c265-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc346   glusterfs-csi            161m
pvc-e974003b-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc347   glusterfs-csi            161m
pvc-e9a98306-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc348   glusterfs-csi            161m
pvc-e9db781a-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc349   glusterfs-csi            161m
pvc-ea0ca12e-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc350   glusterfs-csi            161m
pvc-ea4e65ec-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc351   glusterfs-csi            161m
pvc-ea8940c9-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc352   glusterfs-csi            161m
pvc-eac6e13b-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc353   glusterfs-csi            161m
pvc-eb025328-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc354   glusterfs-csi            161m
pvc-eb3cfc70-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc355   glusterfs-csi            161m
pvc-eb753d84-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc356   glusterfs-csi            161m
pvc-ebaeec38-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc357   glusterfs-csi            161m
pvc-ebe4284a-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc358   glusterfs-csi            161m
pvc-ec4d892d-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc359   glusterfs-csi            161m
pvc-ec7e8960-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc360   glusterfs-csi            161m
pvc-ecb2c4c8-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc361   glusterfs-csi            161m
pvc-ece46cd1-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc362   glusterfs-csi            161m
pvc-ed2b2bf4-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc363   glusterfs-csi            161m
pvc-ed679996-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc364   glusterfs-csi            161m
pvc-eda19a71-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc365   glusterfs-csi            161m
pvc-ede28fb7-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc366   glusterfs-csi            161m
pvc-ee1e2ae7-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc367   glusterfs-csi            161m
pvc-ee580898-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc368   glusterfs-csi            161m
pvc-ee8c8175-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc369   glusterfs-csi            160m
pvc-eec5ba33-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc370   glusterfs-csi            160m
pvc-eefd6639-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc371   glusterfs-csi            160m
pvc-ef3ccbab-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc372   glusterfs-csi            160m
pvc-ef6f67f5-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc373   glusterfs-csi            160m
pvc-efa6b149-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc374   glusterfs-csi            160m
pvc-eff50d01-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc375   glusterfs-csi            160m
pvc-f032865e-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc376   glusterfs-csi            160m
pvc-f0704ba7-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc377   glusterfs-csi            160m
pvc-f0a54792-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc378   glusterfs-csi            160m
pvc-f0e8761d-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc379   glusterfs-csi            160m
pvc-f11c6eb4-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc380   glusterfs-csi            160m
pvc-f1596b01-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc381   glusterfs-csi            160m
pvc-f18e3794-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc382   glusterfs-csi            160m
pvc-f1c9a356-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc383   glusterfs-csi            160m
pvc-f2116107-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc384   glusterfs-csi            160m
pvc-f245f301-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc385   glusterfs-csi            160m
pvc-f27845cf-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc386   glusterfs-csi            160m
pvc-f2aa5dd0-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc387   glusterfs-csi            160m
pvc-f2de1a45-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc388   glusterfs-csi            160m
pvc-f31ad3a0-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc389   glusterfs-csi            160m
pvc-f3571380-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc390   glusterfs-csi            160m
pvc-f38c906a-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc391   glusterfs-csi            160m
pvc-f3c70e36-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc392   glusterfs-csi            160m
pvc-f3fd0185-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc393   glusterfs-csi            160m
pvc-f435bd86-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc394   glusterfs-csi            160m
pvc-f46c072d-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc395   glusterfs-csi            160m
pvc-f4a41a57-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc396   glusterfs-csi            160m
pvc-f4f16507-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc397   glusterfs-csi            160m
pvc-f52629e6-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc398   glusterfs-csi            160m
pvc-f579d124-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc399   glusterfs-csi            160m
pvc-f5ace6be-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc400   glusterfs-csi            160m
pvc-f5e4966c-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc401   glusterfs-csi            159m
pvc-f61edafd-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc402   glusterfs-csi            159m
pvc-f65ee8e6-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc403   glusterfs-csi            159m
pvc-f6a095ab-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc404   glusterfs-csi            159m
pvc-f6d9a254-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc405   glusterfs-csi            159m
pvc-f71723e9-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc406   glusterfs-csi            159m
pvc-f74e7059-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc407   glusterfs-csi            159m
pvc-f785d856-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc408   glusterfs-csi            159m
pvc-f7c626e8-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc409   glusterfs-csi            159m
pvc-f8055c64-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc410   glusterfs-csi            159m
pvc-f83c924f-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc411   glusterfs-csi            159m
pvc-f87128c4-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc412   glusterfs-csi            159m
pvc-f8a3b77a-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc413   glusterfs-csi            159m
pvc-f8dfafa8-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc414   glusterfs-csi            159m
pvc-f91d65fd-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc415   glusterfs-csi            159m
pvc-f95de037-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc416   glusterfs-csi            159m
pvc-f99c7405-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc417   glusterfs-csi            159m
pvc-f9e311df-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc418   glusterfs-csi            159m
pvc-fa1a1458-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc419   glusterfs-csi            159m
pvc-fa4c730a-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc420   glusterfs-csi            159m
pvc-fa94080a-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc421   glusterfs-csi            159m
pvc-facf830c-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc422   glusterfs-csi            159m
pvc-fb085a68-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc423   glusterfs-csi            159m
pvc-fb3ef599-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc424   glusterfs-csi            159m
pvc-fb75a2eb-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc425   glusterfs-csi            159m
pvc-fbaefd13-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc426   glusterfs-csi            159m
pvc-fbe3029b-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc427   glusterfs-csi            159m
pvc-fc1f3566-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc428   glusterfs-csi            159m
pvc-fc5bfb2b-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc429   glusterfs-csi            159m
pvc-fca0c955-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc430   glusterfs-csi            159m
pvc-fcda03b1-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc431   glusterfs-csi            159m
pvc-fd1429b7-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc432   glusterfs-csi            159m
pvc-fd4ce5da-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc433   glusterfs-csi            158m
pvc-fd814b9c-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc434   glusterfs-csi            158m
pvc-fdcafa15-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc435   glusterfs-csi            158m
pvc-fdfea413-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc436   glusterfs-csi            158m
pvc-fe36c259-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc437   glusterfs-csi            158m
pvc-fe6db081-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc438   glusterfs-csi            158m
pvc-fea7362a-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc439   glusterfs-csi            158m
pvc-feeb3d20-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc440   glusterfs-csi            158m
pvc-ff2b45c4-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc441   glusterfs-csi            158m
pvc-ff66625e-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc442   glusterfs-csi            158m
pvc-ffa3f25d-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc443   glusterfs-csi            158m
pvc-ffdb5fbb-feb5-11e8-9256-525400058ab6   2Gi        RWX            Delete           Released   default/gcs-pvc444   glusterfs-csi            158m
[vagrant@kube1 ~]$

Seems like i am hitting the below issue and very consistently
gluster/gcs#25

@ghost ghost removed the bug label Dec 13, 2018
aravindavk pushed a commit that referenced this issue Dec 13, 2018
@Madhu-1 @aravindavk
Volume Create: Add a check to validate if volume exists
05fee9f 
during volume create, before starting
of transaction,add a check to validate
volume already exists or not.

Partial Fixes: #1364

Signed-off-by: Madhu Rajanna <mrajanna@redhat.com>
@Madhu-1
Copy link
Member Author

Madhu-1 commented Dec 13, 2018

E1205 07:53:57.951456 1 controller.go:701] error syncing claim "default/gcs-app636": failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: Transaction step vol-create.PrepareBricks failed on peer 74d48f37-812e-4459-b97b-10983c261312 with error: exit status 32; mount: mount /dev/mapper/vg--dev--vdd-brick_pvc--df1f7786--f862--11e8--bb41--525400770e06_s1_b2 on /run/glusterd2/bricks/pvc-df1f7786-f862-11e8-bb41-525400770e06/subvol1/brick2 failed: Cannot allocate memory
I1205 07:53:57.951826 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-app636", UID:"df1f7786-f862-11e8-bb41-525400770e06", APIVersion:"v1", ResourceVersion:"53238", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: Transaction step vol-create.PrepareBricks failed on peer 74d48f37-812e-4459-b97b-10983c261312 with error: exit status 32; mount: mount /dev/mapper/vg--dev--vdd-brick_pvc--df1f7786--f862--11e8--bb41--525400770e06_s1_b2 on /run/glusterd2/bricks/pvc-df1f7786-f862-11e8-bb41-525400770e06/subvol1/brick2 failed: Cannot allocate memory
W1205 07:53:58.113959 1 controller.go:686] Retrying syncing claim "default/gcs-app637" because failures 0 < threshold 15
E1205 07:53:58.114000 1 controller.go:701] error syncing claim "default/gcs-app637": failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: Transaction step vol-create.PrepareBricks failed on peer 74d48f37-812e-4459-b97b-10983c261312 with error: exit status 5; fork failed: Cannot allocate memory
Check of pool vg-dev-vdd/tp_pvc-e556c971-f862-11e8-bb41-525400770e06_s1_b2 failed (status:-1). Manual repair required!
Failed to activate thin pool vg-dev-vdd/tp_pvc-e556c971-f862-11e8-bb41-525400770e06_s1_b2.
I1205 07:53:58.114323 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-app637", UID:"e556c971-f862-11e8-bb41-525400770e06", APIVersion:"v1", ResourceVersion:"53244", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: Transaction step vol-create.PrepareBricks failed on peer 74d48f37-812e-4459-b97b-10983c261312 with error: exit status 5; fork failed: Cannot allocate memory
Check of pool vg-dev-vdd/tp_pvc-e556c971-f862-11e8-bb41-525400770e06_s1_b2 failed (status:-1). Manual repair required!
Failed to activate thin pool vg-dev-vdd/tp_pvc-e556c971-f862-11e8-bb41-525400770e06_s1_b2.

@aravindavk i have seen a different error message, i don't think this has been fixed

@Madhu-1
Copy link
Member Author

Madhu-1 commented Dec 17, 2018

as a chat with @ksandha seems like he is also hitting the same issue , so reopening the issue

@Madhu-1 Madhu-1 reopened this Dec 17, 2018
@ksandha ksandha mentioned this issue Dec 17, 2018
Closed
@atinmu atinmu added the GCS/0.5 GCS 0.5 release label Dec 17, 2018
@atinmu
Copy link
Contributor

atinmu commented Jan 9, 2019

We haven't seen this in our multiple iterations of recent scale testing environment.

@ksandha
Copy link

ksandha commented Jan 10, 2019

@atinmu I will again try this scenario tonight and update you by tomorrows scrum. Last time when we tried to scale we were able to scale upto 716 PVC.

@atinmu atinmu removed the GCS/0.5 GCS 0.5 release label Jan 11, 2019
@ksandha ksandha mentioned this issue Jan 11, 2019
Closed
@ksandha
Copy link

ksandha commented Jan 11, 2019

Created the parallel PVC creation on non brick mux environment. Below is the state of cluster once the pvc reached 548

[vagrant@kube1 ~]$ kubectl get pods -n gcs
NAME                                   READY   STATUS        RESTARTS   AGE
anthill-58b9b9b6f-cnn2s                1/1     Running       0          9h
anthill-58b9b9b6f-dr2f6                1/1     Terminating   0          47h
csi-attacher-glusterfsplugin-0         2/2     Terminating   0          47h
csi-nodeplugin-glusterfsplugin-ckfpz   2/2     Running       0          47h
csi-nodeplugin-glusterfsplugin-qm8lw   2/2     Running       0          47h
csi-nodeplugin-glusterfsplugin-z5j5l   2/2     Running       0          47h
csi-provisioner-glusterfsplugin-0      3/3     Terminating   0          47h
etcd-l2456qnlkf                        0/1     Error         0          47h
etcd-operator-77bfcd6595-kxp9f         1/1     Running       0          9h
etcd-operator-77bfcd6595-n962c         1/1     Terminating   0          47h
etcd-wx7l5mzl4n                        0/1     Terminating   0          10h
etcd-zmzth8pg2g                        0/1     Terminating   0          47h
gluster-kube1-0                        1/1     Terminating   0          47h
gluster-kube2-0                        1/1     Running       1          47h
gluster-kube3-0                        1/1     Terminating   1          47h
[vagrant@kube1 ~]$

@rishubhjain
Copy link
Contributor

Can you provide the glusterd2 logs as well as etcd pod logs

@ksandha
Copy link

ksandha commented Jan 11, 2019

@rishubhjain can you please login to rhsqa-virt05.lab.eng.blr.redhat.com (root/GCS-karan/deploy)

I see the glusterd2 logs are 78MB in size which I wont be able to attach it here.

@atinmu atinmu removed the GCS/1.0 Issue is blocker for Gluster for Container Storage label Jan 17, 2019
@atinmu
Copy link
Contributor

atinmu commented Jan 17, 2019

I believe this is already fixed in the latest master (or GCS 0.6) of GCS.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@aravindavk aravindavk

Labels
priority: high
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@aravindavk @rishubhjain @atinmu @Madhu-1 @ksandha @rmadaka