Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MongoDB Application Deployment fails upon using xfs as fstype in packet #2078

Closed
nsathyaseelan opened this issue Oct 12, 2018 · 2 comments
Closed

MongoDB Application Deployment fails upon using xfs as fstype in packet #2078

nsathyaseelan opened this issue Oct 12, 2018 · 2 comments
Labels
No ISSUE activity

Comments

@nsathyaseelan
Copy link
Contributor

BUG REPORT
What happened:
-Tried to deploy MongoDB statefulset with 3 replicas.
-Application pods remains in container creating state upon using XFS as fstype in packet cloud.
(Used openebs-mongodb as a storageclass which has xfs as FStype)

Application pod description:

iscsi: failed to sendtargets to portal 10.106.6.8:3260 output: iscsiadm: cannot make connection to 10.106.6.8: Connection refused
iscsiadm: cannot make connection to 10.106.6.8: Connection refused
iscsiadm: cannot make connection to 10.106.6.8: Connection refused
iscsiadm: cannot make connection to 10.106.6.8: Connection refused
iscsiadm: cannot make connection to 10.106.6.8: Connection refused
iscsiadm: cannot make connection to 10.106.6.8: Connection refused
iscsiadm: connection login retries (reopen_max) 5 exceeded
iscsiadm: No portals found
, err exit status 21
  Warning  FailedMount  35m (x4 over 37m)  kubelet, node-sharp-bohr02  MountVolume.MountDevice failed for volume "pvc-edf28c48-cddc-11e8-878c-0cc47a205cf8" : executable file not found in $PATH
  Warning  FailedMount  33m                kubelet, node-sharp-bohr02  MountVolume.MountDevice failed for volume "pvc-edf28c48-cddc-11e8-878c-0cc47a205cf8" : failed to mount the volume as "xfs", it already contains ext4. Mount error: mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-default/10.106.6.8:3260-iqn.2016-09.com.openebs.jiva:pvc-edf28c48-cddc-11e8-878c-0cc47a205cf8-lun-0 --scope -- mount -t xfs -o defaults /dev/dm-0 /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-default/10.106.6.8:3260-iqn.2016-09.com.openebs.jiva:pvc-edf28c48-cddc-11e8-878c-0cc47a205cf8-lun-0
Output: Running scope as unit run-r3cc9052e4617403f8e66360e800ebdcc.scope.
mount: /dev/mapper/33000000000000001 is already mounted or /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-default/10.106.6.8:3260-iqn.2016-09.com.openebs.jiva:pvc-edf28c48-cddc-11e8-878c-0cc47a205cf8-lun-0 busy
       /dev/mapper/33000000000000001 is already mounted on /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-default/10.103.68.7:3260-iqn.2016-09.com.openebs.jiva:pvc-6343f263-cddd-11e8-878c-0cc47a205cf8-lun-0
       /dev/mapper/33000000000000001 is already mounted on /var/lib/kubelet/pods/63473b4c-cddd-11e8-878c-0cc47a205cf8/volumes/kubernetes.io~iscsi/pvc-6343f263-cddd-11e8-878c-0cc47a205cf8
  Warning  FailedMount  31m  kubelet, node-sharp-bohr02  MountVolume.MountDevice failed for volume "pvc-edf28c48-cddc-11e8-878c-0cc47a205cf8" : failed to mount the volume as "xfs", it already contains ext4. Mount error: mount failed: exit status 32

Controller Logs:-

time="2018-10-12T05:09:10Z" level=info msg="REPLICATION_FACTOR: 3" 
time="2018-10-12T05:09:10Z" level=info msg="Starting controller with frontendIP: , and clusterIP: 10.106.6.8" 
time="2018-10-12T05:09:10Z" level=info msg="resetting controller" 
time="2018-10-12T05:09:10Z" level=info msg="Listening on :9501" 
time="2018-10-12T05:09:38Z" level=info msg="List Replicas" 
time="2018-10-12T05:09:38Z" level=info msg="List Replicas" 
time="2018-10-12T05:09:38Z" level=info msg="Register Replica for address 10.1.3.10" 
time="2018-10-12T05:09:38Z" level=info msg="Register Replica, Address: 10.1.3.10 Uptime: 35.046473855s State: closed Type: Backend RevisionCount: 0" 
time="2018-10-12T05:09:38Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1" 
10.1.3.10 - - [12/Oct/2018:05:09:38 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2018-10-12T05:09:42Z" level=info msg="List Replicas" 
time="2018-10-12T05:09:42Z" level=info msg="List Replicas" 
time="2018-10-12T05:09:42Z" level=info msg="Register Replica for address 10.1.1.9" 
time="2018-10-12T05:09:42Z" level=info msg="Register Replica, Address: 10.1.1.9 Uptime: 35.033802011s State: closed Type: Backend RevisionCount: 0" 
time="2018-10-12T05:09:42Z" level=info msg="Replica 10.1.3.10 signalled to start, no of registered replicas are 2 and replica count is 3" 
10.1.1.9 - - [12/Oct/2018:05:09:42 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2018-10-12T05:09:42Z" level=info msg="resetting controller" 
time="2018-10-12T05:09:42Z" level=info msg="Connecting to remote: 10.1.3.10:9502" 
time="2018-10-12T05:09:42Z" level=info msg="Opening: 10.1.3.10:9502" 
time="2018-10-12T05:09:42Z" level=info msg="check if replica tcp://10.1.3.10:9502 is already added" 
time="2018-10-12T05:09:42Z" level=info msg="check if any WO replica available" 
time="2018-10-12T05:09:42Z" level=info msg="Adding backend: tcp://10.1.3.10:9502" 
time="2018-10-12T05:09:42Z" level=info msg="replicator reset false" 
time="2018-10-12T05:09:42Z" level=info msg="buildreadwriters: prev: 0 0 cur: 1 0" 
time="2018-10-12T05:09:42Z" level=info msg="Start monitoring tcp://10.1.3.10:9502" 
time="2018-10-12T05:09:42Z" level=info msg="Get backend tcp://10.1.3.10:9502 clone status" 
time="2018-10-12T05:09:42Z" level=error msg="Waiting for replica to update CloneStatus to Completed/NA, retry after 2s" 
time="2018-10-12T05:09:43Z" level=info msg="List Replicas" 
time="2018-10-12T05:09:44Z" level=info msg="Get backend tcp://10.1.3.10:9502 clone status" 
time="2018-10-12T05:09:44Z" level=info msg="Set replica tcp://10.1.3.10:9502 to mode RW" 
time="2018-10-12T05:09:44Z" level=info msg="addr tcp://10.1.3.10:9502 m: RW prev: WO in setmode" 
time="2018-10-12T05:09:44Z" level=info msg="replicator reset false" 
time="2018-10-12T05:09:44Z" level=info msg="buildreadwriters: prev: 0 0 cur: 1 1" 
time="2018-10-12T05:09:44Z" level=info msg="Get backend tcp://10.1.3.10:9502 revision counter 0" 
time="2018-10-12T05:09:44Z" level=info msg="sending add signal to 10.1.1.9" 
time="2018-10-12T05:09:44Z" level=info msg="Create Replica for address tcp://10.1.1.9:9502" 
time="2018-10-12T05:09:44Z" level=info msg="Update volume status" 
time="2018-10-12T05:09:44Z" level=info msg="controller readonly p:true c:true rcount:1 rw_count:1" 
time="2018-10-12T05:09:44Z" level=info msg="backends len: 1" 
time="2018-10-12T05:09:44Z" level=info msg="Start SCSI target" 
time="2018-10-12T05:09:44Z" level=info msg="SCSI device created" 
time="2018-10-12T05:09:44Z" level=info msg="Listening ..." 
10.1.3.10 - - [12/Oct/2018:05:09:42 +0000] "POST /v1/volumes/cHZjLWVkZjI4YzQ4LWNkZGMtMTFlOC04NzhjLTBjYzQ3YTIwNWNmOA==?action=start HTTP/1.1" 200 645
time="2018-10-12T05:09:44Z" level=info msg="check if replica tcp://10.1.1.9:9502 is already added" 
time="2018-10-12T05:09:44Z" level=info msg="check if any WO replica available" 
time="2018-10-12T05:09:44Z" level=info msg="verify replication factor" 
time="2018-10-12T05:09:44Z" level=info msg="Connecting to remote: 10.1.1.9:9502" 
time="2018-10-12T05:09:44Z" level=info msg="List Replicas" 
time="2018-10-12T05:09:44Z" level=info msg="Opening: 10.1.1.9:9502" 
time="2018-10-12T05:09:44Z" level=info msg="Create Replica for address tcp://10.1.2.10:9502" 
time="2018-10-12T05:09:44Z" level=info msg="check if replica tcp://10.1.2.10:9502 is already added" 
time="2018-10-12T05:09:44Z" level=info msg="check if any WO replica available" 
time="2018-10-12T05:09:44Z" level=info msg="verify replication factor" 
time="2018-10-12T05:09:44Z" level=info msg="Connecting to remote: 10.1.2.10:9502" 
time="2018-10-12T05:09:44Z" level=info msg="Opening: 10.1.2.10:9502" 
time="2018-10-12T05:09:44Z" level=info msg="check if replica tcp://10.1.1.9:9502 is already added" 
time="2018-10-12T05:09:44Z" level=info msg="check if any WO replica available" 
time="2018-10-12T05:09:44Z" level=info msg="Snapshot: 10.1.3.10:9502 3efaa9d4-2bbb-4def-ab26-914931f22d9e UserCreated false Created at 2018-10-12T05:09:44Z" 
time="2018-10-12T05:09:45Z" level=info msg="successfully taken snapshots cnt 1" 
time="2018-10-12T05:09:45Z" level=info msg="Snapshot: 10.1.1.9:9502 3efaa9d4-2bbb-4def-ab26-914931f22d9e UserCreated false Created at 2018-10-12T05:09:44Z" 
time="2018-10-12T05:09:45Z" level=info msg="Adding backend: tcp://10.1.1.9:9502" 
time="2018-10-12T05:09:45Z" level=info msg="replicator reset false" 
time="2018-10-12T05:09:45Z" level=info msg="buildreadwriters: prev: 0 1 cur: 2 1" 
time="2018-10-12T05:09:45Z" level=info msg="controller readonly p:true c:true rcount:2 rw_count:1" 
time="2018-10-12T05:09:45Z" level=info msg="backends len: 2" 
time="2018-10-12T05:09:45Z" level=info msg="Start monitoring tcp://10.1.1.9:9502" 
time="2018-10-12T05:09:45Z" level=info msg="check if replica tcp://10.1.2.10:9502 is already added" 
time="2018-10-12T05:09:45Z" level=info msg="check if any WO replica available" 
time="2018-10-12T05:09:45Z" level=warning msg="can have only one WO replica at a time, found WO replica: %stcp://10.1.1.9:9502" 
10.1.1.9 - - [12/Oct/2018:05:09:44 +0000] "POST /v1/replicas HTTP/1.1" 200 402
time="2018-10-12T05:09:45Z" level=info msg="addReplicaNoLock tcp://10.1.2.10:9502 from addReplica failed can only have one WO replica at a time, found WO Replica: tcp://10.1.1.9:9502" 
time="2018-10-12T05:09:45Z" level=error msg="Error in request: can only have one WO replica at a time, found WO Replica: tcp://10.1.1.9:9502" 
10.1.2.10 - - [12/Oct/2018:05:09:44 +0000] "POST /v1/replicas HTTP/1.1" 500 221
time="2018-10-12T05:09:45Z" level=info msg="List Replicas" 
time="2018-10-12T05:09:45Z" level=info msg="List Replicas" 
time="2018-10-12T05:09:45Z" level=info msg="Get Replica for id tcp://10.1.1.9:9502" 
10.1.1.9 - - [12/Oct/2018:05:09:45 +0000] "GET /v1/replicas/dGNwOi8vMTAuMS4xLjk6OTUwMg== HTTP/1.1" 200 402
time="2018-10-12T05:09:45Z" level=info msg="Prepare Rebuild Replica for id tcp://10.1.1.9:9502"
time="2018-10-12T05:09:45Z" level=info msg="Set revision counter of 10.1.1.9:9502 to : 0" 
time="2018-10-12T05:09:45Z" level=info msg="Set backend tcp://10.1.1.9:9502 revision counter to 0" 
time="2018-10-12T05:09:45Z" level=info msg="Synchronizing volume-head-001.img.meta@tcp://10.1.3.10:9502 to volume-head-001.img.meta@10.1.1.9:9700" 
time="2018-10-12T05:09:46Z" level=info msg="Done synchronizing volume-head-001.img.meta to volume-head-001.img.meta@10.1.1.9:9700" 
10.1.1.9 - - [12/Oct/2018:05:09:45 +0000] "POST /v1/replicas/dGNwOi8vMTAuMS4xLjk6OTUwMg==?action=preparerebuild HTTP/1.1" 200 224
time="2018-10-12T05:09:47Z" level=info msg="Create Replica for address tcp://10.1.2.10:9502" 
time="2018-10-12T05:09:47Z" level=info msg="check if replica tcp://10.1.2.10:9502 is already added" 
time="2018-10-12T05:09:47Z" level=info msg="check if any WO replica available" 
time="2018-10-12T05:09:47Z" level=warning msg="can have only one WO replica at a time, found WO replica: %stcp://10.1.1.9:9502" 
time="2018-10-12T05:09:47Z" level=error msg="Error in request: can only have one WO replica at a time, found WO Replica: tcp://10.1.1.9:9502" 
10.1.2.10 - - [12/Oct/2018:05:09:47 +0000] "POST /v1/replicas HTTP/1.1" 500 221
time="2018-10-12T05:09:48Z" level=error msg="replying TypeErr for seq 2 of type 6 on addr 10.1.2.10:9503" 
time="2018-10-12T05:09:49Z" level=info msg="Create Replica for address tcp://10.1.2.10:9502" 
time="2018-10-12T05:09:49Z" level=info msg="check if replica tcp://10.1.2.10:9502 is already added" 
time="2018-10-12T05:09:49Z" level=info msg="check if any WO replica available" 
time="2018-10-12T05:09:49Z" level=warning msg="can have only one WO replica at a time, found WO replica: %stcp://10.1.1.9:9502" 
time="2018-10-12T05:09:49Z" level=error msg="Error in request: can only have one WO replica at a time, found WO Replica: tcp://10.1.1.9:9502" 
10.1.2.10 - - [12/Oct/2018:05:09:49 +0000] "POST /v1/replicas HTTP/1.1" 500 221
time="2018-10-12T05:09:50Z" level=info msg="Exiting rpc loop for 10.1.2.10:9503 with err ping failure: replica state closed" 
time="2018-10-12T05:09:50Z" level=info msg="Exiting rpc writer, RemoteAddr:10.1.2.10:9503" 
time="2018-10-12T05:09:50Z" level=info msg="Get Replica for id tcp://10.1.1.9:9502" 
10.1.1.9 - - [12/Oct/2018:05:09:50 +0000] "GET /v1/replicas/dGNwOi8vMTAuMS4xLjk6OTUwMg== HTTP/1.1" 200 402
time="2018-10-12T05:09:50Z" level=info msg="Verify Rebuild Replica for id tcp://10.1.1.9:9502" 
time="2018-10-12T05:09:50Z" level=info msg="chain [volume-head-001.img volume-snap-3efaa9d4-2bbb-4def-ab26-914931f22d9e.img] from rw replica tcp://10.1.3.10:9502" 
time="2018-10-12T05:09:50Z" level=info msg="chain [volume-head-001.img volume-snap-3efaa9d4-2bbb-4def-ab26-914931f22d9e.img] from wo replica tcp://10.1.1.9:9502" 
time="2018-10-12T05:09:50Z" level=info msg="Get backend tcp://10.1.3.10:9502 revision counter 0" 
time="2018-10-12T05:09:50Z" level=info msg="rw replica tcp://10.1.3.10:9502 revision counter 0" 
time="2018-10-12T05:09:50Z" level=info msg="Set revision counter of 10.1.1.9:9502 to : 0" 
time="2018-10-12T05:09:50Z" level=info msg="Set backend tcp://10.1.1.9:9502 revision counter to 0" 
time="2018-10-12T05:09:50Z" level=info msg="Set replica tcp://10.1.1.9:9502 to mode RW" 
time="2018-10-12T05:09:50Z" level=info msg="addr tcp://10.1.1.9:9502 m: RW prev: WO in setmode" 
time="2018-10-12T05:09:50Z" level=info msg="replicator reset false" 
time="2018-10-12T05:09:50Z" level=info msg="buildreadwriters: prev: 0 1 cur: 2 2" 
time="2018-10-12T05:09:50Z" level=info msg="controller readonly p:true c:false rcount:2 rw_count:2" 
time="2018-10-12T05:09:50Z" level=info msg="backends len: 2" 
time="2018-10-12T05:09:50Z" level=info msg="Get Replica for id tcp://10.1.1.9:9502" 
10.1.1.9 - - [12/Oct/2018:05:09:50 +0000] "POST /v1/replicas/dGNwOi8vMTAuMS4xLjk6OTUwMg==?action=verifyrebuild HTTP/1.1" 200 402
time="2018-10-12T05:09:51Z" level=info msg="Create Replica for address tcp://10.1.2.10:9502" 
time="2018-10-12T05:09:51Z" level=info msg="check if replica tcp://10.1.2.10:9502 is already added" 
time="2018-10-12T05:09:51Z" level=info msg="check if any WO replica available"
time="2018-10-12T05:09:51Z" level=info msg="verify replication factor" 
time="2018-10-12T05:09:51Z" level=info msg="Connecting to remote: 10.1.2.10:9502" 
time="2018-10-12T05:09:51Z" level=info msg="Opening: 10.1.2.10:9502" 
time="2018-10-12T05:09:51Z" level=info msg="check if replica tcp://10.1.2.10:9502 is already added" 
time="2018-10-12T05:09:51Z" level=info msg="check if any WO replica available" 
time="2018-10-12T05:09:51Z" level=info msg="Snapshot: 10.1.1.9:9502 ff10975d-65e3-4a96-a1c5-e17547fb8558 UserCreated false Created at 2018-10-12T05:09:51Z" 
time="2018-10-12T05:09:51Z" level=info msg="Snapshot: 10.1.3.10:9502 ff10975d-65e3-4a96-a1c5-e17547fb8558 UserCreated false Created at 2018-10-12T05:09:51Z" 
time="2018-10-12T05:09:51Z" level=info msg="successfully taken snapshots cnt 2" 
time="2018-10-12T05:09:51Z" level=info msg="Snapshot: 10.1.2.10:9502 ff10975d-65e3-4a96-a1c5-e17547fb8558 UserCreated false Created at 2018-10-12T05:09:51Z" 
time="2018-10-12T05:09:51Z" level=info msg="Adding backend: tcp://10.1.2.10:9502" 
time="2018-10-12T05:09:51Z" level=info msg="replicator reset false" 
time="2018-10-12T05:09:51Z" level=info msg="buildreadwriters: prev: 0 2 cur: 3 2" 
time="2018-10-12T05:09:51Z" level=info msg="controller readonly p:false c:false rcount:3 rw_count:2" 
time="2018-10-12T05:09:51Z" level=info msg="backends len: 3" 
10.1.2.10 - - [12/Oct/2018:05:09:51 +0000] "POST /v1/replicas HTTP/1.1" 200 403
time="2018-10-12T05:09:51Z" level=info msg="Start monitoring tcp://10.1.2.10:9502" 
time="2018-10-12T05:09:51Z" level=info msg="List Replicas" 
time="2018-10-12T05:09:51Z" level=info msg="List Replicas" 
time="2018-10-12T05:09:51Z" level=info msg="Get Replica for id tcp://10.1.2.10:9502" 
10.1.2.10 - - [12/Oct/2018:05:09:51 +0000] "GET /v1/replicas/dGNwOi8vMTAuMS4yLjEwOjk1MDI= HTTP/1.1" 200 403
time="2018-10-12T05:09:51Z" level=info msg="Prepare Rebuild Replica for id tcp://10.1.2.10:9502" 
time="2018-10-12T05:09:51Z" level=info msg="Set revision counter of 10.1.2.10:9502 to : 0" 
time="2018-10-12T05:09:51Z" level=info msg="Set backend tcp://10.1.2.10:9502 revision counter to 0" 
time="2018-10-12T05:09:51Z" level=info msg="Synchronizing volume-head-002.img.meta@tcp://10.1.1.9:9502 to volume-head-001.img.meta@10.1.2.10:9700" 
time="2018-10-12T05:09:51Z" level=info msg="10.1.2.11:3260" 
time="2018-10-12T05:09:51Z" level=info msg="Accepting ..." 
time="2018-10-12T05:09:51Z" level=info msg="connection is connected from 147.75.205.47:53592...\n" 
time="2018-10-12T05:09:51Z" level=info msg="Listening ..." 
time="2018-10-12T05:09:51Z" level=warning msg="unexpected connection state: full feature" 
time="2018-10-12T05:09:51Z" level=error msg=EOF 
time="2018-10-12T05:09:51Z" level=info msg="10.1.2.11:3260" 
time="2018-10-12T05:09:51Z" level=info msg="Accepting ..." 
time="2018-10-12T05:09:51Z" level=info msg="connection is connected from 147.75.205.47:53594...\n" 
time="2018-10-12T05:09:51Z" level=info msg="Listening ..." 
time="2018-10-12T05:09:52Z" level=info msg="Done synchronizing volume-head-002.img.meta to volume-head-001.img.meta@10.1.2.10:9700" 
10.1.2.10 - - [12/Oct/2018:05:09:51 +0000] "POST /v1/replicas/dGNwOi8vMTAuMS4yLjEwOjk1MDI=?action=preparerebuild HTTP/1.1" 200 281
time="2018-10-12T05:09:52Z" level=info msg="New Session initiator name:iqn.1993-08.org.debian:01:db7d7cb92692,target name:iqn.2016-09.com.openebs.jiva:pvc-edf28c48-cddc-11e8-878c-0cc47a205cf8,ISID:0x23d020000" 
time="2018-10-12T05:09:52Z" level=error msg="non support" 
time="2018-10-12T05:09:52Z" level=warning msg="check condition" 
time="2018-10-12T05:09:57Z" level=info msg="Get Replica for id tcp://10.1.2.10:9502" 
10.1.2.10 - - [12/Oct/2018:05:09:57 +0000] "GET /v1/replicas/dGNwOi8vMTAuMS4yLjEwOjk1MDI= HTTP/1.1" 200 403
time="2018-10-12T05:09:57Z" level=info msg="Verify Rebuild Replica for id tcp://10.1.2.10:9502" 
time="2018-10-12T05:09:57Z" level=info msg="chain [volume-head-002.img volume-snap-ff10975d-65e3-4a96-a1c5-e17547fb8558.img volume-snap-3efaa9d4-2bbb-4def-ab26-914931f22d9e.img] from rw replica tcp://10.1.1.9:9502" 
time="2018-10-12T05:09:57Z" level=info msg="chain [volume-head-001.img volume-snap-ff10975d-65e3-4a96-a1c5-e17547fb8558.img volume-snap-3efaa9d4-2bbb-4def-ab26-914931f22d9e.img] from wo replica tcp://10.1.2.10:9502" 
time="2018-10-12T05:09:57Z" level=info msg="Get backend tcp://10.1.1.9:9502 revision counter 0" 
time="2018-10-12T05:09:57Z" level=info msg="rw replica tcp://10.1.1.9:9502 revision counter 0" 
time="2018-10-12T05:09:57Z" level=info msg="Set revision counter of 10.1.2.10:9502 to : 0" 
time="2018-10-12T05:09:57Z" level=info msg="Set backend tcp://10.1.2.10:9502 revision counter to 0" 
time="2018-10-12T05:09:57Z" level=info msg="Set replica tcp://10.1.2.10:9502 to mode RW" 
time="2018-10-12T05:09:57Z" level=info msg="addr tcp://10.1.2.10:9502 m: RW prev: WO in setmode" 
time="2018-10-12T05:09:57Z" level=info msg="replicator reset false" 
time="2018-10-12T05:09:57Z" level=info msg="buildreadwriters: prev: 0 2 cur: 3 3" 
time="2018-10-12T05:09:57Z" level=info msg="controller readonly p:false c:false rcount:3 rw_count:3" 
time="2018-10-12T05:09:57Z" level=info msg="backends len: 3" 
time="2018-10-12T05:09:57Z" level=info msg="Get Replica for id tcp://10.1.2.10:9502" 
10.1.2.10 - - [12/Oct/2018:05:09:57 +0000] "POST /v1/replicas/dGNwOi8vMTAuMS4yLjEwOjk1MDI=?action=verifyrebuild HTTP/1.1" 200 403

Replica Pod logs:-

time="2018-10-12T05:09:03Z" level=info msg="Creating volume /openebs, size 5368709120/512" 
time="2018-10-12T05:09:03Z" level=info msg="Starting replica having replicaType: , frontendIP: 10.106.6.8, size: 5G, dir: /openebs" 
time="2018-10-12T05:09:03Z" level=info msg="Setting replicaAddr: 10.1.3.10:9502, controlAddr: 10.1.3.10:9502, dataAddr: 10.1.3.10:9503, syncAddr: 10.1.3.10:9504" 
time="2018-10-12T05:09:03Z" level=info msg="Waiting for s.Replica() to be non nil" 
time="2018-10-12T05:09:03Z" level=info msg="Listening on data 10.1.3.10:9503" 
time="2018-10-12T05:09:03Z" level=info msg="Listening on control 10.1.3.10:9502" 
time="2018-10-12T05:09:03Z" level=info msg="Listening on sync 10.1.3.10:9504 start: 9700 end: 9800" 
time="2018-10-12T05:09:05Z" level=info msg="Waiting for s.Replica() to be non nil" 
time="2018-10-12T05:09:07Z" level=info msg="Waiting for s.Replica() to be non nil" 
time="2018-10-12T05:09:09Z" level=info msg="Waiting for s.Replica() to be non nil" 
time="2018-10-12T05:09:11Z" level=info msg="Waiting for s.Replica() to be non nil" 
time="2018-10-12T05:09:13Z" level=info msg="Waiting for s.Replica() to be non nil" 
time="2018-10-12T05:09:15Z" level=info msg="Waiting for s.Replica() to be non nil" 
time="2018-10-12T05:09:17Z" level=info msg="Waiting for s.Replica() to be non nil" 
time="2018-10-12T05:09:19Z" level=info msg="Waiting for s.Replica() to be non nil" 
time="2018-10-12T05:09:21Z" level=info msg="Waiting for s.Replica() to be non nil" 
time="2018-10-12T05:09:23Z" level=info msg="Waiting for s.Replica() to be non nil" 
time="2018-10-12T05:09:25Z" level=info msg="Waiting for s.Replica() to be non nil" 
time="2018-10-12T05:09:27Z" level=info msg="Waiting for s.Replica() to be non nil" 
time="2018-10-12T05:09:29Z" level=info msg="Waiting for s.Replica() to be non nil" 
time="2018-10-12T05:09:31Z" level=info msg="Waiting for s.Replica() to be non nil" 
time="2018-10-12T05:09:33Z" level=info msg="Waiting for s.Replica() to be non nil" 
time="2018-10-12T05:09:33Z" level=info msg="Replicastate:  err:Get http://10.106.6.8:9501/v1/replicas: dial tcp 10.106.6.8:9501: i/o timeout" 
time="2018-10-12T05:09:35Z" level=info msg="Waiting for s.Replica() to be non nil" 
time="2018-10-12T05:09:37Z" level=info msg="Waiting for s.Replica() to be non nil" 
time="2018-10-12T05:09:38Z" level=info msg="Replicastate:  err:<nil>" 
time="2018-10-12T05:09:38Z" level=info msg="Closing replica" 
time="2018-10-12T05:09:38Z" level=info msg="Close replica failed, s.r not set" 
time="2018-10-12T05:09:38Z" level=info msg="Addreplica tcp://10.1.3.10:9502" 
time="2018-10-12T05:09:38Z" level=info msg="Get Volume info from controller" 
time="2018-10-12T05:09:38Z" level=info msg="Register replica at controller" 
time="2018-10-12T05:09:39Z" level=info msg="Waiting for s.Replica() to be non nil" 
time="2018-10-12T05:09:41Z" level=info msg="Waiting for s.Replica() to be non nil" 
time="2018-10-12T05:09:42Z" level=info msg="StartReplica with value start" 
time="2018-10-12T05:09:42Z" level=info msg="Received start from controller" 
10.1.2.11 - - [12/Oct/2018:05:09:42 +0000] "POST /v1/replicas/1?action=start HTTP/1.1" 200 988
time="2018-10-12T05:09:42Z" level=info msg=GetReplica 
time="2018-10-12T05:09:42Z" level=info msg="GetReplica for id 1" 
time="2018-10-12T05:09:42Z" level=info msg="New connection from: 10.1.2.11:55090" 
time="2018-10-12T05:09:42Z" level=info msg="Got signal: 'open', proceed to open replica" 
time="2018-10-12T05:09:42Z" level=info msg="Opening volume /openebs, size 5368709120/512" 
10.1.2.11 - - [12/Oct/2018:05:09:42 +0000] "POST /v1/replicas/1?action=open HTTP/1.1" 200 1452
time="2018-10-12T05:09:42Z" level=info msg=GetReplica 
time="2018-10-12T05:09:42Z" level=info msg="GetReplica for id 1" 
time="2018-10-12T05:09:42Z" level=info msg=GetReplica 
time="2018-10-12T05:09:42Z" level=info msg="GetReplica for id 1" 
time="2018-10-12T05:09:42Z" level=info msg=GetReplica 
time="2018-10-12T05:09:42Z" level=info msg="GetReplica for id 1" 
time="2018-10-12T05:09:43Z" level=info msg="Set clone status as NA" 
time="2018-10-12T05:09:44Z" level=info msg=GetReplica 
time="2018-10-12T05:09:44Z" level=info msg="GetReplica for id 1" 
time="2018-10-12T05:09:44Z" level=info msg=GetReplica 
time="2018-10-12T05:09:44Z" level=info msg="GetReplica for id 1" 
time="2018-10-12T05:09:44Z" level=info msg="Waiting on MonitorChannel" 
time="2018-10-12T05:09:44Z" level=info msg=GetReplica 
time="2018-10-12T05:09:44Z" level=info msg="GetReplica for id 1" 
time="2018-10-12T05:09:44Z" level=info msg="SnapshotReplica name: 3efaa9d4-2bbb-4def-ab26-914931f22d9e created: 2018-10-12T05:09:44Z" 
time="2018-10-12T05:09:44Z" level=info msg="Snapshotting [3efaa9d4-2bbb-4def-ab26-914931f22d9e] volume, user created false, created time 2018-10-12T05:09:44Z" 
10.1.2.11 - - [12/Oct/2018:05:09:44 +0000] "POST /v1/replicas/1?action=snapshot HTTP/1.1" 200 1774
time="2018-10-12T05:09:45Z" level=info msg=GetReplica 
time="2018-10-12T05:09:45Z" level=info msg="GetReplica for id 1" 
time="2018-10-12T05:09:45Z" level=info msg="Running ssync [ssync -host 10.1.1.9 -timeout 7 -port 9700 volume-head-001.img.meta]" 
time="2018-10-12T05:09:45Z" level=info msg="Syncing volume-head-001.img.meta to 10.1.1.9:9700...\n" 
time="2018-10-12T05:09:45Z" level=info msg="source file size: 164, setting up directIo: false" 
time="2018-10-12T05:09:45Z" level=info msg="Ssync client: exit code 0" 
time="2018-10-12T05:09:45Z" level=info msg="Done running ssync [ssync -host 10.1.1.9 -timeout 7 -port 9700 volume-head-001.img.meta]" 
time="2018-10-12T05:09:46Z" level=info msg="Running ssync [ssync -host 10.1.1.9 -timeout 7 -port 9701 volume-snap-3efaa9d4-2bbb-4def-ab26-914931f22d9e.img]" 
time="2018-10-12T05:09:46Z" level=info msg="Syncing volume-snap-3efaa9d4-2bbb-4def-ab26-914931f22d9e.img to 10.1.1.9:9701...\n" 
time="2018-10-12T05:09:46Z" level=info msg="source file size: 5368709120, setting up directIo: true" 
time="2018-10-12T05:09:46Z" level=warning msg="Failed to open server: 10.1.1.9:9701, Retrying..." 
time="2018-10-12T05:09:46Z" level=warning msg="Failed to open server: 10.1.1.9:9701, Retrying..." 
time="2018-10-12T05:09:47Z" level=info msg="The file is a hole: [       0: 1310720](1310720)" 
time="2018-10-12T05:09:47Z" level=info msg="Ssync client: exit code 0" 
time="2018-10-12T05:09:47Z" level=info msg="Done running ssync [ssync -host 10.1.1.9 -timeout 7 -port 9701 volume-snap-3efaa9d4-2bbb-4def-ab26-914931f22d9e.img]" 
time="2018-10-12T05:09:48Z" level=info msg="Running ssync [ssync -host 10.1.1.9 -timeout 7 -port 9702 volume-snap-3efaa9d4-2bbb-4def-ab26-914931f22d9e.img.meta]" 
time="2018-10-12T05:09:48Z" level=info msg="Syncing volume-snap-3efaa9d4-2bbb-4def-ab26-914931f22d9e.img.meta to 10.1.1.9:9702...\n" 
time="2018-10-12T05:09:48Z" level=info msg="source file size: 112, setting up directIo: false" 
time="2018-10-12T05:09:49Z" level=info msg="Ssync client: exit code 0" 
time="2018-10-12T05:09:49Z" level=info msg="Done running ssync [ssync -host 10.1.1.9 -timeout 7 -port 9702 volume-snap-3efaa9d4-2bbb-4def-ab26-914931f22d9e.img.meta]" 
time="2018-10-12T05:09:50Z" level=info msg=GetReplica 
time="2018-10-12T05:09:50Z" level=info msg="GetReplica for id 1" 
time="2018-10-12T05:09:50Z" level=info msg=GetReplica 
time="2018-10-12T05:09:50Z" level=info msg="GetReplica for id 1" 
time="2018-10-12T05:09:51Z" level=info msg=GetReplica 
time="2018-10-12T05:09:51Z" level=info msg="GetReplica for id 1" 
time="2018-10-12T05:09:51Z" level=info msg="SnapshotReplica name: ff10975d-65e3-4a96-a1c5-e17547fb8558 created: 2018-10-12T05:09:51Z" 
time="2018-10-12T05:09:51Z" level=info msg="Snapshotting [ff10975d-65e3-4a96-a1c5-e17547fb8558] volume, user created false, created time 2018-10-12T05:09:51Z" 
10.1.2.11 - - [12/Oct/2018:05:09:51 +0000] "POST /v1/replicas/1?action=snapshot HTTP/1.1" 200 2160
time="2018-10-12T05:09:52Z" level=info msg="Running ssync [ssync -host 10.1.2.10 -timeout 7 -port 9701 volume-snap-3efaa9d4-2bbb-4def-ab26-914931f22d9e.img]" 
time="2018-10-12T05:09:52Z" level=info msg="Syncing volume-snap-3efaa9d4-2bbb-4def-ab26-914931f22d9e.img to 10.1.2.10:9701...\n" 
time="2018-10-12T05:09:52Z" level=info msg="source file size: 5368709120, setting up directIo: true" 
time="2018-10-12T05:09:52Z" level=info msg="The file is a hole: [       0: 1310720](1310720)" 
time="2018-10-12T05:09:52Z" level=info msg="Ssync client: exit code 0" 
time="2018-10-12T05:09:52Z" level=info msg="Done running ssync [ssync -host 10.1.2.10 -timeout 7 -port 9701 volume-snap-3efaa9d4-2bbb-4def-ab26-914931f22d9e.img]" 
time="2018-10-12T05:09:52Z" level=info msg="Running ssync [ssync -host 10.1.2.10 -timeout 7 -port 9702 volume-snap-3efaa9d4-2bbb-4def-ab26-914931f22d9e.img.meta]" 
time="2018-10-12T05:09:52Z" level=info msg="Syncing volume-snap-3efaa9d4-2bbb-4def-ab26-914931f22d9e.img.meta to 10.1.2.10:9702...\n" 
time="2018-10-12T05:09:52Z" level=info msg="source file size: 112, setting up directIo: false" 
time="2018-10-12T05:09:52Z" level=warning msg="Failed to open server: 10.1.2.10:9702, Retrying..." 
time="2018-10-12T05:09:52Z" level=warning msg="Failed to open server: 10.1.2.10:9702, Retrying..." 
time="2018-10-12T05:09:53Z" level=info msg="Ssync client: exit code 0" 
time="2018-10-12T05:09:53Z" level=info msg="Done running ssync [ssync -host 10.1.2.10 -timeout 7 -port 9702 volume-snap-3efaa9d4-2bbb-4def-ab26-914931f22d9e.img.meta]" 
time="2018-10-12T05:09:55Z" level=info msg="Running ssync [ssync -host 10.1.2.10 -timeout 7 -port 9703 volume-snap-ff10975d-65e3-4a96-a1c5-e17547fb8558.img]" 
time="2018-10-12T05:09:55Z" level=info msg="Syncing volume-snap-ff10975d-65e3-4a96-a1c5-e17547fb8558.img to 10.1.2.10:9703...\n" 
time="2018-10-12T05:09:55Z" level=info msg="source file size: 5368709120, setting up directIo: true" 
time="2018-10-12T05:09:55Z" level=info msg="The file is a hole: [       0: 1310720](1310720)" 
time="2018-10-12T05:09:55Z" level=info msg="Ssync client: exit code 0" 
time="2018-10-12T05:09:55Z" level=info msg="Done running ssync [ssync -host 10.1.2.10 -timeout 7 -port 9703 volume-snap-ff10975d-65e3-4a96-a1c5-e17547fb8558.img]" 
time="2018-10-12T05:09:56Z" level=info msg="Running ssync [ssync -host 10.1.2.10 -timeout 7 -port 9704 volume-snap-ff10975d-65e3-4a96-a1c5-e17547fb8558.img.meta]" 
time="2018-10-12T05:09:56Z" level=info msg="Syncing volume-snap-ff10975d-65e3-4a96-a1c5-e17547fb8558.img.meta to 10.1.2.10:9704...\n" 
time="2018-10-12T05:09:56Z" level=info msg="source file size: 164, setting up directIo: false" 
time="2018-10-12T05:09:56Z" level=warning msg="Failed to open server: 10.1.2.10:9704, Retrying..." 
time="2018-10-12T05:09:56Z" level=info msg="Ssync client: exit code 0" 
time="2018-10-12T05:09:56Z" level=info msg="Done running ssync [ssync -host 10.1.2.10 -timeout 7 -port 9704 volume-snap-ff10975d-65e3-4a96-a1c5-e17547fb8558.img.meta]"

Anything else we need to know?:
Packet cloud - t1.small.x86
kubernetes version 1.10.0
openebs v0.7.x-ci
@Yaxian
Copy link

Yaxian commented Jan 17, 2019 •
edited

same issue.
kubernetes version 1.13.0
openebs v0.8
iscsiadm version 2.0-873

pod mongodb log:

  Warning  FailedScheduling        7m33s (x7 over 7m33s)  default-scheduler                 pod has unbound immediate PersistentVolumeClaims (repeated 3 times)
  Normal   Scheduled               7m33s                  default-scheduler                 Successfully assigned default/mongodb-2 to node0
  Normal   SuccessfulAttachVolume  7m33s                  attachdetach-controller           AttachVolume.Attach succeeded for volume "pvc-14d2391c-1a62-11e9-a3ee-00163e0ec7ac"
  Warning  FailedMount             7m1s (x4 over 7m20s)   kubelet, node0 
MountVolume.WaitForAttach failed for volume "pvc-14d2391c-1a62-11e9-a3ee-00163e0ec7ac" : failed to get any path for iscsi disk, last err seen:
iscsi: failed to sendtargets to portal 10.109.65.57:3260 output: iscsiadm: cannot make connection to 10.109.65.57: Connection refused
iscsiadm: cannot make connection to 10.109.65.57: Connection refused
iscsiadm: cannot make connection to 10.109.65.57: Connection refused
iscsiadm: cannot make connection to 10.109.65.57: Connection refused
iscsiadm: cannot make connection to 10.109.65.57: Connection refused
iscsiadm: cannot make connection to 10.109.65.57: Connection refused
iscsiadm: connection login retries (reopen_max) 5 exceeded
iscsiadm: No portals found
, err exit status 21
  Warning  FailedMount  59s (x3 over 5m30s)  kubelet, node0  Unable to mount volumes for pod "mongodb-2_default(14d2c465-1a62-11e9-a3ee-00163e0ec7ac)": timeout expired waiting for volumes to attach or mount for pod "default"/"mongodb-2". list of unmounted volumes=[mongo-persistent-storage]. list of unattached volumes=[mongo-persistent-storage default-token-gcrkp]
  Warning  FailedMount  46s (x7 over 6m52s)  kubelet, node0  MountVolume.MountDevice failed for volume "pvc-14d2391c-1a62-11e9-a3ee-00163e0ec7ac" : executable file not found in $PATH

Pod ctrl log

time="2019-01-17T14:13:48Z" level=info msg="REPLICATION_FACTOR: 1"
time="2019-01-17T14:13:48Z" level=info msg="Starting controller with frontendIP: , and clusterIP: 10.109.65.57"
time="2019-01-17T14:13:48Z" level=info msg="resetting controller"
time="2019-01-17T14:13:48Z" level=info msg="Listening on :9501"
time="2019-01-17T14:14:14Z" level=info msg="List Replicas"
time="2019-01-17T14:14:14Z" level=info msg="List Replicas"
time="2019-01-17T14:14:14Z" level=info msg="Register Replica for address 10.244.2.86"
time="2019-01-17T14:14:14Z" level=info msg="Register Replica, Address: 10.244.2.86 Uptime: 35.13982523s State: closed Type: Backend RevisionCount: 0"
time="2019-01-17T14:14:14Z" level=info msg="Replica 10.244.2.86 signalled to start, no of registered replicas are 1 and replica count is 1"
10.244.2.86 - - [17/Jan/2019:14:14:14 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2019-01-17T14:14:14Z" level=info msg="resetting controller"
time="2019-01-17T14:14:14Z" level=info msg="Connecting to remote: 10.244.2.86:9502"
time="2019-01-17T14:14:14Z" level=info msg="Opening: 10.244.2.86:9502"
time="2019-01-17T14:14:14Z" level=info msg="check if replica tcp://10.244.2.86:9502 is already added"
time="2019-01-17T14:14:14Z" level=info msg="check if any WO replica available"
time="2019-01-17T14:14:14Z" level=info msg="Adding backend: tcp://10.244.2.86:9502"
time="2019-01-17T14:14:14Z" level=info msg="replicator reset false"
time="2019-01-17T14:14:14Z" level=info msg="buildreadwriters: prev: 0 0 cur: 1 0"
time="2019-01-17T14:14:14Z" level=info msg="Start monitoring tcp://10.244.2.86:9502"
time="2019-01-17T14:14:14Z" level=info msg="Get backend tcp://10.244.2.86:9502 clone status"
time="2019-01-17T14:14:14Z" level=error msg="Waiting for replica to update CloneStatus to Completed/NA, retry after 2s"
time="2019-01-17T14:14:16Z" level=info msg="Get backend tcp://10.244.2.86:9502 clone status"
time="2019-01-17T14:14:16Z" level=info msg="Set replica tcp://10.244.2.86:9502 to mode RW"
time="2019-01-17T14:14:16Z" level=info msg="addr tcp://10.244.2.86:9502 m: RW prev: WO in setmode"
time="2019-01-17T14:14:16Z" level=info msg="replicator reset false"
time="2019-01-17T14:14:16Z" level=info msg="buildreadwriters: prev: 0 0 cur: 1 1"
time="2019-01-17T14:14:16Z" level=info msg="Get backend tcp://10.244.2.86:9502 revision counter 0"
time="2019-01-17T14:14:16Z" level=info msg="Update volume status"
time="2019-01-17T14:14:16Z" level=info msg="controller readonly p:true c:false rcount:1 rw_count:1"
time="2019-01-17T14:14:16Z" level=info msg="backends len: 1"
time="2019-01-17T14:14:16Z" level=info msg="Start SCSI target"
time="2019-01-17T14:14:16Z" level=info msg="SCSI device created"
time="2019-01-17T14:14:16Z" level=info msg="Listening ..."
10.244.2.86 - - [17/Jan/2019:14:14:14 +0000] "POST /v1/volumes/cHZjLTE0ZDIzOTFjLTFhNjItMTFlOS1hM2VlLTAwMTYzZTBlYzdhYw==?action=start HTTP/1.1" 200 654
time="2019-01-17T14:14:17Z" level=info msg="10.244.0.7:3260"
time="2019-01-17T14:14:17Z" level=info msg="Accepting ..."
time="2019-01-17T14:14:17Z" level=info msg="Listening ..."
time="2019-01-17T14:14:17Z" level=info msg="New Session initiator name:iqn.1993-08.org.debian:01:51999edb23e9,target name:iqn.2016-09.com.openebs.jiva:pvc-14d2391c-1a62-11e9-a3ee-00163e0ec7ac,ISID:0x23d15
0000"
time="2019-01-17T14:14:17Z" level=error msg="non support"
time="2019-01-17T14:14:17Z" level=warning msg="check condition"
127.0.0.1 - - [17/Jan/2019:14:14:23 +0000] "GET /v1/stats HTTP/1.1" 200 562
127.0.0.1 - - [17/Jan/2019:14:15:23 +0000] "GET /v1/stats HTTP/1.1" 200 567
127.0.0.1 - - [17/Jan/2019:14:16:23 +0000] "GET /v1/stats HTTP/1.1" 200 568

Pod rep

time="2019-01-17T14:13:38Z" level=info msg="Creating volume /openebs, size 214748364800/512"
time="2019-01-17T14:13:39Z" level=info msg="Starting replica having replicaType: , frontendIP: 10.109.65.57, size: 200G, dir: /openebs"
time="2019-01-17T14:13:39Z" level=info msg="Setting replicaAddr: 10.244.2.86:9502, controlAddr: 10.244.2.86:9502, dataAddr: 10.244.2.86:9503, syncAddr: 10.244.2.86:9504"
time="2019-01-17T14:13:39Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2019-01-17T14:13:39Z" level=info msg="Listening on data 10.244.2.86:9503"
time="2019-01-17T14:13:39Z" level=info msg="Listening on control 10.244.2.86:9502"
time="2019-01-17T14:13:39Z" level=info msg="Listening on sync 10.244.2.86:9504 start: 9700 end: 9800"
time="2019-01-17T14:13:41Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2019-01-17T14:13:43Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2019-01-17T14:13:45Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2019-01-17T14:13:47Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2019-01-17T14:13:49Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2019-01-17T14:13:51Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2019-01-17T14:13:53Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2019-01-17T14:13:55Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2019-01-17T14:13:57Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2019-01-17T14:13:59Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2019-01-17T14:14:01Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2019-01-17T14:14:03Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2019-01-17T14:14:05Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2019-01-17T14:14:07Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2019-01-17T14:14:09Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2019-01-17T14:14:09Z" level=info msg="Replicastate:  err:Get http://10.109.65.57:9501/v1/replicas: dial tcp 10.109.65.57:9501: i/o timeout"
time="2019-01-17T14:14:11Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2019-01-17T14:14:13Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2019-01-17T14:14:14Z" level=info msg="Replicastate:  err:<nil>"
time="2019-01-17T14:14:14Z" level=info msg="Closing replica"
time="2019-01-17T14:14:14Z" level=info msg="Close replica failed, s.r not set"
time="2019-01-17T14:14:14Z" level=info msg="Addreplica tcp://10.244.2.86:9502"
time="2019-01-17T14:14:14Z" level=info msg="Get Volume info from controller"
time="2019-01-17T14:14:14Z" level=info msg="Register replica at controller"
10.244.0.7 - - [17/Jan/2019:14:14:14 +0000] "POST /v1/replicas/1?action=start HTTP/1.1" 200 1010
time="2019-01-17T14:14:14Z" level=info msg="StartReplica with value start"
time="2019-01-17T14:14:14Z" level=info msg="Received start from controller"
time="2019-01-17T14:14:14Z" level=info msg=GetReplica
time="2019-01-17T14:14:14Z" level=info msg="GetReplica for id 1"
time="2019-01-17T14:14:14Z" level=info msg="New connection from: 10.244.0.7:39438"
time="2019-01-17T14:14:14Z" level=info msg="Got signal: 'open', proceed to open replica"
time="2019-01-17T14:14:14Z" level=info msg="Opening volume /openebs, size 214748364800/512"
10.244.0.7 - - [17/Jan/2019:14:14:14 +0000] "POST /v1/replicas/1?action=open HTTP/1.1" 200 1561
time="2019-01-17T14:14:14Z" level=info msg=GetReplica
time="2019-01-17T14:14:14Z" level=info msg="GetReplica for id 1"
time="2019-01-17T14:14:14Z" level=info msg=GetReplica
time="2019-01-17T14:14:14Z" level=info msg="GetReplica for id 1"
time="2019-01-17T14:14:14Z" level=info msg=GetReplica
time="2019-01-17T14:14:14Z" level=info msg="GetReplica for id 1"
time="2019-01-17T14:14:15Z" level=info msg="Set clone status as NA"
time="2019-01-17T14:14:16Z" level=info msg=GetReplica
time="2019-01-17T14:14:16Z" level=info msg="GetReplica for id 1"
time="2019-01-17T14:14:16Z" level=info msg=GetReplica
time="2019-01-17T14:14:16Z" level=info msg="GetReplica for id 1"
time="2019-01-17T14:14:16Z" level=info msg="Waiting on MonitorChannel"
time="2019-01-17T14:14:23Z" level=info msg="Get Volume Usage"
time="2019-01-17T14:14:23Z" level=info msg=GetUsage
10.244.0.7 - - [17/Jan/2019:14:14:23 +0000] "GET /v1/replicas/1/volusage HTTP/1.1" 200 181
time="2019-01-17T14:15:23Z" level=info msg="Get Volume Usage"
time="2019-01-17T14:15:23Z" level=info msg=GetUsage
10.244.0.7 - - [17/Jan/2019:14:15:23 +0000] "GET /v1/replicas/1/volusage HTTP/1.1" 200 181
time="2019-01-17T14:16:23Z" level=info msg="Get Volume Usage"
time="2019-01-17T14:16:23Z" level=info msg=GetUsage
10.244.0.7 - - [17/Jan/2019:14:16:23 +0000] "GET /v1/replicas/1/volusage HTTP/1.1" 200 181
time="2019-01-17T14:17:23Z" level=info msg="Get Volume Usage"
time="2019-01-17T14:17:23Z" level=info msg=GetUsage
10.244.0.7 - - [17/Jan/2019:14:17:23 +0000] "GET /v1/replicas/1/volusage HTTP/1.1" 200 181
time="2019-01-17T14:18:23Z" level=info msg="Get Volume Usage"
time="2019-01-17T14:18:23Z" level=info msg=GetUsage
10.244.0.7 - - [17/Jan/2019:14:18:23 +0000] "GET /v1/replicas/1/volusage HTTP/1.1" 200 181
time="2019-01-17T14:19:23Z" level=info msg="Get Volume Usage"
time="2019-01-17T14:19:23Z" level=info msg=GetUsage
10.244.0.7 - - [17/Jan/2019:14:19:23 +0000] "GET /v1/replicas/1/volusage HTTP/1.1" 200 181

@github-actions
Copy link

Issues go stale after 30d of inactivity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
No ISSUE activity
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Yaxian @nsathyaseelan