Name:           centos-toolbox-2186589236-n1q0t
Namespace:      kube-system
Node:           kube-1.sbezverk.cisco.com/192.168.80.232
Start Time:     Thu, 22 Dec 2016 10:16:38 -0500
Labels:         pod-template-hash=2186589236
                service=centos-toolbox
Status:         Pending
IP:
Controllers:    ReplicaSet/centos-toolbox-2186589236
Containers:
  centos-toolbox:
    Container ID:
    Image:              docker.io/sbezverk/centos-tool-box:latest
    Image ID:
    Port:
    Command:
      sleep
      86480
    State:                      Waiting
      Reason:                   ContainerCreating
    Ready:                      False
    Restart Count:              0
    Environment Variables:      <none>
Conditions:
  Type          Status
  Initialized   True 
  Ready         False 
  PodScheduled  True 
Volumes:
  iscsi-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  mariadb
    ReadOnly:   false
  default-token-2jkht:
    Type:       Secret (a volume populated by a Secret)
    SecretName: default-token-2jkht
QoS Tier:       BestEffort
Events:
  FirstSeen     LastSeen        Count   From                                    SubobjectPath   Type            Reason          Message
  ---------     --------        -----   ----                                    -------------   --------        ------          -------
  2m            2m              1       {default-scheduler }                                    Normal          Scheduled       Successfully assigned centos-toolbox-2186589236-n1q0t to kube-1.sbezverk.cisco.com
  1m            12s             7       {kubelet kube-1.sbezverk.cisco.com}                     Warning         FailedMount     MountVolume.SetUp failed for volume "kubernetes.io/iscsi/a272d8c6-c859-11e6-a647-525400ebb7ee-maraidb" (spec.Name: "maraidb") pod "a272d8c6-c859-11e6-a647-525400ebb7ee" (UID: "a272d8c6-c859-11e6-a647-525400ebb7ee") with: Could not attach disk: Timeout after 10s
  1s            1s              1       {kubelet kube-1.sbezverk.cisco.com}                     Warning         FailedMount     Unable to mount volumes for pod "centos-toolbox-2186589236-n1q0t_kube-system(a272d8c6-c859-11e6-a647-525400ebb7ee)": timeout expired waiting for volumes to attach/mount for pod "centos-toolbox-2186589236-n1q0t"/"kube-system". list of unattached/unmounted volumes=[iscsi-storage]
  1s            1s              1       {kubelet kube-1.sbezverk.cisco.com}                     Warning         FailedSync      Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "centos-toolbox-2186589236-n1q0t"/"kube-system". list of unattached/unmounted volumes=[iscsi-storage]
  

On compute node I can see the iscsi session and corresponding disk device /dev/sda and /dev/sda1 got created:

iscsiadm -m session 
tcp: [11] 192.168.80.233:3260,1 iqn.2003-01.org.linux-iscsi.ceph-node.x8664:sn.1fe689f1449d (non-flash)

[342571.213546] scsi host12: iSCSI Initiator over TCP/IP
[342571.229766] scsi 12:0:0:0: Direct-Access     LIO-ORG  disk-1           4.0  PQ: 0 ANSI: 5
[342571.241847] scsi 12:0:0:0: alua: supports implicit and explicit TPGS
[342571.242396] scsi 12:0:0:0: alua: port group 00 rel port 01
[342571.243159] scsi 12:0:0:0: alua: port group 00 state A non-preferred supports TOlUSNA
[342571.243162] scsi 12:0:0:0: alua: Attached
[342571.243601] sd 12:0:0:0: Attached scsi generic sg1 type 0
[342571.243645] sd 12:0:0:0: [sda] 21481472 512-byte logical blocks: (10.9 GB/10.2 GiB)
[342571.245145] sd 12:0:0:0: [sda] Write Protect is off
[342571.245149] sd 12:0:0:0: [sda] Mode Sense: 43 00 10 08
[342571.245339] sd 12:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
[342571.248146]  sda: sda1
[342571.249220] sd 12:0:0:0: [sda] Attached SCSI disk

Kubelet generates these errors:

Dec 22 10:22:36 kube-1 kubelet: E1222 10:22:36.691419     781 iscsi.go:214] iscsi: failed to setup
Dec 22 10:22:36 kube-1 kubelet: E1222 10:22:36.691474     781 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/iscsi/a272d8c6-c859-11e6-a647-525400ebb7ee-maraidb\" (\"a272d8c6-c859-11e6-a647-525400ebb7ee\")" failed. No retries permitted until 2016-12-22 10:24:36.691449237 -0500 EST (durationBeforeRetry 2m0s). Error: MountVolume.SetUp failed for volume "kubernetes.io/iscsi/a272d8c6-c859-11e6-a647-525400ebb7ee-maraidb" (spec.Name: "maraidb") pod "a272d8c6-c859-11e6-a647-525400ebb7ee" (UID: "a272d8c6-c859-11e6-a647-525400ebb7ee") with: Could not attach disk: Timeout after 10s
Dec 22 10:23:01 kube-1 kubelet: E1222 10:23:01.893044     781 kubelet.go:1522] Unable to mount volumes for pod "centos-toolbox-2186589236-n1q0t_kube-system(a272d8c6-c859-11e6-a647-525400ebb7ee)": timeout expired waiting for volumes to attach/mount for pod "centos-toolbox-2186589236-n1q0t"/"kube-system". list of unattached/unmounted volumes=[iscsi-storage]; skipping pod
Dec 22 10:23:01 kube-1 kubelet: E1222 10:23:01.893093     781 pod_workers.go:184] Error syncing pod a272d8c6-c859-11e6-a647-525400ebb7ee, skipping: timeout expired waiting for volumes to attach/mount for pod "centos-toolbox-2186589236-n1q0t"/"kube-system". list of unattached/unmounted volumes=[iscsi-storage]


Here is the PV,PVC and storage-class objects:

kubectl describe  pv maraidb
Name:           maraidb
Labels:         <none>
StorageClass:   iscsi-storage
Status:         Bound
Claim:          kube-system/mariadb
Reclaim Policy: Recycle
Access Modes:   RWO
Capacity:       10Gi
Message:
Source:
    Type:               ISCSI (an ISCSI Disk resource that is attached to a kubelet's host machine and then exposed to the pod)
    TargetPortal:       192.168.80.233:3260
    IQN:                iqn.2003-01.org.linux-iscsi.ceph-node.x8664:sn.1fe689f1449d
    Lun:                1
    ISCSIInterface      default
    FSType:             xfs
    ReadOnly:           false
	
kubectl describe  pvc mariadb
Name:           mariadb
Namespace:      kube-system
StorageClass:   iscsi-storage
Status:         Bound
Volume:         maraidb
Labels:         <none>
Capacity:       10Gi
Access Modes:   RWO
No events.