[WIP] Add deployment artifacts for iscsi csi driver deployment

[humblec]

Fix: #4

Marking this PR in WIP as the container image should be pushed in a repo and more testing...etc has to be done here

Signed-off-by: Humble Chirammal hchiramm@redhat.com
-->

[obnoxxx]

@humblec :

Fix: #4

I think you mean "Fix: #44". 😄

[humblec]

@humblec :

Fix: #4

I think you mean "Fix: #44". smile

Sorry, I missed this comment. :)

@obnoxxx I meant #4 itself. This is a PR to make the driver deployment in Linux which is missing at the moment. This task has to be taken care to complete #4.

[humblec]

@andyzhangx addressed the comments. Thanks for the review, ptal
@msau42 would like to push some images for the deployment to k8s.gcr.io repo images. Any special permissions needed?

[msau42]

would like to push some images for the deployment to k8s.gcr.io repo images. Any special permissions needed?

We only promote release tagged images to k8s.gcr.io. Once we've tested the staging canary image, then we can cut a release, and then promote the tagged image.

[obnoxxx]

@humblec :

Fix: #4

I think you mean "Fix: #44". smile

Sorry, I missed this comment. :)

@obnoxxx I meant #4 itself. This is a PR to make the driver deployment in Linux which is missing at the moment. This task has to be taken care to complete #4.

Ugh, thanks for clarifying! I hadn't realized that this driver is not actually ready/released yet... 😄

[humblec]

@humblec :

Fix: #4

I think you mean "Fix: #44". smile

Sorry, I missed this comment. :)
@obnoxxx I meant #4 itself. This is a PR to make the driver deployment in Linux which is missing at the moment. This task has to be taken care to complete #4.

Ugh, thanks for clarifying! I hadn't realized that this driver is not actually ready/released yet... smile

Yep, more or less there is more work to do like #49 , E2E to get these things in shape for its initial version. :)

[humblec]

would like to push some images for the deployment to k8s.gcr.io repo images. Any special permissions needed?

We only promote release tagged images to k8s.gcr.io. Once we've tested the staging canary image, then we can cut a release, and then promote the tagged image.

Sure.. Thanks @msau42 .

The deployment part based on this PR from local image is good!

For ex:

[hchiramm@localhost csi-driver-iscsi]$ kubectl get all -n kube-system
NAME                       READY     STATUS    RESTARTS   AGE
pod/csi-iscsi-node-7gwkp   3/3       Running   0          15m
pod/csi-iscsi-node-8dvcj   3/3       Running   0          15m
pod/csi-iscsi-node-b26rh   3/3       Running   0          15m
NAME              TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                        AGE
service/kubelet   ClusterIP   None         <none>        10250/TCP,10255/TCP,4194/TCP   3h43m
NAME                            DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/csi-iscsi-node   3         3         3         3            3           kubernetes.io/os=linux   15m
[hchiramm@localhost csi-driver-iscsi]$ 

[hchiramm@localhost csi-driver-iscsi]$ kubectl get csidriver
NAME               ATTACHREQUIRED   PODINFOONMOUNT   STORAGECAPACITY   TOKENREQUESTS   REQUIRESREPUBLISH   MODES        AGE
iscsi.csi.k8s.io   false            false            false             <unset>         false               Persistent   16m
[hchiramm@localhost csi-driver-iscsi]$ 

[hchiramm@localhost csi-driver-iscsi]$ kubectl get pv,pvc
NAME                              CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                                                 STORAGECLASS   REASON    AGE
persistentvolume/pv-name   1Gi        RWO            Delete           Bound     test/iscsi-pvc                            3m22s

NAME                              STATUS    VOLUME           CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/iscsi-pvc   Bound     pv-name   1Gi        RWO                           4s
[hchiramm@localhost csi-driver-iscsi]$ 

[humblec]

@msau42 Is this #45 (comment) good enough to have a staging image and to follow rest of the image promotion process? or will we do some more experiments for completing the mount process and proceed further. I am yet to try that though

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: humblec

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [humblec]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[msau42]

I think you can put in the k8s staging canary image for now. And before you tag a release, update this to the official repo and tag.

[msau42]

do some more experiments for completing the mount process and proceed further

what other work is there remaining before cutting a release?

[humblec]

do some more experiments for completing the mount process and proceed further

what other work is there remaining before cutting a release?

The mount part has not tested @msau42.

[humblec]

do some more experiments for completing the mount process and proceed further

what other work is there remaining before cutting a release?

The mount part has not tested @msau42.

Experimented on mount part and here is the summary:

kube-system   csi-iscsi-node-7w4c9               3/3     Running             0          4m25s

Driver details:

I0531 15:29:44.156799       8 utils.go:48] GRPC request: {}
I0531 15:29:44.156895       8 identityserver.go:16] Using default GetPluginInfo
I0531 15:29:44.157351       8 utils.go:53] GRPC response: {"name":"iscsi.csi.k8s.io","vendor_version":"1.0.0"}
[hchiramm@localhost csi-driver-iscsi]$ kubectl describe pod task-pv-pod -n default

The mount fails with an error from the library:

I0531 15:34:45.604036       8 utils.go:47] GRPC call: /csi.v1.Node/NodePublishVolume
I0531 15:34:45.604185       8 utils.go:48] GRPC request: {"target_path":"/var/lib/kubelet/pods/dd31590b-22fc-45d1-b77b-5b76a5bb1ad9/volumes/kubernetes.io~csi/static-pv-name/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":1}},"volume_context":{"discoveryCHAPAuth":"false","iqn":"iqn.2015-06.com.example.test:target1","iscsiInterface":"default","lun":"1","portals":"[]","targetPortal":"127.0.0.1:3260"},"volume_id":"hum-iscsi-share"}
I0531 15:34:45.604504       8 mount_linux.go:163] Detected OS without systemd
E0531 15:34:46.613172       8 utils.go:51] GRPC error: rpc error: code = Internal desc = connect reported success, but no path returned
[hchiramm@localhost csi-driver-iscsi]$ kubectl describe pod task-pv-pod -n default
Name:         task-pv-pod
Namespace:    default
Priority:     0
Node:         minikube/192.168.39.82
Start Time:   Mon, 31 May 2021 21:04:33 +0530
Labels:       <none>
Annotations:  <none>
Status:       Pending
IP:           
IPs:          <none>
Containers:
  task-pv-container:
    Container ID:   
    Image:          nginx
    Image ID:       
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /usr/share/nginx/html from task-pv-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-5qmrf (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  task-pv-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  iscsi-pvc
    ReadOnly:   false
  default-token-5qmrf:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-5qmrf
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age                From               Message
  ----     ------       ----               ----               -------
  Normal   Scheduled    49s                default-scheduler  Successfully assigned default/task-pv-pod to minikube
  Warning  FailedMount  10s (x7 over 48s)  kubelet, minikube  MountVolume.SetUp failed for volume "static-pv-name" : rpc error: code = Internal desc = connect reported success, but no path returned
[hchiramm@localhost csi-driver-iscsi]$ 

Above is coming from AttachDisk()

Considering this driver has not gone through any real testing, I was expecting the same. Will continue digging more on this and get back, however considering the plugin discovery and attempt to mount has initiated, I think we can go ahead with the deployment part which is covered in this PR @msau42 , wdyt?.

[humblec]

@pohly @msau42 is there any restrictions on the base image ( only alpine ..etc) we can use for the driver Dockerfile ?

[pohly]

is there any restrictions on the base image ( only alpine ..etc) we can use for the driver Dockerfile ?

Technically you can use whatever works. From a legal perspective it depends. I don't know if the CNCF has an official position on this, but at least Alpine might not be a good choice because it strips license texts (at least by default, if I remember correctly). This is a concern for some people.

Debian might be a better alternative. No trademarks to worry about, strong emphasis on license compliance, community driven, ...

[andyzhangx]

@pohly @msau42 is there any restrictions on the base image ( only alpine ..etc) we can use for the driver Dockerfile ?

for base image, could refer to https://github.com/kubernetes-csi/csi-driver-nfs/blob/master/Dockerfile#L17

[humblec]

is there any restrictions on the base image ( only alpine ..etc) we can use for the driver Dockerfile ?

Technically you can use whatever works. From a legal perspective it depends. I don't know if the CNCF has an official position on this, but at least Alpine might not be a good choice because it strips license texts (at least by default, if I remember correctly). This is a concern for some people.

Debian might be a better alternative. No trademarks to worry about, strong emphasis on license compliance, community driven, ...

Thanks @pohly for the insight!.

I have been spending good amount of time this week on this experiment as this iscsi depends a lot on the distribution of container and host versions, binary paths..etc.

Experiments were going on with centos, alpine base images..etc. will continue experiments on other distros as well.

@andyzhangx Thanks, sure, interestingly while I was looking for base images in various containers of kubernetes repo, I could see almost all distros in place. :),

[msau42]

For reference, discussion on why k8s chose debian as its base image

[humblec]

Eventhough I am making good progress by fixing issues one by one, it looks like I have to continue this process some more time to get this in shape :). Lets keep this PR open while these experiments are going on.

[humblec]

Finally, able to get the mount working.. 👍

I0603 07:11:36.907283       7 mount_linux.go:405] Attempting to determine if disk "/dev/disk/by-path/ip-10.70.53.171:3260-iscsi-iqn.2015-06.com.example.test:target1-lun-1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/by-path/ip-10.70.53.171:3260-iscsi-iqn.2015-06.com.example.test:target1-lun-1])
I0603 07:11:36.925643       7 mount_linux.go:408] Output: "DEVNAME=/dev/disk/by-path/ip-10.70.53.171:3260-iscsi-iqn.2015-06.com.example.test:target1-lun-1\nTYPE=ext4\n", err: <nil>
I0603 07:11:36.925721       7 mount_linux.go:298] Checking for issues with fsck on disk: /dev/disk/by-path/ip-10.70.53.171:3260-iscsi-iqn.2015-06.com.example.test:target1-lun-1
I0603 07:11:37.028261       7 mount_linux.go:394] Attempting to mount disk /dev/disk/by-path/ip-10.70.53.171:3260-iscsi-iqn.2015-06.com.example.test:target1-lun-1 in ext4 format at /var/lib/kubelet/pods/e64ff6a0-fc0a-4f32-a947-04cd873974ce/volumes/kubernetes.io~csi/static-pv-name/mount
I0603 07:11:37.028292       7 mount_linux.go:146] Mounting cmd (mount) with arguments (-t ext4 -o rw,defaults /dev/disk/by-path/ip-10.70.53.171:3260-iscsi-iqn.2015-06.com.example.test:target1-lun-1 /var/lib/kubelet/pods/e64ff6a0-fc0a-4f32-a947-04cd873974ce/volumes/kubernetes.io~csi/static-pv-name/mount)
I0603 07:11:37.086370       7 utils.go:53] GRPC response: {}
I0603 07:11:41.058886       7 utils.go:47] GRPC call: /csi.v1.Identity/Probe
I0603 07:11:41.058968       7 utils.go:48] GRPC request: {}
I0603 07:11:41.059147       7 utils.go:53] GRPC response: {}
[root@dhcp53-171 csi-driver-iscsi]# kubectl get pods |grep task
task-pv-pod                                                       1/1     Running             0          39s
[root@dhcp53-171 csi-driver-iscsi]# 

Unmount is failing though, looking into the same.

[humblec]

DetachDisk code flow:

unmount -> then reading the json from targetPath -> then building the connector again.

	if err = c.mounter.Unmount(targetPath); err != nil {
		klog.Errorf("iscsi detach disk: failed to unmount: %s\nError: %v", targetPath, err)
		return err
	}
	cnt--
	if cnt != 0 {
		return nil
	}

	// load iscsi disk config from json file
	file := path.Join(targetPath, c.iscsiDisk.VolName+".json")
	connector, err := iscsiLib.GetConnectorFromFile(file)
	if err != nil {
		klog.Errorf("iscsi detach disk: failed to get iscsi config from path %s Error: %v", targetPath, err)
		return err
	}

	iscsiLib.Disconnect(connector.TargetIqn, connector.TargetPortals)

The unmount fails while mounter execute detach disk due to the pod volume directory is not empty.

Jun  3 12:45:27 localhost kubelet[902]: I0603 12:45:27.489050     902 reconciler.go:196] operationExecutor.UnmountVolume started for volume "task-pv-storage" (UniqueName: "kubernetes.io/csi/iscsi.csi.k8s.io^hum-iscsi-share") pod "e64ff6a0-fc0a-4f32-a947-04cd873974ce" (UID: "e64ff6a0-fc0a-4f32-a947-04cd873974ce")

Jun  3 12:45:27 localhost kubelet[902]: E0603 12:45:27.558572     902 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/iscsi.csi.k8s.io^hum-iscsi-share podName:e64ff6a0-fc0a-4f32-a947-04cd873974ce nodeName:}" failed. No retries permitted until 2021-06-03 12:45:28.058453563 +0530 IST m=+6203994.880810285 (durationBeforeRetry 500ms). Error: "UnmountVolume.TearDown failed for volume \"task-pv-storage\" (UniqueName: \"kubernetes.io/csi/iscsi.csi.k8s.io^hum-iscsi-share\") pod \"e64ff6a0-fc0a-4f32-a947-04cd873974ce\" (UID: \"e64ff6a0-fc0a-4f32-a947-04cd873974ce\") : kubernetes.io/csi: mounter.TearDownAt failed to clean mount dir [/var/lib/kubelet/pods/e64ff6a0-fc0a-4f32-a947-04cd873974ce/volumes/kubernetes.io~csi/static-pv-name/mount]: kubernetes.io/csi: failed to remove dir [/var/lib/kubelet/pods/e64ff6a0-fc0a-4f32-a947-04cd873974ce/volumes/kubernetes.io~csi/static-pv-name/mount]: remove /var/lib/kubelet/pods/e64ff6a0-fc0a-4f32-a947-04cd873974ce/volumes/kubernetes.io~csi/static-pv-name/mount: directory not empty"

So the metadata file residing inside the pod volume mount which cause the unmount to fail.

In this case, the pod mount is done with bidirectional mount propagation:

        - name: pods-mount-dir
          hostPath:
            path: /var/lib/kubelet/pods
            type: Directory
    .......
        - name: pods-mount-dir
             mountPath: /var/lib/kubelet/pods
             mountPropagation: "Bidirectional"

Trying to figure out why we have unmount failure here ? the introduction of share metadata file was to reconstruct the disconnect details for cleanup or reliable nodeunpublish. But in general flow of events of unmount called at DetachDisk() time , this is failing saying unmount is attempted on a non empty directory as seen above which is confusing. OR the failure here is due to some other bind mount inside this path ?

@j-griffith @msau42 @pohly @andyzhangx @jsafrane any thoughts/pointers on this ?

[humblec]

Some more experiments were carried out without the metadata file inside the path by removing the persistConnector() calls from the driver to roll out the other possibilities of unmount failure. In absence of it, the nodeUnpublish() complete successfully:

I0609 12:04:08.200136       8 utils.go:47] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I0609 12:04:08.200153       8 utils.go:48] GRPC request: {"target_path":"/var/lib/kubelet/pods/57b8e330-ed76-4350-8fb6-d7a07fedd6b7/volumes/kubernetes.io~csi/static-pv-name/mount","volume_id":"hum-iscsi-share"}
I0609 12:04:08.200286       8 mount_linux.go:163] Detected OS without systemd
I0609 12:04:08.201834       8 mount_linux.go:238] Unmounting /var/lib/kubelet/pods/57b8e330-ed76-4350-8fb6-d7a07fedd6b7/volumes/kubernetes.io~csi/static-pv-name/mount
I0609 12:04:08.207536       8 utils.go:53] GRPC response: {}

So, this kind of confirm the unmount failure is caused by the presence of metadata file and not because of some other bind mounts in the same path. However we want this metadata for successful complete disconnect of the share from the kubelet/node in a later code path.

We have seen this kind of behaviour with RWO and blockMode PVCs , but in this case it is RWO + Filesystem Mode PVC which confuse me. @gnufied has fixed this issue to an extend, however I believe some of the corner cases ( block mode cases) were left out:

kubernetes/kubernetes#82190
kubernetes/kubernetes#82340
container-storage-interface/spec#385

Any help is appreciated as I see this is the final blocker for this driver to be in a consumable state.

[msau42]

I'm not sure I completely understand your question, but if you want the metadata file for NodeUnstage, then it should go under the staging directory, not the publish path.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[humblec]

/remove-lifecycle stale

[k8s-ci-robot]

@humblec: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[humblec]

This issue has been addressed via #70 .. closing this PR for the same reason.