Skip to content

Conversation

@prankulmahajan
Copy link
Member

@prankulmahajan prankulmahajan commented Aug 16, 2023

Coverage:

PASS
	github.com/IBM/ibm-vpc-file-csi-driver/pkg/ibmcsidriver	coverage: 86.2% of statements
ok  	github.com/IBM/ibm-vpc-file-csi-driver/pkg/ibmcsidriver	3.235s	coverage: 86.2% of statements
go tool cover -html=cover.out -o=cover.html  # Uncomment this line when UT in place.

Test results: https://github.ibm.com/alchemy-containers/armada-storage/issues/5533

@prankulmahajan prankulmahajan changed the title initial code EIT Aug 16, 2023
@sameshai sameshai force-pushed the eni branch 4 times, most recently from fdc1e02 to df16c5b Compare September 18, 2023 16:05
@sameshai sameshai force-pushed the eni branch 3 times, most recently from a51b875 to acff792 Compare September 19, 2023 11:05
@sameshai sameshai force-pushed the eni branch 7 times, most recently from 3ab95d3 to 641dec5 Compare October 8, 2023 17:59
@sameshai sameshai force-pushed the eni branch 2 times, most recently from d7bf83b to c58842a Compare October 16, 2023 16:00
@prankulmahajan prankulmahajan changed the base branch from eni to master October 18, 2023 14:35
prankulmahajan and others added 9 commits December 5, 2023 00:26
Signed-off-by: Sameer Shaikh <sameer.shaikh@ibm.com>
Signed-off-by: Sameer Shaikh <sameer.shaikh@ibm.com>
Signed-off-by: Sameer Shaikh <sameer.shaikh@ibm.com>
Signed-off-by: Sameer Shaikh <sameer.shaikh@ibm.com>
Signed-off-by: Sameer Shaikh <sameer.shaikh@ibm.com>
Signed-off-by: Sameer Shaikh <sameer.shaikh@ibm.com>
Signed-off-by: Sameer Shaikh <sameer.shaikh@ibm.com>
Signed-off-by: Sameer Shaikh <sameer.shaikh@ibm.com>
Signed-off-by: Sameer Shaikh <sameer.shaikh@ibm.com>
@contsto
Copy link

contsto commented May 6, 2024

👍 Coverage increased from [%] to [81.15%]

@contsto
Copy link

contsto commented May 8, 2024

👍 Coverage increased from [%] to [81.15%]

sameshai
sameshai previously approved these changes May 10, 2024
arahamad
arahamad previously approved these changes May 10, 2024
Copy link
Collaborator

@arahamad arahamad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check my comments as well

@prankulmahajan prankulmahajan dismissed stale reviews from arahamad and sameshai via 6b66a5b May 13, 2024 05:47
@contsto
Copy link

contsto commented May 13, 2024

👍 Coverage increased from [%] to [80.975%]

@contsto
Copy link

contsto commented May 13, 2024

👍 Coverage increased from [%] to [80.3625%]

@contsto
Copy link

contsto commented May 13, 2024

👍 Coverage increased from [%] to [81.225%]

@contsto
Copy link

contsto commented May 13, 2024

👍 Coverage increased from [%] to [83.2125%]

sameshai
sameshai previously approved these changes May 13, 2024
@contsto
Copy link

contsto commented May 14, 2024

👍 Coverage increased from [%] to [83.2125%]

1 similar comment
@contsto
Copy link

contsto commented May 14, 2024

👍 Coverage increased from [%] to [83.2125%]

Copy link
Collaborator

@arahamad arahamad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, please test also

@prankulmahajan
Copy link
Member Author

Testing before merge:
Images used:
icr.io/prankulmahajan/ibm-vpc-file-csi-driver:eit
icr.io/prankulmahajan/ibm-vpc-package-deployer:offline

Positive Test Case (IKS with --disable-outbound-traffic-protection)


EIT test cases

  • Install EIT packages on system using deployer image + file resources (contains logic to install offline packages with skipping installation/uninstallation in case that package is already present in the system)
❯ kubectl get pods -n kube-system -o wide| grep file
ibm-vpc-file-csi-controller-74d8b768f8-f4qjf         5/5     Running   0               23s     172.17.221.81    10.240.0.5     <none>           <none>
ibm-vpc-file-csi-controller-74d8b768f8-hcqnf         5/5     Running   0               23s     172.17.235.200   10.240.0.254   <none>           <none>
ibm-vpc-file-csi-node-qtn6c                          4/4     Running   0               25s     10.240.0.5       10.240.0.5     <none>           <none>
ibm-vpc-file-csi-node-v64cz                          4/4     Running   0               25s     10.240.0.254     10.240.0.254   <none>           <none>

❯ kubectl get pods -n kube-system -o wide| grep insta
installer-8vdjp                                      1/1     Running   0             49s     10.240.0.254     10.240.0.254   <none>           <none>
installer-vpfzv                                      1/1     Running   0             49s     10.240.0.5       10.240.0.5     <none>           <none>
  • PVC creation and mount
❯ kubectl get pvc
NAME     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        VOLUMEATTRIBUTESCLASS   AGE
my-pvc   Bound    pvc-97b56a6a-6bfc-4d06-81a8-14e2fff6fa39   20Gi       RWX            ibmc-vpc-file-eit   <unset>                 56s

❯ kubectl describe pv | grep isEITEnabled
                           isEITEnabled=true

❯ kubectl get pods
NAME                      READY   STATUS    RESTARTS   AGE
my-dep-5d98cb9588-264zn   1/1     Running   0          35s
my-dep-5d98cb9588-9cws5   1/1     Running   0          35s

❯ kubectl exec my-dep-5d98cb9588-264zn -it -- bash
root@my-dep-5d98cb9588-264zn:/# 
root@my-dep-5d98cb9588-264zn:/# df -h
Filesystem                                         Size  Used Avail Use% Mounted on
overlay                                             98G  6.3G   87G   7% /
tmpfs                                               64M     0   64M   0% /dev
tmpfs                                              7.8G     0  7.8G   0% /sys/fs/cgroup
10.240.0.18:/0cbfca07_0aa1_4d7f_9afd_7b59a02c912b   20G  320K   20G   1% /mount-path
/dev/vda2                                           98G  6.3G   87G   7% /etc/hosts
shm                                                 64M     0   64M   0% /dev/shm
tmpfs                                               13G   16K   13G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                                              7.8G     0  7.8G   0% /proc/acpi
tmpfs                                              7.8G     0  7.8G   0% /proc/scsi
tmpfs                                              7.8G     0  7.8G   0% /sys/firmware
root@my-dep-5d98cb9588-264zn:/# cd mount-path/
root@my-dep-5d98cb9588-264zn:/mount-path# 
root@my-dep-5d98cb9588-264zn:/mount-path# ls
root@my-dep-5d98cb9588-264zn:/mount-path# echo "HI" > eit_file.txt
root@my-dep-5d98cb9588-264zn:/mount-path# ls -la
total 8
drwxrwxr-x 2 nobody 4294967294 4096 May 15 08:02 .
drwxr-xr-x 1 root   root       4096 May 15 07:58 ..
-rw-r--r-- 1 nobody 4294967294    3 May 15 08:02 eit_file.txt
root@my-dep-5d98cb9588-264zn:/mount-path# 
root@my-dep-5d98cb9588-264zn:/mount-path# cat eit_file.txt 
HI
root@my-dep-5d98cb9588-264zn:/mount-path# exit
exit
  • Multiple scaleUp scaleDown with r/w operations
❯ kubectl scale deployment my-dep --replicas=4
deployment.apps/my-dep scaled
❯ 
❯ 
❯ kubectl get pods -w
NAME                      READY   STATUS    RESTARTS   AGE
my-dep-5d98cb9588-264zn   1/1     Running   0          5m59s
my-dep-5d98cb9588-46z5r   1/1     Running   0          6s
my-dep-5d98cb9588-9cws5   1/1     Running   0          5m59s
my-dep-5d98cb9588-9pwkf   1/1     Running   0          6s

❯ kubectl exec my-dep-5d98cb9588-46z5r -it -- bash
root@my-dep-5d98cb9588-46z5r:/# cat mount-path/eit_file.txt 
HI

❯ kubectl scale deployment my-dep --replicas=1
deployment.apps/my-dep scaled
❯ 
❯ 
❯ kubectl get pods
NAME                      READY   STATUS    RESTARTS   AGE
my-dep-5d98cb9588-264zn   1/1     Running   0          7m3s
  • Multi node mount -- Done in previous step
  • Cross zone mount
# Adding another node in us-south2
❯ ibmcloud ks workers -c prankul-iks
OK
ID                                                      Primary IP     Flavor     State    Status   Zone         Version
test-cp265fs20gu9km5st1u0-prankuliks-default-0000011d   10.240.0.254   bx2.4x16   normal   Ready    us-south-1   1.30.0_1517
test-cp265fs20gu9km5st1u0-prankuliks-default-000002a2   10.240.0.5     bx2.4x16   normal   Ready    us-south-1   1.30.0_1517
test-cp265fs20gu9km5st1u0-prankuliks-default-0000031c   10.240.64.4    bx2.4x16   normal   Ready    us-south-2   1.30.0_1517
test-cp265fs20gu9km5st1u0-prankuliks-default-00000461   10.240.64.5    bx2.4x16   normal   Ready    us-south-2   1.30.0_1517

❯ kubectl scale deployment my-dep --replicas=5
deployment.apps/my-dep scaled

❯ kubectl get pods -o wide
NAME                      READY   STATUS    RESTARTS   AGE   IP               NODE           NOMINATED NODE   READINESS GATES
my-dep-5d98cb9588-264zn   1/1     Running   0          17m   172.17.235.201   10.240.0.254   <none>           <none>
my-dep-5d98cb9588-2rqfg   1/1     Running   0          27s   172.17.195.5     10.240.64.4    <none>           <none>
my-dep-5d98cb9588-2s6pt   1/1     Running   0          27s   172.17.202.4     10.240.64.5    <none>           <none>
my-dep-5d98cb9588-b9m2g   1/1     Running   0          27s   172.17.221.84    10.240.0.5     <none>           <none>
my-dep-5d98cb9588-llm82   1/1     Running   0          27s   172.17.195.4     10.240.64.4    <none>           <none>

❯ kubectl exec my-dep-5d98cb9588-2rqfg -it -- bash
root@my-dep-5d98cb9588-2rqfg:/# cat mount-path/eit_file.txt 
HI

Non-EIT test cases

  • Create non EIT based PVC and mount + r/w
❯ kubectl get pvc
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        VOLUMEATTRIBUTESCLASS   AGE
my-pvc       Bound    pvc-97b56a6a-6bfc-4d06-81a8-14e2fff6fa39   20Gi       RWX            ibmc-vpc-file-eit   <unset>                 23m
my-pvc-eni   Bound    pvc-96dce036-d913-4652-bca9-3299486bf075   20Gi       RWX            ibmc-vpc-file-dp2   <unset>                 51s

❯ kubectl get pods | grep eni
eni-app-f96ffd6dd-869fj   1/1     Running   0          15s
eni-app-f96ffd6dd-9xfc9   1/1     Running   0          15s

❯ kubectl exec eni-app-f96ffd6dd-869fj -it -- bash
root@eni-app-f96ffd6dd-869fj:/# df -h
Filesystem                                         Size  Used Avail Use% Mounted on
overlay                                             98G  7.3G   86G   8% /
tmpfs                                               64M     0   64M   0% /dev
tmpfs                                              7.8G     0  7.8G   0% /sys/fs/cgroup
10.240.0.19:/2010ed3a_3377_4223_8d67_b5ed95c8438c   20G  320K   20G   1% /mount-path-eni
/dev/vda2                                           98G  7.3G   86G   8% /etc/hosts
shm                                                 64M     0   64M   0% /dev/shm
tmpfs                                               13G   16K   13G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                                              7.8G     0  7.8G   0% /proc/acpi
tmpfs                                              7.8G     0  7.8G   0% /proc/scsi
tmpfs                                              7.8G     0  7.8G   0% /sys/firmware
root@eni-app-f96ffd6dd-869fj:/# cd mount-path-eni/
root@eni-app-f96ffd6dd-869fj:/mount-path-eni# echo "Hi" > text.txt
root@eni-app-f96ffd6dd-869fj:/mount-path-eni# ls -la
total 8
drwxrwxr-x 2 nobody 4294967294 4096 May 15 08:19 .
drwxr-xr-x 1 root   root       4096 May 15 08:18 ..
-rw-r--r-- 1 nobody 4294967294    3 May 15 08:19 text.txt
root@eni-app-f96ffd6dd-869fj:/mount-path-eni# cat tes
cat: tes: No such file or directory
root@eni-app-f96ffd6dd-869fj:/mount-path-eni# cat text.txt 
Hi
root@eni-app-f96ffd6dd-869fj:/mount-path-eni# exit
exit
  • Delete EIT packages and create non-EIT-based PVC and mount r/w
❯ kubectl get pods -n kube-system -o wide| grep insta
uninstaller-7ns68                                     1/1     Running   0               43s     10.240.64.5      10.240.64.5    <none>           <none>
uninstaller-7w6jt                                     1/1     Running   0               43s     10.240.0.254     10.240.0.254   <none>           <none>
uninstaller-h9kh8                                     1/1     Running   0               43s     10.240.64.4      10.240.64.4    <none>           <none>
uninstaller-rshw9                                     1/1     Running   0               43s     10.240.0.5       10.240.0.5     <none>           <none>

# r/w existing mount
❯ kubectl exec eni-app-f96ffd6dd-869fj -it -- bash
root@eni-app-f96ffd6dd-869fj:/# 
root@eni-app-f96ffd6dd-869fj:/# cat mount-path-eni/text.txt 
Hi

# Delete pod and mount again
❯ kubectl get pods | grep eni
eni-app-f96ffd6dd-vr56x   1/1     Running   0          5s
eni-app-f96ffd6dd-wwgxc   1/1     Running   0          5s
❯ 
❯ 
❯ kubectl exec eni-app-f96ffd6dd-vr56x -it -- bash
root@eni-app-f96ffd6dd-vr56x:/# df -h
Filesystem                                         Size  Used Avail Use% Mounted on
overlay                                             98G  5.7G   88G   7% /
tmpfs                                               64M     0   64M   0% /dev
tmpfs                                              7.8G     0  7.8G   0% /sys/fs/cgroup
10.240.0.19:/2010ed3a_3377_4223_8d67_b5ed95c8438c   20G  320K   20G   1% /mount-path-eni
/dev/vda2                                           98G  5.7G   88G   7% /etc/hosts
shm                                                 64M     0   64M   0% /dev/shm
tmpfs                                               13G   16K   13G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                                              7.8G     0  7.8G   0% /proc/acpi
tmpfs                                              7.8G     0  7.8G   0% /proc/scsi
tmpfs                                              7.8G     0  7.8G   0% /sys/firmware
root@eni-app-f96ffd6dd-vr56x:/# cd mount-path-eni/
root@eni-app-f96ffd6dd-vr56x:/mount-path-eni# ls
text.txt
root@eni-app-f96ffd6dd-vr56x:/mount-path-eni# exit
exit

# Check status of it pods as MH/MHC package is deleted
❯ kubectl get pods | grep my-dep
my-dep-5d98cb9588-264zn   1/1     Running   0          26m
my-dep-5d98cb9588-2rqfg   1/1     Running   0          9m41s
my-dep-5d98cb9588-2s6pt   1/1     Running   0          9m41s
my-dep-5d98cb9588-b9m2g   1/1     Running   0          9m41s
my-dep-5d98cb9588-llm82   1/1     Running   0          9m41s

# inside pod things hang
❯ kubectl exec my-dep-5d98cb9588-264zn -it -- bash
root@my-dep-5d98cb9588-264zn:/# 
root@my-dep-5d98cb9588-264zn:/# df -h

^C
root@my-dep-5d98cb9588-264zn:/# 

Positive Test Case (ROKS 4.15 with --disable-outbound-traffic-protection and metadata service not enabled)


EIT test cases

  • Install EIT packages on system using deployer image (contains logic to install offline packages with skipping installation/uninstallation in case that package is already present in the system)
❯ kubectl get pods -n kube-system -o wide| grep file
ibm-vpc-file-csi-controller-84f8d5d57-j9x5w   5/5     Running   0          19s   172.17.114.119   10.240.0.239   <none>           <none>
ibm-vpc-file-csi-controller-84f8d5d57-p9v2g   5/5     Running   0          19s   172.17.91.117    10.240.0.238   <none>           <none>
ibm-vpc-file-csi-node-7nzz6                   4/4     Running   0          19s   10.240.0.239     10.240.0.239   <none>           <none>
ibm-vpc-file-csi-node-cl6pc                   4/4     Running   0          20s   10.240.0.238     10.240.0.238   <none>           <none>

❯ kubectl get pods -n kube-system -o wide| grep insta
installer-6857m                               1/1     Running   0          33s    10.240.0.238     10.240.0.238   <none>           <none>
installer-ds9lh                               1/1     Running   0          33s    10.240.0.239     10.240.0.239   <none>           <none>
  • PVC creation and confirm ipsec status once mounted
❯ kubectl get pvc
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        AGE
my-pvc-eit   Bound    pvc-5b1b61a8-8e2c-4cff-a328-955d1772cf9d   20Gi       RWX            ibmc-vpc-file-eit   54s

❯ kubectl describe pv | grep isEITEnabled
                           isEITEnabled=true

❯ kubectl get pods -w
NAME                      READY   STATUS              RESTARTS   AGE
my-dep-689885b44f-54jbn   0/1     ContainerCreating   0          39s
my-dep-689885b44f-lkdpf   0/1     ContainerCreating   0          39s

# After ~90 sec  will get error is pod describe that metadata service is not enabled for this cluster
❯ kubectl get events --field-selector involvedObject.name=my-dep-689885b44f-54jbn
LAST SEEN   TYPE      REASON        OBJECT                        MESSAGE
2m6s        Normal    Scheduled     pod/my-dep-689885b44f-54jbn   Successfully assigned default/my-dep-689885b44f-54jbn to 10.240.0.239
30s         Warning   FailedMount   pod/my-dep-689885b44f-54jbn   MountVolume.SetUp failed for volume "pvc-5b1b61a8-8e2c-4cff-a328-955d1772cf9d" : rpc error: code = Internal desc = {RequestID: a058d2ec-d093-4a22-a8cc-9379194203c6 , Code: MetadataServiceNotEnabled, Description: Failed to mount target., BackendError: Response from mount-helper-container -> Exit Status Code: exit status 1 ,ResponseCode: 500, Action: Metadata service might not be enabled for worker node. Make sure to use IKS>=1.30 or ROKS>=4.16 cluster.}

# Delete deployment and wait for pod to be deleted
❯ kubectl delete deployment my-dep
deployment.apps "my-dep" deleted
❯ kubectl get pods
No resources found in default namespace.
  • Multiple scaleUp scaleDown with r/w operations -- NA as there isn't any ROKS cluster available where ibmshare mounting is possible
  • Multi node mount -- NA as there isn't any ROKS cluster available where ibmshare mounting is possible
  • Cross zone mount -- NA as there isn't any ROKS cluster available where ibmshare mounting is possible

Non-EIT test cases

  • Create non EIT based PVC and mount + r/w
❯ kubectl get pvc
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        AGE
my-pvc       Bound    pvc-223108e4-b436-4099-ab6e-83807f43e253   20Gi       RWX            ibmc-vpc-file-dp2   47s
my-pvc-eit   Bound    pvc-5b1b61a8-8e2c-4cff-a328-955d1772cf9d   20Gi       RWX            ibmc-vpc-file-eit   9m1s
❯ kubectl get pods
NAME                      READY   STATUS    RESTARTS   AGE
my-dep-7d449d8b5d-ptbfk   1/1     Running   0          13s
my-dep-7d449d8b5d-rqr6v   1/1     Running   0          13s
❯ kubectl exec my-dep-7d449d8b5d-ptbfk -it -- bash
root@my-dep-7d449d8b5d-ptbfk:/# df -h   
Filesystem                                         Size  Used Avail Use% Mounted on
overlay                                             98G   17G   77G  18% /
tmpfs                                               64M     0   64M   0% /dev
tmpfs                                              7.9G     0  7.9G   0% /sys/fs/cgroup
shm                                                 64M     0   64M   0% /dev/shm
/dev/vda2                                           98G   17G   77G  18% /etc/hosts
10.240.0.17:/e4a0cca5_05b9_4bc7_b9bb_4cb2c443f676   20G  320K   20G   1% /mount-path
tmpfs                                               13G   24K   13G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                                              7.9G     0  7.9G   0% /proc/acpi
tmpfs                                              7.9G     0  7.9G   0% /proc/scsi
tmpfs                                              7.9G     0  7.9G   0% /sys/firmware
root@my-dep-7d449d8b5d-ptbfk:/# cd mount-path/
root@my-dep-7d449d8b5d-ptbfk:/mount-path# echo "HI" > test.txt
root@my-dep-7d449d8b5d-ptbfk:/mount-path# ls -la
total 12
drwxrwxr-x. 2 nobody nogroup 4096 May 15 07:43 .
dr-xr-xr-x. 1 root   root    4096 May 15 07:42 ..
-rw-r--r--. 1 nobody nogroup    3 May 15 07:43 test.txt
root@my-dep-7d449d8b5d-ptbfk:/mount-path# exit
exit
  • Delete EIT packages and create non-EIT-based PVC and mount r/w
❯ kubectl get pods -n kube-system -o wide| grep uninst
uninstaller-fvjcv                             1/1     Running   0          28s   10.240.0.238     10.240.0.238   <none>           <none>
uninstaller-x5lmh                             1/1     Running   0          28s   10.240.0.239     10.240.0.239   <none>           <none>

# Check existing non-eit mount r/w
❯ kubectl exec my-dep-7d449d8b5d-ptbfk -it -- bash
root@my-dep-7d449d8b5d-ptbfk:/# 
root@my-dep-7d449d8b5d-ptbfk:/# cd mount-path/
root@my-dep-7d449d8b5d-ptbfk:/mount-path# cat test.txt 
HI
root@my-dep-7d449d8b5d-ptbfk:/mount-path# ls -la
total 12
drwxrwxr-x. 2 nobody nogroup 4096 May 15 07:43 .
dr-xr-xr-x. 1 root   root    4096 May 15 07:42 ..
-rw-r--r--. 1 nobody nogroup    3 May 15 07:43 test.txt

❯ kubectl delete -f dep.yaml
deployment.apps "my-dep" deleted
❯ kubectl get pods
No resources found in default namespace.

# Mount again
❯ kubectl get pods
NAME                      READY   STATUS    RESTARTS   AGE
my-dep-7d449d8b5d-lthcw   1/1     Running   0          11s
my-dep-7d449d8b5d-mkhwl   1/1     Running   0          11s
❯ 
❯ 
❯ kubectl exec my-dep-7d449d8b5d-lthcw -it -- bash
root@my-dep-7d449d8b5d-lthcw:/# cd mount-path/
root@my-dep-7d449d8b5d-lthcw:/mount-path# ls
test.txt
root@my-dep-7d449d8b5d-lthcw:/mount-path# cat test.txt 
HI

@prankulmahajan prankulmahajan merged commit 789e547 into master May 15, 2024
@prankulmahajan prankulmahajan deleted the eit branch May 20, 2024 12:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants