Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Uninstall hangs with NFS mount points (using longhorn for example) #8520

Closed
smutel opened this issue Oct 3, 2023 · 2 comments
Closed

Uninstall hangs with NFS mount points (using longhorn for example) #8520

smutel opened this issue Oct 3, 2023 · 2 comments
Assignees
@brandond @fmoral2
Milestone
v1.28.4+k3s1

Comments

@smutel
Copy link
Contributor

smutel commented Oct 3, 2023

Environmental Info:
K3s Version:
v1.27.5+k3s1

Node(s) CPU architecture, OS, and Version:

Linux 5.10.0-25-amd64 #1 SMP Debian 5.10.191-1 (2023-08-16) x86_64 GNU/Linux

Cluster Configuration:

3 servers, 4 agents

Describe the bug:

When using k3s with storage system like longhorn, the uninstall script hangs when trying to unmount filesystems managed by NFS.

Steps To Reproduce:

  • Installed K3s
  • Install longhorn on top of it
  • Create a pod with a storage located in longhorn
  • Uninstall K3s

Expected behavior:

K3s uninstalled

Actual behavior:

Unable to uninstall K3s until all longhorn volumes are destroyed
@smutel smutel mentioned this issue Oct 3, 2023
Merged
@brandond
Copy link
Contributor

brandond commented Oct 3, 2023

for NFS with longhorn, I assume you need to create a RWX volume, not just a standard LH volume?

@brandond brandond added this to the v1.28.4+k3s1 milestone Nov 14, 2023
@brandond brandond self-assigned this Nov 14, 2023
This was referenced Nov 14, 2023
Closed
Closed
Closed
@fmoral2
Copy link
Contributor

fmoral2 commented Nov 22, 2023

Validated on Version:

-$   k3s version v1.28.3+k3s-6d3a92a6 (6d3a92a6)

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"

Cluster Configuration:
1 node server

Steps to validate the fix

  1. Install k3s
  2. Install daemonset with nfs and iscsi (https://github.com/longhorn/longhorn/blob/master/deploy/prerequisite/longhorn-nfs-installation.yaml and https://github.com/longhorn/longhorn/blob/master/deploy/prerequisite/longhorn-iscsi-installation.yaml)
  3. Deploy Longhorn using Rancher-chart or using manifest https://github.com/longhorn/longhorn/blob/v1.5.3/deploy/longhorn.yaml
  4. deploy https://github.com/longhorn/longhorn/blob/v1.5.3/examples/rwx/rwx-nginx-deployment.yaml to create a pod with rwx volume.
  5. Validate nodes and pods are up and ok
  6. Validate pv and pvc are created and bound
  7. Run uninstall script
  8. Validate k3s no longet installed

Reproduction Issue:

#$ k3s -v
k3s version v1.28.3+k3s-0011eb5e (0011eb5e)
 

#$  k get pv -A
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM              STORAGECLASS   REASON   AGE
pvc-9816dc19-dd55-4141-b08f-cab375127c0f   1Gi        RWX            Delete           Bound    default/rwx-test   longhorn                117s

~$ k get pvc -A
NAMESPACE   NAME       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
default     rwx-test   Bound    pvc-9816dc19-dd55-4141-b08f-cab375127c0f   1Gi        RWX            longhorn       2m23s

~$ k get pods -A
NAMESPACE         NAME                                                     READY   STATUS      RESTARTS   AGE
default           longhorn-iscsi-installation-vshdg                        1/1     Running     0          3m19s
default           longhorn-nfs-installation-r88zj                          1/1     Running     0          3m46s
default           rwx-test-79d5494fd8-54fgw                                2/2     Running     0          111s
default           rwx-test-79d5494fd8-vp2v9                                2/2     Running     0          111s
default           rwx-test-79d5494fd8-zxgjg                                2/2     Running     0          111s
kube-system       coredns-6799fbcd5-lnvj6                                  1/1     Running     0          5m34s
kube-system       helm-install-traefik-crd-xhwz5                           0/1     Completed   0          5m35s
kube-system       helm-install-traefik-zt2pw                               0/1     Completed   1          5m35s
kube-system       local-path-provisioner-84db5d44d9-pkpwz                  1/1     Running     0          5m34s
kube-system       metrics-server-67c658944b-7wqlt                          1/1     Running     0          5m34s
kube-system       svclb-traefik-a352b0cc-t444r                             2/2     Running     0          5m18s
kube-system       traefik-f4564c4f4-pnjlg                                  1/1     Running     0          5m18s
longhorn-system   csi-attacher-79b44f5d-7kbbn                              1/1     Running     0          111s
longhorn-system   csi-attacher-79b44f5d-rf2d5                              1/1     Running     0          111s
longhorn-system   csi-attacher-79b44f5d-rktjh                              1/1     Running     0          111s
longhorn-system   csi-provisioner-c5bb4fff7-5fntj                          1/1     Running     0          111s
longhorn-system   csi-provisioner-c5bb4fff7-jznz4                          1/1     Running     0          111s
longhorn-system   csi-provisioner-c5bb4fff7-w8fr9                          1/1     Running     0          111s
longhorn-system   csi-resizer-8cc975c7f-gbxs6                              1/1     Running     0          111s
longhorn-system   csi-resizer-8cc975c7f-kqnvm                              1/1     Running     0          111s
longhorn-system   csi-resizer-8cc975c7f-wnl8d                              1/1     Running     0          111s
longhorn-system   csi-snapshotter-58bb8475bc-86lhg                         1/1     Running     0          110s
longhorn-system   csi-snapshotter-58bb8475bc-8hc8x                         1/1     Running     0          110s
longhorn-system   csi-snapshotter-58bb8475bc-fdhmd                         1/1     Running     0          110s
longhorn-system   engine-image-ei-68f17757-kbxbf                           1/1     Running     0          2m9s
longhorn-system   instance-manager-61b800d6da62f1226ea40566b5d3d24e        1/1     Running     0          2m9s
longhorn-system   longhorn-csi-plugin-hz9cr                                3/3     Running     0          109s
longhorn-system   longhorn-driver-deployer-54d5cddccc-nb5kk                1/1     Running     0          2m21s
longhorn-system   longhorn-manager-ktqg9                                   1/1     Running     0          2m21s
longhorn-system   longhorn-ui-5b974686f-4tknj                              1/1     Running     0          2m21s
longhorn-system   longhorn-ui-5b974686f-7rsll                              1/1     Running     0          2m21s


 ~$ /usr/local/bin/k3s-uninstall.sh


stuck here =>
+ do_unmount_and_remove /var/lib/rancher/k3s
+ set +x
+ do_unmount_and_remove /var/lib/kubelet/pods
+ set +x
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/effcf53f-661f-4344-bb57-2919584a6d57/volumes/kubernetes.io~projected/kube-api-access-jw2xc
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/ef7dcca7-7aaf-4857-910a-8ff71db0c788/volumes/kubernetes.io~projected/kube-api-access-qhvk9
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/e525235b-4e36-426c-9874-579ba11ff7b6/volumes/kubernetes.io~projected/kube-api-access-xs8zs
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/e46edee6-1292-4558-9c72-ebac3c460ba5/volumes/kubernetes.io~projected/kube-api-access-zprtr
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/e2ed2d2a-13e6-4acd-a03d-e8503fe08699/volumes/kubernetes.io~secret/longhorn-grpc-tls
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/e2ed2d2a-13e6-4acd-a03d-e8503fe08699/volumes/kubernetes.io~projected/kube-api-access-vx5jd
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/e12b0d9d-94d0-48e5-98b0-4582e906b37a/volumes/kubernetes.io~projected/kube-api-access-fvhlf
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/d8edbdfb-9c7a-490c-980f-f108b64efc15/volumes/kubernetes.io~projected/kube-api-access-6nxsm
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/cab83c05-902f-4d71-9cbb-374f527e10dd/volumes/kubernetes.io~projected/kube-api-access-nsxlx
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/cab83c05-902f-4d71-9cbb-374f527e10dd/volumes/kubernetes.io~csi/pvc-9816dc19-dd55-4141-b08f-cab375127c0f/mount
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/c0e6113e-b435-46ad-9fed-0fbf0e4e80aa/volumes/kubernetes.io~projected/kube-api-access-xcgj7
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/b7f9c299-5da9-4d29-a760-957beb6ca288/volumes/kubernetes.io~projected/kube-api-access-fjpvk
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/b41d6a2a-ca8e-466c-870c-f31f42b28b98/volumes/kubernetes.io~projected/kube-api-access-s4qrz
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/b41d6a2a-ca8e-466c-870c-f31f42b28b98/volumes/kubernetes.io~csi/pvc-9816dc19-dd55-4141-b08f-cab375127c0f/mount
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/a1c4c9ff-7f14-4742-aa6f-3d2eb037b9a5/volumes/kubernetes.io~projected/kube-api-access-nx4dd
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/7a8848f0-a6f8-4e31-bb46-a20933a595c7/volumes/kubernetes.io~projected/kube-api-access-v7528
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/70eb8296-499d-4a31-b1ef-f3e4d8e052bc/volumes/kubernetes.io~projected/kube-api-access-9dwhp
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/70eb8296-499d-4a31-b1ef-f3e4d8e052bc/volumes/kubernetes.io~csi/pvc-9816dc19-dd55-4141-b08f-cab375127c0f/mount
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/6f4bdf5e-557d-4a7e-945a-3d43a6da39fe/volumes/kubernetes.io~projected/kube-api-access-fb67s
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/6c14f024-253f-4ace-923b-dd41c4ccc576/volumes/kubernetes.io~projected/kube-api-access-6gljw
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/6b02b512-1f85-4293-acf7-d2492da07ba7/volumes/kubernetes.io~projected/kube-api-access-hx5b7
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/69f14bba-4b3d-46d0-8a62-1cf029509787/volumes/kubernetes.io~secret/longhorn-grpc-tls
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/69f14bba-4b3d-46d0-8a62-1cf029509787/volumes/kubernetes.io~projected/kube-api-access-tdktk
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/65d72495-181b-45a5-9867-7cefcc7a515e/volumes/kubernetes.io~projected/kube-api-access-qsfv9
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/637c8b3a-2a62-4873-a8de-55f522987f6d/volumes/kubernetes.io~projected/kube-api-access-7dw87
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/54435a11-c9d9-48a2-86ac-15006a1fbdd8/volumes/kubernetes.io~projected/kube-api-access-6kljc
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/3ef82823-3f52-4060-b406-cce7400393a5/volumes/kubernetes.io~projected/kube-api-access-dlx9m
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/3cc05f3f-c77f-4e74-93e0-e88fbaa2a013/volumes/kubernetes.io~projected/kube-api-access-f874k
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/3aa703d0-0dab-434e-8c64-f636e1609c23/volumes/kubernetes.io~projected/kube-api-access-q6h4m
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/329d2788-d569-44d1-8e9d-f010d7084ff1/volumes/kubernetes.io~projected/kube-api-access-kcsfs
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/2d8944f4-2bb6-43ab-9e9a-e0eebc66929b/volumes/kubernetes.io~projected/kube-api-access-jdgxj
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/1a2df9b0-2f4d-4db2-97b7-6bf156d83b76/volumes/kubernetes.io~projected/kube-api-access-6f898
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/0cf75b40-76a1-4ba7-a03d-15121e7bbe68/volumes/kubernetes.io~projected/kube-api-access-7pdz7
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/pods/07575c5b-655d-419a-a2be-7d481015c3ca/volumes/kubernetes.io~projected/kube-api-access-lm9kl
+ do_unmount_and_remove /var/lib/kubelet/plugins
+ set +x
sh -c 'umount -f "$0" && rm -rf "$0"' /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/cdb38f0bf0d253e9a2d067f8865ab6ecb0a1533f091b3514f0f377778c803244/globalmount

Validation Results:

k3s -v
k3s version v1.28.3+k3s-6d3a92a6 (6d3a92a6)
go version go1.20.10

 
k get pvc -A
NAMESPACE   NAME       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
default     rwx-test   Bound    pvc-ef6fbb94-e7eb-48c4-87d3-05de87b19458   1Gi        RWX            longhorn       2m31s


$ k get pv -A
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM              STORAGECLASS   REASON   AGE
pvc-ef6fbb94-e7eb-48c4-87d3-05de87b19458   1Gi        RWX            Delete           Bound    default/rwx-test   longhorn                108s


 k get pods -A
NAMESPACE         NAME                                                     READY   STATUS      RESTARTS   AGE
default           longhorn-iscsi-installation-kkbvk                        1/1     Running     0          3m6s
default           longhorn-nfs-installation-bdx7f                          1/1     Running     0          3m26s
default           rwx-test-79d5494fd8-586tl                                2/2     Running     0          2m11s
default           rwx-test-79d5494fd8-869c9                                2/2     Running     0          2m11s
default           rwx-test-79d5494fd8-v2g99                                2/2     Running     0          2m11s
kube-system       coredns-6799fbcd5-9mbhm                                  1/1     Running     0          3m30s
kube-system       helm-install-traefik-crd-4pkg9                           0/1     Completed   0          3m30s
kube-system       helm-install-traefik-zvjlx                               0/1     Completed   1          3m30s
kube-system       local-path-provisioner-84db5d44d9-hx895                  1/1     Running     0          3m30s
kube-system       metrics-server-67c658944b-z6jnk                          1/1     Running     0          3m30s
kube-system       svclb-traefik-a989f811-8w2k9                             2/2     Running     0          2m57s
kube-system       traefik-f4564c4f4-jcpzv                                  1/1     Running     0          2m57s
longhorn-system   csi-attacher-79b44f5d-kl2zm                              1/1     Running     0          95s
longhorn-system   csi-attacher-79b44f5d-mtml5                              1/1     Running     0          95s
longhorn-system   csi-attacher-79b44f5d-r2dsl                              1/1     Running     0          95s
longhorn-system   csi-provisioner-c5bb4fff7-bxjt6                          1/1     Running     0          95s
longhorn-system   csi-provisioner-c5bb4fff7-dbbps                          1/1     Running     0          95s
longhorn-system   csi-provisioner-c5bb4fff7-qqwpq                          1/1     Running     0          95s
longhorn-system   csi-resizer-8cc975c7f-6pv9p                              1/1     Running     0          94s
longhorn-system   csi-resizer-8cc975c7f-blkj4                              1/1     Running     0          94s
longhorn-system   csi-resizer-8cc975c7f-tgs54                              1/1     Running     0          94s
longhorn-system   csi-snapshotter-58bb8475bc-88528                         1/1     Running     0          94s
longhorn-system   csi-snapshotter-58bb8475bc-qgklh                         1/1     Running     0          94s
longhorn-system   csi-snapshotter-58bb8475bc-vjzws                         1/1     Running     0          94s
longhorn-system   engine-image-ei-68f17757-j79sz                           1/1     Running     0          2m4s
longhorn-system   instance-manager-a87b39431cfe738389e16278129d89a3        1/1     Running     0          2m4s
longhorn-system   longhorn-csi-plugin-rbw45                                3/3     Running     0          93s
longhorn-system   longhorn-driver-deployer-54d5cddccc-w7ssd                1/1     Running     0          2m29s
longhorn-system   longhorn-manager-2gw2w                                   1/1     Running     0          2m29s
longhorn-system   longhorn-ui-5b974686f-csccq                              1/1     Running     0          2m29s
longhorn-system   longhorn-ui-5b974686f-svsbf                              1/1     Running     0          2m29s
longhorn-system   share-manager-pvc-ef6fbb94-e7eb-48c4-87d3-05de87b19458   1/1     Running     0          39s




 ~$ /usr/local/bin/k3s-uninstall.sh


 ....
 + type rpm-ostree
+ type zypper
+ remove_uninstall
+ rm -f /usr/local/bin/k3s-uninstall.sh

~$ k3s -v
-bash: /usr/local/bin/k3s: No such file or directory

@fmoral2 fmoral2 closed this as completed Nov 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@brandond brandond

@fmoral2 fmoral2

Labels
None yet
Projects
K3s Development
Archived in project
Milestone
v1.28.4+k3s1
Development

No branches or pull requests
3 participants
@brandond @smutel @fmoral2