Duplicated PVs and lost PVC after bunch of scylla pods recreations caused by K8S scylla node recreations

[vponomaryov]

Describe the bug

After several attempts to recreate K8S scylla node we get following failure event:

Warning   FailedScheduling     pod/sct-cluster-us-east1-b-us-east1-1    
        0/9 nodes are available: 9 persistentvolumeclaim "data-sct-cluster-us-east1-b-us-east1-1" not found.

Which results in non-scheduled pod due to absent PVC.

List of PVs:

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                                                          STORAGECLASS       REASON   AGE    VOLUMEMODE
local-pv-29aa8bcb                          3537Gi     RWO            Delete           Available                                                                  local-raid-disks            49m    Filesystem
local-pv-39c354bd                          3537Gi     RWO            Delete           Available                                                                  local-raid-disks            42m    Filesystem
local-pv-5ef5c4bc                          3537Gi     RWO            Delete           Bound       scylla/data-sct-cluster-us-east1-b-us-east1-2                  local-raid-disks            58m    Filesystem
local-pv-716fa622                          3537Gi     RWO            Delete           Released    scylla/data-sct-cluster-us-east1-b-us-east1-1                  local-raid-disks            152m   Filesystem
local-pv-87153eaf                          3537Gi     RWO            Delete           Released    scylla/data-sct-cluster-us-east1-b-us-east1-1                  local-raid-disks            101m   Filesystem
local-pv-8acb06d5                          3537Gi     RWO            Delete           Bound       scylla/data-sct-cluster-us-east1-b-us-east1-0                  local-raid-disks            66m    Filesystem
local-pv-8d372df5                          3537Gi     RWO            Delete           Released    scylla/data-sct-cluster-us-east1-b-us-east1-1                  local-raid-disks            74m    Filesystem
local-pv-c967a2a4                          3537Gi     RWO            Delete           Released    scylla/data-sct-cluster-us-east1-b-us-east1-2                  local-raid-disks            93m    Filesystem
local-pv-cf5ab3cd                          3537Gi     RWO            Delete           Released    scylla/data-sct-cluster-us-east1-b-us-east1-0                  local-raid-disks            95m    Filesystem
local-pv-ee2540c                           3537Gi     RWO            Delete           Released    scylla/data-sct-cluster-us-east1-b-us-east1-0                  local-raid-disks            153m   Filesystem
pvc-8cf39e10-2b33-40e0-8bd7-47fd0b10437c   10Gi       RWO            Delete           Bound       scylla-manager/data-scylla-manager-manager-dc-manager-rack-0   gp2                         155m   Filesystem
pvc-d8aa1672-e927-45f7-b45a-c5399a987831   10Gi       RWO            Delete           Bound       minio/minio                                                    gp2                         155m   Filesystem

List of PVCs:

NAMESPACE        NAME                                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE    VOLUMEMODE
minio            minio                                           Bound    pvc-d8aa1672-e927-45f7-b45a-c5399a987831   10Gi       RWO            gp2                156m   Filesystem
scylla-manager   data-scylla-manager-manager-dc-manager-rack-0   Bound    pvc-8cf39e10-2b33-40e0-8bd7-47fd0b10437c   10Gi       RWO            gp2                155m   Filesystem
scylla           data-sct-cluster-us-east1-b-us-east1-0          Bound    local-pv-8acb06d5                          3537Gi     RWO            local-raid-disks   58m    Filesystem
scylla           data-sct-cluster-us-east1-b-us-east1-2          Bound    local-pv-5ef5c4bc                          3537Gi     RWO            local-raid-disks   51m    Filesystem

List of Pods:

NAMESPACE         NAME                                                     READY   STATUS             RESTARTS   AGE    IP           NODE                                        NOMINATED NODE   READINESS GATES
...
scylla            sct-cluster-us-east1-b-us-east1-0                        2/2     Running            0          58m    10.0.2.39    ip-10-0-2-39.eu-north-1.compute.internal    <none>           <none>
scylla            sct-cluster-us-east1-b-us-east1-1                        0/2     Pending            0          45m    <none>       <none>                                      <none>           <none>
scylla            sct-cluster-us-east1-b-us-east1-2                        2/2     Running            1          51m    10.0.1.156   ip-10-0-1-156.eu-north-1.compute.internal   <none>           <none>

To Reproduce
Steps to reproduce the behavior:

1. Deploy operator
2. Deploy scylla
3. Terminate a K8S scylla node
4. Mark damaged's scylla pod's service to be replaced
5. Wait till it comes up
6. See error

Expected behavior
No duplicated PVs for first several such attempts and none of PVC gets lost which is dependency for scylla pods workability.

Logs
EKS failure logs:
kubernetes-99684374.tar.gz
GKE failure logs:
kubernetes-71b476cb.tar.gz

Environment:

• Platform: EKS / GKE
• Kubernetes version: v1.20.7-eks-8be107 / v1.19.11-gke.2100
• Scylla version: 4.4.3
• Scylla-operator version: e.g.: v1.4.0-alpha.0-66-g8e21e88

Additional context
Fails on both backends the same way - on EKS and GKE.

[vponomaryov]

This one looks like combo of #663 and #643

[tnozicka]

yeah, the PVC is missing but it should be created by statefulset controller, not us. I'll try to spin GKE to see if I can reproduce it but our e2e for replace works fine, which is weird.

[tnozicka]

hm, it worked fine for me with GKE v1.19.11-gke.2100:

• kubectl drain gke-tnozicka-test-0-default-pool-55574ae2-jd54 --delete-emptydir-data --ignore-daemonsets
• kubectl patch service/basic-us-east-1-us-east-1a-1 --type strategic -p '{"metadata": {"labels":{"scylla/replace": ""}}}'

Can you always reproduce it? If yes, can you write down the exact steps using kubectl?

[vponomaryov]

@tnozicka such bug appears when we terminate K8S node, not drain.

[tnozicka]

terminate K8S node

terminate in what way? delete the GCE instance or delete the kubernetes node object or both in sequence? using the exact CLI commands in the reproducer steps would avoid those ambiguities

[tnozicka]

worked fine even if I just delete the GCE instance like

gcloud compute instance-groups managed delete-instances $( gcloud container clusters describe tnozicka-test-0 --format json | jq  --raw-output '.instanceGroupUrls[0]' | rev | cut -d'/' -f 1 | rev ) --instances gke-tnozicka-test-0-default-pool-9ebea7ec-mmq1