Set volume health to inaccessible when PVC not found in CNS #944

xing-yang · 2021-06-02T21:54:46Z

What this PR does / why we need it:
Set volume health to inaccessible when PVC not found in CNS

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Testing done:
https://container-dp.svc.eng.vmware.com/view/Pre-Checkin-CSI/job/csi-wcp-pre-check-in/29/testReport/(root)/CNS%20CSI%20Driver%20End-to-End%20Tests/

13 tests passed, 1 failed.

CNS CSI Driver End-to-End Tests.Basic Static Provisioning [csi-supervisor] Verify static provisioning workflow - when DuplicateFCD is used

/home/worker/workspace/csi-wcp-pre-check-in@2/Results/29/vsphere-csi-driver/tests/e2e/csi_static_provisioning_basic.go:882
Unexpected error:
<*errors.errorString | 0xc000eaed10>: {
s: "PersistentVolume static-pv-561ef975-ada8-4c07-a801-5a5a5c78f4ad still exists within 3m0s",
}
PersistentVolume static-pv-561ef975-ada8-4c07-a801-5a5a5c78f4ad still exists within 3m0s
occurred
/home/worker/workspace/csi-wcp-pre-check-in@2/Results/29/vsphere-csi-driver/tests/e2e/csi_static_provisioning_basic.go:959

The test failure is not caused by the change.

Special notes for your reviewer:

Release note:

Set volume health to inaccessible when PVC is not found in CNS

xing-yang · 2021-06-02T21:56:13Z

jtest wcp

xing-yang · 2021-06-02T21:56:24Z

jtest gc

svcbot-qecnsdp · 2021-06-02T21:59:08Z

Started WCP block pipeline...

svcbot-qecnsdp · 2021-06-02T22:04:55Z

Started GC block pipeline...

svcbot-qecnsdp · 2021-06-02T22:08:29Z

WCP build status: FAILURE 
Stage before exit: null

xing-yang · 2021-06-02T22:14:58Z

jtest wcp

svcbot-qecnsdp · 2021-06-02T22:17:10Z

Started WCP block pipeline...

svcbot-qecnsdp · 2021-06-02T22:23:12Z

WCP build status: FAILURE 
Stage before exit: null

svcbot-qecnsdp · 2021-06-03T01:03:28Z

GC build status: FAILURE 
Stage before exit: deploy-gc-testbed

svcbot-qecnsdp · 2021-06-03T03:34:29Z

Started WCP block pipeline...

svcbot-qecnsdp · 2021-06-03T03:49:30Z

Build ID: 18
WCP build status: FAILURE 
Stage before exit: null 
Jenkins E2E Test Results: 
Ran 14 of 187 Specs in 82.929 seconds
FAIL! -- 0 Passed | 14 Failed | 0 Pending | 173 Skipped
--- FAIL: TestE2E (83.06s)
FAIL

Ginkgo ran 1 suite in 1m51.461080604s
Test Suite Failed
make: Leaving directory `/home/worker/workspace/csi-wcp-pre-check-in/Results/18/vsphere-csi-driver`

svcbot-qecnsdp · 2021-06-03T03:59:12Z

Started WCP block pipeline...

svcbot-qecnsdp · 2021-06-03T04:28:08Z

Build ID: 19
WCP build status: FAILURE 
Stage before exit: null

svcbot-qecnsdp · 2021-06-03T05:14:56Z

Started WCP block pipeline...

svcbot-qecnsdp · 2021-06-03T06:42:22Z

Build ID: 20
WCP build status: FAILURE 
Stage before exit: null 
Jenkins E2E Test Results: 
Ran 14 of 187 Specs in 4483.011 seconds
FAIL! -- 0 Passed | 14 Failed | 0 Pending | 173 Skipped
--- FAIL: TestE2E (4483.14s)
FAIL

Ginkgo ran 1 suite in 1h15m7.870977408s
Test Suite Failed
make: Leaving directory `/home/worker/workspace/csi-wcp-pre-check-in/Results/20/vsphere-csi-driver`

svcbot-qecnsdp · 2021-06-03T17:39:26Z

Started WCP block pipeline...

svcbot-qecnsdp · 2021-06-03T18:07:22Z

Build ID: 21
WCP build status: FAILURE 
Stage before exit: null

svcbot-qecnsdp · 2021-06-03T18:19:35Z

Started WCP block pipeline...

svcbot-qecnsdp · 2021-06-03T19:02:14Z

Build ID: 22
WCP build status: FAILURE 
Stage before exit: null

svcbot-qecnsdp · 2021-06-04T00:27:05Z

Started WCP block pipeline...

svcbot-qecnsdp · 2021-06-04T01:58:23Z

Build ID: 26
WCP build status: FAILURE 
Stage before exit: null 
Jenkins E2E Test Results: 
Ran 14 of 187 Specs in 4673.084 seconds
FAIL! -- 1 Passed | 13 Failed | 0 Pending | 173 Skipped
--- FAIL: TestE2E (4673.22s)
FAIL

Ginkgo ran 1 suite in 1h18m22.430803379s
Test Suite Failed
make: Leaving directory `/home/worker/workspace/csi-wcp-pre-check-in/Results/26/vsphere-csi-driver`

svcbot-qecnsdp · 2021-06-04T03:06:01Z

Started WCP block pipeline...

svcbot-qecnsdp · 2021-06-04T03:22:31Z

Build ID: 27
WCP build status: FAILURE 
Stage before exit: null 
Jenkins E2E Test Results: 
Ran 14 of 187 Specs in 106.904 seconds
FAIL! -- 0 Passed | 14 Failed | 0 Pending | 173 Skipped
--- FAIL: TestE2E (107.02s)
FAIL

Ginkgo ran 1 suite in 2m14.416897446s
Test Suite Failed
make: Leaving directory `/home/worker/workspace/csi-wcp-pre-check-in/Results/27/vsphere-csi-driver`

svcbot-qecnsdp · 2021-06-04T13:36:41Z

Started WCP block pipeline...

svcbot-qecnsdp · 2021-06-04T15:36:33Z

Build ID: 30
WCP build status: SUCCESS 
Stage before exit: finally 
Jenkins E2E Test Results: 

Ran 14 of 187 Specs in 3253.154 seconds
SUCCESS! -- 14 Passed | 0 Failed | 0 Pending | 173 Skipped
PASS

Ginkgo ran 1 suite in 54m43.233742796s
Test Suite Passed
make: Leaving directory `/home/worker/workspace/csi-wcp-pre-check-in/Results/30/vsphere-csi-driver`

pkg/syncer/volume_health.go

lipingxue · 2021-06-04T17:03:18Z

@xing-yang Could you run some test to make sure when PVC not found in CNS, the volume health status is set to inaccessible?

xing-yang · 2021-06-04T17:11:04Z

@xing-yang Could you run some test to make sure when PVC not found in CNS, the volume health status is set to inaccessible?

This is a rare corner case. I don't know how to reproduce this case. Any suggestions?

svcbot-qecnsdp · 2021-06-04T18:21:03Z

Started WCP block pipeline...

svcbot-qecnsdp · 2021-06-04T18:29:37Z

Build ID: 31
WCP build status: SUCCESS 
Stage before exit: finally 
Jenkins E2E Test Results: 

Ran 1 of 187 Specs in 358.073 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 186 Skipped
PASS

Ginkgo ran 1 suite in 6m26.377712229s
Test Suite Passed
make: Leaving directory `/home/worker/workspace/csi-wcp-pre-check-in/Results/31/vsphere-csi-driver`

lipingxue · 2021-06-04T18:35:43Z

@xing-yang Could you run some test to make sure when PVC not found in CNS, the volume health status is set to inaccessible?

This is a rare corner case. I don't know how to reproduce this case. Any suggestions?

In the code comment, I saw "When a Datastore is removed from VC (like vSAN direct disk decommisson with noAction does)" then PVC will not be found in CNS. So is there any bug that this PR try to fix?
If it is, we may ask the QE who filed this bug to verify with this fix.

Maybe we can try to delete the FCD disk out of band to simulate the case that PVC not found in CNS?

pkg/syncer/volume_health.go

subramanian-neelakantan · 2021-06-05T06:32:23Z

@xing-yang Could you run some test to make sure when PVC not found in CNS, the volume health status is set to inaccessible?

This is a rare corner case. I don't know how to reproduce this case. Any suggestions?

In the code comment, I saw "When a Datastore is removed from VC (like vSAN direct disk decommisson with noAction does)" then PVC will not be found in CNS. So is there any bug that this PR try to fix?
If it is, we may ask the QE who filed this bug to verify with this fix.

Maybe we can try to delete the FCD disk out of band to simulate the case that PVC not found in CNS?

We have two cases to verify manually. Assuming that you can change the frequency of volume health poller from the default 5 mins, here are the two cases:
Case 1. Set volume health syncer interval > 15 mins so that CNS cache DB sync happens before volume health syncer.
Case 2. Set volume health syncer interval < 15 mins so that volume health syncer runs before CNS cache DB is updated.

Run tests for both cases as follows.

Create a PVC. Wait for its health to show as "accessible".
Delete the underlying FCD out-of-band. This can be done from the https://<vc_ip/vslm/mob using the VStorageObjectManager#VslmDeleteVStorageObject_Task() API.
Expectations in each case:
- Case 1 (health syncer > 15 mins):
  * (Without fix) PVC remains "accessible" forever
  * (With fix) PVC goes to "inaccessible" when health syncer runs
- Case 2 (health syncer < 15 mins):
  * (Without fix) PVC goes to "" health state and then remains that way forever
  * (With fix) PVC goes to "" health state and after 15 mins goes to "inaccessible"

svcbot-qecnsdp · 2021-06-07T16:01:09Z

Started WCP block pipeline...

svcbot-qecnsdp · 2021-06-07T16:26:45Z

Build ID: 32
WCP build status: FAILURE 
Stage before exit: null 
Jenkins E2E Test Results: 
Ran 14 of 187 Specs in 732.107 seconds
FAIL! -- 0 Passed | 14 Failed | 0 Pending | 173 Skipped
--- FAIL: TestE2E (732.24s)
FAIL

Ginkgo ran 1 suite in 12m38.978136957s
Test Suite Failed
make: Leaving directory `/home/worker/workspace/csi-wcp-pre-check-in/Results/32/vsphere-csi-driver`

svcbot-qecnsdp · 2021-06-07T16:54:19Z

Started WCP block pipeline...

svcbot-qecnsdp · 2021-06-07T17:10:38Z

Build ID: 33
WCP build status: FAILURE 
Stage before exit: null 
Jenkins E2E Test Results: 
Ran 14 of 187 Specs in 126.179 seconds
FAIL! -- 0 Passed | 14 Failed | 0 Pending | 173 Skipped
--- FAIL: TestE2E (126.30s)
FAIL

Ginkgo ran 1 suite in 2m35.553567031s
Test Suite Failed
make: Leaving directory `/home/worker/workspace/csi-wcp-pre-check-in/Results/33/vsphere-csi-driver`

xing-yang · 2021-06-07T20:29:28Z

Manually ran the following test:

Default volume heath sync is 2 min and full sync is also 2 min

Environment:
  CLUSTER_FLAVOR:                  WORKLOAD
  FULL_SYNC_INTERVAL_MINUTES:      2
  VOLUME_HEALTH_INTERVAL_MINUTES:  2
  POD_POLL_INTERVAL_SECONDS:       2

Created a new volume, no health status yet

kubectl describe pvc -n e2e-test-namespace

Name: block-pvc
Namespace: e2e-test-namespace
StorageClass: shared-ds-policy
Status: Bound
Volume: pvc-af778591-5423-43ea-8731-99142edb54b9
Labels:
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: csi.vsphere.vmware.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 1Gi
Access Modes: RWO
VolumeMode: Filesystem
Used By:
Events:
Type Reason Age From Message

Normal ExternalProvisioning 25s persistentvolume-controller waiting for a volume to be created, either by external provisioner "csi.vsphere.vmware.com" or manually created by system administrator
Normal Provisioning 25s csi.vsphere.vmware.com_4220a295c5e4dcd867ee94daa3038e09_415ba9cd-0cc9-4eee-ad16-e52419fd8251 External provisioner is provisioning volume for claim "e2e-test-namespace/block-pvc"
Normal ProvisioningSucceeded 21s csi.vsphere.vmware.com_4220a295c5e4dcd867ee94daa3038e09_415ba9cd-0cc9-4eee-ad16-e52419fd8251 Successfully provisioned volume pvc-af778591-5423-43ea-8731-99142edb54b9

Health status shows up now

kubectl describe pvc -n e2e-test-namespace

Name: block-pvc
Namespace: e2e-test-namespace
StorageClass: shared-ds-policy
Status: Bound
Volume: pvc-af778591-5423-43ea-8731-99142edb54b9
Labels:
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: csi.vsphere.vmware.com
volumehealth.storage.kubernetes.io/health: accessible
volumehealth.storage.kubernetes.io/health-timestamp: Mon Jun 7 19:38:08 UTC 2021
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 1Gi
Access Modes: RWO
VolumeMode: Filesystem
Used By:

Corresponding logs:

2021-06-07T19:38:08.528Z DEBUG syncer/volume_health.go:122 updateVolumeHealthStatus: update volume health annotation for pvc e2e-test-namespace/block-pvc from old value to new value accessible {"TraceId": "3f7dc2ef-843a-476a-b1cb-0ec6515b55fc"}
2021-06-07T19:38:08.528Z INFO syncer/volume_health.go:127 updateVolumeHealthStatus: set annotation for health to accessible at time Mon Jun 7 19:38:08 UTC 2021 for pvc e2e-test-namespace/block-pvc {"TraceId": "3f7dc2ef-843a-476a-b1cb-0ec6515b55fc"}

Delete FCD at 19:44:23 UTC 2021; Health status stayed as accessible until it is updated later

kubectl describe pvc -n e2e-test-namespace

Name: block-pvc
Namespace: e2e-test-namespace
StorageClass: shared-ds-policy
Status: Bound
Volume: pvc-af778591-5423-43ea-8731-99142edb54b9
Labels:
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: csi.vsphere.vmware.com
volumehealth.storage.kubernetes.io/health: accessible
volumehealth.storage.kubernetes.io/health-timestamp: Mon Jun 7 19:38:08 UTC 2021
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 1Gi
Access Modes: RWO
VolumeMode: Filesystem
Used By:

Changed to “inaccessible” 4 minutes after FCD is deleted

It coincided with the ending of a FullSync period.

kubectl describe pvc -n e2e-test-namespace

Name: block-pvc
Namespace: e2e-test-namespace
StorageClass: shared-ds-policy
Status: Bound
Volume: pvc-af778591-5423-43ea-8731-99142edb54b9
Labels:
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: csi.vsphere.vmware.com
volumehealth.storage.kubernetes.io/health: inaccessible
volumehealth.storage.kubernetes.io/health-timestamp: Mon Jun 7 19:48:08 UTC 2021
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 1Gi
Access Modes: RWO
VolumeMode: Filesystem
Used By:

Corresponding logs:

2021-06-07T19:48:07.994Z WARN syncer/fullsync.go:367 could not find any volume which is present in both k8s and in CNS {"TraceId": "0d895233-57fa-4639-9db9-d4201a3a8572"}

2021-06-07T19:48:07.998Z INFO syncer/fullsync.go:430 FullSync: Volume with id: "bd9536ef-6696-4383-a5ba-aefa2a56dd84" and name: "pvc-af778591-5423-43ea-8731-99142edb54b9" is added to cnsCreationMap {"TraceId": "0d895233-57fa-4639-9db9-d4201a3a8572"}

2021-06-07T19:48:08.000Z INFO syncer/fullsync.go:137 FullSync: end {"TraceId": "0d895233-57fa-4639-9db9-d4201a3a8572"}

2021-06-07T19:48:08.006Z DEBUG syncer/volume_health.go:122 updateVolumeHealthStatus: update volume health annotation for pvc e2e-test-namespace/block-pvc from old value accessible to new value inaccessible {"TraceId": "0d895233-57fa-4639-9db9-d4201a3a8572"}
2021-06-07T19:48:08.007Z INFO syncer/volume_health.go:127 updateVolumeHealthStatus: set annotation for health to inaccessible at time Mon Jun 7 19:48:08 UTC 2021 for pvc e2e-test-namespace/block-pvc {"TraceId": "0d895233-57fa-4639-9db9-d4201a3a8572"}

xing-yang · 2021-06-07T20:35:59Z

@subramanian-neelakantan @SandeepPissay @lipingxue Comments are addressed. PTAL.

svcbot-qecnsdp · 2021-06-07T20:41:04Z

Started WCP block pipeline...

svcbot-qecnsdp · 2021-06-07T21:48:33Z

Build ID: 35
WCP build status: SUCCESS 
Stage before exit: finally 
Jenkins E2E Test Results: 

Ran 14 of 187 Specs in 3233.858 seconds
SUCCESS! -- 14 Passed | 0 Failed | 0 Pending | 173 Skipped
PASS

Ginkgo ran 1 suite in 54m25.137123102s
Test Suite Passed
make: Leaving directory `/home/worker/workspace/csi-wcp-pre-check-in/Results/35/vsphere-csi-driver`

subramanian-neelakantan · 2021-06-08T04:36:25Z

Looks good to me.

/approve

k8s-ci-robot · 2021-06-08T04:36:30Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: subramanian-neelakantan, xing-yang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [xing-yang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

SandeepPissay

The changes looks good to me. I have few comments on the logging.

pkg/syncer/volume_health.go

xing-yang · 2021-06-08T17:18:36Z

@SandeepPissay comments are addressed. PTAL.

SandeepPissay · 2021-06-08T17:53:45Z

/lgtm

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jun 2, 2021

k8s-ci-robot requested review from chethanv28 and RaunakShah June 2, 2021 21:54

k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 2, 2021

SandeepPissay reviewed Jun 4, 2021

View reviewed changes

pkg/syncer/volume_health.go Show resolved Hide resolved

SandeepPissay reviewed Jun 4, 2021

View reviewed changes

pkg/syncer/volume_health.go Outdated Show resolved Hide resolved

SandeepPissay reviewed Jun 4, 2021

View reviewed changes

pkg/syncer/volume_health.go Outdated Show resolved Hide resolved

pkg/syncer/volume_health.go Outdated Show resolved Hide resolved

subramanian-neelakantan reviewed Jun 5, 2021

View reviewed changes

pkg/syncer/volume_health.go Show resolved Hide resolved

pkg/syncer/volume_health.go Outdated Show resolved Hide resolved

pkg/syncer/volume_health.go Outdated Show resolved Hide resolved

xing-yang force-pushed the syncer_health branch from a27e022 to a29f2f8 Compare June 7, 2021 15:56

SandeepPissay reviewed Jun 8, 2021

View reviewed changes

pkg/syncer/volume_health.go Outdated Show resolved Hide resolved

pkg/syncer/volume_health.go Outdated Show resolved Hide resolved

pkg/syncer/volume_health.go Outdated Show resolved Hide resolved

pkg/syncer/volume_health.go Outdated Show resolved Hide resolved

Set volume health to inaccessible when PVC not found in CNS

4c75ef7

xing-yang force-pushed the syncer_health branch from a29f2f8 to 4c75ef7 Compare June 8, 2021 17:07

k8s-ci-robot assigned SandeepPissay Jun 8, 2021

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 8, 2021

k8s-ci-robot merged commit fe6484f into kubernetes-sigs:master Jun 8, 2021

Set volume health to inaccessible when PVC not found in CNS #944

Set volume health to inaccessible when PVC not found in CNS #944

Conversation

xing-yang commented Jun 2, 2021 • edited Loading

xing-yang commented Jun 2, 2021

xing-yang commented Jun 2, 2021

svcbot-qecnsdp commented Jun 2, 2021

svcbot-qecnsdp commented Jun 2, 2021

svcbot-qecnsdp commented Jun 2, 2021

xing-yang commented Jun 2, 2021

svcbot-qecnsdp commented Jun 2, 2021

svcbot-qecnsdp commented Jun 2, 2021

svcbot-qecnsdp commented Jun 3, 2021

svcbot-qecnsdp commented Jun 3, 2021

svcbot-qecnsdp commented Jun 3, 2021

svcbot-qecnsdp commented Jun 3, 2021

svcbot-qecnsdp commented Jun 3, 2021

svcbot-qecnsdp commented Jun 3, 2021

svcbot-qecnsdp commented Jun 3, 2021

svcbot-qecnsdp commented Jun 3, 2021

svcbot-qecnsdp commented Jun 3, 2021

svcbot-qecnsdp commented Jun 3, 2021

svcbot-qecnsdp commented Jun 3, 2021

svcbot-qecnsdp commented Jun 4, 2021

svcbot-qecnsdp commented Jun 4, 2021

svcbot-qecnsdp commented Jun 4, 2021

svcbot-qecnsdp commented Jun 4, 2021

svcbot-qecnsdp commented Jun 4, 2021

svcbot-qecnsdp commented Jun 4, 2021

lipingxue commented Jun 4, 2021

xing-yang commented Jun 4, 2021

svcbot-qecnsdp commented Jun 4, 2021

svcbot-qecnsdp commented Jun 4, 2021

lipingxue commented Jun 4, 2021 • edited Loading

subramanian-neelakantan commented Jun 5, 2021

svcbot-qecnsdp commented Jun 7, 2021

svcbot-qecnsdp commented Jun 7, 2021

svcbot-qecnsdp commented Jun 7, 2021

svcbot-qecnsdp commented Jun 7, 2021

xing-yang commented Jun 7, 2021 • edited Loading

Created a new volume, no health status yet

Health status shows up now

Delete FCD at 19:44:23 UTC 2021; Health status stayed as accessible until it is updated later

Changed to “inaccessible” 4 minutes after FCD is deleted

xing-yang commented Jun 7, 2021

svcbot-qecnsdp commented Jun 7, 2021

svcbot-qecnsdp commented Jun 7, 2021

subramanian-neelakantan commented Jun 8, 2021

k8s-ci-robot commented Jun 8, 2021

SandeepPissay left a comment

Choose a reason for hiding this comment

xing-yang commented Jun 8, 2021

SandeepPissay commented Jun 8, 2021

xing-yang commented Jun 2, 2021 •

edited

Loading

lipingxue commented Jun 4, 2021 •

edited

Loading

xing-yang commented Jun 7, 2021 •

edited

Loading