Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add check for CSI volume spec in PVUpdated method #1013

Merged
merged 3 commits into from Jun 24, 2021
Merged

Add check for CSI volume spec in PVUpdated method #1013

merged 3 commits into from Jun 24, 2021

Conversation

chethanv28
Copy link
Collaborator

@chethanv28 chethanv28 commented Jun 23, 2021

What this PR does / why we need it:
This PR is adding CSI check for volume spec in metadatasyncer. Currently we are assuming that the volumes are always CSI volumes, however for migrated volumes the spec does not contain CSI field and hence a crash was observed during CSI migration testing.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Testing done:
Before:

1. Create in-tree volumes
2. Deploy CSI driver
3. Enable CSI migration FSS
4. Enable Migration flags on kube-controller-manager and kubelet

k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.21.1/pkg/util/runtime/runtime.go:55 +0x109
panic(0x203bdc0, 0x385fbb0)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
sigs.k8s.io/vsphere-csi-driver/pkg/syncer.csiPVUpdated(0x27e8c78, 0xc0010a8090, 0xc0008ff900, 0xc00024b900, 0xc0000ec000)
	/build/pkg/syncer/metadatasyncer.go:989 +0x304

Performed same steps after the fix and no crash was observed

&
Running e2e pipelines

Special notes for your reviewer:

Release note:

Add check for volume spec in PVUpdated

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 23, 2021
@k8s-ci-robot k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jun 23, 2021
@chethanv28 chethanv28 changed the title Add check for volume spec to avoid NPE Add check for volume spec in PVUpdated method Jun 23, 2021
@svcbot-qecnsdp
Copy link

Started Vanilla block pre-checkin pipeline... Build Number: 105

@chethanv28 chethanv28 changed the title Add check for volume spec in PVUpdated method Add check for CSI volume spec in PVUpdated method Jun 23, 2021
@chethanv28
Copy link
Collaborator Author

This change needs to be cherry-picked to 2.3 & 2.2.
cc @divyenpatel Can you help add labels for the same

@gohilankit
Copy link
Contributor

Change looks good to me.
/approve

pkg/syncer/metadatasyncer.go Outdated Show resolved Hide resolved
@svcbot-qecnsdp
Copy link

Build ID: 105
Block vanilla build status: FAILURE 
Stage before exit: e2e-tests 
Jenkins E2E Test Results: 
Ran 45 of 195 Specs in 11417.785 seconds
FAIL! -- 44 Passed | 1 Failed | 0 Pending | 150 Skipped
--- FAIL: TestE2E (11417.87s)
FAIL

Ginkgo ran 1 suite in 3h10m57.042729493s
Test Suite Failed
make: Leaving directory `/home/worker/workspace/csi-block-vanilla-pre-check-in/105/vsphere-csi-driver`

@chethanv28
Copy link
Collaborator Author

Build ID: 105
Block vanilla build status: FAILURE 
Stage before exit: e2e-tests 
Jenkins E2E Test Results: 
Ran 45 of 195 Specs in 11417.785 seconds
FAIL! -- 44 Passed | 1 Failed | 0 Pending | 150 Skipped
--- FAIL: TestE2E (11417.87s)
FAIL

Ginkgo ran 1 suite in 3h10m57.042729493s
Test Suite Failed
make: Leaving directory `/home/worker/workspace/csi-block-vanilla-pre-check-in/105/vsphere-csi-driver`
[csi-block-vanilla] [csi-supervisor] statefulset
/home/worker/workspace/csi-block-vanilla-pre-check-in/105/vsphere-csi-driver/tests/e2e/statefulsets.go:56
  Statefulset testing with parallel podManagementPolicy [It]
  /home/worker/workspace/csi-block-vanilla-pre-check-in/105/vsphere-csi-driver/tests/e2e/statefulsets.go:266

  Unexpected error:
      <*errors.StatusError | 0xc0004dec80>: {
          ErrStatus: {
              TypeMeta: {Kind: "", APIVersion: ""},
              ListMeta: {
                  SelfLink: "",
                  ResourceVersion: "",
                  Continue: "",
                  RemainingItemCount: nil,
              },
              Status: "Failure",
              Message: "etcdserver: request timed out",
              Reason: "",
              Details: nil,
              Code: 500,
          },
      }
      etcdserver: request timed out
  occurred

The 1 failure is a known intermittent failure which is tracked internally.

cc: @divyenpatel
Can you help with approvals & merge the PR

Copy link
Member

@divyenpatel divyenpatel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve
/ok-to-test

@k8s-ci-robot k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Jun 24, 2021
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chethanv28, divyenpatel, gohilankit

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [chethanv28,divyenpatel]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@divyenpatel divyenpatel added the release-2.3.0-candidate Indicates PR needs to be cherry-picked for 2.3.0 release label Jun 24, 2021
@divyenpatel
Copy link
Member

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 24, 2021
@k8s-ci-robot k8s-ci-robot merged commit 9e01c67 into kubernetes-sigs:master Jun 24, 2021
k8s-ci-robot pushed a commit that referenced this pull request Jun 24, 2021
* Add check for volume spec to avoid NPE

* Rename the variable used for checking if it is a dynamic PV

* Add an extra check to ensure code block is for static CSI volume
@divyenpatel divyenpatel added the release-2.3.0-cherry-picked Indicates PR is cherry-picked for 2.3.0 release label Jun 24, 2021
@svcbot-qecnsdp
Copy link

Build ID: 24
WCP build status: FAILURE 
Stage before exit: null 

@svcbot-qecnsdp
Copy link

Build ID: 25
WCP build status: FAILURE 
Stage before exit: null 

divyenpatel added a commit that referenced this pull request Jul 13, 2021
…k-2.2

Cherry-pick PR (#1013) to rel 2.2 - Add check for CSI volume spec in PVUpdated method
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-2.2.2-candidate release-2.2.2-cherry-picked release-2.3.0-candidate Indicates PR needs to be cherry-picked for 2.3.0 release release-2.3.0-cherry-picked Indicates PR is cherry-picked for 2.3.0 release size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants