prevent calling register volume in parallel for same vmdk

[divyenpatel]

What this PR does / why we need it:

We have a bug in the FCD which might generate different FCD IDs for the same vmdk. If this happens, migrated volume is not guaranteed to be used with the cached Volume ID in the VolumeMigrationService.

This PR prevents calling Register Volume in parallel for the same vmdk.

This PR also prevents calling RegisterVolume from the CSI Controller Pod. Only the syncer container will be able to perform volume registration after this PR is merged. If for any reason, volume registration fails, the syncer container can attempt to re-register the volume during a full sync cycle. If vSphere CSI controller gets any call to operate on the migrated volume before it is registered, it will return an error, and CSI driver will retry the operation in the next attempt.

Testing done:
Verified enabling Migration when syncer is down, so migrated PV did not register before creating the Pod.
During Pod Creation, Controller Publish failed with an expected error.

Events:
  Type     Reason                  Age                From                     Message
  ----     ------                  ----               ----                     -------
  Normal   Scheduled               60s                default-scheduler        Successfully assigned default/pvpod to k8s-node-470-1636402267
  Warning  FailedAttachVolume      19s (x6 over 37s)  attachdetach-controller  AttachVolume.Attach failed for volume "pvc-ab2abaf9-6704-436e-b958-05411e9a86af" : rpc error: code = Internal desc = failed to get VolumeID from volumeMigrationService for volumePath: "[vsanDatastore] 83b38961-4a3f-c694-0552-0200abf20dd7/kubernetes-dynamic-pvc-ab2abaf9-6704-436e-b958-05411e9a86af.vmdk"
  Normal   SuccessfulAttachVolume  2s                 attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-ab2abaf9-6704-436e-b958-05411e9a86af"

and later when the syncer container has registered vmdk as FCD, attach operation succeeded.

cc: @bandyopadhyays-vmware @chethanv28

Release note:

prevent calling register volume in parallel for same vmdk

[divyenpatel]

@chethanv28 I have updated "testing done" section on the PR. Can you review this PR?

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chethanv28, divyenpatel

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [chethanv28,divyenpatel]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[divyenpatel]

@chethanv28 can we merge this PR?

[svcbot-qecnsdp]

Started vanilla Block pipeline... Build Number: 104

[svcbot-qecnsdp]

Block vanilla build status: FAILURE 
Stage before exit: e2e-tests 
Jenkins E2E Test Results: 

Ran 13 of 251 Specs in 6378.089 seconds
FAIL! -- 0 Passed | 13 Failed | 0 Pending | 238 Skipped
--- FAIL: TestE2E (6378.20s)
FAIL

Ginkgo ran 1 suite in 1h53m15.08720705s
Test Suite Failed

[divyenpatel]

Block vanilla build status: FAILURE 
Stage before exit: e2e-tests 
Jenkins E2E Test Results: 

Ran 13 of 251 Specs in 6378.089 seconds
FAIL! -- 0 Passed | 13 Failed | 0 Pending | 238 Skipped
--- FAIL: TestE2E (6378.20s)
FAIL

Ginkgo ran 1 suite in 1h53m15.08720705s
Test Suite Failed

These failures are not related to the change. On the setup csi-migration was disabled.
All failures are related to parsing storage class parameters failed with error: invalid param: “csimigration” and value: “true”.

@chethanv28 we can merge this PR safely.

[chethanv28]

/lgtm