Fix flake with e2e test that checks detach while mount in progress

[gnufied]

A volume can show up as in-use even before it gets attached
to the node.

Fixes #71426

/sig storage
/kind bug

cc @nikopen

NONE

[gnufied]

while root cause is accurate, I need to tweak some other timings here to ensure this does not flake.

[gnufied]

/hold

[nikopen]

/priority critical-urgent
/milestone v1.13

[AishSundar]

/test pull-kubernetes-e2e-gce-device-plugin-gpu

[gnufied]

So I have found another problem with this e2e - sometimes a pod can be deleted by e2e even before the kubelet starts mounting it. When this happens, the test will fail because it does not enter code path we are trying to test. To circumvent it, I had to add another sleep after pod creation. This fixes it but still isn't ideal solution.

/hold cancel

[saad-ali]

@gnufied let me know when this is ready to review. If it is not possible to fix soon, we can consider moving it to flaky.

[gnufied]

@saad-ali it is ready for review, but there is a timing issue that is kinda hard to fix because we don't know state of kubelet from outside. if a pod gets deleted before volume-manager starts mounting it, we run into case where the e2e will not reproduce the bug. I have added a sleep to prevent that and it should lessen that - but it is not a strong guarantee.

[saad-ali]

Ack. Ok let's get this merged. If it is still flaky, we can just mark the test flaky.

/approve
/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gnufied, saad-ali

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• test/e2e/storage/OWNERS [gnufied,saad-ali]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment