Local PV Stress test: don't fail on deleting missing PV

[tsmetana]

What type of PR is this?

/kind bug
/kind flake

What this PR does / why we need it:

The [sig-storage] PersistentVolumes-local Stress with local volumes [Serial] should be able to process many pods and reuse local volumes can sometimes fail with the following error:

  Aug  3 01:59:38.544: INFO: 63/64 pods finished
  Aug  3 01:59:41.141: INFO: Deleting pod pod-68b21e1a-75c8-492a-9a2d-dd9b9b20373a
  Aug  3 01:59:41.248: INFO: Deleting PersistentVolumeClaim "pvc-wjzgs"
  Aug  3 01:59:41.344: INFO: Deleting PersistentVolumeClaim "pvc-7229l"
  Aug  3 01:59:41.444: INFO: Deleting PersistentVolumeClaim "pvc-98gqk"
  Aug  3 01:59:41.543: INFO: 64/64 pods finished
  Aug  3 01:59:41.544: INFO: pvc is nil
  Aug  3 01:59:41.544: INFO: Deleting PersistentVolume "local-pvcv26j"
  Aug  3 01:59:41.643: INFO: pvc is nil
  Aug  3 01:59:41.643: INFO: Deleting PersistentVolume "local-pvkrxnx"
  Aug  3 01:59:41.743: INFO: pvc is nil
  Aug  3 01:59:41.743: INFO: Deleting PersistentVolume "local-pvg96ss"
  Aug  3 01:59:41.844: INFO: pvc is nil
  Aug  3 01:59:41.844: INFO: Deleting PersistentVolume "local-pvmt9gg"
  Aug  3 01:59:41.891: INFO: Unexpected error: 
      <*errors.StatusError | 0xc0026648c0>: 
      persistentvolumes "local-pvg96ss" not found
      {
          ErrStatus: 
              code: 404
              details:
                kind: persistentvolumes
                name: local-pvg96ss
              message: persistentvolumes "local-pvg96ss" not found
              metadata: {}
              reason: NotFound
              status: Failure,
      }

This causes the test to fail, however the 404 error seems to be bogus. The test periodically creates and deletes local volumes reusing their name. At the end of the test all the volumes are being cleaned up but the test loop itself is still running and it may happen that the deletion of the volume from the test itself tries to delete the already cleaned up volume.

There is a check in the test that tries to skip the missing volume but it happens too early, so the test loop races with the cleanup. The volume might be deleted after the test and then fail when trying to delete the volume for the second time. (The pvc is nil log proves the cleanup code is running already.)

Seems like the simplest fix is to simply ignore the missing volume in the test (it was meant to be skipped anyway).

Special notes for your reviewer:

This looks to be difficult to reproduce, since the failure happens only occassionally (up to 10 % of the test runs on a stressed system).

Does this PR introduce a user-facing change?

No, it's a test fix.

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

N/A

[k8s-ci-robot]

Please note that we're already in Test Freeze for the release-1.28 branch. This means every merged PR will be automatically fast-forwarded via the periodic ci-fast-forward job to the release branch of the upcoming v1.28.0 release.

Fast forwards are scheduled to happen every 6 hours, whereas the most recent run was: Thu Aug 3 10:25:55 UTC 2023.

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[xing-yang]

/assign @jsafrane

[jsafrane]

/lgtm
/approve

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: e5680ed94ede7ee8a02221521305468702b68930

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jsafrane, tsmetana

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• test/e2e/storage/OWNERS [jsafrane]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jsafrane]

/retest
unrelated scheduler flake