br: fix redundant pvc and pv in metadata

[WangLe1321]

What problem does this PR solve?

Fix the bug that if there are redundant pvc or pv of backup tidb cluster, the backup can succeed but the restore can fail.

The redundant pvc or pv means:

• redundant pvc: there are labels in the pvc indicating that the pvc belongs to the backup tidb cluster but there is no tikv pod using it
• redundant pv: there are labels in the pv indicating that the pv belongs to the backup tidb cluster but there is no tikv pvc using it.

Closes #5459

What is changed and how does it work?

At first we retrieve all the pvcs and pvs that may belong to the backup tidb cluster. Then we filter pvc by the tikv pod and filter pv by the pvc.

Code changes

• Has Go code change
• Has CI related scripts change

Tests

• Unit test
• E2E test
• Manual test
• No code

Side effects

• Breaking backward compatibility
• Other side effects:

Related changes

• Need to cherry-pick to the release branch
• Need to update the documentation

Release Notes

Please refer to Release Notes Language Style Guide before writing the release note.

[codecov-commenter]

Codecov Report

❗ No coverage uploaded for pull request base (master@d6e665e). Click here to learn what that means.
The diff coverage is 63.88%.

Additional details and impacted files

@@            Coverage Diff            @@
##             master    #5461   +/-   ##
=========================================
  Coverage          ?   61.57%           
=========================================
  Files             ?      230           
  Lines             ?    29176           
  Branches          ?        0           
=========================================
  Hits              ?    17965           
  Misses            ?     9436           
  Partials          ?     1775           

Flag	Coverage Δ
unittest	61.57% <63.88%> (?)

[BornChanger]

/retest

[BornChanger]

/run-pull-e2e-kind-br

[BornChanger]

/run-pull-e2e-kind-br

[BornChanger]

/run-pull-e2e-kind-br

[WangLe1321]

/run-pull-e2e-kind-br

[WangLe1321]

/run-pull-e2e-kind-br

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: BornChanger

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [BornChanger]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2023-12-13 10:47:32.286898122 +0000 UTC m=+439543.324125049: ☑️ agreed by BornChanger.
• 2023-12-14 08:41:08.999004326 +0000 UTC m=+518360.036231253: ✖️🔁 reset by ti-chi-bot[bot].
• 2023-12-18 09:25:58.379246748 +0000 UTC m=+866649.416473670: ☑️ agreed by BornChanger.
• 2023-12-18 11:00:38.138970445 +0000 UTC m=+872329.176197372: ✖️🔁 reset by BornChanger.
• 2023-12-18 11:26:27.265847106 +0000 UTC m=+873878.303074032: ☑️ agreed by BornChanger.
• 2023-12-18 14:19:53.57556508 +0000 UTC m=+884284.612792022: ✖️🔁 reset by ti-chi-bot[bot].
• 2023-12-19 04:15:35.504918492 +0000 UTC m=+934426.542145435: ☑️ agreed by BornChanger.
• 2023-12-19 06:01:13.271291739 +0000 UTC m=+940764.308518666: ✖️🔁 reset by ti-chi-bot[bot].
• 2023-12-19 06:05:49.59544129 +0000 UTC m=+941040.632668217: ☑️ agreed by BornChanger.
• 2023-12-19 06:14:58.446639329 +0000 UTC m=+941589.483866277: ✖️🔁 reset by ti-chi-bot[bot].
• 2023-12-19 08:42:59.577994299 +0000 UTC m=+950470.615221226: ☑️ agreed by BornChanger.

[csuzhangxc]

/cherry-pick release-1.5

[ti-chi-bot]

@csuzhangxc: new pull request created to branch release-1.5: #5475.

In response to this:

/cherry-pick release-1.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.