fix: Make sure pvcs have correct value of the label instanceRole and 'Role'

[YanniHu1996]

closes: #3810

[github-actions]

❗ By default, the pull request is configured to backport to all release branches.

• To stop backporting this pr, remove the label: backport-requested ◀️ or add the label 'do not backport'
• To stop backporting this pr to a certain release branch, remove the specific branch label: release-x.y

[YanniHu1996]

the e2e from fork
https://github.com/EnterpriseDB/cloudnative-pg/actions/runs/8059176715

[armru]

/test limit=local

[github-actions]

@armru, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/8140875763

[mnencia]

/test

[github-actions]

@mnencia, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/8185064309

[mnencia]

/ok-to-merge

[mnencia]

This patch solves the PVC label issue. However, there is still a way the cluster could fail when you delete all the instances in sequence. If the timing is just right, you can end up in a situation where only one not-ready pod remains (a former primary that needs to run pg_rewind).

The cluster remains stuck in a phase like Instance Status Extraction Error: HTTP communication issue, and the operator logs the following message over and over:

{"level":"info","ts":"2024-03-08T18:00:54Z","msg":"Cannot update target primary: operation cannot be fulfilled. An immediate retry will be scheduled","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-example","namespace":"default"},"namespace":"default","name":"cluster-example","reconcileID":"4e0c2cca-d287-45ff-87c5-edd28b494b53","uuid":"cde5207a-dd75-11ee-9130-62304a1b1e3f","error":"unable to evaluate failover logic, unable to fetch the instances status"}

However, the situation is easy to recover: if you delete the not-ready pod, the operator recreates the primary first, and the cluster becomes healthy again.

Given that this patch improves the system's resiliency, I will merge it and open a new issue to address this corner case.