Update VM controller error handling

[machadovilaca]

What this PR does / why we need it:

Sometimes some resources might not exist yet and we see a huge amount of logs stating that an error created the VM metrics.

This PR:

• adds 5 seconds backoff between failed reconcile loops of VM controller to reduce the number of logs
• removes 'error' wording from logs when the VM controller does not create the VM-related metrics, since it might not represent more serious problems, and might deceive users
• skips metric creation when VM-related resources are still in Provisioning status

Which issue(s) this PR fixes:

jira-ticket: https://issues.redhat.com/browse/CNV-35806

Special notes for your reviewer:

Release note:

Update VM controller error handling

[akrejcir]

If the PVC or PV does not exist, do we want to try checking again? Will these resources be created eventually?

[machadovilaca]

If the PVC or PV does not exist, do we want to try checking again? Will these resources be created eventually?

The PVC is associated to the VM, is just not created yet, the backoff ensure we will try the reconcile loop again in a few seconds

[akrejcir]

Instead of retrying later, can you check the .status.printableStatus field?
The status Provisioning is set when not all resources are created yet:
https://github.com/kubevirt/kubevirt/blob/9d9885c655d465657f515c8f73c23b00c9f16898/staging/src/kubevirt.io/api/core/v1/types.go#L1511-L1513

If a VM has this printableStatus, it can be skipped. Later, if the .status filed changes, the VM will trigger reconciliation again.
This way we will avoid reconciliation looping while waiting for the resources to be created.

[machadovilaca]

updated @akrejcir

[akrejcir]

Thanks. Can you add more text to the commit message?

[sonarcloud]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

[machadovilaca]

Thanks. Can you add more text to the commit message?

done!

[machadovilaca]

/retest

[akrejcir]

/approve

[kubevirt-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: akrejcir

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [akrejcir]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[avlitman]

/lgtm

[machadovilaca]

/cherry-pick release-v0.19

[kubevirt-bot]

@machadovilaca: new pull request created: #857

In response to this:

/cherry-pick release-v0.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.