PoC recovering reconcile panics as errors

[davidvossel]

We've encountered two recent virt-controller crashes (#5849 and #5694) that were caused by resources entering into unexpected states. The result is that a crash in one of our reconcile loops crashes all of virt-controller, which can cause the entire virt-controller process to enter into a crash loop.

Using the built in golang recover() function, we can catch and handle panics that occur during our reconcile loops and simply treat them as errors which then get rate limited.

This PR exists to illustrate how this can be done for the sake of discussion.

NONE

[kubevirt-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please ask for approval from davidvossel after the PR has been reviewed.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[davidvossel]

I'm not convinced treating reconcile panics as errors is a good idea. For now i think we should still crash, however we may choose to introduce a small delay in the crash handler to allow other controllers to perform some work before exitting. This would potentially mitigate a situation where a crash occurs which would get resolved if only another controller had a chance to perform some work