[bug]: ReconcileAsync race condition #977
Description
Describe the bug
Due to the fact that the TimedEntityQueue is processed separately from the ResourceWatcher, this can cause a race condition in ReconcileAsync where the method is called for the same entity at the same time.
To reproduce
I have a controller that monitors a CRD and provisions several resources. I have another controller that monitors the provisioned resources and requeues the related CRD resource so that it can respond to a change in status on the provisioned resources. This can cause a scenario where an event on the CRD could be in the middle of processing while a requeued event could begin processing on the same entity. This seems to violate the documented behavior.
Expected behavior
I would expect the ResourceWatcher and the TimedEntityQueue to be publishing to the same queue for processing so that it can adhere to the documented behavior of If a reconciliation is currently running for a resource, new reconciliation requests for the same resource will be queued.
Screenshots
No response
Additional Context
No response