Race Condition in Parallel Reconciliation Prevents Finalizer Addition to Referenced Objects #371
Description
What happened?
In the provider-kubernetes, a race condition during parallel resource reconciliation prevents finalizers from being reliably added to referenced objects. This can lead to orphaned resources upon deletion and an inconsistent state.
The expected behavior is that all necessary finalizers are added to all referenced objects, even if initial attempts conflict.
The actual behavior is that the reconciliation logic has a flaw that causes it to fail silently. The flow is as follows:
- The provider starts reconciling multiple resources that reference the same object at the same time.
- During reconciliation, a resource first adds a finalizer to itself. This step succeeds.
- Immediately after, it attempts to add its finalizer to the shared referenced objects. Due to the parallel reconciliations, this frequently causes a conflict, and the reconciliation request is correctly re-queued.
- On the next reconciliation attempt, the controller sees that the resource's own finalizer has already been set and returns early.
- This early return means the controller never retries adding the finalizer to the referenced objects that it failed to update in the first attempt.
As a result, only the resources that won the initial race add their finalizers to the referenced objects. The others will never have their finalizers added, breaking the garbage collection mechanism.
How can we reproduce it?
- Define multiple Object resources in provider-kubernetes where several objects contains references (
dependsOn) to a single object. - Arrange for these Object resources to be created and reconciled.
- Inspect the referenced object.
- Observe the metadata.finalizers array on the object. You will likely see that finalizers from some of the Object resources are missing, indicating that they lost the race and will never be added due to the bug.
What environment did it happen in?
Crossplane version: 1.19.1
provider-kubernetes version: v0.18.0
Kubernetes version: Client Version: v1.33.1, Kustomize Version: v5.6.0 Server Version: v1.33.1
Kubernetes distribution: Kind