Finalizer Removal Using Retried Update with Optimistic Locking

[csviri]

Currently in v 3.0.1 the finalizer on cleanup is removed by an update using optimistic locking (OL) . This is causing issues in case of patch, see: #1245 (comment)

But also in cases it can be a problem if there are multiple finalizers and a additional is removed while the cleanup is ongoing.

An alternative solution is to use (JSON) PATCH without optimistick locking so to remove just a finalizer in one place:

{"op": "remove", "path": "/metadata/finalizers/0"}

The problem with this if there are multiple finalizers, and there is a parallel update happening, it can either fail or lead to inconsistent state (thus removes wrong finalizer).

1. if there are two finalizers, controller needs to remove the second one (targetfinalizer), but while cleanup running the first removed from this list the request will fail with 422:

metadata:
  finalizers: 
    - otherfinalizer
    - targetfinalizer

The patch would fail: {"op": "remove", "path": "/metadata/finalizers/1"} since there is not element with index 1.

2. If there are 3 finalizers, controller needs to remove the middle, but the first one is removed by its own controller, then the patch will remove the wrong finalizer (third one from original state), so :

metadata:
  finalizers: 
    - otherfinalizer
    - targetfinalizer
    - thirdfinalizer

in case otherfinalizer removed the patch:
{"op": "remove", "path": "/metadata/finalizers/1"}
would remove thirdfinalizer not the desired targetfinalizer.

Solution

Update (or patch) with optimistick locking should be used, and if the request fails on a conflict the resource should be read again in place and finalizer removed again and update with OL tried again. (with possible multiple retries).
That will make sure that the cleanup is not called multiple times, and there is no wrong finalizer removed.