Database Deletion Fails When Secret Deletion Fails

[eberlep]

Probably also cause initially by #205 and the recreation of the same databases with different IDs.

When trying to delete the recreated databases, deletion fails and the state Terminating, even though the corresponding postgresql ressource was already deleted.

Btw: This is the perfect opportunity to test if those databases (which still exist but do have a deletion date) are actually being excluded from accounting.

postgreslet-f5ccbb4c9-jsw7j postgreslet 2021-06-10T16:02:05.081Z        INFO    controllers.Postgres    reconciling     {"postgres": "metal-extension-postgres/8c4974ad-876e-4348-93ce-03800069d4ea"}
postgreslet-f5ccbb4c9-jsw7j postgreslet 2021-06-10T16:02:05.081Z        INFO    controllers.Postgres    postgres fetched        {"postgres": "metal-extension-postgres/8c4974ad-876e-4348-93ce-03800069d4ea", "postgres": {"apiVersion": "database.fits.cloud/v1", "kind": "Postgres", "namespace": "metal-extension-postgres", "name": "8c4974ad-876e-4348-93ce-03800069d4ea"}}
postgreslet-f5ccbb4c9-jsw7j postgreslet 2021-06-10T16:02:05.081Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Postgres","namespace":"metal-extension-postgres","name":"064cf825-c4ce-45e2-a112-c84672c20d34","uid":"94707cba-bf6a-4a90-a2a3-58edef620bfd","apiVersion":"database.fits.cloud/v1","resourceVersion":"670728530"}, "reason": "Error", "message": "failed to delete user passwords secret: %!w(*errors.StatusError=&{{{ } {   <nil>} Failure secrets \"064cf825-c4ce-45e2-a112-c84672c20d34-passwords\" not found NotFound 0xc0003645a0 404}})"}
postgreslet-f5ccbb4c9-jsw7j postgreslet 2021-06-10T16:02:05.096Z        INFO    controllers.Postgres    instance being deleted  {"postgres": "metal-extension-postgres/8c4974ad-876e-4348-93ce-03800069d4ea"}
postgreslet-f5ccbb4c9-jsw7j postgreslet 2021-06-10T16:02:05.100Z        INFO    controllers.Postgres    corresponding CRD ClusterwideNetworkPolicy deleted      {"postgres": "metal-extension-postgres/8c4974ad-876e-4348-93ce-03800069d4ea"}
postgreslet-f5ccbb4c9-jsw7j postgreslet 2021-06-10T16:02:05.103Z        INFO    controllers.Postgres    corresponding Service of type LoadBalancer deleted      {"postgres": "metal-extension-postgres/8c4974ad-876e-4348-93ce-03800069d4ea"}
postgreslet-f5ccbb4c9-jsw7j postgreslet 2021-06-10T16:02:05.103Z        INFO    controllers.Postgres    owned zalando postgresql deleted        {"postgres": "metal-extension-postgres/8c4974ad-876e-4348-93ce-03800069d4ea"}
postgreslet-f5ccbb4c9-jsw7j postgreslet 2021-06-10T16:02:05.109Z        INFO    OperatorManager operator deletable
postgreslet-f5ccbb4c9-jsw7j postgreslet 2021-06-10T16:02:05.122Z        ERROR   controller-runtime.manager.controller.postgres  Reconciler error        {"reconciler group": "database.fits.cloud", "reconciler kind": "Postgres", "name": "8c4974ad-876e-4348-93ce-03800069d4ea", "namespace": "metal-extension-postgres", "error": "failed to delete user passwords secret: secrets \"8c4974ad-876e-4348-93ce-03800069d4ea-passwords\" not found"}
postgreslet-f5ccbb4c9-jsw7j postgreslet sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
postgreslet-f5ccbb4c9-jsw7j postgreslet         /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253
postgreslet-f5ccbb4c9-jsw7j postgreslet sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
postgreslet-f5ccbb4c9-jsw7j postgreslet         /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216
postgreslet-f5ccbb4c9-jsw7j postgreslet k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
postgreslet-f5ccbb4c9-jsw7j postgreslet         /go/pkg/mod/k8s.io/apimachinery@v0.20.6/pkg/util/wait/wait.go:185
postgreslet-f5ccbb4c9-jsw7j postgreslet k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
postgreslet-f5ccbb4c9-jsw7j postgreslet         /go/pkg/mod/k8s.io/apimachinery@v0.20.6/pkg/util/wait/wait.go:155
postgreslet-f5ccbb4c9-jsw7j postgreslet k8s.io/apimachinery/pkg/util/wait.BackoffUntil
postgreslet-f5ccbb4c9-jsw7j postgreslet         /go/pkg/mod/k8s.io/apimachinery@v0.20.6/pkg/util/wait/wait.go:156
postgreslet-f5ccbb4c9-jsw7j postgreslet k8s.io/apimachinery/pkg/util/wait.JitterUntil
postgreslet-f5ccbb4c9-jsw7j postgreslet         /go/pkg/mod/k8s.io/apimachinery@v0.20.6/pkg/util/wait/wait.go:133
postgreslet-f5ccbb4c9-jsw7j postgreslet k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
postgreslet-f5ccbb4c9-jsw7j postgreslet         /go/pkg/mod/k8s.io/apimachinery@v0.20.6/pkg/util/wait/wait.go:185
postgreslet-f5ccbb4c9-jsw7j postgreslet k8s.io/apimachinery/pkg/util/wait.UntilWithContext
postgreslet-f5ccbb4c9-jsw7j postgreslet         /go/pkg/mod/k8s.io/apimachinery@v0.20.6/pkg/util/wait/wait.go:99
postgreslet-f5ccbb4c9-jsw7j postgreslet 2021-06-10T16:02:05.122Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Postgres","namespace":"metal-extension-postgres","name":"8c4974ad-876e-4348-93ce-03800069d4ea","uid":"8b107500-0c3c-420f-ab6c-298faef42020","apiVersion":"database.fits.cloud/v1","resourceVersion":"670725168"}, "reason": "Error", "message": "failed to delete user passwords secret: %!w(*errors.StatusError=&{{{ } {   <nil>} Failure secrets \"8c4974ad-876e-4348-93ce-03800069d4ea-passwords\" not found NotFound 0xc000bc8b40 404}})"}

Bonus question: Why was that secret not created? Probably also caused by #205.

[eberlep]

Update: Postgres databases in the state "Terminating" are not billed.

[eberlep]

Upon further investigation, the secret is being created properly when the custom ressource is being created. It is also being deleted properly once the custom resource is deleted. However, because of the very nature of the reconcile loop, the secret is being deleted every time the finalizer is still present and the reconcile loop is triggered.

As the secret deletions throws a 404/not found when trying to delete a secret we already deleted in a previous run, we never reach the code that removes the finalizer and completes the deletion.