K8ssandra-operator memory leak after failed MedusaBackupJob

[adziura-tcloud]

What happened?
After upgrading K8ssandra-operator from 1.7.0 to 1.16.0 I noticed periodic restarts of k8ssandra-operator due to memory leak.
It happened twice right after failed MedusaBackupJob

spec:
  backupType: full
  cassandraDatacenter: gke-dc1
status:
  finished:
  - example-test-1-gke-dc1-r1-sts-2
  - example-test-1-gke-dc1-r1-sts-3
  - example-test-1-gke-dc1-r1-sts-1
  inProgress:
  - example-test-1-gke-dc1-r1-sts-0
  startTime: "2024-05-12T00:30:15Z"

MedusaBackupJob failed because of POD OOM restart (we are using pretty small instances on test env)

Also MedusaBackupSchedule is not working after failed backup - not creating new MedusaBackupJobs

Did you expect to see something different?
Failed backup should not affect operator and next backups.

How to reproduce it (as minimally and precisely as possible):
I think killing one POD during the backup should work

• K8ssandra Operator version: 1.16.0

• Kubernetes version information: 1.29

• Kubernetes cluster kind: GKE

• K8ssandra Operator Logs: I see a lot of MedusaBackupJob is still being processed INFO messages

2024-05-13T07:07:34.916Z	INFO	MedusaBackupJob is still being processed	{"controller": "medusabackupjob", "controllerGroup": "medusa.k8ssandra.io", "controllerKind": "MedusaBackupJob", "MedusaBackupJob": {"name":"daily-1715473800","namespace":"cassandra"}, "namespace": "cassandra", "name": "daily-1715473800", "reconcileID": "ace193e0-a3b6-4935-80f7-736b7fbae2a0", "medusabackupjob": "cassandra/daily-1715473800", "Backup": {"namespace": "cassandra", "name": "daily-1715473800"}}