Failing to list Medusa backups if there are too many of them

[burmanm]

What happened?

When there are too many backups, the gRPC connection fails with the error:

│ 2023-10-23T20:41:06.011Z  ERROR  failed to list backups  {"controller": "medusatask", "controllerGroup": "medusa.k8ssandra.io", "controllerKind": "MedusaTask", "MedusaTask": {"name":"medusa-backup-sync-cluster2-main-1698093327","namespace":"k8s
│ sandra-failover"}, "namespace": "k8ssandra-failover", "name": "medusa-backup-sync-cluster2-main-1698093327", "reconcileID": "ce6efd88-b6c8-406b-bcb1-b7f7b1cd4a2e", "MedusaTask": "k8ssandra-failover/medusa-backup-sync-cluster2-main-1698093327", "cassd
│ c": "main", "CassandraPod": "cluster2-main-az-1c-sts-1", "error": "failed to get backups: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (10903823 vs. 4194304)"}

This is because our gRPC setting allows the reply to be a maximum of 4MB, which isn't enough.

Did you expect to see something different?

We should probably split the request to multiple parts, or something else. Otherwise we'll be chasing some arbitary number and keep increasing the size forever (and it will start causing network transfer errors). We might also consider compression on this connection.

How to reproduce it (as minimally and precisely as possible):

Environment

• K8ssandra Operator version:

  Insert image tag or Git SHA here

• Kubernetes version information:

  kubectl version

• Kubernetes cluster kind:

  insert how you created your cluster: kops, bootkube, etc.

• Manifests:

insert manifests relevant to the issue

• K8ssandra Operator Logs:

insert K8ssandra Operator logs relevant to the issue here

Anything else we need to know?:

[KEDLogic]

This issue is actually for our cluster. I had mistakenly created a ticket on medusa-operator. It has some additional info that may be relevent: k8ssandra/medusa-operator#71

Thank you burmanm for posting this :)