Add custom annotation to all managed resources

[michael-ylb]

Please, answer some short questions which should help us to understand your problem / question better?

• Which image of the operator are you using? e.g. registry.opensource.zalan.do/acid/postgres-operator:v1.10.1

v1.10.1

• Where do you run it - cloud or metal? Kubernetes or OpenShift? [AWS K8s | GCP ... | Bare Metal K8s]

Bare metal, Kubernetes

• Are you running Postgres Operator in production? [yes | no]

yes

• Type of issue? [Bug report, question, feature request, etc.]

feature request

Request
There should be a configurable option in order to provide a custom annotation, that is being attached to all managed resources of the operator (e.g. statefulset, services, PDB).
Currently, this is only possible for either pods, the operator itself or inheriting from the cluster resource (which is not allowed to be annotated in this case).

Explanation
We are running Kubernetes and use ArgoCD to manage our resources.
ArgoCD tracks its resources using the "app.kubernetes.io/instance" label. This means, every resource that has this label is being recognized by ArgoCD as part of the application with the given value.
Resources, that are NOT managed by ArgoCD, but still have this label, are getting pruned on sync. This is very bad.

Now there are two options:

1. Don't inherit the "app.kubernetes.io/instance" label. This means, operator resources are not shown in ArgoCD. This is not too big of a problem, but also, this label has to be carefully removed from any (automatically created) selectors targetting the database pods (network policies, servicemonitors, ...). This costs us much time and trouble when working with the operator in our environment.

2. Inherit the "app.kubernetes.io/instance" label. This would be the correct way. But then, ArgoCD rejects all the new resources and wants to prune them away (because of the bug, where the ownerReferences are missing). As a workaround, we could prevent that by adding some specific annotations to the resources - but this is not possible.
   Adding custom pod annotations still gets the statefulset pruned.
   Inheriting cluster annotation would be wrong (since the cluster resource itself IS managed by ArgoCD) and also violates poka yoke (missing/removing this annotation gets the whole cluster annihilated in seconds; not a lot of fun).

[FxKu]

For labels this can be done with cluster_labels. Sounds like you need a similar option but for annotations. Maybe check how cluster_labels is implemented to come up with a PR for annotations.

It's interesting that we already have so many different options for annotations but users keep finding new combinations they need 😃