Multitenancy: Allow to watch ExternalSecrets using annotation

[aabouzaid]

Hello,

This PR is 2 of 2 PRs to address support for multitenancy. The first PR is #548

This PR adds a new option that allows to scope KES access by ExternalSecrets using annotation.
i.e. each ExternalSecret can tell which KES is responsible to manage that ExternalSecret.

This feature doesn't fix any security concerns like #548, but in combination with #548 it allows to have namespace scoped KES and global KES at the same time. So if there are 2 KES, there is no need to set namespaces for both of them as described in the other PR.

The feature implementation is done, and also the unit test. I just need to update the docs.

[aabouzaid]

This PR is ready for review :-)

[aabouzaid]

I've updated the docs with a recommendation to disable the CRD manager to avoid having multiple deployments fighting over the CRD.

[aabouzaid]

@Flydiverny, is there any blocker for this PR? If not, I would be happy to fix the conflicts to make sure it's ready to be merged.

[Flydiverny]

@moolen Do you know if we had any discussions for the CRD to have something like an ingressClassName field on secrets if running multiple operators? Or if that makes sense 🤔

What I'm thinking maybe it would be better to have an optional field on the CRD resource instead of using an annotation and if possible align it with the crd-spec?

@aabouzaid I think there shouldn't be any blockers but the question above I think is something we could discuss first :)

[moolen]

Yes, we have a mechanic for that with the SecretStore.spec.controller field. It is similar to the well-known ingress.spec.ingressClassName concept. It defines which controller reconciles this particular resource. A little bit of the concept is outlined here: https://github.com/external-secrets/crd-spec/blob/main/Spec.md

We factor out a SecretStore (how to fetch the data) from the ExternalSecret (what data to fetch). For the KES implementation i think we can go with a ExternalSecret.spec.controller field.

[Flydiverny]

Ah! I was sure I had seen it in there, but didn't locate it in the ExternalSecret CRD! Makes sense its in the SecretStore.

ExternalSecret.spec.controller does indeed sound like the best alternative then.

@aabouzaid WDYT? Up for updating it to using controller field instead of an annotation?

[aabouzaid]

@Flydiverny sounds good, I will work on it asap 👍

[Flydiverny]

Closing as #701 was merged