Add ability to provide per-namespace classes

[mkmik]

The telegraf.influxdata.com/class annotation allows a user to choose which class out of the pre-configured classes telegraf-operator is configured with.

However, there are many reasonable uses of a k8s cluster that involve N teams sharing the same cluster, with one team handling the "infrastructure" part and N-1 teams handling each their own stuff in their own namespace(s). It's quite a pain when a new service needs a custom telegraf config and you need to update the central secret that contains all the classes definition for the telegraf operator.

Proposal

1. A CRD called TelegrafClass, which contains a telegraf config. It also contains references to Secrets which can be mapped to env vars, so that the body of the config doesn't have to be a secret.\
2. A CRD called ClusterTelegrafClass which would be the non-namespaced version of (1) (similar to Role vs ClusterRole)
3. A pod with a telegraf class name referenced in telegraf.influxdata.com/class will match a class defined in a TelegrafClass CR with the same name found in the same namespace or, if that's not found, it will match a ClusterTelegrafClass resource and if that's not found either, it will fallback to the current behaviour (the classes files mounted in the telegraf-operator container)

[athornton]

Agreed that something like this would be useful. We run the rubin-science-platform where, effectively, each service in the cluster runs in its own namespace, and when individual users spin up their JupyterLab-and-Dask environments, they get their own namespace, which is destroyed when the user stops running the lab.

I have observed that deleting namespaces hangs when the telegraf-operator is installed, and also that spawning users into individual namespaces (as JupyterHub/kubespawner does if one enables user namespace support) doesn't work: the request to add the pod is never seen by K8s (although the namespace is created--I suspect this may be a sequencing problem, in that when the create-pod request happens, the namespace into which that pod will be put isn't finished being created, and that may confuse the pod creation interception), so clearly telegraf-operator is not yet engineered for the use case where namespaces are plentiful and ephemeral.