Thanks for coming up with a possible solution here @CatherineF-dev @chrischdi
Some thoughts on this approach and the design:

I feel like this approach definitely improves the situation, coming with a drawback of added complexity to kube-state-metrics which might make it more difficult to maintain though.

I would prefer a solution that mimics a similar design as prometheus and prometheus-operator. This will reduce complexity and simplify things, as well as follow the unix philosphy "Make each program do one thing well".

kube-state-metrics-operator could support two CRDs:

• CustomResourceMonitor, which matches your design
• KubeStateMetrics, which sets up new instances of kube-state-metrics

A couple of use cases:

• Large fleet of nodes and you need to separate kube-state-metrics out into multiple deployments (e.g. one is a daemonset collecting pod metrics, while the other one is a deployment collecting api resource info, while the third one is dealing with custom-resources)
• Multi-tenancy and you want to have one kube-state-metrics deployment per namespace
• The operator further could support an AdmissionWebhook to make sure no invalid CustomResourceMonitor gets added and takes kube-state-metrics down

What do you think?

I think we can split it into kube-state-metrics and kube-state-metrics-operator in the future if it's needed. Because

1. It's a breaking change (users need to deploy kube-state-metrics-operator) and needs to be in a major release (3.x).
2. Prometheus-operator manages many components (prometheus, alert-manager and related monitoring components) and many CRDs. kube-state-metrics' complexity seems fine so far.
3. Scaling model still needs some improvements, which might affect the design of kube-state-metrics-operator.
4. A new operator also brings some maintenance work.

One possible plan is:

• v2.10: add CustomResourceMonitor feat: add CustomResourceMonitor CRD (WIP) #2049
• v3.x: discuss whether we need to introduce kube-state-metrics-operator

Given kube-state-metrics-operator is a breaking change, we can migrate CustomResourceMonitor CRD into kube-state-metrics-operator in v3.x

Could you elaborate why you think kube-state-metrics-operator is a breaking change to kube-state-metrics?

I would have assumed the operator is maintained in a separate repository, similar to https://github.com/prometheus-operator/prometheus-operator and https://github.com/prometheus/prometheus - completely decoupled.

This would allow to work independently on it, be on different release cycles.

Could you elaborate why you think kube-state-metrics-operator is a breaking change to kube-state-metrics?

I thought KubeStateMetrics, which sets up new instances of kube-state-metrics is related to auto-scaling, which will be moved into operator.

Could you elaborate why you think kube-state-metrics-operator is a breaking change to kube-state-metrics?

I thought KubeStateMetrics, which sets up new instances of kube-state-metrics is related to auto-scaling, which will be moved into operator.

Ah understood. Let me explain this better:
KubeStateMetrics is a CRD similar to https://github.com/prometheus-operator/prometheus-operator/blob/main/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml used by the operator to spin up new instances of kube-state-metrics.

Okay, I think both KubeStateMetrics and auto-sharding should be moved into kube-state-metrics-operator. So that kube-state-metrics-operator responsibility is managing pods.

CustomResourceMonitor is similar with other metrics, which maybe can be kept in kube-state-metrics. So that kube-state-metrics responsibility is collecting metrics.

Some perspectives I hope are helpful in deciding where to put this feature.

• Configuring components through a (potentially giant) yaml blob in a config file that is mounted via a config map is not really friendly UX. I don't get any help from my IDE on linting or auto-completion; it's just a big blob of text.
• CRDs are so much more friendly and the natural UX that Kubernetes engineers are familiar with.
• CRDs are easier to deal with when you are providing metrics configs for API types coming from multiple components / projects / repos. Each one gets a separate CR which you can decide to deploy or ignore as you like based on labels.
• CRD generation and validation is easier to automate.
  • FR: CustomResourceMonitor CRs could be generated from go type definitions using comment tags (this would be awesome)!

For these reasons, CRDs should be the first class API for configuring custom resource monitoring. I wouldn't want to have to deploy an entire separate operator pod (wasting resources) just to get a friendly UX.

/assign @CatherineF-dev
/triage accepted