Remove prometheus CRDs

[jkroepke]

What this PR does / why we need it:

Remove Prometheus CRDs from loki chart.

Why?

CRDs are not fully managed by helm. If loki chart is installed before kube-prometheus-stack helm chart, ServiceMonitor and PrometheusRule CRD are not in sync/still outdated.

In umbrella chart scenarios, where kube-prometheus-stack and loki are bundled to one helm charts, the versions of the installed CRDs maybe unexpectable. (No one knows, if the CRD are picked from loki or kube-prometheus-stack first)

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Checklist

• Reviewed the CONTRIBUTING.md guide
• Documentation added
• Tests updated
• CHANGELOG.md updated
• Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md

[jkroepke]

I remove my fork by accident, it replaces #7480

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[jeschkies]

Thanks for the input. I'll just reject it so we can discuss our options. Ideally we have an out of the box solution. The CRDs are required by the Agent Operator. Would it help if you could disable the CRDs?

[jkroepke]

Would it help if you could disable the CRDs?

There is no away to disable single CRDs. I could run helm install --skip-crds, but that would break the Installation of Grafana Agent Operator, since the CRDs manadory.

On top, there are issues with having the loki chart as part of an umbrella chart. For example together with the kube-prometheus-stack.

---

At the end, I do not get the point here. What the point to create service Monitors without having kube-prometheus-stack installed? If kube-prometheus-stack is present, the CRDs also exists.

[trevorwhitney]

I for one am ok with this change. Prometheus operator is a prerequisite since it is currently the only option for PrometheusRule resources. It should install the ServiceMonitor CRD. Furthermore the Grafana Agent will also install the ServiceMonitor CRD. I am opinionated about having self monitoring be enabled by default, but this PR achieves that with the API capabilities check, so I'm good with it. I'm of course open to discussion though.

[jeschkies]

Sorry. I misinterpreted this change. I thought you've removed the ServiceMonitor CRD kind: ServiceMonitor when you are removing kind: CustomResourceDefiniton.

[trevorwhitney]

@jeschkies ahh, yes, definitely still need our ServiceMonitors :)

[jkroepke]

No, I only want to remove the CRDs, since they should be owned by the kube-prometheus-stack helm chart.

The kind: ServiceMonitor manifeste are totally fine here.

---

The grafana-agent-operator helm chart has the same problem. The helm chart also contains CRDs from kube-prometheus-stack. Only the kube-prometheus-stack should hold and deploy the CRDs, because the deployed controller (prometheus-operator) will watch such CRs...

[trevorwhitney]

@jkroepke I think the Grafana Agent chart is a slightly different problem, because the Grafana Agent can act on service monitor resources independent of a prometheus operator. So, in a deployment where you only are using Grafana Agent, that chart needs to include those CRDs

[jkroepke]

@jkroepke I think the Grafana Agent chart is a slightly different problem, because the Grafana Agent can act on service monitor resources independent of a prometheus operator. So, in a deployment where you only are using Grafana Agent, that chart needs to include those CRDs

In such scenarios, think about an additional sub chart and put the Prometheus Operator CRD in. This way, you can toggle the additional CRDs using the subchart.condition toggle.

Unlike Helm, ArgoCD and other GitOps tools manage the CRDs manifest. If kube-promethus-stack and grafana-operator is installed in the same cluster using ArgoCD, the CRDs always ouf of sync on one application side.

With the additional subchart and condition, the ServiceMonitor CRD can be toggled.