prometheus-kube-stack

[Daxcor69]

Version of Helm and Kubernetes:

Helm Version:

$ helm version
version.BuildInfo{Version:"v3.5.0", GitCommit:"32c22239423b3b4ba6706d450bd044baffdcf9e6", GitTreeState:"clean", GoVersion:"go1.15.6"}

Kubernetes Version:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.1", GitCommit:"c4d752765b3bbac2237bf87cf0b1c2e307844666", GitTreeState:"clean", BuildDate:"2020-12-18T12:09:25Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:41:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

Which chart: helm-charts/charts/kube-prometheus-stack/

Which version of the chart: kube-prometheus-stack-13.0.0

What happened: error on install

What you expected to happen: no errors

How to reproduce it (as minimally and precisely as possible): helm install prometheus-operator prometheus-community/kube-prometheus-stack --namespace prometheus-operator --dry-run

Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: [ValidationError(Alertmanager.spec): unknown field "alertmanagerConfigNamespaceSelector" in com.coreos.monitoring.v1.Alertmanager.spec, ValidationError(Alertmanager.spec): unknown field "alertmanagerConfigSelector" in com.coreos.monitoring.v1.Alertmanager.spec]

This is on a managed kubernetes provided by digitalocean.

[yashrsharma44]

While performing the command - helm install prometheus prometheus-community/kube-prometheus-stack, I am getting the same error.

$ helm install prometheus prometheus-community/kube-prometheus-stack
Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: [ValidationError(Alertmanager.spec): unknown field "alertmanagerConfigNamespaceSelector" in com.coreos.monitoring.v1.Alertmanager.spec, ValidationError(Alertmanager.spec): unknown field "alertmanagerConfigSelector" in com.coreos.monitoring.v1.Alertmanager.spec]

[kbreit]

I'm seeing this same error.

Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: [ValidationError(Alertmanager.spec): unknown field "alertmanagerConfigNamespaceSelector" in com.coreos.monitoring.v1.Alertmanager.spec, ValidationError(Alertmanager.spec): unknown field "alertmanagerConfigSelector" in com.coreos.monitoring.v1.Alertmanager.spec]

[PSjoe]

I ran into a similar issue when upgrading a cluster. At least for me, I had old CRDs. Helm doesn't uninstall nor update CRDs. If you had a deployment previously and tore it down, it's possible you've got some outdated ones sitting around.

For me, I had to download the current CRDs and manually install them with kubectl which seemed to fix things.

[kbreit]

@PSjoe Do you know which CRDs you had to remove?

[kbreit]

This worked for me. kubectl get crd | grep coreos will show all the CRDs related to Prometheus. Output the YAML files (kubectl get crd something.coreos -o yaml > file.yml and delete the resource using that file.

[PSjoe]

I didn't remove any CRDs, as I had an existing installation. It seems that if you delete a CRD and there are bits that depend on it, those bits get deleted. (I could be wrong. I wasn't specifically testing for that. I just sort of noticed.) So I just installed them, which seems to have been enough to update them in place.

[Daxcor69]

Hello, listening to the comments above, I went in and removed every bit of the previous installation. When I try again, I get this error.

helm install prometheus-operator prometheus-community/kube-prometheus-stack --dry-run

Error: unable to build kubernetes objects from release manifest: [unable to recognize "": no matches for kind "Alertmanager" in
 version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "Prometheus" in version 
"monitoring.coreos.com/v1", unable to recognize "": no matches for kind "PrometheusRule" in version 
"monitoring.coreos.com/v1", unable to recognize "": no matches for kind "ServiceMonitor" in version 
"monitoring.coreos.com/v1"]

The error has changed, but it still references the alert manager in someway. Thoughts?

[peterton]

The README.md says to upgrade 2 CRDs: monitoring.coreos.com_alertmanagerconfigs.yaml and monitoring.coreos.com_prometheuses.yaml (https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/README.md)

I had to also upgrade monitoring.coreos.com_alertmanagers.yaml to make it work.

[rodricarvalho]

Hi,

I've run into that problem as well. However, if you update/upgrade version by version you can manage that issue smoothly.
E.g:
1st:
helm upgrade --install prometheus -n monitoring prometheus-community/kube-prometheus-stack --atomic --debug -f prometheus-operator/values.yaml --version 9.4.10 --reset-values
2nd:
helm upgrade --install prometheus -n monitoring prometheus-community/kube-prometheus-stack --atomic --debug -f prometheus-operator/values.yaml --version 10.0.0 --reset-values

3rd:
If you starting to see Bad gateway request at Grafana UI, don’t worry. Just delete the new prometheus-grafana-xxx pod to get a new grafana pointing to the correct prometheus datasource .

4th:
Update your CRD(s) to keep moving forward with the upgrade process

kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.42.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.42.0/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.42.0/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.42.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.42.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.42.0/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.42.0/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml

PS: I am re-using the existed values file used to deploy the previous version of Prometheus is not mandatory.

I hope this helps you guys.

[viatcheslavmogilevsky]

the following commands helped me to upgrade release (in addition to https://github.com/prometheus-community/helm-charts/blame/61e7d540b686fa0df428933a42136f7084223ee2/charts/kube-prometheus-stack/README.md#L91-L92 ):

kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.45.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml && \
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.45.0/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml && \
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.45.0/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml && \
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.45.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml && \
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.45.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml && \
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.45.0/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml && \
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.45.0/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale]

This issue is being automatically closed due to inactivity.

[LinTechSo]

$ helm install loki ./
error:
Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"

Hi all. i hit same issue. any update?
using loki-stack (loki version 2.4.1)
on kubernetes version 1.18 and also 1.23

[martinbe1io]

same here