Error: Internal error occurred: failed calling webhook

[ravizeewee]

Hello There,

I am tring to install prometheus but getting below issue:

Error: Internal error occurred: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://monitoring-prometheus-oper-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: service "monitoring-prometheus-oper-operator" not found

Helm Version:
version.BuildInfo{Version:"v3.1.1", GitCommit:"afe70585407b420d0097d07b21c47dc511525ac8", GitTreeState:"clean", GoVersion:"go1.13.8"}

Kubernetes Version
Client Version: v1.17.3
Server Version: v1.15.7

Thanks

[baelish]

I'm having a similar issue. I think mine is caused by the fullnameOverride: po setting causing the creation of a service called po-operator and the webhook still using the default service name prometheus-operator-operator

error message when applying a rules file:

Error from server (InternalError): error when creating "system-deployments/prometheus-operator/vaion-rules.yaml": Internal error occurred: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.prometheus-operator.svc:443/admission-prometheusrules/mutate?timeout=30s: service "prometheus-operator-operator" not found

Chart version: 8.9.3
Helm version: v3.0.3
Kubernetes version: v1.14.10-gke.17

This worked fine before I added the fullnameOverride: po which I require to work around an issue with gke ingresses.

[baelish]

@ravizeewee you may want to add [stable/prometheus-operator] to the title so it get's the right people's attention.

[baelish]

In my case it ended up being caused by some webhook configurations still hanging around from the old release. The below commands and a re-deploy fixed it for me.

kubectl delete validatingwebhookconfigurations.admissionregistration.k8s.io prometheus-operator-admission
kubectl delete mutatingwebhookconfigurations.admissionregistration.k8s.io prometheus-operator-admission

[BahmaniAlireza]

if you have run before without disabling the webhook, you must manually delete the following kinds:

kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io //delete all objects
kubectl get MutatingWebhookConfiguration //delete all objects

and after that run:

helm install --name prometheus-operator stable/prometheus-operator
--set prometheusOperator.admissionWebhooks.enabled=false
--set prometheusOperator.admissionWebhooks.patch.enabled=false
--set prometheusOperator.tlsProxy.enabled=false

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale]

This issue is being automatically closed due to inactivity.