Admission Webhooks not working in v0.40.0+

[plnordquist]

What happened?
When I setup a MutatingWebhookConfiguration or a ValidatingWebhookConfiguration against the prometheusrule webhook paths with v0.40.0 or v0.41.1 of the operator, it does not return the appropriate output to the Kubernetes API Server. The output is missing the apiVersion=admission.k8s.io/v1 and kind=AdmissionReview when returning data from the web call. Kubernetes fails to create prometheusrules with the error Error from server (InternalError): error when creating "test.yaml": Internal error occurred: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": expected webhook response of admission.k8s.io/v1, Kind=AdmissionReview, got /, Kind=.

Did you expect to see something different?
The webhooks should appropriately respond to the API server when attempting to create or update prometheus rules.

How to reproduce it (as minimally and precisely as possible):

1. Deploy the prometheus operator with tls enabled
2. Configure Validating and Mutating Webhook configurations against the deployed operator
3. Attempt to create a prometheus rule

Environment

• Prometheus Operator version:

  quay.io/prometheus-operator/prometheus-operator:v0.41.1

• Kubernetes version information:

  Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.11", GitCommit:"ea5f00d93211b7c80247bf607cfa422ad6fb5347", GitTreeState:"clean", BuildDate:"2020-08-13T15:20:25Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.11", GitCommit:"ea5f00d93211b7c80247bf607cfa422ad6fb5347", GitTreeState:"clean", BuildDate:"2020-08-13T15:11:47Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

  kubeadm

• Manifests:

# Sample rule used for testing
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    prometheus: example
    role: alert-rules
  name: prometheus-example-rules
  namespace: prometheus-operator
spec:
  groups:
  - name: ./example.rules
    rules:
    - alert: ExampleAlert
      expr: vector(1)

• Prometheus Operator Logs:

v0.40.0 mutate:
2020-09-04T12:31:53.796002717-07:00 level=debug ts=2020-09-04T19:31:53.795812259Z caller=admission.go:122 component=admissionwebhook msg="Received request" content="{\"kind\":\"AdmissionReview\",\"apiVersion\":\"admission.k8s.io/v1\",\"request\":{\"uid\":\"8bb5aa62-2708-44a0-a01d-6f42a8576886\",\"kind\":{\"group\":\"monitoring.coreos.com\",\"version\":\"v1\",\"kind\":\"PrometheusRule\"},\"resource\":{\"group\":\"monitoring.coreos.com\",\"version\":\"v1\",\"resource\":\"prometheusrules\"},\"requestKind\":{\"group\":\"monitoring.coreos.com\",\"version\":\"v1\",\"kind\":\"PrometheusRule\"},\"requestResource\":{\"group\":\"monitoring.coreos.com\",\"version\":\"v1\",\"resource\":\"prometheusrules\"},\"name\":\"prometheus-example-rules\",\"namespace\":\"prometheus-operator\",\"operation\":\"CREATE\",\"userInfo\":{\"username\":\"kubernetes-admin\",\"groups\":[\"system:masters\",\"system:authenticated\"]},\"object\":{\"apiVersion\":\"monitoring.coreos.com/v1\",\"kind\":\"PrometheusRule\",\"metadata\":{\"annotations\":{\"kubectl.kubernetes.io/last-applied-configuration\":\"{\\\"apiVersion\\\":\\\"monitoring.coreos.com/v1\\\",\\\"kind\\\":\\\"PrometheusRule\\\",\\\"metadata\\\":{\\\"annotations\\\":{},\\\"labels\\\":{\\\"prometheus\\\":\\\"example\\\",\\\"role\\\":\\\"alert-rules\\\"},\\\"name\\\":\\\"prometheus-example-rules\\\",\\\"namespace\\\":\\\"prometheus-operator\\\"},\\\"spec\\\":{\\\"groups\\\":[{\\\"name\\\":\\\"./example.rules\\\",\\\"rules\\\":[{\\\"alert\\\":\\\"ExampleAlert\\\",\\\"expr\\\":\\\"vector(1)\\\"}]}]}}\\n\"},\"creationTimestamp\":null,\"labels\":{\"prometheus\":\"example\",\"role\":\"alert-rules\"},\"name\":\"prometheus-example-rules\",\"namespace\":\"prometheus-operator\"},\"spec\":{\"groups\":[{\"name\":\"./example.rules\",\"rules\":[{\"alert\":\"ExampleAlert\",\"expr\":\"vector(1)\"}]}]}},\"oldObject\":null,\"dryRun\":false,\"options\":{\"kind\":\"CreateOptions\",\"apiVersion\":\"meta.k8s.io/v1\"}}}\n"
2020-09-04T12:31:53.797426151-07:00 level=debug ts=2020-09-04T19:31:53.797299028Z caller=admission.go:151 component=admissionwebhook msg="Mutating prometheusrules"
2020-09-04T12:31:53.798168515-07:00 level=debug ts=2020-09-04T19:31:53.798059717Z caller=admission.go:138 component=admissionwebhook msg="sending response" content="{\"response\":{\"uid\":\"8bb5aa62-2708-44a0-a01d-6f42a8576886\",\"allowed\":true,\"patch\":\"W3sgIm9wIjogImFkZCIsICJwYXRoIjogIi9tZXRhZGF0YS9hbm5vdGF0aW9ucy9wcm9tZXRoZXVzLW9wZXJhdG9yLXZhbGlkYXRlZCIsICJ2YWx1ZSI6ICJ0cnVlIiB9XQ==\",\"patchType\":\"JSONPatch\"}}"

v0.41.1 mutate:
2020-09-04T13:58:41.956471855-07:00 level=debug ts=2020-09-04T20:58:41.956332281Z caller=admission.go:122 component=admissionwebhook msg="Received request" content="{\"kind\":\"AdmissionReview\",\"apiVersion\":\"admission.k8s.io/v1\",\"request\":{\"uid\":\"c9b2aa62-c6d6-465b-86b9-30563e2ddc4f\",\"kind\":{\"group\":\"monitoring.coreos.com\",\"version\":\"v1\",\"kind\":\"PrometheusRule\"},\"resource\":{\"group\":\"monitoring.coreos.com\",\"version\":\"v1\",\"resource\":\"prometheusrules\"},\"requestKind\":{\"group\":\"monitoring.coreos.com\",\"version\":\"v1\",\"kind\":\"PrometheusRule\"},\"requestResource\":{\"group\":\"monitoring.coreos.com\",\"version\":\"v1\",\"resource\":\"prometheusrules\"},\"name\":\"prometheus-example-rules\",\"namespace\":\"prometheus-operator\",\"operation\":\"CREATE\",\"userInfo\":{\"username\":\"kubernetes-admin\",\"groups\":[\"system:masters\",\"system:authenticated\"]},\"object\":{\"apiVersion\":\"monitoring.coreos.com/v1\",\"kind\":\"PrometheusRule\",\"metadata\":{\"annotations\":{\"kubectl.kubernetes.io/last-applied-configuration\":\"{\\\"apiVersion\\\":\\\"monitoring.coreos.com/v1\\\",\\\"kind\\\":\\\"PrometheusRule\\\",\\\"metadata\\\":{\\\"annotations\\\":{},\\\"labels\\\":{\\\"prometheus\\\":\\\"example\\\",\\\"role\\\":\\\"alert-rules\\\"},\\\"name\\\":\\\"prometheus-example-rules\\\",\\\"namespace\\\":\\\"prometheus-operator\\\"},\\\"spec\\\":{\\\"groups\\\":[{\\\"name\\\":\\\"./example.rules\\\",\\\"rules\\\":[{\\\"alert\\\":\\\"ExampleAlert\\\",\\\"expr\\\":\\\"vector(1)\\\"}]}]}}\\n\"},\"creationTimestamp\":null,\"labels\":{\"prometheus\":\"example\",\"role\":\"alert-rules\"},\"name\":\"prometheus-example-rules\",\"namespace\":\"prometheus-operator\"},\"spec\":{\"groups\":[{\"name\":\"./example.rules\",\"rules\":[{\"alert\":\"ExampleAlert\",\"expr\":\"vector(1)\"}]}]}},\"oldObject\":null,\"dryRun\":false,\"options\":{\"kind\":\"CreateOptions\",\"apiVersion\":\"meta.k8s.io/v1\"}}}\n"
2020-09-04T13:58:41.957385819-07:00 level=debug ts=2020-09-04T20:58:41.957331777Z caller=admission.go:151 component=admissionwebhook msg="Mutating prometheusrules"
2020-09-04T13:58:41.957823935-07:00 level=debug ts=2020-09-04T20:58:41.957772335Z caller=admission.go:138 component=admissionwebhook msg="sending response" content="{\"response\":{\"uid\":\"c9b2aa62-c6d6-465b-86b9-30563e2ddc4f\",\"allowed\":true,\"patch\":\"W3sgIm9wIjogImFkZCIsICJwYXRoIjogIi9tZXRhZGF0YS9hbm5vdGF0aW9ucy9wcm9tZXRoZXVzLW9wZXJhdG9yLXZhbGlkYXRlZCIsICJ2YWx1ZSI6ICJ0cnVlIiB9XQ==\",\"patchType\":\"JSONPatch\"}}"

v0.41.1 validate:
2020-09-04T14:12:32.996219579-07:00 level=debug ts=2020-09-04T21:12:32.996013926Z caller=admission.go:122 component=admissionwebhook msg="Received request" content="{\"kind\":\"AdmissionReview\",\"apiVersion\":\"admission.k8s.io/v1\",\"request\":{\"uid\":\"fea692c2-d538-4c87-abfa-dfd59ecfc49a\",\"kind\":{\"group\":\"monitoring.coreos.com\",\"version\":\"v1\",\"kind\":\"PrometheusRule\"},\"resource\":{\"group\":\"monitoring.coreos.com\",\"version\":\"v1\",\"resource\":\"prometheusrules\"},\"requestKind\":{\"group\":\"monitoring.coreos.com\",\"version\":\"v1\",\"kind\":\"PrometheusRule\"},\"requestResource\":{\"group\":\"monitoring.coreos.com\",\"version\":\"v1\",\"resource\":\"prometheusrules\"},\"name\":\"prometheus-example-rules\",\"namespace\":\"prometheus-operator\",\"operation\":\"CREATE\",\"userInfo\":{\"username\":\"kubernetes-admin\",\"groups\":[\"system:masters\",\"system:authenticated\"]},\"object\":{\"apiVersion\":\"monitoring.coreos.com/v1\",\"kind\":\"PrometheusRule\",\"metadata\":{\"annotations\":{\"kubectl.kubernetes.io/last-applied-configuration\":\"{\\\"apiVersion\\\":\\\"monitoring.coreos.com/v1\\\",\\\"kind\\\":\\\"PrometheusRule\\\",\\\"metadata\\\":{\\\"annotations\\\":{},\\\"labels\\\":{\\\"prometheus\\\":\\\"example\\\",\\\"role\\\":\\\"alert-rules\\\"},\\\"name\\\":\\\"prometheus-example-rules\\\",\\\"namespace\\\":\\\"prometheus-operator\\\"},\\\"spec\\\":{\\\"groups\\\":[{\\\"name\\\":\\\"./example.rules\\\",\\\"rules\\\":[{\\\"alert\\\":\\\"ExampleAlert\\\",\\\"expr\\\":\\\"vector(1)\\\"}]}]}}\\n\"},\"creationTimestamp\":\"2020-09-04T21:12:32Z\",\"generation\":1,\"labels\":{\"prometheus\":\"example\",\"role\":\"alert-rules\"},\"name\":\"prometheus-example-rules\",\"namespace\":\"prometheus-operator\",\"uid\":\"1cf2f54b-ed20-443b-a981-a2882ccfee4e\"},\"spec\":{\"groups\":[{\"name\":\"./example.rules\",\"rules\":[{\"alert\":\"ExampleAlert\",\"expr\":\"vector(1)\"}]}]}},\"oldObject\":null,\"dryRun\":false,\"options\":{\"kind\":\"CreateOptions\",\"apiVersion\":\"meta.k8s.io/v1\"}}}\n"
2020-09-04T14:12:32.997703190-07:00 level=debug ts=2020-09-04T21:12:32.997574803Z caller=admission.go:186 component=admissionwebhook msg="Validating prometheusrules"
2020-09-04T14:12:32.998976239-07:00 level=debug ts=2020-09-04T21:12:32.998872345Z caller=admission.go:138 component=admissionwebhook msg="sending response" content="{\"response\":{\"uid\":\"fea692c2-d538-4c87-abfa-dfd59ecfc49a\",\"allowed\":true}}"

Anything else we need to know?:
I have been trying to create an Operator Lifecycle Manager Cluster Service Version for the prometheus operator for v0.40.0 or v0.41.1 that might help to reproduce this issue but I don't have it publicly available. If it would help, I can look into making that resource publicly available.

[brancz]

cc @s-urbaniak @paulfantom

[s-urbaniak]

I will have a look, that looks suspicious indeed 🤔