Internal error occurred: failed calling webhook "webhook.serving.knative.dev": failed to call webhook: Post "https://webhook.knative-serving.svc:443/defaulting?timeout=10s": context deadline exceeded

[Si-ren]

Ask your question here:

**there is an e rror when i create knative service **
kubernetes: v1.23.7
Knative Install type : yaml https://knative.dev/docs/install/yaml-install/serving/install-serving-with-yaml/
network layer :istio

[21:37:00 root@k8s-master01 ~/k8s/knative-in-practise-main/serving/revision-and-route]#**cat 001-hello-world.yaml**
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: hello
spec:
  template:
    spec:
      containers:
        - image: ikubernetes/helloworld-go
          ports:
            - containerPort: 8080
          env:
            - name: TARGET
              value: "World"

[21:37:09 root@k8s-master01 ~/k8s/knative-in-practise-main/serving/revision-and-route]#k**ubectl create -f ./001-hello-world.yaml**
^[[AError from server (InternalError): error when creating "./001-hello-world.yaml": Internal error occurred: failed calling webhook "webhook.serving.knative.dev": failed to call webhook: Post "https://webhook.knative-serving.svc:443/defaulting?timeout=10s": context deadline exceeded

----------------env------------------

[21:44:44 root@k8s-master01 ~/k8s/knative-in-practise-main/serving/revision-and-route]#**kubectl get pod -A**
NAMESPACE         NAME                                      READY   STATUS    RESTARTS      AGE
default           busybox                                   1/1     Running   5 (47m ago)   5h48m
default           client-21723                              1/1     Running   0             35m
istio-system      istio-ingressgateway-5c9d78c775-bhp4m     1/1     Running   0             41m
istio-system      istio-ingressgateway-5c9d78c775-kfzsq     1/1     Running   0             41m
istio-system      istio-ingressgateway-5c9d78c775-vh94r     1/1     Running   0             41m
istio-system      istiod-579df55f96-dk5v8                   1/1     Running   0             40m
istio-system      istiod-579df55f96-dt52l                   1/1     Running   0             40m
istio-system      istiod-579df55f96-x6ds8                   1/1     Running   0             41m
knative-serving   activator-67688f67c6-lrbpb                1/1     Running   0             43m
knative-serving   autoscaler-58f7dfdb67-c4tnb               1/1     Running   0             43m
knative-serving   controller-6ddd5b667d-td5c5               1/1     Running   0             43m
knative-serving   domain-mapping-9657f967f-kcgfv            1/1     Running   0             43m
knative-serving   domainmapping-webhook-f5bfc7479-tzrmr     1/1     Running   0             43m
knative-serving   net-istio-controller-998b46c7f-9c6f4      1/1     Running   0             40m
knative-serving   net-istio-webhook-687bb6b995-5f8lb        1/1     Running   0             40m
knative-serving   webhook-5c4bff9565-7v9dj                  1/1     Running   0             43m
kube-system       calico-kube-controllers-6b77fff45-vswjv   1/1     Running   0             4h26m
kube-system       calico-node-b5qs2                         1/1     Running   0             4h26m
kube-system       calico-node-t8cmx                         1/1     Running   0             4h26m
kube-system       coredns-5c99bdb8b8-cv7mt                  1/1     Running   0             6h5m
kube-system       coredns-5c99bdb8b8-rwrbs                  1/1     Running   0             6h5m
kube-system       etcd-k8s-master01                         1/1     Running   3             6h6m
kube-system       kube-apiserver-k8s-master01               1/1     Running   0             6h6m
kube-system       kube-controller-manager-k8s-master01      1/1     Running   0             6h6m
kube-system       kube-proxy-lt5jf                          1/1     Running   0             6h5m
kube-system       kube-proxy-nbnnn                          1/1     Running   0             6h5m
kube-system       kube-scheduler-k8s-master01               1/1     Running   0             6h6m
kube-system       metrics-server-7cf8b65d65-5tzg7           1/1     Running   0             5h39m

----------------kube-api logs-----------------------

Trace[1084738683]: [6.966253515s] [6.966253515s] END
I0618 13:38:40.657347       1 trace.go:205] Trace[537590635]: "Call validating webhook" configuration:istio-validator-istio-system,webhook:rev.validation.istio.io,resource:networking.istio.io/v1alpha3, Resource=gateways,subresource:,operation:CREATE,UID:c719c5c7-4f82-4b57-983f-cdea3446c98f (18-Jun-2022 13:38:33.585) (total time: 7071ms):
Trace[537590635]: [7.071570708s] [7.071570708s] END
W0618 13:38:40.657385       1 dispatcher.go:142] Failed calling webhook, failing open rev.validation.istio.io: failed calling webhook "rev.validation.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/validate?timeout=10s": EOF
E0618 13:38:40.657405       1 dispatcher.go:149] failed calling webhook "rev.validation.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/validate?timeout=10s": EOF
I0618 13:38:40.658657       1 trace.go:205] Trace[1512351775]: "Create" url:/apis/networking.istio.io/v1alpha3/namespaces/istio-system/gateways,user-agent:pilot-discovery/1.13.4,audit-id:4236c778-b0d1-4f41-804d-4e2128073c06,client:192.168.137.201,accept:application/json, */*,protocol:HTTP/2.0 (18-Jun-2022 13:38:33.585) (total time: 7073ms):
Trace[1512351775]: ---"Object stored in database" 7073ms (13:38:40.658)
Trace[1512351775]: [7.073442557s] [7.073442557s] END
I0618 13:38:58.363242       1 trace.go:205] Trace[1700291169]: "Call validating webhook" configuration:istio-validator-istio-system,webhook:rev.validation.istio.io,resource:networking.istio.io/v1alpha3, Resource=gateways,subresource:,operation:CREATE,UID:d375d375-e8f7-4e2d-b9ef-77f6315cb8e5 (18-Jun-2022 13:38:51.539) (total time: 6823ms):
Trace[1700291169]: [6.823558801s] [6.823558801s] END
W0618 13:38:58.363263       1 dispatcher.go:142] Failed calling webhook, failing open rev.validation.istio.io: failed calling webhook "rev.validation.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/validate?timeout=10s": EOF
E0618 13:38:58.363282       1 dispatcher.go:149] failed calling webhook "rev.validation.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/validate?timeout=10s": EOF
I0618 13:38:58.364438       1 trace.go:205] Trace[1221173442]: "Create" url:/apis/networking.istio.io/v1alpha3/namespaces/istio-system/gateways,user-agent:pilot-discovery/1.13.4,audit-id:56c3f13b-39ca-4f40-ac7b-f5d0d94cd641,client:172.168.32.139,accept:application/json, */*,protocol:HTTP/2.0 (18-Jun-2022 13:38:51.539) (total time: 6825ms):
Trace[1221173442]: ---"Object stored in database" 6825ms (13:38:58.364)
Trace[1221173442]: [6.825369624s] [6.825369624s] END

-----------------webhook logs---------------------

{"severity":"INFO","timestamp":"2022-06-18T13:01:45.434180086Z","logger":"webhook.ValidationWebhook","caller":"validation/reconcile_config.go:173","message":"Updating webhook","commit":"36ee6f2","knative.dev/pod":"webhook-5c4bff9565-7v9dj","knative.dev/traceid":"103139a8-74c7-4c6a-94c8-a8c4deb71bbc","knative.dev/key":"validation.webhook.serving.knative.dev"}
{"severity":"INFO","timestamp":"2022-06-18T13:01:45.434631382Z","logger":"webhook.DefaultingWebhook","caller":"controller/controller.go:550","message":"Reconcile succeeded","commit":"36ee6f2","knative.dev/pod":"webhook-5c4bff9565-7v9dj","knative.dev/traceid":"2dfa8a08-b80e-4059-b386-7123510ed5d9","knative.dev/key":"webhook.serving.knative.dev","duration":"42.586596ms"}
{"severity":"INFO","timestamp":"2022-06-18T13:01:45.438726684Z","logger":"webhook.ConfigMapWebhook","caller":"configmaps/configmaps.go:168","message":"Webhook is valid","commit":"36ee6f2","knative.dev/pod":"webhook-5c4bff9565-7v9dj","knative.dev/traceid":"b3d91c6c-7252-46f9-9424-1aa962580760","knative.dev/key":"config.webhook.serving.knative.dev"}
{"severity":"INFO","timestamp":"2022-06-18T13:01:45.43878326Z","logger":"webhook.ConfigMapWebhook","caller":"controller/controller.go:550","message":"Reconcile succeeded","commit":"36ee6f2","knative.dev/pod":"webhook-5c4bff9565-7v9dj","knative.dev/traceid":"b3d91c6c-7252-46f9-9424-1aa962580760","knative.dev/key":"config.webhook.serving.knative.dev","duration":"6.274745ms"}
{"severity":"INFO","timestamp":"2022-06-18T13:01:45.447070005Z","logger":"webhook.DefaultingWebhook","caller":"defaulting/defaulting.go:253","message":"Updating webhook","commit":"36ee6f2","knative.dev/pod":"webhook-5c4bff9565-7v9dj","knative.dev/traceid":"92a7d367-94c4-431e-a5a3-23408b33a58e","knative.dev/key":"webhook.serving.knative.dev"}
{"severity":"INFO","timestamp":"2022-06-18T13:01:45.447900191Z","logger":"webhook.ValidationWebhook","caller":"controller/controller.go:550","message":"Reconcile succeeded","commit":"36ee6f2","knative.dev/pod":"webhook-5c4bff9565-7v9dj","knative.dev/traceid":"103139a8-74c7-4c6a-94c8-a8c4deb71bbc","knative.dev/key":"validation.webhook.serving.knative.dev","duration":"55.173963ms"}
{"severity":"INFO","timestamp":"2022-06-18T13:01:45.450752735Z","logger":"webhook.DefaultingWebhook","caller":"controller/controller.go:550","message":"Reconcile succeeded","commit":"36ee6f2","knative.dev/pod":"webhook-5c4bff9565-7v9dj","knative.dev/traceid":"92a7d367-94c4-431e-a5a3-23408b33a58e","knative.dev/key":"webhook.serving.knative.dev","duration":"16.076148ms"}
{"severity":"INFO","timestamp":"2022-06-18T13:01:45.498473389Z","logger":"webhook.ValidationWebhook","caller":"validation/reconcile_config.go:173","message":"Updating webhook","commit":"36ee6f2","knative.dev/pod":"webhook-5c4bff9565-7v9dj","knative.dev/traceid":"c3fd1d11-88e2-4996-99f1-206e8d0aad7a","knative.dev/key":"validation.webhook.serving.knative.dev"}
{"severity":"INFO","timestamp":"2022-06-18T13:01:45.541219039Z","logger":"webhook.ValidationWebhook","caller":"controller/controller.go:550","message":"Reconcile succeeded","commit":"36ee6f2","knative.dev/pod":"webhook-5c4bff9565-7v9dj","knative.dev/traceid":"c3fd1d11-88e2-4996-99f1-206e8d0aad7a","knative.dev/key":"validation.webhook.serving.knative.dev","duration":"93.139912ms"}

[nyarly]

We're having the same problem in a Kourier Knative deployment right now. This deployment has been stable for quite a while.

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[Wouter0100]

I have had this just now as well and it surely wasn't the first time. Twice actually. Kourier with Knative Serving.

│ Error: cannot patch "x" with kind Service: Internal error occurred: failed calling webhook "validation.webhook.serving.knative.dev": failed to call webhook: Post "https://webhook.knative-serving.svc:443/resource-validation?timeout=10s": context deadline exceeded

│ Error: cannot patch "x" with kind DomainMapping: Internal error occurred: failed calling webhook "validation.webhook.domainmapping.serving.knative.dev": failed to call webhook: Post "https://domainmapping-webhook.knative-serving.svc/resource-validation?timeout=10s": context deadline exceeded

/remove-lifecycle stale

[RileySeaburg]

I'm getting this issue on GKE autopilot as well.

[hecklawert]

Same issue here. I installed it on K8s 1.22 using Knative Operator and I get the same error trying to deploy a Service.

[dprotaso]

Hey folks - can you post a gist link with logs.

Generally we've seen errors like this when the API server is under-provisioned. If you have a way to reproduce this that would be even better.

[Wouter0100]

Thanks for responding, @dprotaso. That's also what I expected, but unfortunately I do not manage the deployment of the API server. We use Scaleway's managed control plane offering named Kapsule. Unfortunately no way to reproduce consistently.

Will get back to you with logs.

[hecklawert]

Thanks for your reply @dprotaso . In my case, I don't have access to API as this is an EKS instance with one t3.medium worker node.

Here you can find the logs.
https://gist.github.com/hecklawert/93a20902bb1e8c9b0dd52fe896d43580

[hecklawert]

I think I've found the problem. My instance of EKS was deployed with the prefix delegation enabled on the CNI plugin (I did this to have capacity for 110 pods per node instead of 17) using this TF module as example.

Using an EKS without prefix delegation Knative is working as expected.

[paulgrav]

We run our clusters on GKE. The webhook call is made from the apiserver. The webhook pod listens on 8443 whilst the service listens on 443. When making the webhook call the GKE apiserver tries to hit the webhook pod on 8443. Only ports 443 and 10250 are open between the apiserver and the GKE nodes.

I solved this issue by creating a firewall rule that allowed the apiserver to talk to the nodes over 8443.

It took me a while to figure this out. If you are seeing failed to call webhook: Post "https://webhook.knative-serving.svc:443/defaulting?timeout=10s": context deadline exceeded then my suspicion would be that there’s a firewall in place somewhere. I couldn’t see anything in the webhook logs that suggested that the calls were making it to the pod. I looked in the the APIserver logs in the GKE log console, I could see the calls being made there but there wasn’t much more information than Internal server error.

[Si-ren]

check proxy