error when creating "test1": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": EOF

[hmi12]

Describe the bug:
When create new certificates, got below error:
Error from server (InternalError): error when creating "test1": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": EOF

The logs from webhook pod:

I0303 02:55:28.259166 1 logs.go:59] http: TLS handshake error from 10.244.0.6:57282: EOF
I0303 02:58:40.170490 1 logs.go:59] http: TLS handshake error from 10.231.96.16:24108: EOF
I0306 03:19:15.623237 1 logs.go:59] http: TLS handshake error from 10.244.1.5:40462: EOF
I0306 03:19:15.625800 1 logs.go:59] http: TLS handshake error from 10.244.0.6:39672: EOF
I0306 03:19:15.631995 1 logs.go:59] http: TLS handshake error from 10.244.1.5:40468: EOF
I0306 03:19:15.649874 1 logs.go:59] http: TLS handshake error from 10.244.1.5:40482: EOF
I0306 03:19:16.009029 1 logs.go:59] http: TLS handshake error from 10.244.1.5:40488: EOF
I0306 03:19:16.080735 1 logs.go:59] http: TLS handshake error from 10.244.0.6:39676: EOF
I0306 04:00:54.816144 1 logs.go:59] http: TLS handshake error from 10.244.1.5:52224: EOF

Environment details::

• Kubernetes version: 1.23.9
• Cloud-provider/provisioner: Azure/aks

/kind bug

[irbekrm]

I'd recommend that you take a look at https://cert-manager.io/docs/troubleshooting/webhook/ and update the issue description with any relevant findings.

[jetstack-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

[KiruyaMomochi]

Although I don't think my solution may help you, I will still add it here for reference.

I faced a similar issue when deploying an on-premise Kubernetes cluster using kubeadm. The cert-manager API check (cmctl check api) always resulted in the following error:

Not ready: Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": context deadline exceeded

After investigating the kube-apiserver pod, I discovered the root cause. It turned out that I had set the http_proxy environment variable during the cluster deployment, causing the pod IPs to be proxied within kube-apiserver. Consequently, all requests from the API server to the webhook endpoint were not reaching their destination, and when using tcpdump with a cluster IP filter, no traffic was captured.

To resolve this issue, either add the Pod CIDR to the no_proxy variable in the manifests located under /etc/kubernetes/manifests/*.yaml, or completely remove all proxy environment variables from these files.

---

This problem could be rare, as most people may not set http_proxy / https_proxy in this way. But if @irbekrm don't mind, I can still update https://cert-manager.io/docs/troubleshooting/webhook/ to add this scenario.

[jetstack-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle rotten
/remove-lifecycle stale

[jetstack-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to jetstack.
/close

[jetstack-bot]

@jetstack-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to jetstack.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.