Error X509 : Internal error occurred: failed calling webhook "inferenceservice.kfserving-webhook-server.pod-mutator"

[kdecyllene]

/kind bug

Hi,

We use kubeflow for our business and recently had a problem or unfortunately nothing is working anymore. Currently we have this error:

Reissued from statefulset/jupytermain: create Pod jupytermain-0 in StatefulSet jupytermain failed error: Internal error occurred: failed calling webhook "inferenceservice.kfserving-webhook-server.pod-mutator": Post "https://kfserving-webhook-server-service.kubeflow.svc:443/mutate-pods?timeout=30s": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0

Kfserver-manager generate message : kfserver-controller-manager: bad certificate

Status certificate manager :

[root@kubemaster ~]# kubectl get pods -A | grep cert
cert-manager cert-manager-cainjector-7c755ff968-c5znk 1/1 Running 20 69d
cert-manager cert-manager-df69887f8-f5ff6 1/1 Running 20 69d
cert-manager cert-manager-webhook-b487b785f-wfks9 1/1 Running 15 69

Environment:

• Istio Version: ?
• kfctl version : v1.0.2-0-ga476281
• Knative Version: gcr.io/knative-releases/knative.dev/serving/cmd/webhook@sha256:1ef3328282f31704b5802c1136bd117e8598fd9f437df8209ca87366c5ce9fcb
• KFServing Version: kfserving-controller:0.2.2
• KFServing-ingressgateway : proxyv2:1.1.6
• Tsensor : gcr.io/kubeflow-images-public/tensorflow-2.1.0-notebook-cpu:1.0.0
• Kubeflow version: ?
• Kfdef:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm] : k8s_istio
• Kubernetes version: (use kubectl version): 1.19.3
• OS (e.g. from /etc/os-release): Centos 7

[xybuty]

I got a same error.

[yuzisun]

@kdecyllene looks like there is similar issue for katib kubeflow/katib#1395, related to kubernetes 1.19.

[yuzisun]

Fixed in #1661