apply ingress rule error after install ingress-nginx: x509 certificate is not valid ingress-nginx-controller-admission.ingress-nginx.svc

[liminghua999]

k8s cluster install by binary  （i  also try v1.18.0）
[root@m-etc-1 ssl-nginx-webhook]# kubectl get no
NAME      STATUS   ROLES    AGE   VERSION
m-etc-1   Ready    <none>   20h   v1.18.6
m-etc-2   Ready    <none>   20h   v1.18.6
m-etc-3   Ready    <none>   20h   v1.18.6
n-1       Ready    <none>   20h   v1.18.6
n-2       Ready    <none>   20h   v1.18.6
n-3       Ready    <none>   20h   v1.18.6
slb-1     Ready    <none>   20h   v1.18.6
slb-2     Ready    <none>   20h   v1.18.6


and calico install is ok
and coredns install is OK
and ingress-nginx install is OK  
ingress-nginx deploy file:
https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider//baremetal/deploy.yaml

[root@m-etc-1 cfg]# kubectl api-versions | grep admissionregistration.k8s.io
admissionregistration.k8s.io/v1
admissionregistration.k8s.io/v1beta1

AND:  --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,PersistentVolumeClaimResize,PodPreset 


[root@m-etc-1 ssl-nginx-webhook]# kubectl get all -n ingress-nginx
NAME                                       READY   STATUS      RESTARTS   AGE
pod/ingress-nginx-admission-create-v27qd   0/1     Completed   0          20h
pod/ingress-nginx-admission-patch-599bf    0/1     Completed   0          20h
pod/ingress-nginx-controller-dsg2j         1/1     Running     1          14h
pod/ingress-nginx-controller-jwjxk         1/1     Running     1          14h

NAME                                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
service/ingress-nginx-controller             ClusterIP   10.244.81.145    <none>        80/TCP,443/TCP   20h
service/ingress-nginx-controller-admission   ClusterIP   10.244.170.231   <none>        443/TCP          20h

NAME                                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/ingress-nginx-controller   2         2         2       2            2           in=ingress      20h

NAME                                       COMPLETIONS   DURATION   AGE
job.batch/ingress-nginx-admission-create   1/1           5s         20h
job.batch/ingress-nginx-admission-patch    1/1           5s         20h

I try apply ingress rule,but I get error:
[root@m-etc-1 ~]# kubectl apply -f ingress-nginx.yml
Error from server (InternalError): error when creating "ingress-nginx.yml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s: x509: certificate is valid for kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster, kubernetes.default.svc.cluster.local, not ingress-nginx-controller-admission.ingress-nginx.svc

WHY???

[liminghua999]

No one meet this error??
I try the k8s cluster (install by kubeadm)

k8s version: v1.18.6
docker version: 19.03.12 
os: centos 7.6
helm3

apply ingress rule still report same error ;

If I disable webhooks ,then I apply ingress rule,it is OK;

[21ki]

kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission

[costela]

@liminghua999 can you please check if the output of these two commands match?

$ kubectl -n ingress-nginx get validatingwebhookconfigurations ingress-nginx-admission -ojsonpath='{.webhooks[0].clientConfig.caBundle}'

$ kubectl -n ingress-nginx get secret ingress-nginx-admission -ojsonpath='{.data.ca}'

[21ki]

[root@master01 ~]# kubectl -n ingress-nginx get validatingwebhookconfigurations ingress-nginx-admission -ojsonpath='{.webhooks[0].clientConfig.caBundle}'
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJOekNCMzZBREFnRUNBaEVBc1Fib3U4OWlSVmJZazJVT2RPRGh3REFLQmdncWhrak9QUVFEQWpBQU1DQVgKRFRJd01Ea3pNREF5TVRJd01Gb1lEekl4TWpBd09UQTJNREl4TWpBd1dqQUFNRmt3RXdZSEtvWkl6ajBDQVFZSQpLb1pJemowREFRY0RRZ0FFZ3lWQkZGUnN5dFlIb3N1N24xRVVsekJ5aTRJaEtKYmx0WFRieElTYmlDLzJWRk9ZClI0NzJSczBHMnhCS1NkR3NtaGZOK1ZkTG1EREdNeEE2UE55MVVhTTRNRFl3RGdZRFZSMFBBUUgvQkFRREFnSUUKTUJNR0ExVWRKUVFNTUFvR0NDc0dBUVVGQndNQk1BOEdBMVVkRXdFQi93UUZNQU1CQWY4d0NnWUlLb1pJemowRQpBd0lEUndBd1JBSWdGK1JxUW5oRzgxbVp1TEFKc0RaNUozN3Y1VUI0U2trUXpDb21ya2dYWTgwQ0lGOW4vQ002CjBHV2hFOXRFVVFleFV2MTZTR1NabGVpSEFoMXE5dElEVlh0bwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
[root@master01 ~]# kubectl -n ingress-nginx get secret ingress-nginx-admission -ojsonpath='{.data.ca}'
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJOekNCMzZBREFnRUNBaEVBc1Fib3U4OWlSVmJZazJVT2RPRGh3REFLQmdncWhrak9QUVFEQWpBQU1DQVgKRFRJd01Ea3pNREF5TVRJd01Gb1lEekl4TWpBd09UQTJNREl4TWpBd1dqQUFNRmt3RXdZSEtvWkl6ajBDQVFZSQpLb1pJemowREFRY0RRZ0FFZ3lWQkZGUnN5dFlIb3N1N24xRVVsekJ5aTRJaEtKYmx0WFRieElTYmlDLzJWRk9ZClI0NzJSczBHMnhCS1NkR3NtaGZOK1ZkTG1EREdNeEE2UE55MVVhTTRNRFl3RGdZRFZSMFBBUUgvQkFRREFnSUUKTUJNR0ExVWRKUVFNTUFvR0NDc0dBUVVGQndNQk1BOEdBMVVkRXdFQi93UUZNQU1CQWY4d0NnWUlLb1pJemowRQpBd0lEUndBd1JBSWdGK1JxUW5oRzgxbVp1TEFKc0RaNUozN3Y1VUI0U2trUXpDb21ya2dYWTgwQ0lGOW4vQ002CjBHV2hFOXRFVVFleFV2MTZTR1NabGVpSEFoMXE5dElEVlh0bwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==

[md-waldron]

I am seeing this problem also:

Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "https://ingress-nginx-controller-admission.kube-system.svc:443/extensions/v1beta1/ingresses?timeout=30s": x509: certificate signed by unknown authority

This is in minikube with Ingress-nginx installed using minikube addons enable ingress

I can 'resolve' it using kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission but this does not seem like a solution, its just a hack to get it working.

[ghfalcon7]

I'm running into the same issue, did anyone solve this?

[marvinnitz18]

I'm running into the same issue, did anyone solve this?

Me too, is it new ?

[KaivalyaDabhadkar]

I am seeing this problem also:

Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "https://ingress-nginx-controller-admission.kube-system.svc:443/extensions/v1beta1/ingresses?timeout=30s": x509: certificate signed by unknown authority

This is in minikube with Ingress-nginx installed using minikube addons enable ingress

I can 'resolve' it using kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission but this does not seem like a solution, its just a hack to get it working.

Hi , I am also facing this same issue, did you find any resolution for this other than deleting the Webhook configuration?

[md-waldron]

No, not yet. It's annoyingly, but the work around is not too troublesome, so we have not spent more time trying to fix it.

…

On Tue, 24 Nov 2020, 11:45 KaivalyaDabhadkar, ***@***.***> wrote: I am seeing this problem also: Internal error occurred: failed calling webhook " validate.nginx.ingress.kubernetes.io": Post " https://ingress-nginx-controller-admission.kube-system.svc:443/extensions/v1beta1/ingresses?timeout=30s": x509: certificate signed by unknown authority This is in minikube with Ingress-nginx installed using minikube addons enable ingress I can 'resolve' it using kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission but this does not seem like a solution, its just a hack to get it working. Hi , I am also facing this same issue, did you find any resolution for this other than deleting the Webhook configuration? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5968 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBOUR5V5CFA2BFTIMLELFLSROMFZANCNFSM4PTALKBA> .

[sschne]

@md-waldron What version of minikube, kubernetes and which minikube driver are you using? Can you describe what you are doing to reproduce this issue?

[md-waldron]

@sschne I updated my docker and k8s recently and I no longer seem to have the issue. Here is the startup I get when starting minikube, seems all the version info you wanted is reported in that:

😄 minikube v1.14.0 on Darwin 11.0.1
✨ Using the virtualbox driver based on existing profile
👍 Starting control plane node minikube in cluster minikube
🔄 Restarting existing virtualbox VM for "minikube" ...
🐳 Preparing Kubernetes v1.19.2 on Docker 19.03.12 ...
🔎 Verifying Kubernetes components...
🔎 Verifying ingress addon...
🌟 Enabled addons: storage-provisioner, default-storageclass, dashboard, ingress
🏄 Done! kubectl is now configured to use "minikube" by default

[cloud-66]

in kubernetes 1.17.13 i have the same issue and don't know how to solve it.

[raider444]

Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"d360454c9bcd1634cf4cc52d1867af5491dc9c5f", GitTreeState:"clean", BuildDate:"2020-11-11T13:09:17Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

docker://19.3.13

The same problem.

[codeclown]

Borrowing from answers above, here's what resolved this for me. Issue started when I deleted an entire namespace that had contained nginx-ingress, and then tried to reinstall everything via helm. Kept getting upon helm upgrade --install:

Error: UPGRADE FAILED: failed to create resource: Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "https://foobar-ingress-nginx-controller-admission.foobar.svc:443/networking/v1beta1/ingresses?timeout=10s": x509: certificate signed by unknown authority (possibly because of "x509: ECDSA verification failure" while trying to verify candidate authority certificate "nil1")

Solution:

# Find name of the ingress-nginx-admission resource
kubectl get -A ValidatingWebhookConfiguration
# Delete it
kubectl delete -A ValidatingWebhookConfiguration <name>
# Example:
kubectl delete -A ValidatingWebhookConfiguration foobar-ingress-nginx-admission

[timd73]

I'm encountering this issue, and I don't have a valid workaround, because my ingress-nginx is deployed with fluxcd, so every time I delete the ValidatingWebhookConfiguration, it is recreated.

I am not sure if it's related, but I followed @costela's instructions, and this command has no output:

kubectl -n ingress-nginx get validatingwebhookconfigurations ingress-nginx-admission -ojsonpath='{.webhooks[0].clientConfig.caBundle}'

This command

kubectl -n ingress-nginx get validatingwebhookconfigurations ingress-nginx-admission -ojsonpath='{.webhooks[0].clientConfig}'

shows

{"service":{"name":"ingress-nginx-controller-admission","namespace":"ingress-nginx","path":"/networking/v1beta1/ingresses","port":443}}

Is this the problem, no caBundle? Can someone help solve this?

Or is there a workaround to delete the ValidatingWebhookConfiguration from the source yaml, which I took from here? Can someone explain how exactly to do that?

[damienleger]

My fix for this issue without deleting the validatingwebhookconfigurations. Inspired by #5968 (comment)

CA=$(kubectl -n ingress-nginx get secret ingress-nginx-admission -ojsonpath='{.data.ca}')
kubectl patch validatingwebhookconfigurations ingress-nginx-admission --type='json' -p='[{"op": "add", "path": "/webhooks/0/clientConfig/caBundle", "value":"'$CA'"}]'

[imranrazakhan]

@liminghua999 Why you close this issue? I am still getting it with K8S 1.21.1 and 0.46.0 ingress.

[kristopher-bredemeier-gcx]

I ran into the the same problem. I fixed it by deleting and reapplying the ingress-nginx-admission-patch job.

[juozasget]

Running the ingress-nginx-admission-patch job fixed it for me as suggested by @kristopher-bredemeier-gcx

If you are looking for an easy way to trigger the ingress-nginx-admission-patch job. Helm chart version upgrade will trigger it.
apiVersion: batch/v1 kind: Job metadata: name: ingress-nginx-admission-patch annotations: helm.sh/hook: post-install,post-upgrade

[dannystaple]

I've seen this where I'd accidentally introduced quotes into the class-name for the controller in the args for Deployment-ingress-nginx-controller.yml - doing some substitution so I could consider multiple controllers (on different network segments). Don't put quotes there, and avoid underscores.

The log for the ingress controller deployment will show: Invalid value: "ingress-controller-leader-\"nginx\"": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters,

[marv254]

My fix for this issue without deleting the validatingwebhookconfigurations. Inspired by #5968 (comment)

CA=$(kubectl -n ingress-nginx get secret ingress-nginx-admission -ojsonpath='{.data.ca}')
kubectl patch validatingwebhookconfigurations ingress-nginx-admission --type='json' -p='[{"op": "add", "path": "/webhooks/0/clientConfig/caBundle", "value":"'$CA'"}]'

[xiazemin]

Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": an error on the server ("") has prevented the request from succeeding

[felipess19]

the solution of @marv254 works for me.
thks.

[beastob]

Inspired by @marv254 's comment.

I had a similar issue with missing caBundle in the validatingwebhookconfigurations/ingress-nginx-admission, and it was installed via ArgoCD and a customized ingress-nginx helm chart.

Since ingress-nginx-admission-patch has an annotation "helm.sh/hook": post-install, caBundle will only appear in the validatingwebhookconfigurations after the Chart is fully installed.

In my case, my custom chart contains some resources that required the admission webhook with the caBundle, and that caused Helm chart to never reaches the 'post-install' stage.

My solution is to add the annotation "helm.sh/hook": post-install to my custom resource as well.

[danivendetta]

Hi

I've having the same problem. And based in the comment

My fix for this issue without deleting the validatingwebhookconfigurations. Inspired by #5968 (comment)

CA=$(kubectl -n ingress-nginx get secret ingress-nginx-admission -ojsonpath='{.data.ca}')
kubectl patch validatingwebhookconfigurations ingress-nginx-admission --type='json' -p='[{"op": "add", "path": "/webhooks/0/clientConfig/caBundle", "value":"'$CA'"}]'

I've been investigating. I don't use Helm, and I've had to configure each component individually.

There're two Jobs for nginx-admission-webhooks, create and patch. And Based in the Image's documentation we have a command line with their especifications:

https://github.com/jet/kube-webhook-certgen#patch

  --patch-validating              If true, patch validatingwebhookconfiguration (default true)

Doing that I've got a correct validatingwebhookconfiguration config.

I hope that help you.

[damienleger]

@danivendetta you put the same link twice, typo I think.

[danivendetta]

@danivendetta you put the same link twice, typo I think.

@damienleger Thanks, I've edit the post.

[dpankros]

I'm just adding this in case someone else hits the same issue as me. In my case, we use Hasicorp Consul to secure our k8s mesh. As part of that process, I had used to the consul.hashicorp.com/transparent-proxy-exclude-inbound-port annotation on Nginx to exclude inbound traffic but encrypt the mesh traffic, but I did not include 8443, which the admission webhook uses. When the webhook tried to access the controller via the admission service it failed because the port was not excluded and was therefore hitting the envoy sidecar (serving up tls for the service mesh). When I added 8443 to the exclude inbound port annotation, the error went away.

[tontondematt]

Hi

I've having the same problem. And based in the comment

My fix for this issue without deleting the validatingwebhookconfigurations. Inspired by #5968 (comment)

CA=$(kubectl -n ingress-nginx get secret ingress-nginx-admission -ojsonpath='{.data.ca}')
kubectl patch validatingwebhookconfigurations ingress-nginx-admission --type='json' -p='[{"op": "add", "path": "/webhooks/0/clientConfig/caBundle", "value":"'$CA'"}]'

I've been investigating. I don't use Helm, and I've had to configure each component individually.

There're two Jobs for nginx-admission-webhooks, create and patch. And Based in the Image's documentation we have a command line with their especifications:

https://github.com/jet/kube-webhook-certgen#patch

  --patch-validating              If true, patch validatingwebhookconfiguration (default true)

Doing that I've got a correct validatingwebhookconfiguration config.

I hope that help you.

can you please detail what you have done

[mrnonz]

Should we reach a conclusion about the root cause? and how can this be avoided in the future?

[AnthonyWC]

Ran into this issue as well; in my case I deployed multiple ingress controllers in different namespaces and created 2 different ValidatingWebhookConfiguration (as they are non name-spaced cluster wide).

What fixed it (for me) was deleting both ValidatingWebhookConfiguration and then re-apply CA patches for each one.

[hnikt-jonasfh]

We also ran into this issue when reinstalling the nginx ingress via a customized Helm chart. The ingress deployment went fine, but any ingress object would get the x509 signed by unknown certificate authority error, and the validationwebhookconfiguration had no caBundle prior to fixing.
The patching solution as explained by @damienleger fixed this for us, although we do have to run a replace sync for the ingresses that failed due to our configuration of Argo CD, but that's not related to the issue discussed here. I also wonder why this issue is closed when it is still a live issue.

Our customized Helm chart uses nginx ingress helm chart version 4.4.x, according to the dependencies. Controller image tagged at version 1.5.1.

[dbeltman]

For anyone banging their heads against this:

TLDR: Metallb/Loadbalancer was not running/working correctly causing the service not to be available. TLS/x509errors threw me on a wild goose chase.

My only indication was that the service created by nginx-ingress was not put in a ready state ( nothing apparent except ArgoCD saying it wasnt ).
When investigating i saw that i had a config error in my metallb chart values. After fixing that the service finally became available, all errors disappeared and i could create ingresses again.
(be sure to terminate sync-action in flux/argo if applicable and retry sync)

[Jeansen]

My fix for this issue without deleting the validatingwebhookconfigurations. Inspired by #5968 (comment)

CA=$(kubectl -n ingress-nginx get secret ingress-nginx-admission -ojsonpath='{.data.ca}')
kubectl patch validatingwebhookconfigurations ingress-nginx-admission --type='json' -p='[{"op": "add", "path": "/webhooks/0/clientConfig/caBundle", "value":"'$CA'"}]'

I faced the same issue after having kille the Ingress controller pod. When it got recreated a deployed Jaeger operator complaind not being able to call the ingress-nginx-controller-admission webhook on port 443. After I applied the patch, all was fine again.

[leewoobin789]

this seems to occur even in the latest version.
is there any planned prevention mechanism for the race condition that ValidatingWebhookConfiguration is missing the CA?
patching ValidatingWebhookConfiguration manually sounds like a quick fix, not a promising solution.

[micheljung]

Also facing this issue with 4.8.4. Can this be reopened?