Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s: context deadline exceeded #5583

Closed
luppie opened this issue May 20, 2020 · 23 comments
Closed

Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s: context deadline exceeded #5583

luppie opened this issue May 20, 2020 · 23 comments
Assignees
@longwuyuan
Labels
kind/support Categorizes issue or PR as a support question.

Comments

@luppie
Copy link

luppie commented May 20, 2020 •
edited

Hi!

I've installed kubernetes cluster on three servers: l000d01kms001 (K8s master) and l000d01ksl001 & l000d01ksl001 - all basen Centos 7.5

Installed software:

docker-ce-cli-19.03.6-3.el7.x86_64
docker-ce-19.03.6-3.el7.x86_64
kubelet-1.17.3-0.x86_64
kubernetes-cni-0.7.5-0.x86_64
kubeadm-1.17.3-0.x86_64
kubectl-1.17.3-0.x86_64`

I've installed MetalLB (https://metallb.universe.tf) and then deployed Ingress-Nginx from URL https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-0.32.0/deploy/static/provider/baremetal/deploy.yaml

When trying to deploy endpoint.yaml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: test-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:

  • http:
    paths:
    • path: /testpath
      pathType: Prefix
      backend:
      serviceName: hellok8s-service
      servicePort: 8080

I'm getting error as below:

[root@l000d01kms001 KubeHelm]# kubectl apply -f endpoint.yaml --validate=false
Error from server (InternalError): error when creating "endpoint.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s: context deadline exceeded

`[root@l000d01kms001 KubeHelm]# kubectl logs pod/ingress-nginx-admission-create-44q78 -n ingress-nginx
^C
[root@l000d01kms001 KubeHelm]# kubectl logs pod/ingress-nginx-admission-patch-gqwbr -n ingress-nginx
{"level":"info","msg":"patching webhook configurations 'ingress-nginx-admission' mutating=false, validating=true, failurePolicy=Fail","source":"k8s/k8s.go:38","time":"2020-05-20T20:29:52Z"}
{"level":"info","msg":"Patched hook(s)","source":"k8s/k8s.go:91","time":"2020-05-20T20:29:52Z"}
[root@l000d01kms001 KubeHelm]# kubectl logs pod/ingress-nginx-controller-f8d756996-rhmjl -n ingress-nginx

NGINX Ingress controller
Release: 0.32.0
Build: git-446845114
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.17.10

I0520 20:29:56.807345 6 flags.go:204] Watching for Ingress class: nginx
W0520 20:29:56.807691 6 flags.go:249] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
W0520 20:29:56.807752 6 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0520 20:29:56.807947 6 main.go:220] Creating API client for https://10.96.0.1:443
I0520 20:29:56.815151 6 main.go:264] Running in Kubernetes cluster version v1.17 (v1.17.3) - git (clean) commit 06ad960bfd03b39c8310aaf92d1e7c12ce618213 - platform linux/amd64
I0520 20:29:57.013592 6 main.go:105] SSL fake certificate created /etc/ingress-controller/ssl/default-fake-certificate.pem
I0520 20:29:57.018776 6 ssl.go:528] loading tls certificate from certificate path /usr/local/certificates/cert and key path /usr/local/certificates/key
I0520 20:29:57.043694 6 nginx.go:263] Starting NGINX Ingress controller
I0520 20:29:57.047238 6 event.go:278] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"a204e9f1-6905-44c8-9fd0-e0aae4355e46", APIVersion:"v1", ResourceVersion:"2794479", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0520 20:29:58.244113 6 nginx.go:307] Starting NGINX process
I0520 20:29:58.244155 6 leaderelection.go:242] attempting to acquire leader lease ingress-nginx/ingress-controller-leader-nginx...
I0520 20:29:58.244567 6 nginx.go:327] Starting validation webhook on :8443 with keys /usr/local/certificates/cert /usr/local/certificates/key
I0520 20:29:58.244740 6 controller.go:139] Configuration changes detected, backend reload required.
I0520 20:29:58.248112 6 leaderelection.go:252] successfully acquired lease ingress-nginx/ingress-controller-leader-nginx
I0520 20:29:58.248192 6 status.go:86] new leader elected: ingress-nginx-controller-f8d756996-rhmjl
I0520 20:29:58.306048 6 controller.go:155] Backend successfully reloaded.
I0520 20:29:58.306087 6 controller.go:164] Initial sync, sleeping for 1 second.
[root@l000d01kms001 KubeHelm]#`
@luppie luppie added the kind/bug Categorizes issue or PR as related to a bug. label May 20, 2020
@aledbf
Copy link
Member

aledbf commented May 20, 2020

@luppie when you deploy the ingress controller for the first time two Jobs are executed in order to generate the SSL certificate for the validation webhook. For this reason, it can take up to 90 seconds the start of the controller. After you start you can run the command

kubectl wait --namespace ingress-nginx \
  --for=condition=ready pod \
  --selector=app.kubernetes.io/component=controller \
  --timeout=90s

when that finish the ingress controller webhook is ready and you should not see that error

@aledbf aledbf added kind/support Categorizes issue or PR as a support question. and removed kind/bug Categorizes issue or PR as related to a bug. labels May 20, 2020
@luppie
Copy link
Author

luppie commented May 21, 2020

@aledbf - this command returns
`[root@l000d01kms001 KubeHelm]# kubectl wait --namespace ingress-nginx \

--for=condition=ready pod
--selector=app.kubernetes.io/component=controller
--timeout=90s
pod/ingress-nginx-controller-f8d756996-rhmjl condition met
[root@l000d01kms001 KubeHelm]#`

Error still occurs
[root@l000d01kms001 KubeHelm]# kubectl apply -f endpoint.yaml --validate=false Error from server (InternalError): error when creating "endpoint.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s: context deadline exceeded [root@l000d01kms001 KubeHelm]#

@aledbf
Copy link
Member

aledbf commented May 21, 2020

this command returns
Error still occurs

Ok, that is strange.

pathType: Prefix

Checking the example and the versions of the packages you used, the only "error" I see in the field pathType. That field is available since k8s v1.18.

@aledbf
Copy link
Member

aledbf commented May 21, 2020 •
edited

@luppie I cannot reproduce this issue

kind create cluster --image=kindest/node:v1.17.5@sha256:ab3f9e6ec5ad8840eeb1f76c89bb7948c77bbf76bcebe1a8b59790b8ae9a283a

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-0.32.0/deploy/static/provider/baremetal/deploy.yaml

kubectl wait --namespace ingress-nginx \
  --for=condition=ready pod \
  --selector=app.kubernetes.io/component=controller \
  --timeout=90s
  
echo "
 apiVersion: networking.k8s.io/v1beta1
 kind: Ingress
 metadata:
   name: test-ingress
   annotations:
     nginx.ingress.kubernetes.io/rewrite-target: /
 spec:
   rules:
   - http:
       paths:
       - path: /testpath
         pathType: Prefix
         backend:
           serviceName: hellok8s-service
           servicePort: 8080
" | kubectl apply -f -

error: error validating "STDIN": error validating data: ValidationError(Ingress.spec.rules[0].http.paths[0]): unknown field "pathType" in io.k8s.api.networking.v1beta1.HTTPIngressPath; if you choose to ignore these errors, turn validation off with --validate=false

In fact, the error I see is the one I mentioned in my previous comment

Removing the line pathType: Prefix I get
ingress.networking.k8s.io/test-ingress created

@luppie luppie closed this as completed May 22, 2020
@luppie
Copy link
Author

luppie commented May 22, 2020

I've reinstalled Kubernetes with Calico network plugin and problem doesn't occur.

@adamxy
Copy link

adamxy commented Jun 23, 2020

kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission

@thirumurthis
Copy link

I got the same error:

Error from server (InternalError): error when creating "ingressresource.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

after deleting the kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission now the ingress rule got created.

@marcboivin
Copy link

@adamxy's solution worked for me to, but why? What was wrong in my config? I'm also using MetalLB. Will this be a recurrent problem?

Regards,

@boxcee
Copy link

boxcee commented Jul 16, 2020 •
edited

Yeah, seems like an odd approach to just delete something.

We are using a private cluster on GKE. Not sure if related.

You can also disable the webhook: controller.admissionWebhooks.enabled.

I am still curious why this doesn't work.

@msgongora
Copy link

@boxcee explanation here: kubernetes/kubernetes/issues/79739

@hunzaboy
Copy link

Run the ingress controller again worked for me.

Related issue : ( might help others )
I also had the issue of not able to access the localhost. So I reset Kubernetes from Docker Dashboard and then it worked.

@Gaadek
Copy link

Gaadek commented Jan 14, 2021

Hello,

For information, I'm (totally) new at using K8S so I apologize if I'm wrong.

I had the same error message when I setup my cluster few days ago. My configuration was not exactly the same, I setup my cluster on a IaaS (Scaleway to not mention it)

During the setup, I realized that I can select a CNI which is "Cilium" by default.
After the cluster setup is setup on my cloud provider, I install Flannel, then nginx-ingress.

In this configuration, the definition of an ingress resource fails with the error message Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=10s: context deadline exceeded

After hours of comparisons with a working cluster, I realized that my issue was due to the CNI. Once I defined "Flannel" instead of "Cilium", everything ran as expected.

I cannot explain why the CNI affect the behavior of an ingress, but in my case, this was the root cause

@kfox1111
Copy link

I'm guessing the "CNI" as defined by your cloud provider simply means they provision their infrastructure to have firewall rules appropriate for the specified CNI backend. So by specifying Cilium and then using Flannel, Flannel didn't have the right firewall rules and was cut off from talking amongst its agents. In most cases ingress-nginx relies on the CNI setup to be sane just like any other Kubernetes app.

@prashantkumarpathak
Copy link

prashantkumarpathak commented Aug 26, 2021 •
edited

Hello there,
Need help!!.

I have installed Nginx Ingress Controller (Not the NGINX Plus) from https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/
My Pods are running well Under NameSpace : nginx-ingress

Now when I am running the following yaml file getting error

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: template-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
    - host: learning.ingress
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nginx
                port:
                  number: 30816

Error from server (InternalError): error when creating "ingress_prashant.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "https://ingress-nginx-controller-admission.ingress.svc:443/networking/v1beta1/ingresses?timeout=10s": service "ingress-nginx-controller-admission" not found

IF I remove the pathType: Prefix, then I am getting
The Ingress "template-ingress" is invalid: spec.rules[0].http.paths[0].pathType: Required value: pathType must be specified

Can some one help me why webhook error comes, All youtube videos followed the https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/ and for them it worked well.

@longwuyuan
Copy link
Contributor

longwuyuan commented Aug 26, 2021 via email

@k8s-ci-robot
Copy link
Contributor

@longwuyuan: Those labels are not set on the issue: kind/bug

In response to this:

What command did you use to install.
/assign
/remove-kind bug

Thanks,
; Long

On Thu, 26 Aug, 2021, 9:39 PM Prashant Kumar Pathak, <
@.***> wrote:

Hello there,
Need help!!.

I have installed Nginx Ingress Controller (Not the NGINX Plus) from https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/
https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/
My Pods are running well Under NameSpace : nginx-ingress

Now when I am running the following yaml file getting error
Error from server (InternalError): error when creating
"ingress_prashant.yaml": Internal error occurred: failed calling webhook
"validate.nginx.ingress.kubernetes.io
http://validate.nginx.ingress.kubernetes.io": Post
"https://ingress-nginx-controller-admission.ingress.svc:443/networking/v1beta1/ingresses?timeout=10s
https://ingress-nginx-controller-admission.ingress.svc:443/networking/v1beta1/ingresses?timeout=10s":
service "ingress-nginx-controller-admission" not found

IF I remove the pathType: Prefix, then I am getting
The Ingress "template-ingress" is invalid:
spec.rules[0].http.paths[0].pathType: Required value: pathType must be
specified

Can some one help me why webhook error comes, All youtube videos followed
the
https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/
and for them it worked well.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#5583 (comment),
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABGZVWQ6FMKWSKZAAYI73Y3T6ZRLHANCNFSM4NGJDNGA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email
.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@prashantkumarpathak
Copy link

What command did you use to install. /assign /remove-kind bug Thanks, ; Long
…
On Thu, 26 Aug, 2021, 9:39 PM Prashant Kumar Pathak, < @.***> wrote: Hello there, Need help!!. I have installed Nginx Ingress Controller (Not the NGINX Plus) from https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/ https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/ My Pods are running well Under NameSpace : nginx-ingress Now when I am running the following yaml file getting error Error from server (InternalError): error when creating "ingress_prashant.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io http://validate.nginx.ingress.kubernetes.io": Post "https://ingress-nginx-controller-admission.ingress.svc:443/networking/v1beta1/ingresses?timeout=10s https://ingress-nginx-controller-admission.ingress.svc:443/networking/v1beta1/ingresses?timeout=10s": service "ingress-nginx-controller-admission" not found IF I remove the pathType: Prefix, then I am getting The Ingress "template-ingress" is invalid: spec.rules[0].http.paths[0].pathType: Required value: pathType must be specified Can some one help me why webhook error comes, All youtube videos followed the https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/ and for them it worked well. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#5583 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABGZVWQ6FMKWSKZAAYI73Y3T6ZRLHANCNFSM4NGJDNGA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

Hello long

I followed the steps mentioned here ( https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/) for installing Ingress Controller.

$ git clone https://github.com/nginxinc/kubernetes-ingress/
$ cd kubernetes-ingress/deployments
$ git checkout v1.12.0

$ kubectl apply -f common/ns-and-sa.yaml

$ kubectl apply -f rbac/rbac.yaml

$ kubectl apply -f rbac/ap-rbac.yaml
$ kubectl apply -f common/default-server-secret.yaml
$ kubectl apply -f common/nginx-config.yaml
$ kubectl apply -f common/ingress-class.yaml
$ kubectl apply -f common/crds/k8s.nginx.org_virtualservers.yaml
$ kubectl apply -f common/crds/k8s.nginx.org_virtualserverroutes.yaml
$ kubectl apply -f common/crds/k8s.nginx.org_transportservers.yaml
$ kubectl apply -f common/crds/k8s.nginx.org_policies.yaml
$ kubectl apply -f common/crds/k8s.nginx.org_globalconfigurations.yaml


$ kubectl apply -f daemon-set/nginx-ingress.yaml

@longwuyuan
Copy link
Contributor

longwuyuan commented Aug 26, 2021 via email

@prashantkumarpathak
Copy link

https://kubernetes.github.io/ingress-nginx/

ohh, I see now. Can you please suggest me where to look for webhook and how it's related to Ingress.
Thanks for your help

@Ventsislav86
Copy link

Hello I have a similar problem:

kubectl create -f ingress.yaml
Error from server (InternalError): error when creating "ingress.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1/ingresses?timeout=10s": dial tcp 10.111.201.182:443: connect: no route to host

venci@kubemaster:~$ cat ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-dashboard
namespace: kubernetes-dashboard
spec:
rules:

  • host: dashboard.com
    http:
    paths:
    • pathType: Prefix
      path: "/"
      backend:
      service:
      name: kubernetes-dashboard
      port:
      number: 443
      venci@kubemaster:~$

venci@kubemaster:$ kubectl get svc -n kubernetes-dashboard
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
dashboard-metrics-scraper ClusterIP 10.106.141.132 8000/TCP 42h
kubernetes-dashboard ClusterIP 10.104.175.10 443/TCP 42h
venci@kubemaster:$

venci@kubemaster:$ kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.102.145.209 80:31421/TCP,443:32632/TCP 36h
ingress-nginx-controller-admission ClusterIP 10.111.201.182 443/TCP 36h
venci@kubemaster:$

@longwuyuan
Copy link
Contributor

longwuyuan commented Aug 30, 2021 via email

@matteovivona
Copy link

kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission

be careful that if you delete the ValidatingWebhookConfiguration you risk to break the whole nginx config in case of ingress configuration errors.

@thihathura
Copy link

After I did as the mentioned of @thirumurthis .
It is working as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@longwuyuan longwuyuan

Labels
kind/support Categorizes issue or PR as a support question.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests