Controller failed to renew lease (leader election) due to timeout until pod restart #11287
Comments
k8s-ci-robot
added
the
needs-triage
|
This issue is currently awaiting triage. If Ingress contributors determines this is a relevant issue, they will accept it by applying the The Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/remove-kind bug
Any idea why a re-election was triggered /triage needs-information |
k8s-ci-robot
added
triage/needs-information
All there is now is: LAST SEEN TYPE REASON OBJECT MESSAGE
4m25s Normal UpdatedLoadBalancer service/ingress-nginx-controller-internal Updated load balancer with new hosts
4m24s Normal UpdatedLoadBalancer service/ingress-nginx-controller Updated load balancer with new hosts
There was only one controller pod at the time of the incident. I tried looking at host logs for where the pod was scheduled. Between {
"host": "ip-172-32-107-61",
"ident": "rsyslogd",
"message": "imjournal: 182979 messages lost due to rate-limiting",
"az": "eu-west-1a",
"ec2_instance_id": "i-057706eb5156f775d"
}There were some logs from the Perhaps everything just got in to a twist? The controller pod was Running long before and after the time of the incident |
|
It looks like the controller couldn't reach the API server with all the The restart fixed the issue; I'm going to close it for now; if it does happen again, feel free to reopen it. |
|
/close |
|
@strongjz: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
What happened:
We deployed a new version of our application, at the ~same time, the ingress-nginx controller failed to renew a lease in the leader election. The controller was then unable to update the configmap. Our applications stopped receiving traffic.
When examining the logs of the controller, requests were being forwarded to pods that no longer existed.
Issuing the command
kubectl -n ingress-controller rollout restart deployment ingress-nginx-controllerresolved the issue.What you expected to happen:
I would expect the controller to reestablish the lease, without having to recreate the pod. As soon as the pod was recreated, the controller worked as expected.
NGINX Ingress controller version
Kubernetes version
Environment:
Cloud provider or hardware configuration:
EKS 1.29
Install tools:
EKS
Basic cluster related info:
How was the ingress-nginx-controller installed:
We took the 1.9.3 chart and modified for our needs, creating a second loadbalancer service for internal traffic
Controller Deployment Manifest
```yaml apiVersion: apps/v1 kind: Deployment metadata: labels: helm.sh/chart: ingress-nginx-4.8.2 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/version: "1.9.3" app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: controller name: ingress-nginx-controller namespace: ingress-nginx spec: selector: matchLabels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/component: controller replicas: 1 revisionHistoryLimit: 10 minReadySeconds: 0 template: metadata: labels: helm.sh/chart: ingress-nginx-4.8.2 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/version: "1.9.3" app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: controller spec: dnsPolicy: ClusterFirst containers: - name: controller image: "registry.k8s.io/ingress-nginx/controller:v1.9.3@sha256:8fd21d59428507671ce0fb47f818b1d859c92d2ad07bb7c947268d433030ba98" imagePullPolicy: IfNotPresent lifecycle: preStop: exec: command: - /wait-shutdown args: - /nginx-ingress-controller - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller - --election-id=ingress-nginx-leader - --controller-class=k8s.io/ingress-nginx - --ingress-class=nginx - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller - --validating-webhook=:8443 - --validating-webhook-certificate=/usr/local/certificates/cert - --validating-webhook-key=/usr/local/certificates/key - --enable-ssl-passthrough securityContext: capabilities: drop: - ALL add: - NET_BIND_SERVICE runAsUser: 101 allowPrivilegeEscalation: true env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: LD_PRELOAD value: /usr/local/lib/libmimalloc.so livenessProbe: failureThreshold: 5 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 ports: - name: http containerPort: 80 protocol: TCP - name: https containerPort: 443 protocol: TCP - name: metrics containerPort: 10254 protocol: TCP - name: webhook containerPort: 8443 protocol: TCP volumeMounts: - name: webhook-cert mountPath: /usr/local/certificates/ readOnly: true resources: requests: cpu: 100m memory: 90Mi nodeSelector: kubernetes.io/os: linux serviceAccountName: ingress-nginx terminationGracePeriodSeconds: 300 volumes: - name: webhook-cert secret: secretName: ingress-nginx-admission ```kubectl describe ingressclassesName: nginx Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.9.3 helm.sh/chart=ingress-nginx-4.8.2 Annotations: <none> Controller: k8s.io/ingress-nginx Events: <none>kubectl -n <ingresscontrollernamespace> get all -o widekubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>Name: ingress-nginx-controller-696d4ff7c5-f7cgj Namespace: ingress-nginx Priority: 0 Service Account: ingress-nginx Node: ip-172-32-168-225.eu-west-1.compute.internal/172.32.168.225 Start Time: Thu, 18 Apr 2024 19:35:09 +0100 Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.9.3 helm.sh/chart=ingress-nginx-4.8.2 pod-template-hash=696d4ff7c5 Annotations: kubectl.kubernetes.io/restartedAt: 2024-04-18T19:35:09+01:00 Status: Running IP: 172.32.184.183 IPs: IP: 172.32.184.183 Controlled By: ReplicaSet/ingress-nginx-controller-696d4ff7c5 Containers: controller: Container ID: containerd://672b2ac13c8eae21ea4e7003695dbfbf53fe34dfaf6cc16159d1b82492be1571 Image: registry.k8s.io/ingress-nginx/controller:v1.9.3@sha256:8fd21d59428507671ce0fb47f818b1d859c92d2ad07bb7c947268d433030ba98 Image ID: registry.k8s.io/ingress-nginx/controller@sha256:8fd21d59428507671ce0fb47f818b1d859c92d2ad07bb7c947268d433030ba98 Ports: 80/TCP, 443/TCP, 10254/TCP, 8443/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP Args: /nginx-ingress-controller --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller --election-id=ingress-nginx-leader --controller-class=k8s.io/ingress-nginx --ingress-class=nginx --configmap=$(POD_NAMESPACE)/ingress-nginx-controller --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key --enable-ssl-passthrough State: Running Started: Thu, 18 Apr 2024 19:35:16 +0100 Ready: True Restart Count: 0 Requests: cpu: 100m memory: 90Mi Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5 Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: POD_NAME: ingress-nginx-controller-696d4ff7c5-f7cgj (v1:metadata.name) POD_NAMESPACE: ingress-nginx (v1:metadata.namespace) LD_PRELOAD: /usr/local/lib/libmimalloc.so Mounts: /usr/local/certificates/ from webhook-cert (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5fdtk (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: webhook-cert: Type: Secret (a volume populated by a Secret) SecretName: ingress-nginx-admission Optional: false kube-api-access-5fdtk: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: Burstable Node-Selectors: kubernetes.io/os=linux Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: <none>kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>Name: ingress-nginx-controller Namespace: ingress-nginx Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.9.3 helm.sh/chart=ingress-nginx-4.8.2 Annotations: service.beta.kubernetes.io/aws-load-balancer-access-log-enabled: true service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name: xxx service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix: xxx service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: 60 service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: true service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:eu-west-1:xxx:certificate/xxx service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: ELBSecurityPolicy-TLS13-1-2-Res-2021-06 service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=false service.beta.kubernetes.io/aws-load-balancer-type: nlb Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx Type: LoadBalancer IP Family Policy: SingleStack IP Families: IPv4 IP: 10.100.17.246 IPs: 10.100.17.246 LoadBalancer Ingress: adeexxx.elb.eu-west-1.amazonaws.com Port: http 80/TCP TargetPort: http/TCP NodePort: http 30417/TCP Endpoints: 172.32.184.183:80 Port: https 443/TCP TargetPort: http/TCP NodePort: https 30515/TCP Endpoints: 172.32.184.183:80 Session Affinity: None External Traffic Policy: Cluster Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal UpdatedLoadBalancer 2s (x1195 over 6d12h) service-controller Updated load balancer with new hostsHow to reproduce this issue:
Unknown
The text was updated successfully, but these errors were encountered: