Deregistration Delay does not work as expectation when use IP Target-Type Load Balancers

[gaussye]

Describe the bug
A concise description of what the bug is.
According to the EKS best practices document:https://aws.github.io/aws-eks-best-practices/networking/loadbalancing/loadbalancing/

Elastic Load Balancing stops sending requests to targets that are deregistering. By default, Elastic Load Balancing waits 300 seconds before completing the deregistration process, which can help in-flight requests to the target to complete. To change the amount of time that Elastic Load Balancing waits, update the deregistration delay value. The initial state of a deregistering target is draining. After the deregistration delay elapses, the deregistration process completes and the state of the target is unused. If the target is part of an Auto Scaling group, it can be terminated and replaced.

If a deregistering target has no in-flight requests and no active connections, Elastic Load Balancing immediately completes the deregistration process, without waiting for the deregistration delay to elapse.

Requests still send to the terminating pod which is in the status of deregistering in target group. And even though i set deregistering delay to 5 seconds, and the deregistering process takes about 2 minutes

Steps to reproduce

1. Create a nginx deployment with 2 replicas and set pre-stop hook to sleep 150s
   ` containers:

   • image: nginx
     imagePullPolicy: Always
     lifecycle:
     preStop:
     exec:
     command:
     - sh
     - -c
     - echo The app is stopping! && sleep 150
     terminationGracePeriodSeconds: 180`

2. create svc with ip mode, set deregistering delay=5s
   service.beta.kubernetes.io/aws-load-balancer-ip-address-type: dualstack
   service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
   service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
   service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-xxx,subnet-xxx,subnet-xxx
   service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: deregistration_delay.timeout_seconds=5
   service.beta.kubernetes.io/aws-load-balancer-type: external

3. Please note the 2 pods are new created , no traffic and no active connections

4. Execute the testing scripts below:
   a. scale the nginx deployment from 2 to 1, so the pod is in terminating status and target in target goup is deregistering
   b. sleep 10s
   c. access nginx through NLB
   #!/bin/bash
   kubectl scale deployment nginx --replicas=1
   sleep 10
   #./script -worker 10 -count 10000 -url
   while true; do
   echo -n "HTTP response status codes: " >> result.txt
   curl -s -o /dev/null -w "%{http_code}" -m 1 >> result.txt
   date '+ time: %H:%M:%S' >> result.txt
   echo >> result.txt
   sleep 1
   done

5. By checking the log of the pod in terminating status:
   The terminating pod still receives traffics and it lasts around 1 minute(from 08:19:26 to 08:20:29). At 08:21:46 , it receives the signal to shutdown the pod.
   Problems:

6. Since the 2 new pods have no traffic and active connections, when one of the pod goes to terminating status, it should not receive any traffics according to the document. Why i still see the traffics go into the terminating pod.

7. when the pod starts terminating and then test script runs after 10s, but before the scripts running, it exceeds the Deregisteration delay time(5s), NLB targetgroup should complete the deregistering process immediately, and should not receive any traffics. Also it takes more than 2 minutes to finish the deregistering process..

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2023/06/30 08:04:15 [notice] 1#1: using the "epoll" event method
2023/06/30 08:04:15 [notice] 1#1: nginx/1.25.1
2023/06/30 08:04:15 [notice] 1#1: built by gcc 12.2.0 (Debian 12.2.0-14)
2023/06/30 08:04:15 [notice] 1#1: OS: Linux 5.10.179-168.710.amzn2.x86_64
2023/06/30 08:04:15 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2023/06/30 08:04:15 [notice] 1#1: start worker processes
2023/06/30 08:04:15 [notice] 1#1: start worker process 29
2023/06/30 08:04:15 [notice] 1#1: start worker process 30
2023/06/30 08:04:15 [notice] 1#1: start worker process 31
2023/06/30 08:04:15 [notice] 1#1: start worker process 32
2023/06/30 08:04:15 [notice] 1#1: start worker process 33
2023/06/30 08:04:15 [notice] 1#1: start worker process 34
2023/06/30 08:04:15 [notice] 1#1: start worker process 35
2023/06/30 08:04:15 [notice] 1#1: start worker process 36
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:19:26 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:19:29 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:19:34 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:19:35 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:19:38 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:19:40 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:19:41 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:19:43 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:19:44 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:19:45 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:19:46 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:19:47 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:19:48 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:19:50 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:19:51 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:19:52 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:19:54 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:19:55 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:19:58 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:20:07 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:20:08 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:20:09 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:20:10 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:20:12 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:20:13 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:20:14 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:20:16 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:20:17 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:20:19 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:20:20 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:20:21 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:20:22 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:20:24 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:20:26 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:20:27 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2406:da14:f5e:e001::c766 - - [30/Jun/2023:08:20:29 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.0.1" "-"
2023/06/30 08:21:46 [notice] 1#1: signal 3 (SIGQUIT) received, shutting down
2023/06/30 08:21:46 [notice] 30#30: gracefully shutting down
2023/06/30 08:21:46 [notice] 29#29: gracefully shutting down

Expected outcome
A concise description of what you expected to happen.

Environment

• AWS Load Balancer controller version
  v2.5.2
• Kubernetes version
• Using EKS (yes/no), if so version?
  yes, 1.25
  Additional Context:

[gaussye]

anyone can help?

[sosheskaz]

@gaussye This is a long-standing issue which has not been fixed for a few years, and I doubt it will be fixed any time soon.

The original issue (to my knowledge) is this one: #1834

The issue does not seem to apply to ALBs, only NLBs. The main workaround is to "trick" the load balancer into taking your pod out of the rotation by failing health checks, waiting for a bit, then killing the pod. Read my comment on that thread for a detailed explanation of how the workaround works, and there are various starting points for implementations of workarounds in the thread.

[oliviassss]

@gaussye, sorry for the late reply.
By reading through the previous issues and the eks best practice, I think the limitation is from NLB side, from the aws load balancer controller side we could not do much to improve that, unfortunately.
We suggest you reach out to AWS support and cut a ticket to NLB team directly for escalation. Or you could try the wordarounds in the issue #1834 to see if it mitigates.
Closing for duplication.