Pod looses network connection (connection refused errors) during graceful shutdown period

[nilesh-telang]

What happened?

Hi,

Our backend infrastructure uses Kubernetes pods to perform analysis of data. As the flow of data increases and decreases through the day, the Kubernetes auto scaler scales up and scales down the pods.

The analysis operation on a single workload can take up to six minutes. To enable successful completion of in flight scan operation, the pods

1. have a terminationGracePeriodSeconds set to 360 seconds
2. capture the SIGTERM event and prevent the pod from accepting new requests

           SpringApplication springApplication = new SpringApplication(ProductApplication.class);
           springApplication.addListeners(new GracefulShutdownListener());
           
           GracefulShutdownListener -> onApplicationEvent(ContextClosedEvent event) 
           1. Prevent new requests from being sent
           2. Sleep for 6 minutes
   

The issue is that thought the pods are waiting for 6 minutes to complete in flight operation, these operation fail in outward network communication. Thus the pod gets a Connection Refused error when communicating with SNS in the graceful shutdown phase.

Investigation reveal that pods are failing in external network communication with multiple applications during the 'graceful shutdown' phase.

I have gone through multiple tickets /documents in a similar area

1. Network becomes unavailable on terminating pods #44956
2. Graceful terminations within Kubernetes ardanlabs/service#189
3. https://freecontent.manning.com/handling-client-requests-properly-with-kubernetes/

One comment on the same area seems to be #86280 (comment)

I am raising this ticket as we require the ability to create new external connections (SNS) during graceful shutdown.

Primarily I see in the document https://freecontent.manning.com/handling-client-requests-properly-with-kubernetes/, that a B flow is started on pod deletion, which removed pods from ip tables. It seems to me that this may result in the pod not being able to create new network connections. If this is the case we would need a mechanism to delay the B flow until the grace period is done.

I am not sure if sleeping in the pre-stop hook is the recommended mechanism to prevent pods from loosing network connection during the graceful shutdown phase.

What did you expect to happen?

Any Kubernetes pod during it's graceful shutdown period should have unrestricted access to required resources (network) and the ability to create new connections. This ability need not be by default and could also be by setting a configuration.

How can we reproduce it (as minimally and precisely as possible)?

1. Start a pod which perpetually creates a new connection with and updates an external entity
2. Terminate the pod manually

Connection refused should be seen in attempts to create new connection
Connection refused error may be seen for readiness probe.

We have not found this to always be reproducible in our test clusters, however can be seen very frequently in our clusters server continuous data.

Anything else we need to know?

The evidences that we can see indicating that pod is loosing ability to communicate with external components

1. Connection refused error to multiple components, during graceful shut down phase.
2. pod events display connectino refused event for readiness proble a minute after shutdown

38m         Normal    Killing             pod/scan-434f242-fr23e   Stopping container xyz
38m         Normal    Killing             pod/scan-434f242-fr23e   Stopping container pqr
38m         Normal    Killing             pod/scan-434f242-fr23e   Stopping container abc
38m         Warning   FailedPreStopHook   pod/s434f242-fr23e   Exec lifecycle hook ([]) for Container "abc" in Pod "scan-434f242-fr23e_dss(19451a4f-2w32-4221-we32-4a3b0b169a7c)" failed - error: command '' exited with 126: , message: "OCI runtime exec failed: exec failed: container_linux.go:370: starting container process caused: exec: \"\": executable file not found in $PATH: unknown\r\n"
37m         Warning   Unhealthy           pod/scan-434f242-fr23e   Readiness probe failed: Get http://10.111.2.71:8080/actuator/health/readiness: dial tcp 10.203.9.71:8080: connect: connection refused
33m         Warning   Unhealthy           pod/scan-434f242-fr23e   Readiness probe failed: Get http://10.111.2.71:4191/ready: dial tcp 10.203.9.71:4191: connect: connection refused
38m         Warning   Unhealthy           pod/scan-434f242-fr23e   Liveness probe failed: Get http://10.112.2.71:4191/live: dial tcp 10.111.2.71:4191: connect: connection refused
38m         Warning   Unhealthy           pod/scan-434f242-fr23e   Liveness probe failed: Get http://10.111.2.71:8080/actuator/health/liveness: dial tcp 10.203.9.71:8080: connect: connection refused```

This does not always reproduce.

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"}

Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.17-eks-087e67", GitCommit:"087e67e479962798594218dc6d99923f410c145e", GitTreeState:"clean", BuildDate:"2021-07-31T01:39:55Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

Details

AWS

OS version

Details

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# cat /etc/os-release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

Install tools

Details

Container runtime (CRI) and and version (if applicable)

Details

Docker container

Related plugins (CNI, CSI, ...) and versions (if applicable)

Details

[k8s-ci-robot]

@nilesh-telang: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[nilesh-telang]

/sig network

[aojea]

Any Kubernetes pod during it's graceful shutdown period should have unrestricted access to required resources (network) and the ability to create new connections. This ability need not be by default and could also be by setting a configuration.

pods can create connections during shutdown period

a different topic is that if you are trying to connect to a service, the endpoints that are termination are removed from the service, so all the connections to that service will go to the endpoints that are ready, if there are no endpoints those connections will be refused

/kind support

[nilesh-telang]

Thank you @aojea for the clarification and the prompt response.

It is great to know that by design, pods are permitted to create new network connections during the grace period in shut down, as it is important for us to be able to complete tasks (add SNS notifications) that the pod has already started on, during the grace period.

As we are seeing connections to AWS SNS fail (ConnectionRefused) during shutdown, I have added checks for network connectivity of the Pod with multiple components (Kafka, Redis etc) whenever the issue reproduces. This may help us isolate why specifically SNS connections fail during shutdown.

We have an SNS client created using AmazonSNSClientBuilder, as a Bean during start up. The client publishes messages as expected through the pod's lifetime, however for a reason not know yet, the attempts start failing with 'Connection Refused' during the pod's graceful shutdown phase. As this is a standard SNS client, I am assuming that end points to communicate with AWS SNS, exist during the graceful shutdown phase.

I have a few follow up questions please that may help us isolate the problem that we are facing.

1. Command / API in java to verify network health

Could you please let me know if there is any kubernetes command / API that we can invoke from JAVA to verify that pod's entries are present in the ip-tables and/ or verify the pod's networking state in kubernetes. The intent is that for us to identify and log the state of the pod (specific to networking) when the issue reproduces.

2. Request for sharing on sequence with which pods entries are removed form ip tables, when grace period is specified.

I think I may have misunderstood some details shared in https://freecontent.manning.com/handling-client-requests-properly-with-kubernetes/. The document specifies that the pods entries are removed from the ip tables, in a separate asynchronous flow as part of pod shutdown. I wanted to confirm that it is accurate to assume that the steps to remove entries from the ip tables will be delayed for the duration of the pod's grace period .i.e if the pod has a grace period of 6 minutes, the pods entries in the ip table will remain for 6 minutes, post the shutdown initiation.

[aojea]

It is great to know that by design, pods are permitted to create new network connections during the grace period in shut down, as it is important for us to be able to complete tasks (add SNS notifications) that the pod has already started on, during the grace period.

AFAIK they are able to create connection until the network sandbox of the pod is removed, and that is the last thing that happens in the shutdown cycle, but sig-node people will now better

Could you please let me know if there is any kubernetes command / API that we can invoke from JAVA to verify that pod's entries are present in the ip-tables and/ or verify the pod's networking state in kubernetes.

you can exec in the kube-proxy pod or in the node and dump iptables, but that is not network healthz, there can be hundred of things that impact the network

I wanted to confirm that it is accurate to assume that the steps to remove entries from the ip tables will be delayed for the duration of the pod's grace period .i.e if the pod has a grace period of 6 minutes, the pods entries in the ip table will remain for 6 minutes, post the shutdown initiation.

nope, is just the opposite, the entries for the endpoints will be removed as soon as the pod disappears from the endpoint object, the behavior is that "existing" TCP connections to the Service are not cleared.

[nilesh-telang]

Thank you @aojea again for the clarification and the prompt response.

I think your last response clarifies the root cause of what we are observing. You have mentioned that endpoints will be removed as soon as the pod disappears from the endpoint object, the behavior is that "existing" TCP connections to the Service are not cleared.

I believe that this explains what we are observing. During the pod shutdown, I believe the pod is removed from the endpoint object. At this point the pod does not have an active TCP connection with SNS, and thus the connection with SNS is not retained. The fact that pod is removed from the end point may be the reason, that pod is unable to make further communications with SNS.

Retaining the connection with SNS as a keep alive TCP connection may the required route for enabling SNS connections during the shutdown phase. I have also added code to create a new SNSClient when the failure is encountered, to verify if creating a new connection via a new client addresses the issue. I will be running the tests in the coming few days and will udpate the ticket with the results. This will verify the ability to create a new SNS connection during the grace period.

[dcbw]

@nilesh-telang what CNI/network plugin are you using in the cluster?

Also, kubelet and docker logs showing the time sequence of when the container sandbox stop request begins and when it finally ends might help. There are also CRI operation timeouts that could be in play here, but kubelet and docker logs will help figure that out.

[caseydavenport]

If you're using Calico for policy with a non-Calico CNI plugin (e.g., AWS VPC CNI or Azure CNI), there was a known issue that has already been fixed in v3.17.5 and onwards which manifested similar symptoms: projectcalico/calico#4518

This might not be a Kubernetes issue.