New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When using IPVS with graceful termination, connections to services with sessionAffinity are not expired #71809

Closed
lbernail opened this Issue Dec 6, 2018 · 0 comments

Comments

Projects
None yet
3 participants
@lbernail
Contributor

lbernail commented Dec 6, 2018

What happened:
After updating the pods backing a service with sessionAffinity set, all communications to the service stopped working.

What you expected to happen:
Communications should continue to work

How to reproduce it (as minimally and precisely as possible):
1- Create a service with sessionAffinity: ClientIP backed by a deployment of a single pod
2- Connect to the service from a pod
3- Delete the pod and wait for it to recreate

When trying to reconnect to the service all packets will be lost until the real server is removed because the persistency remains regardless of weight=0. In addition, if the client keeps retrying, the number of connections to the real server will never reach 0 and it will not be deleted (until the persistency timeout expires, which by default takes 3h)

Anything else we need to know?:
I did some testing and setting net.ipv4.vs.expire_quiescent_template=1 fixes the issue. I'll create a PR tomorrow (including some changes discussed in #71358 which are related)

Environment:

  • Kubernetes version (use kubectl version): kube-proxy v1.13.0-alpha.3
  • OS (e.g. from /etc/os-release): Ubuntu 18.04.1 LTS
  • Kernel (e.g. uname -a): Linux ip-10-128-212-113 4.15.0-1023-aws #23-Ubuntu SMP Mon Sep 24 16:31:06 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

/area ipvs
/sig network
/kind bug

/assign @m1093782566

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment