k8s1.14 ipvs mode, pod rescheduling，Ipvs cannot remove IP forwarding of old pods

[liupeng0518]

What happened:
In ipvs mode, pod rescheduling sometimes leaves a record:

[root@k8s2 ~]# ipvsadm -ln -t 10.233.51.239:7979
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.233.51.239:7979 rr
  -> 10.233.124.161:7979          Masq    1      0          183       
  -> 10.233.124.171:7979          Masq    0      0          1         
[root@k8s2 ~]# ipvsadm -lnc|grep  10.233.124.171
TCP 00:35  SYN_RECV    10.233.124.160:40001 10.233.51.239:7979 10.233.124.171:7979

IP: 10.233.124.171, This is a remnant of the old pod. When I delete it ，looks normal:

[root@k8s2 ~]# ipvsadm -d -t 10.233.51.239:7979 -r 10.233.124.171:7979

What you expected to happen:
10.233.124.171:7979 auto delete.

How to reproduce it (as minimally and precisely as possible):
pod rescheduling
Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:16Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:16Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
  Bare Metal

• OS (e.g: cat /etc/os-release):
  NAME="CentOS Linux"
  VERSION="7 (Core)"
  ID="centos"
  ID_LIKE="rhel fedora"
  VERSION_ID="7"
  PRETTY_NAME="CentOS Linux 7 (Core)"
  ANSI_COLOR="0;31"
  CPE_NAME="cpe:/o:centos:centos:7"
  HOME_URL="https://www.centos.org/"
  BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

• Kernel (e.g. uname -a):
  Linux k8s1 3.10.0-514.el7.x86_64 Unit test coverage in Kubelet is lousy. (~30%) #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

• Install tools:

• Network plugin and version (if this is a network-related bug):
  calico 3.4.0

• Others:

[gongguan]

Seems fixed by #81482

[athenabot]

/sig network

These SIGs are my best guesses for this issue. Please comment /remove-sig <name> if I am incorrect about one.

🤖 I am a bot run by vllry. 👩‍🔬

[liupeng0518]

Seems fixed by #81482

It looks different, just delete the pod here

[uablrek]

Please note that the "weight" for "10.233.124.171:7979" is 0 (zero). This means that it will not get any new connections but old connections remains and (I assume) will receive a RST if used. This I believe is a part of the "graceful termination" mentioned in #81482.

[liupeng0518]

Please note that the "weight" for "10.233.124.171:7979" is 0 (zero). This means that it will not get any new connections but old connections remains and (I assume) will receive a RST if used. This I believe is a part of the "graceful termination" mentioned in #81482.

OK, thanks..

[thockin]

Is this a bug or should it be closed?

[liupeng0518]

1.14.10 still has this problem.