Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

1.13.1 kube-proxy ipvs don't work when pod ip is changed #72270

Closed
caoban opened this issue Dec 21, 2018 · 8 comments
Closed

1.13.1 kube-proxy ipvs don't work when pod ip is changed #72270

caoban opened this issue Dec 21, 2018 · 8 comments
Labels
area/ipvs kind/bug Categorizes issue or PR as related to a bug. kind/support Categorizes issue or PR as a support question. sig/network Categorizes an issue or PR as relevant to SIG Network. triage/needs-information Indicates an issue needs more information in order to work on it.

Comments

@caoban
Copy link

caoban commented Dec 21, 2018 •
edited

What happened:

In k8s 1.13.1 version。When a Pod is changed, the kube-proxy log(info) show that the LVS rs IP is changed. but using 'ipvsadm -Ln', it's unccorrectly. it's only Part of the node will have the situation
sometimes.
the bad node in kube-proxy log(info) don't have graceful_termination.go:160] Trying to delete rs xxxxxx and graceful_termination.go:173] Deleting rs xxxxx

What you expected to happen:

The nodeport on the corresponding node is inaccessible

How to reproduce it (as minimally and precisely as possible):

in 1.13.1 version，and then update one deployment . use ipvsadm -Ln check nodeport

Anything else we need to know?:

Environment:

  • Kubernetes version (use 1.13.1):
  • Cloud provider or hardware configuration:
  • OS (CentOS Linux):
  • Kernel (3.10.0-514.el7.x86_64 ):
  • Install tools: ipvsadm-1.27-7.el7.x86_64
  • Others:

/kind bug

kube-proxy
@k8s-ci-robot k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Dec 21, 2018
@caoban caoban changed the title 1.13.1 kube-proxy ipvs don't work when pod ip is Changed 1.13.1 kube-proxy ipvs don't work when pod ip is changed Dec 21, 2018
@liggitt
Copy link
Member

liggitt commented Dec 21, 2018

/sig network

@k8s-ci-robot k8s-ci-robot added sig/network Categorizes an issue or PR as relevant to SIG Network. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Dec 21, 2018
@liggitt liggitt added kind/support Categorizes issue or PR as a support question. triage/needs-information Indicates an issue needs more information in order to work on it. labels Dec 23, 2018
@xuezm01
Copy link

xuezm01 commented Jan 21, 2019 •
edited

I have the same problem，Is there a solution that can be solved?
Always try to delete, but the deletion is not successful 。
log:
I0121 03:22:14.390838 1 graceful_termination.go:173] Deleting rs: 10.122.43.247:80/TCP/10.121.254.11:80
I0121 03:22:35.345897 1 graceful_termination.go:160] Trying to delete rs: 10.122.43.247:80/TCP/10.121.48.136:80
I0121 03:22:35.345939 1 graceful_termination.go:173] Deleting rs: 10.122.43.247:80/TCP/10.121.48.136:80
I0121 03:22:36.087533 1 graceful_termination.go:160] Trying to delete rs: 10.122.43.247:80/TCP/10.121.18.140:80
I0121 03:22:48.912235 1 graceful_termination.go:160] Trying to delete rs: 10.122.43.247:80/TCP/10.121.113.71:80
I0121 03:23:00.059719 1 graceful_termination.go:160] Trying to delete rs: 10.122.43.247:80/TCP/10.121.113.71:80
I0121 03:23:00.059881 1 graceful_termination.go:160] Trying to delete rs: 10.122.43.247:80/TCP/10.121.18.140:80
I0121 03:24:00.060147 1 graceful_termination.go:160] Trying to delete rs: 10.122.43.247:80/TCP/10.121.113.71:80
I0121 03:24:00.060298 1 graceful_termination.go:160] Trying to delete rs: 10.122.43.247:80/TCP/10.121.18.140:80
I0121 03:25:00.060508 1 graceful_termination.go:160] Trying to delete rs: 10.122.43.247:80/TCP/10.121.113.71:80
I0121 03:25:00.060641 1 graceful_termination.go:173] Deleting rs: 10.122.43.247:80/TCP/10.121.113.71:80
I0121 03:25:00.060680 1 graceful_termination.go:93] lw: remote out of the list: 10.122.43.247:80/TCP/10.121.113.71:80
I0121 03:25:00.060694 1 graceful_termination.go:160] Trying to delete rs: 10.122.43.247:80/TCP/10.121.18.140:80
I0121 03:25:00.060737 1 graceful_termination.go:173] Deleting rs: 10.122.43.247:80/TCP/10.121.18.140:80
I0121 03:25:00.060776 1 graceful_termination.go:93] lw: remote out of the list: 10.122.43.247:80/TCP/10.121.18.140:80
I0121 03:27:45.592526 1 graceful_termination.go:160] Trying to delete rs: 10.122.97.108:80/TCP/10.121.247.38:8080
I0121 03:27:45.592572 1 graceful_termination.go:173] Deleting rs: 10.122.97.108:80/TCP/10.121.247.38:8080
I0121 03:27:51.396588 1 graceful_termination.go:160] Trying to delete rs: 10.122.139.70:80/TCP/10.121.227.36:80
I0121 03:27:51.396644 1 graceful_termination.go:173] Deleting rs: 10.122.139.70:80/TCP/10.121.227.36:80
I0121 03:28:22.453703 1 graceful_termination.go:160] Trying to delete rs: 10.122.97.108:80/TCP/10.121.82.160:8080
I0121 03:28:22.453759 1 graceful_termination.go:173] Deleting rs: 10.122.97.108:80/TCP/10.121.82.160:8080
I0121 03:28:57.593600 1 graceful_termination.go:160] Trying to delete rs: 10.122.97.108:80/TCP/10.121.129.95:8080
I0121 03:28:57.593645 1 graceful_termination.go:173] Deleting rs: 10.122.97.108:80/TCP/10.121.129.95:8080
I0121 03:35:02.831449 1 graceful_termination.go:160] Trying to delete rs: 10.122.192.195:80/TCP/10.121.153.182:80
I0121 03:35:02.831493 1 graceful_termination.go:173] Deleting rs: 10.122.192.195:80/TCP/10.121.153.182:80
I0121 03:35:18.570425 1 graceful_termination.go:160] Trying to delete rs: 10.122.192.195:80/TCP/10.121.124.218:80
I0121 03:35:18.570471 1 graceful_termination.go:173] Deleting rs: 10.122.192.195:80/TCP/10.121.124.218:80
I0121 03:51:18.831720 1 graceful_termination.go:160] Trying to delete rs: 10.122.131.116:80/TCP/10.121.153.181:80
I0121 03:51:18.831761 1 graceful_termination.go:173] Deleting rs: 10.122.131.116:80/TCP/10.121.153.181:80
I0121 03:53:47.646628 1 graceful_termination.go:160] Trying to delete rs: 10.122.193.188:80/TCP/10.121.200.30:80
I0121 03:54:00.064799 1 graceful_termination.go:160] Trying to delete rs: 10.122.193.188:80/TCP/10.121.200.30:80
I0121 03:54:29.980431 1 graceful_termination.go:160] Trying to delete rs: 10.122.193.188:80/TCP/10.121.232.167:80
I0121 03:55:00.065143 1 graceful_termination.go:160] Trying to delete rs: 10.122.193.188:80/TCP/10.121.200.30:80
I0121 03:55:00.065330 1 graceful_termination.go:160] Trying to delete rs: 10.122.193.188:80/TCP/10.121.232.167:80
I0121 03:55:18.929524 1 graceful_termination.go:160] Trying to delete rs: 10.122.193.188:80/TCP/10.121.169.77:80
I0121 03:56:00.065608 1 graceful_termination.go:160] Trying to delete rs: 10.122.193.188:80/TCP/10.121.232.167:80

@2PacIsAlive
Copy link

might be related to #71071, if so v1.13.2 has the fix

@lbernail
Copy link
Contributor

lbernail commented Mar 5, 2019

Yes there is definitely a problem with v1.13.1
If the issue persists with v1.13.2 let us know

@thockin
Copy link
Member

thockin commented Mar 8, 2019

One better - if this still exists, please reopen

@thockin thockin closed this as completed Mar 8, 2019
@shimron
Copy link

shimron commented Oct 21, 2019

having the same problem in v1.13.8, logs:

I1021 08:29:22.907423 1 graceful_termination.go:94] lw: remote out of the list: 11.233.32.84:6666/TCP/11.233.86.120:6666 I1021 08:29:22.907428 1 graceful_termination.go:161] Trying to delete rs: 11.233.32.84:6666/TCP/11.233.86.81:6666 I1021 08:29:22.907596 1 graceful_termination.go:175] Deleting rs: 11.233.32.84:6666/TCP/11.233.86.81:6666 I1021 08:29:22.907615 1 graceful_termination.go:94] lw: remote out of the list: 11.233.32.84:6666/TCP/11.233.86.81:6666 I1021 08:29:22.907621 1 graceful_termination.go:161] Trying to delete rs: 11.233.32.84:6666/TCP/11.233.80.10:6666 I1021 08:29:22.907774 1 graceful_termination.go:175] Deleting rs: 11.233.32.84:6666/TCP/11.233.80.10:6666 I1021 08:29:22.907787 1 graceful_termination.go:94] lw: remote out of the list: 11.233.32.84:6666/TCP/11.233.80.10:6666 I1021 08:29:22.907792 1 graceful_termination.go:161] Trying to delete rs: 11.233.32.84:6666/TCP/11.233.90.157:6666 I1021 08:29:22.908107 1 graceful_termination.go:175] Deleting rs: 11.233.32.84:6666/TCP/11.233.90.157:6666 I1021 08:29:22.908162 1 graceful_termination.go:94] lw: remote out of the list: 11.233.32.84:6666/TCP/11.233.90.157:6666 I1021 08:29:22.908183 1 graceful_termination.go:161] Trying to delete rs: 11.233.32.84:6666/TCP/11.233.92.228:6666 I1021 08:29:22.908505 1 graceful_termination.go:175] Deleting rs: 11.233.32.84:6666/TCP/11.233.92.228:6666 I1021 08:29:22.908541 1 graceful_termination.go:94] lw: remote out of the list: 11.233.32.84:6666/TCP/11.233.92.228:6666 I1021 08:29:22.908551 1 graceful_termination.go:161] Trying to delete rs: 11.233.32.84:6666/TCP/11.233.115.122:6666 I1021 08:29:22.908677 1 graceful_termination.go:175] Deleting rs: 11.233.32.84:6666/TCP/11.233.115.122:6666 I1021 08:29:22.908696 1 graceful_termination.go:94] lw: remote out of the list: 11.233.32.84:6666/TCP/11.233.115.122:6666 I1021 08:29:22.908703 1 graceful_termination.go:161] Trying to delete rs: 11.233.32.84:6666/TCP/11.233.93.178:6666 I1021 08:29:22.908808 1 graceful_termination.go:175] Deleting rs: 11.233.32.84:6666/TCP/11.233.93.178:6666 I1021 08:29:22.909155 1 graceful_termination.go:94] lw: remote out of the list: 11.233.32.84:6666/TCP/11.233.93.178:6666 I1021 08:29:22.909180 1 graceful_termination.go:161] Trying to delete rs: 11.233.32.84:6666/TCP/11.233.82.7:6666 I1021 08:29:22.909294 1 graceful_termination.go:175] Deleting rs: 11.233.32.84:6666/TCP/11.233.82.7:6666 I1021 08:29:22.909331 1 graceful_termination.go:94] lw: remote out of the list: 11.233.32.84:6666/TCP/11.233.82.7:6666 I1021 08:29:22.909357 1 graceful_termination.go:161] Trying to delete rs: 11.233.32.84:6666/TCP/11.233.102.33:6666 I1021 08:29:22.909501 1 graceful_termination.go:175] Deleting rs: 11.233.32.84:6666/TCP/11.233.102.33:6666 I1021 08:29:22.910176 1 graceful_termination.go:94] lw: remote out of the list: 11.233.32.84:6666/TCP/11.233.102.33:6666 I1021 08:29:22.910185 1 graceful_termination.go:161] Trying to delete rs: 11.233.32.84:6666/TCP/11.233.86.77:6666 I1021 08:29:22.910297 1 graceful_termination.go:175] Deleting rs: 11.233.32.84:6666/TCP/11.233.86.77:6666 I1021 08:29:22.910311 1 graceful_termination.go:94] lw: remote out of the list: 11.233.32.84:6666/TCP/11.233.86.77:6666 I1021 08:29:22.910316 1 graceful_termination.go:161] Trying to delete rs: 11.233.32.84:6666/TCP/11.233.90.152:6666 I1021 08:29:22.910401 1 graceful_termination.go:172] Not deleting, RS 11.233.32.84:6666/TCP/11.233.90.152:6666: 0 ActiveConn, 1 InactiveConn I1021 08:29:22.910409 1 graceful_termination.go:161] Trying to delete rs: 11.233.32.84:6666/TCP/11.233.106.75:6666 I1021 08:29:22.910523 1 graceful_termination.go:175] Deleting rs: 11.233.32.84:6666/TCP/11.233.106.75:6666 I1021 08:29:22.910544 1 graceful_termination.go:94] lw: remote out of the list: 11.233.32.84:6666/TCP/11.233.106.75:6666 I1021 08:29:22.910550 1 graceful_termination.go:161] Trying to delete rs: 11.233.32.84:6666/TCP/11.233.115.102:6666 I1021 08:29:22.910640 1 graceful_termination.go:175] Deleting rs: 11.233.32.84:6666/TCP/11.233.115.102:6666 I1021 08:29:22.910656 1 graceful_termination.go:94] lw: remote out of the list: 11.233.32.84:6666/TCP/11.233.115.102:6666 I1021 08:29:22.910660 1 graceful_termination.go:161] Trying to delete rs: 11.233.32.84:6666/TCP/11.233.115.67:6666 I1021 08:29:22.910746 1 graceful_termination.go:175] Deleting rs: 11.233.32.84:6666/TCP/11.233.115.67:6666 I1021 08:29:22.910759 1 graceful_termination.go:94] lw: remote out of the list: 11.233.32.84:6666/TCP/11.233.115.67:6666 I1021 08:29:22.910763 1 graceful_termination.go:161] Trying to delete rs: 11.233.32.84:6666/TCP/11.233.67.117:6666 I1021 08:29:22.910849 1 graceful_termination.go:175] Deleting rs: 11.233.32.84:6666/TCP/11.233.67.117:6666 I1021 08:29:22.910862 1 graceful_termination.go:94] lw: remote out of the list: 11.233.32.84:6666/TCP/11.233.67.117:6666 I1021 08:29:22.910867 1 graceful_termination.go:161] Trying to delete rs: 11.233.32.84:6666/TCP/11.233.74.117:6666 I1021 08:29:22.910944 1 graceful_termination.go:175] Deleting rs: 11.233.32.84:6666/TCP/11.233.74.117:6666 I1021 08:29:22.910955 1 graceful_termination.go:94] lw: remote out of the list: 11.233.32.84:6666/TCP/11.233.74.117:6666 I1021 08:29:22.910959 1 graceful_termination.go:161] Trying to delete rs: 11.233.32.84:6666/TCP/11.233.109.51:6666 I1021 08:29:22.911058 1 graceful_termination.go:175] Deleting rs: 11.233.32.84:6666/TCP/11.233.109.51:6666 I1021 08:29:22.911078 1 graceful_termination.go:94] lw: remote out of the list: 11.233.32.84:6666/TCP/11.233.109.51:6666 I1021 08:29:22.911088 1 graceful_termination.go:161] Trying to delete rs: 11.233.32.84:6666/TCP/11.233.98.62:6666 I1021 08:29:22.911180 1 graceful_termination.go:175] Deleting rs: 11.233.32.84:6666/TCP/11.233.98.62:6666 I1021 08:29:22.911195 1 graceful_termination.go:94] lw: remote out of the list: 11.233.32.84:6666/TCP/11.233.98.62:6666 I1021 08:29:22.911200 1 graceful_termination.go:161] Trying to delete rs: 11.233.32.84:6666/TCP/11.233.93.132:6666 I1021 08:29:22.911270 1 graceful_termination.go:175] Deleting rs: 11.233.32.84:6666/TCP/11.233.93.132:6666 I1021 08:29:22.911285 1 graceful_termination.go:94] lw: remote out of the list: 11.233.32.84:6666/TCP/11.233.93.132:6666 I1021 08:30:22.911437 1 graceful_termination.go:161] Trying to delete rs: 11.233.32.84:6666/TCP/11.233.90.152:6666 I1021 08:30:22.911622 1 graceful_termination.go:175] Deleting rs: 11.233.32.84:6666/TCP/11.233.90.152:6666 I1021 08:30:22.911645 1 graceful_termination.go:94] lw: remote out of the list: 11.233.32.84:6666/TCP/11.233.90.152:6666

@lbernail
Copy link
Contributor

The logs don't indicate a problem, this is expected with graceful termination
What symptoms are you seeing?

@shimron
Copy link

shimron commented Oct 22, 2019 •
edited

When updating a deployment, requests to the pods through the service ip are timeout on some nodes. After restart the kube-proxy on those nodes, the timeout disappeared.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/ipvs kind/bug Categorizes issue or PR as related to a bug. kind/support Categorizes issue or PR as a support question. sig/network Categorizes an issue or PR as relevant to SIG Network. triage/needs-information Indicates an issue needs more information in order to work on it.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@liggitt @lbernail @shimron @thockin @2PacIsAlive @k8s-ci-robot @xuezm01 @caoban