Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When running in ipvs mode, unable to update kube-proxy to 1.15 #77265

Closed
lbernail opened this issue Apr 30, 2019 · 7 comments · Fixed by #77371
Closed

When running in ipvs mode, unable to update kube-proxy to 1.15 #77265

lbernail opened this issue Apr 30, 2019 · 7 comments · Fixed by #77371
Assignees
Labels
area/ipvs kind/bug Categorizes issue or PR as related to a bug. sig/network Categorizes an issue or PR as relevant to SIG Network.

Comments

@lbernail
Copy link
Contributor

What happened:
When updating to 1.15 in IPVS mode, kube-proxy fails to start

What you expected to happen:
kube-proxy updates ok

How to reproduce it (as minimally and precisely as possible):
Deploy kube-proxy <1.15 in ipvs mode and update to 1.15

Anything else we need to know?:
Error message in kube-proxy:

E0430 10:05:53.783443       1 ipset.go:162] Failed to make sure ip set: &{{KUBE-NODE-PORT-LOCAL-SCTP hash:ip,port inet 1024 65536 0-65535 Kubernetes nodeport SCTP port with externalTrafficPolicy=local} map[] 0xc00032e830} exist, error: error creating ipset KUBE-NODE-PORT-LOCAL-SCTP, error: exit status 1

I think the reason is the change of ipset type (bitmap to hash) in PR #74341. The 2 sets that creates the problem are:

  • KUBE-NODE-PORT-SCTP
  • KUBE-NODE-PORT-LOCAL-SCTP

Removing the sets manually works but is not a great solution for in-place updates.
I think we should fix this before the 1.15 release (not sure what the best approach is: delete the set if it exists with the wrong type? Use a different name to avoid the collision?)

Environment:

  • Kubernetes version (use kubectl version): 1.14 => 1.15
  • Cloud provider or hardware configuration: AWS
  • OS (e.g: cat /etc/os-release): ubuntu 18.04
  • Kernel (e.g. uname -a): 4.4

/sig network
/area ipvs
/assign @m1093782566

@lbernail lbernail added the kind/bug Categorizes issue or PR as related to a bug. label Apr 30, 2019
@k8s-ci-robot k8s-ci-robot added sig/network Categorizes an issue or PR as relevant to SIG Network. area/ipvs labels Apr 30, 2019
@athenabot
Copy link

/triage unresolved

Comment /remove-triage unresolved when the issue is assessed and confirmed.

🤖 I am a bot run by @vllry. 👩‍🔬

@k8s-ci-robot k8s-ci-robot added the triage/unresolved Indicates an issue that can not or will not be resolved. label Apr 30, 2019
@andrewsykim
Copy link
Member

andrewsykim commented May 2, 2019

Great catch! Looks like a breaking change. One solution like you mentioned is to add support for replacing ipsets in pkg/util/ipset if a similar ipset already exists? Thoughts @m1093782566?

@andrewsykim
Copy link
Member

I noticed we pass in -exists for ipsets but that only checks for identical ipset types. Maybe pkg/util/ipset needs to be smarter and do a full diff

@andrewsykim
Copy link
Member

andrewsykim commented May 2, 2019

Use a different name to avoid the collision?)

This could work as well 🤔

@andrewsykim
Copy link
Member

Opened a fix for it here #77371, let's chat there :)

@athenabot
Copy link

@m1093782566 @andrewsykim
If this issue has been triaged, please comment /remove-triage unresolved.

If you aren't able to handle this issue, consider unassigning yourself and/or adding the help-wanted label.

🤖 I am a bot run by vllry. 👩‍🔬

@andrewsykim
Copy link
Member

/remove-triage unresolved

@k8s-ci-robot k8s-ci-robot removed the triage/unresolved Indicates an issue that can not or will not be resolved. label May 7, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/ipvs kind/bug Categorizes issue or PR as related to a bug. sig/network Categorizes an issue or PR as relevant to SIG Network.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants