New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

due to ipset not supporting "comment", kube-proxy is not working in ipvs mode in Kubernetes 1.11.3 #68974

Open
nishantsh77 opened this Issue Sep 22, 2018 · 9 comments

Comments

Projects
None yet
6 participants
@nishantsh77

nishantsh77 commented Sep 22, 2018

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug
/sig network
/area kube-proxy
/area ipvs

What happened:
Kube-proxy not working in ipvs mode in Kubernetes version 1.11.3.

What you expected to happen:
Kube-proxy working in ipvs mode in centos 7.

How to reproduce it (as minimally and precisely as possible):
Initialize Kubernetes using kubeadm init command.

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):

[root@K8s-master ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T18:02:47Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:08:34Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}

  • Cloud provider or hardware configuration:

Bare metal installation (VM).

[root@K8s-master ~]# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
stepping : 1
microcode : 0xffffffff
cpu MHz : 2097.552
cache size : 20480 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm xsaveopt
bogomips : 4195.10
clflush size : 64
cache_alignment : 64
address sizes : 42 bits physical, 48 bits virtual
power management:

  • OS (e.g. from /etc/os-release):

[root@K8s-master ~]# rpm -q centos-release
centos-release-7-3.1611.el7.centos.x86_64

  • Kernel (e.g. uname -a):

[root@K8s-master ~]# uname -a
Linux K8s-master.cluster.k8.local 3.10.0-514.16.1.el7.x86_64 #1 SMP Wed Apr 12 15:04:24 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

  • Install tools:

[root@K8s-master ~]# ipset version
ipset v6.29, protocol version: 6

[root@K8s-master ~]# ipset create foo hash:ip comment
ipset v6.29: Unknown argument: `comment'

  • Others:
    [root@K8s-master ~]# kubectl logs kube-proxy-mskxf -c kube-proxy --namespace=kube-system
    I0921 09:46:16.614722 1 feature_gate.go:230] feature gates: &{map[]}
    I0921 09:46:16.645055 1 server_others.go:183] Using ipvs Proxier.
    W0921 09:46:16.656609 1 proxier.go:355] IPVS scheduler not specified, use rr by default
    I0921 09:46:16.656673 1 server_others.go:210] Tearing down inactive rules.
    I0921 09:46:16.697410 1 server.go:448] Version: v1.11.0
    I0921 09:46:16.706371 1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
    I0921 09:46:16.706433 1 conntrack.go:52] Setting nf_conntrack_max to 131072
    I0921 09:46:16.706468 1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
    I0921 09:46:16.706512 1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
    I0921 09:46:16.709397 1 config.go:202] Starting service config controller
    I0921 09:46:16.709476 1 controller_utils.go:1025] Waiting for caches to sync for service config controller
    I0921 09:46:16.710610 1 config.go:102] Starting endpoints config controller
    I0921 09:46:16.710734 1 controller_utils.go:1025] Waiting for caches to sync for endpoints config controller
    I0921 09:46:16.810690 1 controller_utils.go:1032] Caches are synced for service config controller
    I0921 09:46:16.810877 1 controller_utils.go:1032] Caches are synced for endpoints config controller
    E0921 09:46:16.889058 1 ipset.go:156] Failed to make sure ip set: &{{KUBE-CLUSTER-IP hash:ip,port inet 1024 65536 0-65535 Kubernetes service cluster ip + port for masquerade purpose} map[] 0xc4202da360} exist, error: error creating ipset KUBE-CLUSTER-IP, error: exit status 2
    E0921 09:46:46.734521 1 ipset.go:156] Failed to make sure ip set: &{{KUBE-CLUSTER-IP hash:ip,port inet 1024 65536 0-65535 Kubernetes service cluster ip + port for masquerade purpose} map[] 0xc4202da360} exist, error: error creating ipset KUBE-CLUSTER-IP, error: exit status 2
    E0921 09:47:16.757767 1 ipset.go:156] Failed to make sure ip set: &{{KUBE-LOAD-BALANCER-SOURCE-IP hash:ip,port,ip inet 1024 65536 0-65535 Kubernetes service load balancer ip + port + source IP for packet filter purpose} map[] 0xc4202da360} exist, error: error creating ipset KUBE-LOAD-BALANCER-SOURCE-IP, error: exit status 2
    E0921 09:47:46.780141 1 ipset.go:156] Failed to make sure ip set: &{{KUBE-CLUSTER-IP hash:ip,port inet 1024 65536 0-65535 Kubernetes service cluster ip + port for masquerade purpose} map[] 0xc4202da360} exist, error: error creating ipset KUBE-CLUSTER-IP, error: exit status 2
    E0921 09:48:16.801422 1 ipset.go:156] Failed to make sure ip set: &{{KUBE-LOOP-BACK hash:ip,port,ip inet 1024 65536 0-65535 Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose} map[] 0xc4202da360} exist, error: error creating ipset KUBE-LOOP-BACK, error: exit status 2
@dims

This comment has been minimized.

Show comment
Hide comment
@dims
Member

dims commented Sep 23, 2018

@m1093782566

This comment has been minimized.

Show comment
Hide comment
@m1093782566

m1093782566 Sep 23, 2018

Member
Member

m1093782566 commented Sep 23, 2018

@nishantsh77

This comment has been minimized.

Show comment
Hide comment
@nishantsh77

nishantsh77 Sep 23, 2018

The fix should be merged in 1.11.1 发件人:Davanum Srinivas 收件人:kubernetes/kubernetes, 抄 送:dujun (D),Mention, 时间:2018-09-23 21:29:19 主 题:Re: [kubernetes/kubernetes] due to ipset not supporting "comment", kube-proxy is not working in ipvs mode in Kubernetes 1.11.3 (#68974) cc @m1093782566https://github.com/m1093782566 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#68974 (comment)>, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEuXak-gwXL-2VosOV0Iz3rKSuKeUWJ8ks5ud4ymgaJpZM4W1JII.

Kindly let me know if more information is required.

nishantsh77 commented Sep 23, 2018

The fix should be merged in 1.11.1 发件人:Davanum Srinivas 收件人:kubernetes/kubernetes, 抄 送:dujun (D),Mention, 时间:2018-09-23 21:29:19 主 题:Re: [kubernetes/kubernetes] due to ipset not supporting "comment", kube-proxy is not working in ipvs mode in Kubernetes 1.11.3 (#68974) cc @m1093782566https://github.com/m1093782566 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#68974 (comment)>, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEuXak-gwXL-2VosOV0Iz3rKSuKeUWJ8ks5ud4ymgaJpZM4W1JII.

Kindly let me know if more information is required.

@m1093782566

This comment has been minimized.

Show comment
Hide comment
@m1093782566
Member

m1093782566 commented Sep 24, 2018

xref: #65741

/cc @islinwb

@m1093782566

This comment has been minimized.

Show comment
Hide comment
@m1093782566

m1093782566 Sep 24, 2018

Member

@nishantsh77

Please also note that your server binary version is :"v1.11.0" instead of v.11.3(client).

Member

m1093782566 commented Sep 24, 2018

@nishantsh77

Please also note that your server binary version is :"v1.11.0" instead of v.11.3(client).

@nishantsh77

This comment has been minimized.

Show comment
Hide comment
@nishantsh77

nishantsh77 Sep 24, 2018

@nishantsh77

Please also note that your server binary version is :"v1.11.0" instead of v.11.3(client).

Thanks for mentioning it. What I understood from above discussion that bug shall be fixed in Kubernetes server release "v1.11.1"

nishantsh77 commented Sep 24, 2018

@nishantsh77

Please also note that your server binary version is :"v1.11.0" instead of v.11.3(client).

Thanks for mentioning it. What I understood from above discussion that bug shall be fixed in Kubernetes server release "v1.11.1"

@islinwb

This comment has been minimized.

Show comment
Hide comment
@islinwb

islinwb Sep 25, 2018

Member

About ipset comment:
added in #63585 and removed in #65533.

The problem of creating ipsets with comments only exists in code with tag v1.11.0, v1.11.0-beta.0, v1.11.0-beta.1, v1.11.0-beta.2, v1.11.0-rc.1, v1.11.0-rc.2, v1.11.0-rc.3.

If you ever used the code of the above tag (for example, upgrade to/from), the problem Failed to make sure ip set...error: error creating ipset... would occur. A workaround is manually deleting all these ipsets using ipset destroy [SETNAME].

Member

islinwb commented Sep 25, 2018

About ipset comment:
added in #63585 and removed in #65533.

The problem of creating ipsets with comments only exists in code with tag v1.11.0, v1.11.0-beta.0, v1.11.0-beta.1, v1.11.0-beta.2, v1.11.0-rc.1, v1.11.0-rc.2, v1.11.0-rc.3.

If you ever used the code of the above tag (for example, upgrade to/from), the problem Failed to make sure ip set...error: error creating ipset... would occur. A workaround is manually deleting all these ipsets using ipset destroy [SETNAME].

@m1093782566

This comment has been minimized.

Show comment
Hide comment
@m1093782566

m1093782566 Oct 12, 2018

Member

@nishantsh77

Mind update your issue title to "due to ipset not supporting "comment", kube-proxy is not working in ipvs mode in Kubernetes 1.11.0"? As 1.11.3 works well and your server version is 1.11.0 indeed.

Member

m1093782566 commented Oct 12, 2018

@nishantsh77

Mind update your issue title to "due to ipset not supporting "comment", kube-proxy is not working in ipvs mode in Kubernetes 1.11.0"? As 1.11.3 works well and your server version is 1.11.0 indeed.

@Lion-Wei

This comment has been minimized.

Show comment
Hide comment
@Lion-Wei

Lion-Wei Oct 12, 2018

Contributor

@nishantsh77 Hi, any update or more information? Seems this problem have been fixed in #65741 in 1.11.1. And if you still have this problem, advise from @islinwb will be helpful.

Contributor

Lion-Wei commented Oct 12, 2018

@nishantsh77 Hi, any update or more information? Seems this problem have been fixed in #65741 in 1.11.1. And if you still have this problem, advise from @islinwb will be helpful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment