Add [experimental] option for using IPVS proxy mode #1074

ivanilves · 2017-12-14T17:30:48Z

Add [experimental] option to use IPVS kube-proxy mode instead of [default] iptables one.

Hope to address performance issues of iptables mode for clusters with big number of nodes and services!

(>20 nodes, >5k services is a big cluster to me)

We also hope IPVS will handle UDP traffic better (This needs to be validated, a hope-based guess only)

codecov-io · 2017-12-14T17:50:53Z

Codecov Report

Merging #1074 into master will increase coverage by 0.15%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #1074      +/-   ##
==========================================
+ Coverage   35.13%   35.28%   +0.15%     
==========================================
  Files          59       59              
  Lines        3393     3401       +8     
==========================================
+ Hits         1192     1200       +8     
  Misses       2039     2039              
  Partials      162      162

Impacted Files	Coverage Δ
core/controlplane/config/config.go	`61.18% <100%> (+0.38%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ae23d78...0c88cc0. Read the comment docs.

ivanilves · 2017-12-18T10:13:56Z

Could we review this? /cc @mumoshu @redbaron 🙏

Sorry for being intrusive! 😭

camilb · 2017-12-18T10:22:27Z

Hi @ivanilves did you create a cluster using these changes? From what I'm seeing, in 1.8.4, kube-dns and flannel fail to start. Didn't have time to investigate further.

mumoshu

Thanks a lot for your contribution - I'm really looking forward to try it myself!

Mostly questions/nits. Would you mind addressing these?

mumoshu · 2017-12-18T10:18:39Z

core/controlplane/config/templates/cloud-config-controller

                - mountPath: /etc/kubernetes/kubeconfig
                  name: kubeconfig
                  readOnly: true
                - mountPath: /etc/kubernetes/kube-proxy
                  name: kube-proxy-config
                  readOnly: true
              volumes:
+              - name: lib-modules
+                hostPath:
+                  path: /lib/modules


Can we enclose this volume and the corresponding volumemount inside {{if .Experimental.IPVSProxy.Enabled}} if it is necessary when and only when IPVSProxy is enabled?

mumoshu · 2017-12-18T10:20:58Z

test/integration/maincluster_test.go

+  ipvsProxy:
+    enabled: true
+    scheduler: lc
+    syncPeriod: 900s


Any reference for properly configuring this setting?
At first glance, a sync period of 15min seems like a bit long - does it mean that a newly added pod/svc IP becomes accessible from the entire cluster after at most 15min?

mumoshu · 2017-12-18T10:25:48Z

core/controlplane/config/templates/cluster.yaml

@@ -1261,6 +1261,14 @@ experimental:
  kube2IamSupport:
    enabled: false

+  # Use IPVS kube-proxy mode instead of [default] iptables one (requires Kubernetes 1.8.3+)
+  # This is intended to address performance issues of iptables mode for clusters with big number of nodes and services
+  ipvsProxy:


Could you move this to kubeProxy.ipvsMode?
Sorry for taking your time but we're migrating away from the experimental settings because not only experimental but everything could change while we're in pre v1.0 😉

ivanilves · 2017-12-18T10:41:04Z

@camilb yes we did for both 1.8.4 and 1.8.5. 😟 Could you share some logs? 🙏

@mumoshu thanks for your comments! 👏 I will try to address them all ASAP.

camilb · 2017-12-18T12:09:41Z

Hi @ianilves , I managed to fix kube-dns by specifying these parameters to kube-proxy's DaemonSet:

env:
- name: KUBEPROXY_MODE
  value: ipvs
command:
 - /hyperkube
 - proxy
 - --config=/etc/kubernetes/kube-proxy/kube-proxy-config.yaml
 - --feature-gates=SupportIPVSProxyMode=true

They are mentioned here:

https://github.com/kubernetes/kubernetes/tree/master/pkg/proxy/ipvs#why-kube-proxy-cant-start-ipvs-mode

However, it started to crash again after rebooting the nodes
kube-dns logs

I1218 12:01:17.148395       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
E1218 12:01:17.148904       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.3.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.3.0.1:443: i/o timeout
E1218 12:01:17.149031       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.3.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.3.0.1:443: i/o timeout
I1218 12:01:17.648421       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...

Also:

trying to run a pod in the cluster, wasn't able to resolve any external dns when kube-dns was up
flannel started well in the end
if you can, please also run https://scanner.heptio.com in your cluster and check the results

ivanilves · 2017-12-18T15:57:12Z

OK, at least kube-proxy starts well with settings from PR (Using 1.8.5_coreos.0 with calico)

I1218 12:08:36.732790       1 feature_gate.go:156] feature gates: map[SupportIPVSProxyMode:true]
I1218 12:08:36.744495       1 server_others.go:164] Using ipvs Proxier.
I1218 12:08:36.782612       1 server_others.go:187] Tearing down inactive rules.
I1218 12:08:37.289500       1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
I1218 12:08:37.289633       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I1218 12:08:37.289694       1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I1218 12:08:37.289781       1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I1218 12:08:37.289952       1 config.go:102] Starting endpoints config controller
I1218 12:08:37.290010       1 controller_utils.go:1041] Waiting for caches to sync for endpoints config controller
I1218 12:08:37.290062       1 config.go:202] Starting service config controller
I1218 12:08:37.290103       1 controller_utils.go:1041] Waiting for caches to sync for service config controller
I1218 12:08:37.390208       1 controller_utils.go:1048] Caches are synced for service config controller
I1218 12:08:37.390208       1 controller_utils.go:1048] Caches are synced for endpoints config controller

ivanilves · 2017-12-19T13:07:18Z

FYI @camilb I've run a decent amout of cluster creations and upgrades.

And I may state this:

both your and mine settings for kube-proxy IPVS mode are correct.
... but kube-proxy in IPVS mode starts in a flaky maner - sometimes it outputs this:

E1219 12:27:06.901138       1 proxier.go:1362] Failed to unbind service from dummy interface, error: error unbind address: 172.16.0.10 from dummy interface: kube-ipvs0, err: exit status 2: RTNETLINK answers: Cannot assign requested address

and does not load DNS service (which has IP 172.16.0.10 in my cluster and is UDP). Still I can connect to other cluster services (which are TCP) by curl/telnet and that makes me a little bit puzzled... 🤔

I think it's something with a way how kube-proxy in IPVS mode is getting initialized.
This explains both why it gets "fixed" after restart (e.g. after DaemonSet change) and why it could get broken again after node reboot.

Meanwhile I'm working towards:

fix/workaround of kube-proxy IPVS issue (to add more sense to this PR)
integrating Heptio Sonobuoy in my pipeline (actually this is already done by one brilliant man from my team)

Thank you again for your input!

ivanilves · 2017-12-21T12:09:40Z

Hey @camilb I was running the IPVS thing for the last few days, and what I've found:

on k8s 1.8: kube-proxy in IPVS mode is flacky, I have workaround: delete problematic kube-proxy pods and re-scheduled ones will run OK, will but it's too ugly and wrong to include here. 😱 😱 😱
on k8s 1.9: kube-proxy in IPVS mode is decent, but the hyperkube image lacks ipset binary CoreOS hyperkube v1.9.0_coreos.0: No ipset utility, IPVS proxy mode fails coreos/coreos-kubernetes#915 so it is also broken. However it's broken in a straightforward way 😉

How could I know it is decent ?

I've made my own version of hyperkube ivanilves/hyperkube:v1.9.0_coreos.0 https://hub.docker.com/r/ivanilves/hyperkube/ and running w/o any issue creating/updating/rebooting clusters for the time being 😅

k8s-ci-robot · 2017-12-21T12:43:30Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://github.com/kubernetes/kubernetes/wiki/CLA-FAQ to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please email the CNCF helpdesk: helpdesk@rt.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

ivanilves · 2017-12-21T16:17:26Z

@mumoshu could you take a look on this PR now? (I've changed it)

Due to some concerns of IPVS integration state I would like to not merge it now, but maintain it until kube-aws will have K8s 1.9.1+ as default. Meanwhile I would maintain/rebase it. WDYT?

ivanilves · 2017-12-22T10:34:30Z

BTW Heptio tests ran well: https://scanner.heptio.com/d30af2b8fc3d94de31f341036da6165e/diagnostics/ ☺️

mumoshu · 2017-12-26T03:24:44Z

@ivanilves Hi, thanks a lot for your efforts!

After seeing your great explanations in cluster.yaml about various gotchas to get it running, I'm willing to merge this now to reduce hassle of resolving conflicts again and again.

Would it be ok for you, too? 😃

ivanilves · 2017-12-26T22:38:45Z

@mumoshu YES!!! Please!!! 🙏

mumoshu · 2017-12-26T22:52:37Z

@ivanilves Thanks again for your contribution 👍

camilb · 2018-01-05T12:02:37Z

@ivanilves @mumoshu Tested the Google's hyperkube 1.9.1 image with IPVS in #1104 .
Everything looks fine: https://scanner.heptio.com/59e89093d850dc7b5c8719cb87757646/diagnostics/

mumoshu · 2018-01-05T13:59:51Z

Great!

ivanilves · 2018-01-06T09:16:04Z

👏 👏 👏

Add option for using IPVS proxy mode

cknowles · 2018-04-05T05:25:20Z

These docs mention that ipvs falls back to iptables when kube-proxy is started with the --cluster-cidr option set. Does anyone know if that's specifically just the flag and not the config file?

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Dec 14, 2017

ivanilves mentioned this pull request Dec 14, 2017

IPVS mode for kube-proxy. Implementation #1073

Closed

zmalik approved these changes Dec 14, 2017

View reviewed changes

mumoshu suggested changes Dec 18, 2017

View reviewed changes

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Dec 21, 2017

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. and removed cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Dec 21, 2017

ivanilves added 2 commits December 21, 2017 13:48

Add option for using IPVS proxy mode

f45f22f

Optional functionality is optional

8dafbbc

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Dec 21, 2017

Move Experimental.IPVSProxy => KubeProxy.IPVSMode

0c88cc0

mumoshu approved these changes Dec 26, 2017

View reviewed changes

mumoshu merged commit d15e5cf into kubernetes-retired:master Dec 26, 2017

mumoshu added this to the v0.9.10.rc-1 milestone Dec 26, 2017

camilb mentioned this pull request Jan 22, 2018

K8s version hardcoded? (... or I am wrong) #1070

Closed

kylehodgetts pushed a commit to HotelsDotCom/kube-aws that referenced this pull request Mar 27, 2018

Merge pull request kubernetes-retired#1074 from ivanilves/ipvs-proxy

3cd920d

Add option for using IPVS proxy mode

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add [experimental] option for using IPVS proxy mode #1074

Add [experimental] option for using IPVS proxy mode #1074

ivanilves commented Dec 14, 2017 •

edited

Loading

codecov-io commented Dec 14, 2017 •

edited

Loading

ivanilves commented Dec 18, 2017

camilb commented Dec 18, 2017

mumoshu left a comment

mumoshu Dec 18, 2017

mumoshu Dec 18, 2017

mumoshu Dec 18, 2017

ivanilves commented Dec 18, 2017 •

edited

Loading

camilb commented Dec 18, 2017

ivanilves commented Dec 18, 2017

ivanilves commented Dec 19, 2017 •

edited

Loading

ivanilves commented Dec 21, 2017 •

edited

Loading

k8s-ci-robot commented Dec 21, 2017

ivanilves commented Dec 21, 2017

ivanilves commented Dec 22, 2017 •

edited

Loading

mumoshu commented Dec 26, 2017

ivanilves commented Dec 26, 2017

mumoshu commented Dec 26, 2017

camilb commented Jan 5, 2018

mumoshu commented Jan 5, 2018

ivanilves commented Jan 6, 2018

cknowles commented Apr 5, 2018

Add [experimental] option for using IPVS proxy mode #1074

Add [experimental] option for using IPVS proxy mode #1074

Conversation

ivanilves commented Dec 14, 2017 • edited Loading

codecov-io commented Dec 14, 2017 • edited Loading

Codecov Report

ivanilves commented Dec 18, 2017

camilb commented Dec 18, 2017

mumoshu left a comment

Choose a reason for hiding this comment

mumoshu Dec 18, 2017

Choose a reason for hiding this comment

mumoshu Dec 18, 2017

Choose a reason for hiding this comment

mumoshu Dec 18, 2017

Choose a reason for hiding this comment

ivanilves commented Dec 18, 2017 • edited Loading

camilb commented Dec 18, 2017

ivanilves commented Dec 18, 2017

ivanilves commented Dec 19, 2017 • edited Loading

ivanilves commented Dec 21, 2017 • edited Loading

How could I know it is decent ?

k8s-ci-robot commented Dec 21, 2017

ivanilves commented Dec 21, 2017

ivanilves commented Dec 22, 2017 • edited Loading

mumoshu commented Dec 26, 2017

ivanilves commented Dec 26, 2017

mumoshu commented Dec 26, 2017

camilb commented Jan 5, 2018

mumoshu commented Jan 5, 2018

ivanilves commented Jan 6, 2018

cknowles commented Apr 5, 2018

ivanilves commented Dec 14, 2017 •

edited

Loading

codecov-io commented Dec 14, 2017 •

edited

Loading

ivanilves commented Dec 18, 2017 •

edited

Loading

ivanilves commented Dec 19, 2017 •

edited

Loading

ivanilves commented Dec 21, 2017 •

edited

Loading

ivanilves commented Dec 22, 2017 •

edited

Loading