Support for IPVS CNI Plugin #159
Conversation
|
@ivan4th Any thoughts on the test_src_master failure. It seems to do kubeadm init and shows a message (doesn't say it is an error) and then gets stuck in a kubeadm reset loop. It seems like, with V1.11, the API version should be v1alpha2. |
|
Can we just run kube-proxy in IPVS mode, instead of pulling in kube-router? |
|
@rpothier Never tried that, and it's not listed as a way to setup networking using KubeAdm (whereas using kube-router is listed). Do you know if doing that will provide BGP (which is what kube-router does)? If not, then I'm not sure how it handles routing between nodes. With kube-router, it uses bride CNI plugin, only for IPAM, and no hairpin or static routes need to be setup. In DinD, the YAML for kube-router sets up the CNI config. |
|
Will give kube-proxy with IPVS to see how that works. |
|
Clarification: Probably should have been clearer on this PR description, but partly it is an issue with the ambiguity of kube-router. Although called a CNI plugin, kube-router is really a networking component, which is really a replacement for kube-proxy, plus more. It has both the advantage of IPVS performance and scale, and the added functionality of BGP for dealing with the routing. It really is not a CNI plugin, per se, and uses the bridge plugin, with some different settings. For k-d-c, I defined a different CNI_PLUGIN, so that it doesn't do the bridge plugin setup in dindnet script. Like calico and others, kube-router has a YAML for setup. One thing it does in this YAML, is setup the bridge CNI config file, so functionality in dindnet is not needed (hence the separate CNI_PLUGIN type). So, I'd like to make this available, as a CNI_PLUGIN "kube-router". I'm peeking into running kube-proxy in IPVS mode, as @rpothier mentioned, but, if that is another viable method, I'd like to provide that as a separate option for k-d-c, and not as an alternative for kube-router, because kube-router does much more (BGP) and makes the setup even simpler (and works with external BGP,...). IOW, I'd like to move this PR forward, independent of kube-proxy in IPVS mode. |
|
Talked to @ivan4th and it looks like for master branch Kubernetes, the k-d-c image needs updating. Once that is done, I can rebase this PR to pickup the changes. |
|
@pmichali Sounds good to me. |
Kube-router is a replacement for kube-proxy that uses IPVS for load balancing containers and uses goBGP for creating network. This commit adds support for kube-router, by using a new CNI_PLUGIN setting called "kube-router". This will skip the bridge CNI plugin, and as part of kube-router setup, will install a custom bridge CNI config file. It will remove the kube-proxy daemonset, after starting up the kube-router daemonset. Note: kube-router currently supports only IPv4 (issue open for IPv6 support). A few warnings occur during up/down, but appear to be benign (e.g. kube-proxy warns to use a config file, no subnet to delete iptable rules for IPVS). Updated the "fixed" version scripts. NOTE: Will need to manually install ipset and ipvsadm on host, prior to using this caability.
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: ivan4th, pmichali The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
merged commit 0ddfc20
into
kubernetes-retired:master
Added support for the plugin for IPv4 (does not support IPv6 yet). Can
be used by setting CNI_PLUGIN to "kube-router". As part of the setup,
the kube-proxy daemonset is deleted.
A few warnings occur during up/down, but appear to be benign (e.g.
kube-proxy warns to use a config file, no subnet to delete iptable
rules for IPVS).
Updated the "fixed" version scripts.
Fixes: #158