New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for IPVS CNI Plugin #159

Merged
merged 1 commit into from Jul 20, 2018

Conversation

Projects
None yet
4 participants
@pmichali
Contributor

pmichali commented Jul 10, 2018

Added support for the plugin for IPv4 (does not support IPv6 yet). Can
be used by setting CNI_PLUGIN to "kube-router". As part of the setup,
the kube-proxy daemonset is deleted.

A few warnings occur during up/down, but appear to be benign (e.g.
kube-proxy warns to use a config file, no subnet to delete iptable
rules for IPVS).

Updated the "fixed" version scripts.

Fixes: #158

@pmichali

This comment has been minimized.

Contributor

pmichali commented Jul 11, 2018

@ivan4th Any thoughts on the test_src_master failure. It seems to do kubeadm init and shows a message (doesn't say it is an error) and then gets stuck in a kubeadm reset loop.

your configuration file uses an old API spec: "kubeadm.k8s.io/v1alpha1". Please use kubeadm v1.11 instead and run 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.

It seems like, with V1.11, the API version should be v1alpha2.
Does master need branch need to be updated?

@rpothier

This comment has been minimized.

Contributor

rpothier commented Jul 11, 2018

Can we just run kube-proxy in IPVS mode, instead of pulling in kube-router?
i.e. update kubeadm.conf with
kubeProxy:
config:
mode: ipvs

@pmichali

This comment has been minimized.

Contributor

pmichali commented Jul 11, 2018

@rpothier Never tried that, and it's not listed as a way to setup networking using KubeAdm (whereas using kube-router is listed). Do you know if doing that will provide BGP (which is what kube-router does)? If not, then I'm not sure how it handles routing between nodes.

With kube-router, it uses bride CNI plugin, only for IPAM, and no hairpin or static routes need to be setup. In DinD, the YAML for kube-router sets up the CNI config.

@pmichali

This comment has been minimized.

Contributor

pmichali commented Jul 12, 2018

Will give kube-proxy with IPVS to see how that works.

@pmichali

This comment has been minimized.

Contributor

pmichali commented Jul 13, 2018

Clarification: Probably should have been clearer on this PR description, but partly it is an issue with the ambiguity of kube-router. Although called a CNI plugin, kube-router is really a networking component, which is really a replacement for kube-proxy, plus more. It has both the advantage of IPVS performance and scale, and the added functionality of BGP for dealing with the routing.

It really is not a CNI plugin, per se, and uses the bridge plugin, with some different settings. For k-d-c, I defined a different CNI_PLUGIN, so that it doesn't do the bridge plugin setup in dindnet script. Like calico and others, kube-router has a YAML for setup. One thing it does in this YAML, is setup the bridge CNI config file, so functionality in dindnet is not needed (hence the separate CNI_PLUGIN type).

So, I'd like to make this available, as a CNI_PLUGIN "kube-router". I'm peeking into running kube-proxy in IPVS mode, as @rpothier mentioned, but, if that is another viable method, I'd like to provide that as a separate option for k-d-c, and not as an alternative for kube-router, because kube-router does much more (BGP) and makes the setup even simpler (and works with external BGP,...).

IOW, I'd like to move this PR forward, independent of kube-proxy in IPVS mode.

@pmichali

This comment has been minimized.

Contributor

pmichali commented Jul 13, 2018

Talked to @ivan4th and it looks like for master branch Kubernetes, the k-d-c image needs updating. Once that is done, I can rebase this PR to pickup the changes.

@rpothier

This comment has been minimized.

Contributor

rpothier commented Jul 13, 2018

@pmichali Sounds good to me.

Support for kube-router based IPVS
Kube-router is a replacement for kube-proxy that uses IPVS for load
balancing containers and uses goBGP for creating network. This commit
adds support for kube-router, by using a new CNI_PLUGIN setting called
"kube-router". This will skip the bridge CNI plugin, and as part of
kube-router setup, will install a custom bridge CNI config file. It
will remove the kube-proxy daemonset, after starting up the kube-router
daemonset.

Note: kube-router currently supports only IPv4 (issue open for IPv6
support).

A few warnings occur during up/down, but appear to be benign (e.g.
kube-proxy warns to use a config file, no subnet to delete iptable
rules for IPVS).

Updated the "fixed" version scripts.

NOTE: Will need to manually install ipset and ipvsadm on host, prior
to using this caability.
@ivan4th

This comment has been minimized.

Collaborator

ivan4th commented Jul 20, 2018

/approve
/lgtm

@k8s-ci-robot

This comment has been minimized.

Contributor

k8s-ci-robot commented Jul 20, 2018

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ivan4th, pmichali

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit 0ddfc20 into kubernetes-sigs:master Jul 20, 2018

25 of 26 checks passed

tide Not mergeable. Needs approved, lgtm labels.
Details
ci/circleci: build Your tests passed on CircleCI!
Details
ci/circleci: test_1.10 Your tests passed on CircleCI!
Details
ci/circleci: test_1.10_calico Your tests passed on CircleCI!
Details
ci/circleci: test_1.10_calico_kdd Your tests passed on CircleCI!
Details
ci/circleci: test_1.10_flannel Your tests passed on CircleCI!
Details
ci/circleci: test_1.10_weave Your tests passed on CircleCI!
Details
ci/circleci: test_1.11 Your tests passed on CircleCI!
Details
ci/circleci: test_1.11_calico Your tests passed on CircleCI!
Details
ci/circleci: test_1.11_calico_kdd Your tests passed on CircleCI!
Details
ci/circleci: test_1.11_flannel Your tests passed on CircleCI!
Details
ci/circleci: test_1.11_weave Your tests passed on CircleCI!
Details
ci/circleci: test_1.8 Your tests passed on CircleCI!
Details
ci/circleci: test_1.8_calico Your tests passed on CircleCI!
Details
ci/circleci: test_1.8_calico_kdd Your tests passed on CircleCI!
Details
ci/circleci: test_1.8_flannel Your tests passed on CircleCI!
Details
ci/circleci: test_1.8_weave Your tests passed on CircleCI!
Details
ci/circleci: test_1.9 Your tests passed on CircleCI!
Details
ci/circleci: test_1.9_calico Your tests passed on CircleCI!
Details
ci/circleci: test_1.9_calico_kdd Your tests passed on CircleCI!
Details
ci/circleci: test_1.9_flannel Your tests passed on CircleCI!
Details
ci/circleci: test_1.9_weave Your tests passed on CircleCI!
Details
ci/circleci: test_src_master Your tests passed on CircleCI!
Details
ci/circleci: test_src_release Your tests passed on CircleCI!
Details
cla/linuxfoundation pmichali authorized
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment