Adding traffic shaping support for CNI network driver #63194

m1093782566 · 2018-04-26T11:11:04Z

What this PR does / why we need it:

Adding traffic shaping support for CNI network driver - it's also a sub-task of kubenet deprecation work.

Design document is available here: kubernetes/community#1893

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

/cc @freehan @jingax10 @caseydavenport @dcbw

/sig network
/sig node

Release note:

Support traffic shaping for CNI network driver

m1093782566 · 2018-04-26T11:15:15Z

/assign @thockin

thockin · 2018-05-14T23:42:19Z

This looks OK, but I'll need someone to really try it out. What can we do about testing?

m1093782566 · 2018-05-15T02:23:56Z

I'll need someone to really try it out. What can we do about testing?

My team member @Lion-Wei is working on adding some e2e test cases for this feature.

m1093782566 · 2018-05-21T03:59:26Z

/assign @rramkumar1

Can you take a look when you have chance? :)

Thanks!

dcbw · 2018-05-23T21:42:40Z

/lgtm though as noted we should really have some tests too

dcbw · 2018-05-23T21:40:22Z

pkg/kubelet/dockershim/network/cni/cni.go

@@ -68,6 +69,21 @@ type cniPortMapping struct {
 	HostIP        string `json:"hostIP"`
 }

+// cniBandwidthEntry maps to the standard CNI bandwidth Capability
+// see: https://github.com/containernetworking/cni/blob/master/CONVENTIONS.md


Should probably also mention:
https://github.com/containernetworking/plugins/blob/master/plugins/meta/bandwidth/README.md

Lion-Wei · 2018-05-24T01:35:27Z

@dcbw ,I think we better have some e2e test case about bandwidth, but cni plugin v0.8 have not been released yet, so how to get bandwidth package in our test, that'a a problem. : (

m1093782566 · 2018-05-24T01:57:26Z

As @Lion-Wei mentioned, my team has been working on adding e2e tests for some days but we find

There is no specific job for running CNI-plugins-related test cases. We need to create new test job in kubernetes/test-infra.
If we reuse the test jobs kubeadm-gce-cni-calico and kubeadm-gce-cni-flannel, we should add their --ginkgo.focus like [Feature: Bandwidth]. We need to update the test job in kubernetes/test-infra as well.

So, can we let this PR in first and then update the jobs in kubernetes/test-infra? My team will continue to work on it.

BTW, the functionalities of cni bandwidth plugins is already being tested in containernetworking/plugins repo, so can we just write some UTs in kubernetes repo for now?

m1093782566 · 2018-05-28T03:12:28Z

@thockin @dcbw

I just added some tests, PTAL.

squeed · 2018-05-30T18:15:03Z

pkg/kubelet/dockershim/network/cni/cni.go

+// cniBandwidthEntry maps to the standard CNI bandwidth Capability
+// see: https://github.com/containernetworking/cni/blob/master/CONVENTIONS.md
+type cniBandwidthEntry struct {
+	// IngressRate is the bandwidth rate in Kbps for traffic through container. 0 for no limit. If ingressRate is set, ingressBurst must also be set


This was a bug in the CNI documentation; it is actually bits per second.

squeed · 2018-05-30T18:15:40Z

pkg/kubelet/dockershim/network/cni/cni.go

+		bandwidthParam := cniBandwidthEntry{}
+		if ingress != nil {
+			bandwidthParam.IngressRate = int(ingress.Value() / 1000)
+			bandwidthParam.IngressBurst = int(ingress.Value() / 1000)


This value should be hard-coded to 1600 bits - it's the linux TC default and is what kubenet uses.

Fixed. Thanks!

squeed · 2018-05-30T18:16:01Z

pkg/kubelet/dockershim/network/cni/cni.go

+	if ingress != nil || egress != nil {
+		bandwidthParam := cniBandwidthEntry{}
+		if ingress != nil {
+			bandwidthParam.IngressRate = int(ingress.Value() / 1000)


The documentation was wrong; the units are bits per second.

Done. Thanks!

squeed · 2018-05-30T18:16:09Z

pkg/kubelet/dockershim/network/cni/cni.go

+			bandwidthParam.IngressBurst = int(ingress.Value() / 1000)
+		}
+		if egress != nil {
+			bandwidthParam.EgressRate = int(egress.Value() / 1000)


bits per second here

squeed · 2018-05-30T18:16:18Z

pkg/kubelet/dockershim/network/cni/cni.go

+		}
+		if egress != nil {
+			bandwidthParam.EgressRate = int(egress.Value() / 1000)
+			bandwidthParam.EgressBurst = int(egress.Value() / 1000)


should be the default of 1600

squeed · 2018-05-30T18:16:44Z

pkg/kubelet/dockershim/network/cni/cni.go

+// see: https://github.com/containernetworking/cni/blob/master/CONVENTIONS.md
+type cniBandwidthEntry struct {
+	// IngressRate is the bandwidth rate in Kbps for traffic through container. 0 for no limit. If ingressRate is set, ingressBurst must also be set
+	IngressRate int `json:"ingressRate"`


All of these fields should have omitEmpty

squeed · 2018-05-30T19:12:00Z

I did a manual end-to-end test as part of #64445 and found some bugs - left some comments.

m1093782566 · 2018-05-31T03:04:24Z

@squeed

Thanks for your comments, we have addressed all of them. PTAL.

BTW, I find there are some bandwidth-related changes in #64445 as well. As my friend(@Lion-Wei) and I started this work 2 months ago and have spent lots of time doing it, can we let this PR in first and then rework #64445?

Lion-Wei · 2018-07-09T01:02:20Z

/lgtm
/retest

m1093782566 · 2018-07-09T01:28:56Z

Thanks @Lion-Wei

/hold

for @thockin final review :)

m1093782566 · 2018-07-09T02:16:20Z

/hold cancel

timchenxiaoyu · 2018-07-09T03:00:26Z

does support change bandwith dynamic when pods in runtime ?

m1093782566 · 2018-07-09T03:03:42Z

@timchenxiaoyu

does support change bandwith dynamic when pods in runtime?

Currently, it doesn't support - we configure bandwidth limit in pod start up. I think we can support it if we get better consensus.

thockin · 2018-07-09T05:32:30Z

/lgtm
/approve

k8s-ci-robot · 2018-07-09T05:32:37Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dcbw, Lion-Wei, m1093782566, thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/kubelet/dockershim/network/OWNERS~~ [dcbw,thockin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-github-robot · 2018-07-09T05:35:52Z

/test all [submit-queue is verifying that this PR is safe to merge]

m1093782566 · 2018-07-09T06:06:56Z

THANKS @thockin

k8s-github-robot · 2018-07-09T06:54:25Z

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

k8s-ci-robot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Apr 26, 2018

k8s-ci-robot requested review from caseydavenport, dcbw and freehan April 26, 2018 11:11

k8s-ci-robot added the sig/network Categorizes an issue or PR as relevant to SIG Network. label Apr 26, 2018

k8s-ci-robot requested a review from jingax10 April 26, 2018 11:11

k8s-ci-robot added sig/node Categorizes an issue or PR as relevant to SIG Node. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 26, 2018

m1093782566 force-pushed the cni-ts branch from c0fb9be to 4f4020e Compare April 26, 2018 11:13

k8s-ci-robot assigned thockin Apr 26, 2018

m1093782566 force-pushed the cni-ts branch from 099a25c to b4ea1c2 Compare April 26, 2018 14:04

rosenhouse mentioned this pull request Apr 30, 2018

traffic shaping: take configuration via a runtimeConfig containernetworking/plugins#138

Merged

k8s-ci-robot assigned freehan May 2, 2018

thockin assigned bowei and MrHohn May 14, 2018

k8s-ci-robot assigned rramkumar1 May 21, 2018

dcbw approved these changes May 23, 2018

View reviewed changes

squeed mentioned this pull request May 30, 2018

dockershim/network: pass ipRange CNI capabilities #64445

Merged

squeed suggested changes May 30, 2018

View reviewed changes

k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed lgtm "Looks good to me", indicates that a PR is ready to be merged. labels Jul 8, 2018

m1093782566 force-pushed the cni-ts branch from 25767e8 to fe163e5 Compare July 8, 2018 14:25

k8s-ci-robot assigned Lion-Wei Jul 9, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 9, 2018

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 9, 2018

add cni bandwidth test

34d848e

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 9, 2018

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 9, 2018

m1093782566 force-pushed the cni-ts branch from d5ca499 to 34d848e Compare July 9, 2018 02:24

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 9, 2018

k8s-github-robot merged commit e943d09 into kubernetes:master Jul 9, 2018

Lion-Wei mentioned this pull request Aug 9, 2018

cni networking plugin now support portmapping and bandwidth kubernetes/website#9797

Merged

caseydavenport mentioned this pull request Sep 12, 2018

Traffic Shaping in Calico projectcalico/calico#797

Closed

aanm mentioned this pull request Sep 27, 2018

Add support for CNI traffic shaping cilium/cilium#5718

Closed

brb mentioned this pull request Oct 1, 2018

Support traffic shaping in CNI weaveworks/weave#3422

Open

paulnivin mentioned this pull request Apr 15, 2019

Pass metadata to CNI plugins cri-o/cri-o#2207

Closed

kvaps mentioned this pull request Jul 15, 2020

Support traffic shaping in CNI cloudnativelabs/kube-router#954

Closed

This was referenced Jul 3, 2021

KEP-2819: CNI traffic shaping support kubernetes/enhancements#2808

Closed

CNI traffic shaping support kubernetes/enhancements#2819

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding traffic shaping support for CNI network driver #63194

Adding traffic shaping support for CNI network driver #63194

m1093782566 commented Apr 26, 2018 •

edited

Loading

m1093782566 commented Apr 26, 2018

thockin commented May 14, 2018

m1093782566 commented May 15, 2018

m1093782566 commented May 21, 2018

dcbw commented May 23, 2018

dcbw May 23, 2018

m1093782566 Jun 4, 2018

Lion-Wei commented May 24, 2018

m1093782566 commented May 24, 2018 •

edited

Loading

m1093782566 commented May 28, 2018

squeed May 30, 2018

m1093782566 May 31, 2018

squeed May 30, 2018

m1093782566 May 31, 2018

squeed May 30, 2018

m1093782566 May 31, 2018

squeed May 30, 2018

m1093782566 May 31, 2018

squeed May 30, 2018

m1093782566 May 31, 2018

squeed May 30, 2018

m1093782566 May 31, 2018

squeed commented May 30, 2018

m1093782566 commented May 31, 2018 •

edited

Loading

Lion-Wei commented Jul 9, 2018

m1093782566 commented Jul 9, 2018

m1093782566 commented Jul 9, 2018

timchenxiaoyu commented Jul 9, 2018

m1093782566 commented Jul 9, 2018 •

edited

Loading

thockin commented Jul 9, 2018

k8s-ci-robot commented Jul 9, 2018

k8s-github-robot commented Jul 9, 2018

m1093782566 commented Jul 9, 2018

k8s-github-robot commented Jul 9, 2018

Adding traffic shaping support for CNI network driver #63194

Adding traffic shaping support for CNI network driver #63194

Conversation

m1093782566 commented Apr 26, 2018 • edited Loading

m1093782566 commented Apr 26, 2018

thockin commented May 14, 2018

m1093782566 commented May 15, 2018

m1093782566 commented May 21, 2018

dcbw commented May 23, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Lion-Wei commented May 24, 2018

m1093782566 commented May 24, 2018 • edited Loading

m1093782566 commented May 28, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

squeed commented May 30, 2018

m1093782566 commented May 31, 2018 • edited Loading

Lion-Wei commented Jul 9, 2018

m1093782566 commented Jul 9, 2018

m1093782566 commented Jul 9, 2018

timchenxiaoyu commented Jul 9, 2018

m1093782566 commented Jul 9, 2018 • edited Loading

thockin commented Jul 9, 2018

k8s-ci-robot commented Jul 9, 2018

k8s-github-robot commented Jul 9, 2018

m1093782566 commented Jul 9, 2018

k8s-github-robot commented Jul 9, 2018

m1093782566 commented Apr 26, 2018 •

edited

Loading

m1093782566 commented May 24, 2018 •

edited

Loading

m1093782566 commented May 31, 2018 •

edited

Loading

m1093782566 commented Jul 9, 2018 •

edited

Loading