bandwidth limit doesn't work

[liuxu623]

Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug
What happened:
bandwidth limit doesn't work
What you expected to happen:
bandwidth limit work
How to reproduce it (as minimally and precisely as possible):
https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/#support-traffic-shaping

apiVersion: v1
kind: Pod
metadata:
  name: test
  annotations:
    kubernetes.io/ingress-bandwidth: 1M
    kubernetes.io/egress-bandwidth: 1M
spec:
  containers:
  - name: test
    image: nginx:1.13

Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "4df4f85bbe21bf7e1696d07aae18b544f9ff4e0010d0da14cd503fbe071582a6" network for pod "test": NetworkPlugin cni failed to set up pod "test_default" network: if rate is set, burst must also be set

Anything else we need to know?:
https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/network/cni/cni.go#L378-L385

if ingress != nil {
    bandwidthParam.IngressRate = int(ingress.Value() / 1000)
    bandwidthParam.IngressBurst = 0 // default to no limit
}
if egress != nil {
    bandwidthParam.EgressRate = int(egress.Value() / 1000)
    bandwidthParam.EgressBurst = 0 // default to no limit
}

https://github.com/containernetworking/plugins/blob/master/plugins/meta/bandwidth/main.go#L110-L113

case burst == 0 && rate != 0:
    return fmt.Errorf("if rate is set, burst must also be set")
case rate == 0 && burst != 0:
    return fmt.Errorf("if burst is set, rate must also be set")

Environment:

• Kubernetes version (use kubectl version): v1.12.1
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): Debian 8
• Kernel (e.g. uname -a): 4.9.0-0.bpo.6-amd64
• Install tools:
• Others:

[liuxu623]

/sig network

[zyy17]

@liuxu623 You can check #67438.

[m1093782566]

Does #70149 work for you?

[zyy17]

Why ingress/egress value need to divide 1000 to get the rate ?

if ingress != nil {
    bandwidthParam.IngressRate = int(ingress.Value() / 1000)
    bandwidthParam.IngressBurst = 0 // default to no limit
}
if egress != nil {
    bandwidthParam.EgressRate = int(egress.Value() / 1000)
    bandwidthParam.EgressBurst = 0 // default to no limit
}

I think it is the same unit ( bps, bit per second).

@m1093782566

[dongeunsuh]

Hi, I'm having the same issue while testing traffic shaping feature. Is there any sample yaml file for PODs that limits the bandwidth? (one in https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/#support-traffic-shaping seems not working)

[Alan01252]

The problem appears to be that the conversion doesn't take into account the suffix used in the annotation.

For example the config

      annotations:
        kubernetes.io/ingress-bandwidth: "500M"
        kubernetes.io/egress-bandwidth: "500M"

Causes a tbf to be generated as below:

qdisc tbf 1: dev cali3fa6073a780 root refcnt 2 rate 500Kbit burst 10737417b lat 4123.2s
qdisc ingress ffff: dev cali3fa6073a780 parent ffff:fff1 ----------------
qdisc tbf 1: dev 576e root refcnt 2 rate 500Kbit burst 10737417b lat 4123.2s

I've tested that this then limits the traffic to the pod as expected.

[m1093782566]

Yeah, seems you got the key @Alan01252

BTW, what network plugin are you using?

[wsszh]

@m1093782566 Hello, I got the same problem as @Alan01252 said. Have you solved this problem?

if ingress != nil {
    bandwidthParam.IngressRate = int(ingress.Value() / 1000)
}
if egress != nil {
    bandwidthParam.EgressRate = int(egress.Value() / 1000)
}

I want to ask the same question as @GhostComputing asked. Why ingress/egress value needs to divide 1000 to get the rate? Because of it, I have to set the annotation below to limit ingress/egress bandwidth to 1Mbps:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/ingress-bandwidth: 1000M
    kubernetes.io/egress-bandwidth: 1000M

Then, run the command

tc qdisc show

I can get the expected result:

qdisc tbf 1: dev cali72f390115b5 root refcnt 2 rate 1Mbit burst 27917286b lat 1924.2s 
qdisc ingress ffff: dev cali72f390115b5 parent ffff:fff1 ---------------- 
qdisc tbf 1: dev e544 root refcnt 2 rate 1Mbit burst 27917286b lat 1924.2s

However, if I set the annotation following the document:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/ingress-bandwidth: 1M
    kubernetes.io/egress-bandwidth: 1M

Then run the command:

tc qdisc show

The rate is only 1Kbit as shown below!

qdisc tbf 1: dev cali72f390115b5 root refcnt 2 rate 1Kbit burst 21473b lat 4123.2s 
qdisc ingress ffff: dev cali72f390115b5 parent ffff:fff1 ---------------- 
qdisc tbf 1: dev 831e root refcnt 2 rate 1Kbit burst 21473b lat 4123.2s 

[ggaaooppeenngg]

Why bandwidth is divided by 1000?