KEP-2819: CNI traffic shaping support #2808

uablrek · 2021-07-03T13:24:42Z

One-line PR description: revive old KEP

Issue link: CNI traffic shaping support #2819

Other comments:

Note: This is not a new KEP. It is for regain tracking of an already-implemented feature (in alpha)

This KEP is revived to regain tracking for the traffic shaping feature. It was introduced in v1.9 (I think) but has not left "alpha" since 2018. The intention is to allow the function to graduate. (kubernetes/community#2202 (comment))

The original KEP was not merged or moved to kubernetes/enhancements.

Links

k8s-ci-robot · 2021-07-03T13:24:51Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: uablrek
To complete the pull request process, please assign caseydavenport after the PR has been reviewed.
You can assign the PR to them by writing /assign @caseydavenport in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

keps/sig-network/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

uablrek · 2021-07-03T13:25:09Z

/cc @thockin

sftim · 2021-07-06T13:32:44Z

Should we retitle this to KEP-0012?

uablrek · 2021-07-06T13:41:16Z

If the number is still valid, yes. I don't know how the number is assigned. I read it is the trackíng number, but I don't know what that is.

Now "CNI traffic shaping support" in KEP, issue and PR.

keps/sig-network/2819-CNI-traffic-shaping-support/README.md

sftim · 2021-07-15T16:03:34Z

keps/sig-network/2819-CNI-traffic-shaping-support/README.md

+            "kubernetes.io/ingress-bandwidth": "10M",
+            "kubernetes.io/egress-bandwidth": "10M"


We should add these to https://kubernetes.io/docs/reference/labels-annotations-taints/

Co-authored-by: Tim Bannister <tim@scalefactory.com>

uablrek · 2021-09-02T21:22:40Z

/assign @thockin

k8s-triage-robot · 2021-12-01T21:50:06Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

frittentheke · 2021-12-03T07:59:37Z

Is this still being pursued? Would be sad so see my proprietary implementations in various networks without going via the CNI.

uablrek · 2021-12-03T08:40:13Z

/remove-lifecycle stale

thockin

Given that Dockershim is out (even if it were to be reverted, that would be temporary) do we need to re-position this in terms of CRI? E.g. kubelet passes these annotations to the runtime via CRI and then containerd or CRI-O (or whatever) handle it in terms of CNI ?

thockin · 2022-01-24T23:59:01Z

Is this trying to hit 1.24 ?

uablrek · 2022-01-25T06:37:49Z

Given that Dockershim is out (even if it were to be reverted, that would be temporary) do we need to re-position this in terms of CRI? E.g. kubelet passes these annotations to the runtime via CRI and then containerd or CRI-O (or whatever) handle it in terms of CNI ?

Um, does this only work with Dockershim?

Sorry, I haven't tested this myself only relayed a wish to promote this feature from my stake-holders. I am using cri-o so I would have noticed. If it's only working with dockershim I think we should take a step back and reconsider. I have seen proposals to pass pod annotations to CRI-plugins in a generic way but I cant recall where. Perhaps a KEP? If that is implemented it should be used. If not, this feature (and this KEP) should be deprecated. IMHO this feature should not be implemented with specific code.

uablrek · 2022-01-25T06:38:18Z

Is this trying to hit 1.24 ?

No. Considering the comments above.

lordofire · 2022-04-05T20:13:04Z

Instead of handling all the heavy lifting traffic shaping logics on kubernetes CNI directly, will it makes more sense to just provide a mechanism for kubernetes cni to tag (through fields like TOS, etc) the traffic generated per pod through annotation, and offload the traffic shaping logics to the network layer directly? This might not be a good fit for cloud environment, since we do not have privilege to control the traffic shaping logics, but I think it makes it much easier on on-prem setup.

I'm fine with someone adding that mechanism, but I think it could work out fine as extension APIs rather than as part of core Kubernetes. (If it's purely done by extending Kubernetes, no KEP is needed).

Thanks @sftim for the response. Is there any reference on the "extension APIs" you mentioned? I would like to take a look when I'm free.

squeed · 2022-05-24T15:19:57Z

keps/sig-network/2819-CNI-traffic-shaping-support/README.md

+  "type": "bandwidth",
+  "runtimeConfig": {
+    "trafficShaping": {
+      "ingressRate": "X",


One of the problem with the existing annotation is that they rely on plugin-specific definitions of bandwidth and burst.

We should probably clearly define either

Burst isn't required

Burst has a precise definition (e.g. Bucket size is N percent / ratio of bandwidth)

sftim · 2022-05-24T15:21:09Z

Is there any reference on the "extension APIs" you mentioned?

https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/

squeed · 2022-05-24T15:21:26Z

keps/sig-network/2819-CNI-traffic-shaping-support/README.md

+
+* [add traffic shaping support](https://github.com/kubernetes/kubernetes/pull/63194)
+
+## Drawbacks


The only drawback is a larger feature: bandwidth is not currently a scheduled resource. Do we want to consider adding node-egress-bandwidth to a pod's resources?

This is made complicated by the fact that bandwidth is only accountable as a pod-level resource, rather than a container-level resource (barring serious trickery)

brief outline of an approach

Someone who wants to can define an extended resource and assign that to a Pod. Pick something like “bits per second” because you can only have whole numbers of these.

Nodes can advertise their actual uplink as the available amount of that resource. Or, if we decide to, we could implement some cleverness around a particular extended resource where this doesn't hold (allowing overcommitment).

k8s-triage-robot · 2022-10-05T17:02:39Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

uablrek · 2022-10-06T08:14:44Z

/remove-lifecycle stale

sftim · 2022-12-22T20:52:07Z

keps/sig-network/2819-CNI-traffic-shaping-support/kep.yaml

+    - "@m1093782566"
+owning-sig: sig-network
+participating-sigs:
+status: implemented


Would that mean that this is generally available?

Suggested change

status: implemented

status: implementable

sftim · 2022-12-22T20:52:09Z

keps/sig-network/2819-CNI-traffic-shaping-support/README.md

+
+## Alternatives
+
+None


Let's document some alternative approaches.

Rather than using annotations, we could add the network restrictions to the Pod API.

That could allow the kubelet to report via .status whether the shaping was active. It also enables a richer control over flow shaping.

I'd prefer to see Pods refer indirectly to a custom resource, eg NetworkTrafficQuota, but still have the Pod report enforcement via its .status. (Why? Because you might have nodes where this works and other nodes where it doesn't).

thockin · 2022-12-22T20:54:47Z

The right way to handle this is some first-class API, but I am not sure yet what it looks like. There are KEPs open on QoS and other things.

…

On Thu, Dec 22, 2022 at 12:52 PM Tim Bannister ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In keps/sig-network/2819-CNI-traffic-shaping-support/kep.yaml <#2808 (comment)> : > @@ -0,0 +1,35 @@ +title: CNI traffic shaping support +kep-number: 2819 +authors: + - ***@***.***" + - ***@***.***" +owning-sig: sig-network +participating-sigs: +status: implemented Would that mean that this is generally available? ⬇️ Suggested change -status: implemented +status: implementable ------------------------------ In keps/sig-network/2819-CNI-traffic-shaping-support/README.md <#2808 (comment)> : > +### CNI plugin part + +* [traffic shaping plugin](containernetworking/plugins#96) +* [support runtime config](containernetworking/plugins#138) + +### Kubernetes part + +* [add traffic shaping support](kubernetes/kubernetes#63194) + +## Drawbacks + +None + +## Alternatives + +None Let's document some alternative approaches. Rather than using annotations, we could add the network restrictions to the Pod API. That could allow the kubelet to report via .status whether the shaping was active. It also enables a richer control over flow shaping. I'd prefer to see Pods refer indirectly to a custom resource, eg NetworkTrafficQuota, but still have the Pod report enforcement via its .status. (Why? Because you might have nodes where this works and other nodes where it doesn't). — Reply to this email directly, view it on GitHub <#2808 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVBSYRI4CHWYB6UEQUDWOS5RHANCNFSM47YETHGA> . You are receiving this because you were assigned.Message ID: ***@***.***>

k8s-triage-robot · 2023-03-22T21:37:21Z

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2023-04-21T21:52:37Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle rotten
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

lordofire · 2023-04-27T07:12:22Z

/remove-lifecycle rotten

k8s-triage-robot · 2024-01-18T23:57:03Z

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2024-02-17T23:58:11Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle rotten
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot · 2024-03-19T00:46:00Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Reopen this PR with /reopen
Mark this PR as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot · 2024-03-19T00:46:04Z

@k8s-triage-robot: Closed this PR.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Reopen this PR with /reopen

Mark this PR as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jul 3, 2021

k8s-ci-robot requested review from caseydavenport and dcbw July 3, 2021 13:24

k8s-ci-robot added kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory sig/network Categorizes an issue or PR as relevant to SIG Network. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 3, 2021

k8s-ci-robot requested a review from thockin July 3, 2021 13:25

uablrek mentioned this pull request Jul 10, 2021

CNI traffic shaping support #2819

Closed

4 tasks

uablrek changed the title ~~KEP-0000: cni-bandwidth added~~ KEP-2819: CNI traffic shaping support Jul 10, 2021

Add KEP 2819: CNI traffic shaping support

c706d04

uablrek force-pushed the cni-bandwidth branch from 979f2cb to c706d04 Compare July 10, 2021 07:32

Align title

ac14230

Now "CNI traffic shaping support" in KEP, issue and PR.

sftim reviewed Jul 15, 2021

View reviewed changes

Typo

f39e499

Co-authored-by: Tim Bannister <tim@scalefactory.com>

k8s-ci-robot assigned thockin Sep 2, 2021

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 1, 2021

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 3, 2021

sftim mentioned this pull request Dec 16, 2021

Traffic shaping annotations not documented kubernetes/website#31001

Closed

thockin reviewed Dec 21, 2021

View reviewed changes

squeed reviewed May 24, 2022

View reviewed changes

uablrek mentioned this pull request May 26, 2022

As a user I want to be able to restrict IP families to some pods kubernetes/kubernetes#109814

Open

thockin modified the milestones: v1.25, v1.26 Jul 7, 2022

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 5, 2022

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 6, 2022

marosset removed this from the v1.26 milestone Dec 12, 2022

sftim reviewed Dec 22, 2022

View reviewed changes

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 22, 2023

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Apr 21, 2023

k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Apr 27, 2023

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 18, 2024

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Feb 17, 2024

thockin mentioned this pull request Feb 29, 2024

Allow admin to set the default bandwidth limit per port (network interface) for a container in a pod. kubernetes/kubernetes#123501

Open

k8s-ci-robot closed this Mar 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KEP-2819: CNI traffic shaping support #2808

KEP-2819: CNI traffic shaping support #2808

uablrek commented Jul 3, 2021 •

edited

Loading

k8s-ci-robot commented Jul 3, 2021

uablrek commented Jul 3, 2021

sftim commented Jul 6, 2021

uablrek commented Jul 6, 2021

sftim Jul 15, 2021

uablrek commented Sep 2, 2021

k8s-triage-robot commented Dec 1, 2021

frittentheke commented Dec 3, 2021

uablrek commented Dec 3, 2021

thockin left a comment

thockin commented Jan 24, 2022

uablrek commented Jan 25, 2022 •

edited

Loading

uablrek commented Jan 25, 2022

lordofire commented Apr 5, 2022

squeed May 24, 2022

sftim commented May 24, 2022

squeed May 24, 2022

squeed May 24, 2022

sftim May 24, 2022

k8s-triage-robot commented Oct 5, 2022

uablrek commented Oct 6, 2022

sftim Dec 22, 2022

sftim Dec 22, 2022

thockin commented Dec 22, 2022 via email

k8s-triage-robot commented Mar 22, 2023

k8s-triage-robot commented Apr 21, 2023

lordofire commented Apr 27, 2023

k8s-triage-robot commented Jan 18, 2024

k8s-triage-robot commented Feb 17, 2024

k8s-triage-robot commented Mar 19, 2024

k8s-ci-robot commented Mar 19, 2024

		"kubernetes.io/ingress-bandwidth": "10M",
		"kubernetes.io/egress-bandwidth": "10M"


		* [add traffic shaping support](https://github.com/kubernetes/kubernetes/pull/63194)

		## Drawbacks


		## Alternatives

		None

KEP-2819: CNI traffic shaping support #2808

KEP-2819: CNI traffic shaping support #2808

Conversation

uablrek commented Jul 3, 2021 • edited Loading

Note: This is not a new KEP. It is for regain tracking of an already-implemented feature (in alpha)

Links

k8s-ci-robot commented Jul 3, 2021

uablrek commented Jul 3, 2021

sftim commented Jul 6, 2021

uablrek commented Jul 6, 2021

sftim Jul 15, 2021

Choose a reason for hiding this comment

uablrek commented Sep 2, 2021

k8s-triage-robot commented Dec 1, 2021

frittentheke commented Dec 3, 2021

uablrek commented Dec 3, 2021

thockin left a comment

Choose a reason for hiding this comment

thockin commented Jan 24, 2022

uablrek commented Jan 25, 2022 • edited Loading

uablrek commented Jan 25, 2022

lordofire commented Apr 5, 2022

squeed May 24, 2022

Choose a reason for hiding this comment

sftim commented May 24, 2022

squeed May 24, 2022

Choose a reason for hiding this comment

squeed May 24, 2022

Choose a reason for hiding this comment

sftim May 24, 2022

Choose a reason for hiding this comment

k8s-triage-robot commented Oct 5, 2022

uablrek commented Oct 6, 2022

sftim Dec 22, 2022

Choose a reason for hiding this comment

sftim Dec 22, 2022

Choose a reason for hiding this comment

thockin commented Dec 22, 2022 via email

k8s-triage-robot commented Mar 22, 2023

k8s-triage-robot commented Apr 21, 2023

lordofire commented Apr 27, 2023

k8s-triage-robot commented Jan 18, 2024

k8s-triage-robot commented Feb 17, 2024

k8s-triage-robot commented Mar 19, 2024

k8s-ci-robot commented Mar 19, 2024

uablrek commented Jul 3, 2021 •

edited

Loading

uablrek commented Jan 25, 2022 •

edited

Loading