Update Topology Manager documentation to include the scope feature #24781

k-wiatrzyk · 2020-10-29T16:03:13Z

PR refers to:

KEP's update PR: Update Topology Manager to support pod-level resource alignment enhancements#1752
Implementation PR: Topology Manager Scope (container | pod) Feature kubernetes#92967

Signed-off-by: Krzysztof Wiatrzyk k.wiatrzyk@samsung.com

k8sio-netlify-preview-bot · 2020-10-29T16:03:18Z

Deploy preview for kubernetes-io-vnext-staging processing.

Building with commit 4255393

https://app.netlify.com/sites/kubernetes-io-vnext-staging/deploys/5fbd06620af54a0008a770e9

k8s-ci-robot · 2020-10-29T16:03:21Z

Welcome @k-wiatrzyk!

It looks like this is your first PR to kubernetes/website 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/website has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

annajung · 2020-10-29T17:10:49Z

/milestone 1.20
/sig node
/hold
pending merge of kubernetes/kubernetes#92967

annajung · 2020-10-30T19:46:03Z

based on the comment kubernetes/enhancements#693 (comment)

this PR will be on /hold until an exception has been filed/approved.

annajung · 2020-11-04T14:15:05Z

exception approved
hold for k/k PR

annajung · 2020-11-05T21:11:28Z

/assign

annajung · 2020-11-11T00:41:08Z

Hi @k-wiatrzyk
Friendly reminder that docs ready for review deadline is coming up on Nov 23rd, right after Kubecon NA. We would appreciate it if you could provide some/all docs content for this feature early so that it can be reviewed by Sig Docs before Kubecon and other events happening in November.

If it's ready to be reviewed, could you also remove WIP from the title? Thank you!

k-wiatrzyk · 2020-11-12T15:51:09Z

Hi @annajung
Thanks! And I understand, time is crucial. I will give a final update to the docs tomorrow. The code has been merged so we are on the right track.

annajung · 2020-11-12T23:29:39Z

/hold cancel
k/k pr merged

sftim

Hi @k-wiatrzyk

LGTM!

Here's some optional feedback. What I mean is, I'd be happy to have this feedback accepted, but it's also OK to merge without it.

These changes need technical review; can you recommend someone who can provide that?

sftim · 2020-11-19T12:54:16Z

content/en/docs/tasks/administer-cluster/topology-manager.md

+### container scope
+
+This scope was available before any other scope was implemented, and it remains as the default setting in the kubelet.


Suggested change

### container scope

This scope was available before any other scope was implemented, and it remains as the default setting in the kubelet.

### container scope

The `container` scope is used by default.

+1 to suggested change

sftim · 2020-11-19T12:55:10Z

content/en/docs/tasks/administer-cluster/topology-manager.md

+### pod scope
+


Suggested change

### pod scope

### pod scope

To select the `pod` scope, start the kubelet with the command line option `--topology-manager-scope=pod`.

sftim · 2020-11-19T12:56:33Z

content/en/docs/tasks/administer-cluster/topology-manager.md

+* all containers can be and are allocated to a single NUMA node;
+* all containers can be and are allocated to a shared set of NUMA nodes.
+
+The total amount of particular resource demanded for the entire pod is calculated according to [effective requests/limits](/docs/concepts/workloads/pods/init-containers/#resources) formula, and thus, this total value is equal to the maximum of:


I'm not sure about the link to the init container concept - topology scope is relevant even if you never use init containers. Instead, I recommend duplicating the text here.

To be honest, I'm not sure if duplicating here is needed. The init container link also has other details (i.e. regarding scheduling).

Below the selected part there are two bullet points that describe the formula shortly. Init containers are taken into consideration if they exist in the pod. If not, only the maximum of app containers is considered.

...this total value is equal to the maximum of: * the sum of all app container requests, * the maximum of init container requests,

sftim · 2020-11-19T12:58:40Z

content/en/docs/tasks/administer-cluster/topology-manager.md

+* the maximum of init container requests,
+for a resource.
+
+The scope in tandem with `single-numa-node` Topology Manager Policy is specifically valuable for high-performance applications. By combining both options, we are able to place all containers in a pod on a single NUMA node; hence, the inter-NUMA communication overhead can be eliminated.


Suggested change

The scope in tandem with `single-numa-node` Topology Manager Policy is specifically valuable for high-performance applications. By combining both options, we are able to place all containers in a pod on a single NUMA node; hence, the inter-NUMA communication overhead can be eliminated.

Using the `pod` scope in tandem with `single-numa-node` Topology Manager policy is specifically valuable for workloads that are latency sensitive or for high-throughput applications that perform IPC. By combining both options, you are able to place all containers in a pod onto a single NUMA node; hence, the inter-NUMA communication overhead can be eliminated for that pod.

(this is a bunch of suggestions in one - some might not be technically accurate)

kbhawkey · 2020-11-19T16:13:07Z

content/en/docs/tasks/administer-cluster/topology-manager.md

+
+Within this scope, the Topology Manager performs a number of sequential resource alignments, i.e., for each container (in a pod) a separate alignment is computed. In other words, there is no notion of grouping the containers to a specific set of NUMA nodes, for this particular scope. In effect, the Topology Manager performs an arbitrary alignment of individual containers to NUMA nodes.
+
+The notion of grouping the containers was endorsed and implemented on purpose in the following scope, i.e. the Pod Scope.


nit: s/i.e./for example/
nit: What do you think about removing the capitalization of:
... for example, the pod scope.

Right, capitalization was inconsistent with the rest. Done

kbhawkey · 2020-11-19T16:17:54Z

Hi @k-wiatrzyk .
There are a few things to clean up, otherwise the docs look good. LGTM.

irvifa · 2020-11-19T21:55:05Z

/label tide/merge-method-squash

sftim · 2020-11-19T21:56:46Z

I only see 1 commit
/remove-label tide/merge-method-squash

k-wiatrzyk · 2020-11-20T11:48:41Z

@kbhawkey @sftim
Thanks for the reviews and LGTMs.

Pinged @klueska for a tech review

klueska · 2020-11-20T12:02:40Z

content/en/docs/tasks/administer-cluster/topology-manager.md

+### Topology Manager Scopes
+
+The Topology Manager can deal with the alignment of resources in a couple of distinct scopes:
+
+* `container` (default)
+* `pod`
+
+Either option can be selected at a time of the kubelet startup, with `--topology-manager-scope` flag.
+


I would probably combine the Scope and Policy sections into 1 (i.e. Topology Manager Scopes and Policies). That way you can have a quick intro into what both of them are and how they interact before diving into the details of them.

Thanks, I will prepare such an update today.

I think after this is addressed then this is good to go(?)

I hope so. I won't change any content here, only rearrange stuff.

@irvifa @klueska
I grouped scope and policies in the common section (## Topology Manager Scopes and Policies) and added a short description for it.

annajung · 2020-11-23T16:12:30Z

Hi @klueska @kubernetes/sig-node-pr-reviews can you please provide tech review / lgtm?

klueska · 2020-11-24T12:22:00Z

content/en/docs/tasks/administer-cluster/topology-manager.md

@@ -51,6 +51,52 @@ The hint is then stored in the Topology Manager for use by the *Hint Providers*

 Support for the Topology Manager requires `TopologyManager` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) to be enabled. It is enabled by default starting with Kubernetes 1.18.

+## Topology Manager Scopes and Policies
+


Can you move the intro text from the policy section to here and then reword what you currently have here to the following:

The Topology Manager currently: - Aligns Pods of all QoS classes. - Aligns the requested resources that Hint Provider provides topology hints for. If these conditions are met, the Topology Manager will align the requested resources. In order to customise how this alignment is carried out, the Topology Manager provides two distinct knobs: 'scope` and `policy`. The `scope` defines the granularity at which you would like resource alignment to be performed (e.g. at the `pod` or `container` level). And the `policy` defines the the actual strategy used to carry out the alignment (e.g. `best-effort`, `restricted`, `single-numa-node`, etc.). Details on the various `scopes` and `policies` available today can be found below. {{< note >}} To align CPU resources with other requested resources in a Pod Spec, the CPU Manager should be enabled and proper CPU Manager policy should be configured on a Node. See [control CPU Management Policies](/docs/tasks/administer-cluster/cpu-management-policies/). {{< /note >}}

Sure, applying now.

Done. Thanks!

Signed-off-by: Krzysztof Wiatrzyk <k.wiatrzyk@samsung.com>

klueska · 2020-11-24T14:02:40Z

/approve
/lgtm

k8s-ci-robot · 2020-11-24T14:02:57Z

LGTM label has been added.

Git tree hash: d2fef34dcb088b0cee486253b9857e3d3a441d1b

annajung · 2020-11-24T21:36:15Z

/approve

k8s-ci-robot · 2020-11-24T21:36:40Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: annajung, klueska

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~content/en/OWNERS~~ [annajung]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Oct 29, 2020

k8s-ci-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Oct 29, 2020

k8s-ci-robot requested review from celestehorgan and daminisatya October 29, 2020 16:03

k8s-ci-robot added language/en Issues or PRs related to English language sig/docs Categorizes an issue or PR as relevant to SIG Docs. labels Oct 29, 2020

k-wiatrzyk mentioned this pull request Oct 29, 2020

Node Topology Manager kubernetes/enhancements#693

Closed

11 tasks

k8s-ci-robot added sig/node Categorizes an issue or PR as relevant to SIG Node. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Oct 29, 2020

k8s-ci-robot added this to the 1.20 milestone Oct 29, 2020

pablitoergosum force-pushed the dev-1.20-tm-scope branch from 3f6ae6c to 87d2d40 Compare October 30, 2020 15:23

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Oct 30, 2020

k8s-ci-robot assigned annajung Nov 5, 2020

k-wiatrzyk force-pushed the dev-1.20-tm-scope branch from 87d2d40 to 149d2af Compare November 12, 2020 16:03

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 12, 2020

k-wiatrzyk force-pushed the dev-1.20-tm-scope branch from 149d2af to deb47f6 Compare November 13, 2020 14:01

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Nov 13, 2020

k-wiatrzyk changed the title ~~[WIP] Update Topology Manager documentation to include the scope feature~~ Update Topology Manager documentation to include the scope feature Nov 13, 2020

sftim reviewed Nov 19, 2020

View reviewed changes

kbhawkey reviewed Nov 19, 2020

View reviewed changes

k8s-ci-robot added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label Nov 19, 2020

k8s-ci-robot removed the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label Nov 19, 2020

k-wiatrzyk force-pushed the dev-1.20-tm-scope branch from ee5daf2 to dfe5fa7 Compare November 20, 2020 11:37

klueska reviewed Nov 20, 2020

View reviewed changes

k-wiatrzyk force-pushed the dev-1.20-tm-scope branch 3 times, most recently from 3b54845 to de37944 Compare November 23, 2020 16:00

klueska reviewed Nov 24, 2020

View reviewed changes

k-wiatrzyk force-pushed the dev-1.20-tm-scope branch 2 times, most recently from 064d178 to eaa0810 Compare November 24, 2020 13:07

irvifa requested review from klueska, kbhawkey and sftim November 24, 2020 13:08

Update Topology Manager documentation to include the scope feature

4255393

Signed-off-by: Krzysztof Wiatrzyk <k.wiatrzyk@samsung.com>

k-wiatrzyk force-pushed the dev-1.20-tm-scope branch from eaa0810 to 4255393 Compare November 24, 2020 13:10

k8s-ci-robot assigned klueska Nov 24, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 24, 2020

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 24, 2020

k8s-ci-robot merged commit 6fa3e91 into kubernetes:dev-1.20 Nov 24, 2020

		### container scope

		This scope was available before any other scope was implemented, and it remains as the default setting in the kubelet.

	The scope in tandem with `single-numa-node` Topology Manager Policy is specifically valuable for high-performance applications. By combining both options, we are able to place all containers in a pod on a single NUMA node; hence, the inter-NUMA communication overhead can be eliminated.
	Using the `pod` scope in tandem with `single-numa-node` Topology Manager policy is specifically valuable for workloads that are latency sensitive or for high-throughput applications that perform IPC. By combining both options, you are able to place all containers in a pod onto a single NUMA node; hence, the inter-NUMA communication overhead can be eliminated for that pod.


		Within this scope, the Topology Manager performs a number of sequential resource alignments, i.e., for each container (in a pod) a separate alignment is computed. In other words, there is no notion of grouping the containers to a specific set of NUMA nodes, for this particular scope. In effect, the Topology Manager performs an arbitrary alignment of individual containers to NUMA nodes.

		The notion of grouping the containers was endorsed and implemented on purpose in the following scope, i.e. the Pod Scope.

		@@ -51,6 +51,52 @@ The hint is then stored in the Topology Manager for use by the Hint Providers

		Support for the Topology Manager requires `TopologyManager` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) to be enabled. It is enabled by default starting with Kubernetes 1.18.

		## Topology Manager Scopes and Policies

Update Topology Manager documentation to include the scope feature #24781

Update Topology Manager documentation to include the scope feature #24781

Conversation

k-wiatrzyk commented Oct 29, 2020 • edited

k8sio-netlify-preview-bot commented Oct 29, 2020 • edited

k8s-ci-robot commented Oct 29, 2020

annajung commented Oct 29, 2020

annajung commented Oct 30, 2020

annajung commented Nov 4, 2020

annajung commented Nov 5, 2020

annajung commented Nov 11, 2020

k-wiatrzyk commented Nov 12, 2020

annajung commented Nov 12, 2020

sftim left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kbhawkey commented Nov 19, 2020

irvifa commented Nov 19, 2020

sftim commented Nov 19, 2020

k-wiatrzyk commented Nov 20, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

annajung commented Nov 23, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

klueska commented Nov 24, 2020

k8s-ci-robot commented Nov 24, 2020

annajung commented Nov 24, 2020

k8s-ci-robot commented Nov 24, 2020

k-wiatrzyk commented Oct 29, 2020 •

edited

k8sio-netlify-preview-bot commented Oct 29, 2020 •

edited