node: topologymanager: Improved multi-numa alignment in Topology Manager #112914

PiotrProkop · 2022-10-07T13:01:55Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

We propose an enhancement to the TopologyManager that allows it to favor sets of NUMA nodes with shorter distance between nodes when making admission decisions. The proposed enhancement is only applicable when comparing sets of NUMA nodes that are equal in size.

Does this PR introduce a user-facing change?

Added a --topology-manager-policy-options flag to the kubelet to support fine tuning the topology manager policies. The first policy option, `prefer-closest-numa-nodes`, allows these policies to favor sets of NUMA nodes with shorter distance between nodes when making admission decisions.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

KEP PR: [Improved multi-numa alignment in Topology Manager#3549](KEP-3545: Improved multi-numa alignment in Topology Manager enhancements#3549
Enhancement Issue: Improved multi-numa alignment in Topology Manager#3545

k8s-ci-robot · 2022-10-07T13:02:04Z

Hi @PiotrProkop. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

PiotrProkop · 2022-10-07T13:03:38Z

/assign @klueska

k8s-triage-robot · 2022-10-07T13:24:27Z

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

ffromani · 2022-11-03T09:14:18Z

/lgtm
since last approval from @klueska , trivial rebase plus fix of the FG definition

@PiotrProkop
this is good enough. Next time, however, please try to apply fixes related to comments in their pertaining commits, and not change just the last commit in the series. Doing like this will require a interactive rebase (or rewriting history in general) + a force push (which, however, you will need anyway) but it will lead to a much tidier codebase.

PiotrProkop · 2022-11-03T09:18:00Z

@fromanirh you're right I could just edit appropiate commits, I can rearrange those commits now if you don't mind approving once again.

ffromani · 2022-11-03T09:19:58Z

@fromanirh you're right I could just edit appropiate commits, I can rearrange those commits now if you don't mind approving once again.

your call! I can review again, that's no bother.

PiotrProkop · 2022-11-03T09:27:47Z

@fromanirh PTAL

Signed-off-by: PiotrProkop <pprokop@nvidia.com>

…ManagerOptions Signed-off-by: PiotrProkop <pprokop@nvidia.com>

…numa policy Signed-off-by: PiotrProkop <pprokop@nvidia.com>

ffromani

/lgtm

ffromani · 2022-11-03T10:46:57Z

thanks for applying the cleanups!

derekwaynecarr

@klueska @PiotrProkop thanks for the commented code and test cases.

this looks good to me.

/lgtm
/approve

derekwaynecarr · 2022-11-07T19:20:42Z

cmd/kubelet/app/options/options.go

@@ -519,6 +519,7 @@ func AddKubeletConfigFlags(mainfs *pflag.FlagSet, c *kubeletconfig.KubeletConfig
 	fs.BoolVar(&c.ProtectKernelDefaults, "protect-kernel-defaults", c.ProtectKernelDefaults, "Default kubelet behaviour for kernel tuning. If set, kubelet errors if any of kernel tunables is different than kubelet defaults.")
 	fs.StringVar(&c.ReservedSystemCPUs, "reserved-cpus", c.ReservedSystemCPUs, "A comma-separated list of CPUs or CPU ranges that are reserved for system and kubernetes usage. This specific list will supersede cpu counts in --system-reserved and --kube-reserved.")
 	fs.StringVar(&c.TopologyManagerScope, "topology-manager-scope", c.TopologyManagerScope, "Scope to which topology hints applied. Topology Manager collects hints from Hint Providers and applies them to defined scope to ensure the pod admission. Possible values: 'container', 'pod'.")
+	fs.Var(cliflag.NewMapStringStringNoSplit(&c.TopologyManagerPolicyOptions), "topology-manager-policy-options", "A set of key=value Topology Manager policy options to use, to fine tune their behaviour. If not supplied, keep the default behaviour.")


@liggitt I think its fine to have this flag as its the same practice we followed for cpu manager policy options and it was the approach that was litigated during the kep process for this feature so changing this now doesn't feel right to me and is not different from past precedent.

see: https://github.com/kubernetes/enhancements/pull/3549/files#r980945432

if we get a future kep, i will take this advice into account so we can avoid more flag proliferation.

derekwaynecarr · 2022-11-07T19:24:55Z

pkg/features/kube_features.go

+	// alpha: v1.26
+	//
+	// Allow the usage of options to fine-tune the topology manager policies.
+	TopologyManagerPolicyOptions featuregate.Feature = "TopologyManagerPolicyOptions"


just noting this is the same pattern followed with CPUManagerPolicyOptions

derekwaynecarr · 2022-11-07T19:28:25Z

pkg/kubelet/cm/topologymanager/numa_info.go

+	nodeDir string
+}
+
+func NewNUMAInfo(topology []cadvisorapi.Node) (*NUMAInfo, error) {


thank you for the good set of unit tests!

klueska · 2022-11-07T20:39:59Z

/assign @liggitt
for final approval

liggitt · 2022-11-07T21:06:55Z

/approve
for API config changes

k8s-ci-robot · 2022-11-07T21:07:07Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: derekwaynecarr, klueska, liggitt, PiotrProkop

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~cmd/kubelet/OWNERS~~ [derekwaynecarr,klueska,liggitt]
~~pkg/features/OWNERS~~ [derekwaynecarr,liggitt]
~~pkg/generated/openapi/OWNERS~~ [liggitt]
~~pkg/kubelet/apis/config/OWNERS~~ [liggitt]
~~pkg/kubelet/cm/OWNERS~~ [derekwaynecarr,klueska,liggitt]
~~staging/src/k8s.io/kubelet/config/OWNERS~~ [liggitt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Oct 7, 2022

k8s-ci-robot requested review from saad-ali and sjenning October 7, 2022 13:02

k8s-ci-robot assigned klueska Oct 7, 2022

PiotrProkop changed the title ~~Add topology-manager-policy-options flag in Kubelet~~ node: topologymanager: Improved multi-numa alignment in Topology Manager Oct 7, 2022

k8s-ci-robot added do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. and removed release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Oct 7, 2022

PiotrProkop marked this pull request as draft October 7, 2022 13:46

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Oct 7, 2022

PiotrProkop force-pushed the topology-manager-policies-flag branch from c0a69c9 to 9df2b5f Compare October 7, 2022 14:05

PiotrProkop marked this pull request as ready for review October 7, 2022 14:06

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 7, 2022

k8s-ci-robot assigned ffromani Nov 3, 2022

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 3, 2022

PiotrProkop force-pushed the topology-manager-policies-flag branch from 5a9394a to 669fa12 Compare November 3, 2022 09:26

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 3, 2022

PiotrProkop added 3 commits November 3, 2022 10:45

Improved multi-numa alignment in Topology Manager: add NUMAInfo

58ef3f2

Signed-off-by: PiotrProkop <pprokop@nvidia.com>

Improved multi-numa alignment in Topology Manager: introduce Topology…

d5dd42d

…ManagerOptions Signed-off-by: PiotrProkop <pprokop@nvidia.com>

Improved multi-numa alignment in Topology Manager: implement closest …

75bb437

…numa policy Signed-off-by: PiotrProkop <pprokop@nvidia.com>

PiotrProkop force-pushed the topology-manager-policies-flag branch from 669fa12 to 75bb437 Compare November 3, 2022 09:47

ffromani reviewed Nov 3, 2022

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 3, 2022

derekwaynecarr approved these changes Nov 7, 2022

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 7, 2022

k8s-ci-robot merged commit 243ba08 into kubernetes:master Nov 8, 2022

SIG Node PR Triage automation moved this from Triage to Done Nov 8, 2022

k8s-ci-robot added this to the v1.26 milestone Nov 8, 2022

PiotrProkop mentioned this pull request Nov 8, 2022

topologymanager: document topology manager policy options kubernetes/website#37668

Merged

mkumatag mentioned this pull request Nov 9, 2022

Latest k8s fails with kublet panic runtime error #113781

Closed

klueska mentioned this pull request Nov 9, 2022

Fix discovering numa distance when node ids are not starting from 0 or it's ids are not sequential #113788

Merged

swatisehgal mentioned this pull request Nov 29, 2022

Remove NodeResourceTopology API from staging #96275

Merged

swatisehgal mentioned this pull request Feb 8, 2023

KEP-693: Graduate Kubelet Topology Manager to GA kubernetes/enhancements#3745

Merged

PiotrProkop mentioned this pull request Mar 3, 2023

REQUEST: New membership for PiotrProkop kubernetes/org#4063

Closed

9 tasks

klueska mentioned this pull request Apr 3, 2024

kubelet: Add a flag "MaxAllowableNUMANodes" #124148

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

node: topologymanager: Improved multi-numa alignment in Topology Manager #112914

node: topologymanager: Improved multi-numa alignment in Topology Manager #112914

PiotrProkop commented Oct 7, 2022 •

edited

k8s-ci-robot commented Oct 7, 2022

PiotrProkop commented Oct 7, 2022

k8s-triage-robot commented Oct 7, 2022

ffromani commented Nov 3, 2022

PiotrProkop commented Nov 3, 2022

ffromani commented Nov 3, 2022

PiotrProkop commented Nov 3, 2022

ffromani left a comment

ffromani commented Nov 3, 2022

derekwaynecarr left a comment

derekwaynecarr Nov 7, 2022

derekwaynecarr Nov 7, 2022

derekwaynecarr Nov 7, 2022

klueska commented Nov 7, 2022

liggitt commented Nov 7, 2022

k8s-ci-robot commented Nov 7, 2022

node: topologymanager: Improved multi-numa alignment in Topology Manager #112914

node: topologymanager: Improved multi-numa alignment in Topology Manager #112914

Conversation

PiotrProkop commented Oct 7, 2022 • edited

What type of PR is this?

What this PR does / why we need it:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot commented Oct 7, 2022

PiotrProkop commented Oct 7, 2022

k8s-triage-robot commented Oct 7, 2022

ffromani commented Nov 3, 2022

PiotrProkop commented Nov 3, 2022

ffromani commented Nov 3, 2022

PiotrProkop commented Nov 3, 2022

ffromani left a comment

Choose a reason for hiding this comment

ffromani commented Nov 3, 2022

derekwaynecarr left a comment

Choose a reason for hiding this comment

derekwaynecarr Nov 7, 2022

Choose a reason for hiding this comment

derekwaynecarr Nov 7, 2022

Choose a reason for hiding this comment

derekwaynecarr Nov 7, 2022

Choose a reason for hiding this comment

klueska commented Nov 7, 2022

liggitt commented Nov 7, 2022

k8s-ci-robot commented Nov 7, 2022

PiotrProkop commented Oct 7, 2022 •

edited