Provide functionality to start infra containers on the specified set of CPUs #4459

cynepco3hahue · 2021-01-06T18:43:07Z

What type of PR is this?

/kind api-change
/kind bug
/kind ci
/kind cleanup
/kind dependency-change
/kind deprecation
/kind design
/kind documentation
/kind failing-test

/kind feature

/kind flake
/kind other

What this PR does / why we need it:

Infra containers can start on every online CPU under the system that can be a problem for the latency-sensitive workloads because every context switch can result in latency peaks.

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Provide a new configuration flag to specify CPUs that will be used to run infra containers

openshift-ci-robot · 2021-01-06T18:43:16Z

Hi @cynepco3hahue. Thanks for your PR.

I'm waiting for a cri-o member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

haircommander · 2021-01-06T18:46:51Z

docs/crio.8.md

+**--infra-containers-cpus**="": CPU set to run infra containers, if not specified the CRI-O will use all online CPUs to run infra containers (default: "").
+    You can specify CPUs in the Linux CPU list format.


suggestion: I think --infra-cpus would work too
nit: we should describe that these should be the same as the guaranteed-cpus passed to the kubelet, if the user wants their workloads to truly have a unique cpuset

actually, above we use infra-ctr so we should use that here --infra-ctrs-cpus

How about infra-ctr-cpuset?

cpus is too broad and could also refer to amount of cpu shares reserved or cpu quota if we want to add those in the future. So, I think it may be better to be explicit.

makes sense, done

haircommander · 2021-01-06T18:47:43Z

internal/criocli/criocli.go

@@ -824,3 +839,15 @@ func StringSliceTrySplit(ctx *cli.Context, name string) []string {

 	return values
 }
+
+// cpuSetParse parses the the string


can you update this comment a bit?

Thanks, I forgot to do it, done

haircommander · 2021-01-06T18:53:21Z

internal/oci/oci.go

+func (r *Runtime) GetInfraCPUs() *cpuset.CPUSet {
+	return r.config.InfraContainersCPUs
+}
+


I'm not sure this really makes sense in the runtime object. Either in the server config (pkg/config/config.go) or in a special package makes more sense to me

I removed this function.

@haircommander Does it ok to have the infra_ctr_cpus parameter under the

[runtime] ... infra_ctr_cpus=<something> ...

or should I move this parameter under the separate section(probably in the future we can have the same functionality for the memory pinning and more).

haircommander · 2021-01-06T18:55:31Z

server/container_create_linux.go

+				if infraCPUs != nil {
+					specgen.SetLinuxResourcesCPUCpus(infraCPUs.String())
+				}


we don't want to set this here I believe. what this is doing is settting a new, non-infra container to use infra cpus, if it has cpus set. What we should do is set infra containers to be in those cpus (in server/sandbox_run_linux.go). Or, we can move the infra container to the cpus when the container is created (similar to how a container is in container_update_resources)

does that make sense?

thinking about this more, we should unconditionally set the infra container's cpuset if infra-ctr-cpus is set. we'll have to do that in server/sandbox_run_linux.go

codecov · 2021-01-06T19:13:01Z

Codecov Report

Merging #4459 (10c9e49) into master (6517447) will decrease coverage by 0.05%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master    #4459      +/-   ##
==========================================
- Coverage   40.09%   40.03%   -0.06%     
==========================================
  Files         116      116              
  Lines        9378     9391      +13     
==========================================
  Hits         3760     3760              
- Misses       5212     5224      +12     
- Partials      406      407       +1

haircommander · 2021-01-06T21:06:59Z

/ok-to-test

docs/crio.8.md

TomSweeneyRedHat · 2021-01-07T15:14:50Z

docs/crio.conf.5.md

@@ -225,6 +225,10 @@ the container runtime configuration.
 **drop_infra_ctr**=false
  Determines whether we drop the infra container when a pod does not have a private PID namespace, and does not use a kernel separating runtime (like kata).
  Requies **manage_ns_lifecycle** to be true.
+
+**infra-ctrs-cpus**=""
+    Determines the CPU set to run infra containers, if not specified the CRI-O will use all online CPUs to run infra containers.


Suggested change

Determines the CPU set to run infra containers, if not specified the CRI-O will use all online CPUs to run infra containers.

Determines the CPU set to run infra containers. If not specified, the CRI-O will use all online CPUs to run infra containers.

cynepco3hahue · 2021-01-07T17:19:56Z

/hold
I am still need to check it locally

cynepco3hahue · 2021-01-07T21:06:59Z

/hold cancel

haircommander · 2021-01-11T18:37:24Z

/retest

haircommander · 2021-01-11T19:53:11Z

/retest

cynepco3hahue · 2021-01-12T11:21:59Z

/retest

cynepco3hahue · 2021-01-12T12:56:21Z

/retest

TomSweeneyRedHat · 2021-01-12T14:42:21Z

docs/crio.conf.5.md

@@ -225,6 +225,11 @@ the container runtime configuration.
 **drop_infra_ctr**=false
  Determines whether we drop the infra container when a pod does not have a private PID namespace, and does not use a kernel separating runtime (like kata).
  Requies **manage_ns_lifecycle** to be true.
+
+**infra_ctr_cpuset**=""
+    Determines the CPU set to run infra containers. If not specified, the CRI-O will use all online CPUs to run infra containers.


Suggested change

Determines the CPU set to run infra containers. If not specified, the CRI-O will use all online CPUs to run infra containers.

Determines the CPU set to run infra containers. If not specified, CRI-O will use all online CPUs to run infra containers.

TomSweeneyRedHat · 2021-01-12T14:42:54Z

internal/criocli/criocli.go

@@ -794,6 +798,11 @@ func getCrioFlags(defConf *libconfig.Config) []cli.Flag {
 			EnvVars:   []string{"CONTAINER_VERSION_FILE_PERSIST"},
 			TakesFile: true,
 		},
+		&cli.StringFlag{
+			Name:    "infra-ctr-cpuset",
+			Usage:   "CPU set to run infra containers, if not specified the CRI-O will use all online CPUs to run infra containers (default: '').",


Suggested change

Usage: "CPU set to run infra containers, if not specified the CRI-O will use all online CPUs to run infra containers (default: '').",

Usage: "CPU set to run infra containers, if not specified CRI-O will use all online CPUs to run infra containers (default: '').",

The new flag should provide a way to specify desired set of CPUs used to run infra containers. Signed-off-by: Artyom Lukianov <alukiano@redhat.com>

Signed-off-by: Artyom Lukianov <alukiano@redhat.com>

haircommander · 2021-01-12T16:12:04Z

/retest

haircommander · 2021-01-12T19:40:41Z

/retest

haircommander · 2021-01-12T20:46:41Z

/lgtm

haircommander · 2021-01-12T21:12:51Z

/cherry-pick release-1.20

openshift-cherrypick-robot · 2021-01-12T21:13:00Z

@haircommander: #4459 failed to apply on top of branch "release-1.20":

Applying: Add new infra-containers-cpus to the CLI and config file
Applying: Add shell completion for infra-containers-cpu flag
Using index info to reconstruct a base tree...
M	completions/bash/crio
M	completions/fish/crio.fish
M	completions/zsh/_crio
Falling back to patching base and 3-way merge...
Auto-merging completions/zsh/_crio
CONFLICT (content): Merge conflict in completions/zsh/_crio
Auto-merging completions/fish/crio.fish
Auto-merging completions/bash/crio
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0002 Add shell completion for infra-containers-cpu flag
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-1.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

cynepco3hahue requested review from mrunalp and runcom as code owners January 6, 2021 18:43

openshift-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. kind/feature Categorizes issue or PR as related to a new feature. labels Jan 6, 2021

openshift-ci-robot requested review from haircommander and sameo January 6, 2021 18:43

openshift-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jan 6, 2021

haircommander reviewed Jan 6, 2021

View reviewed changes

openshift-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jan 6, 2021

cynepco3hahue force-pushed the provide_infra_cpu_flags branch from 59e83ac to 63c3495 Compare January 7, 2021 14:23

TomSweeneyRedHat reviewed Jan 7, 2021

View reviewed changes

docs/crio.8.md Show resolved Hide resolved

TomSweeneyRedHat reviewed Jan 7, 2021

View reviewed changes

cynepco3hahue force-pushed the provide_infra_cpu_flags branch 3 times, most recently from ce1fd99 to 428ce99 Compare January 7, 2021 16:20

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 7, 2021

cynepco3hahue force-pushed the provide_infra_cpu_flags branch from 428ce99 to fa1be15 Compare January 7, 2021 21:05

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 7, 2021

cynepco3hahue force-pushed the provide_infra_cpu_flags branch 2 times, most recently from 6417cff to fcf4242 Compare January 7, 2021 21:15

cynepco3hahue force-pushed the provide_infra_cpu_flags branch from b6ce24a to 70c00a9 Compare January 11, 2021 18:02

cynepco3hahue force-pushed the provide_infra_cpu_flags branch from 70c00a9 to f8fa688 Compare January 11, 2021 19:11

cynepco3hahue force-pushed the provide_infra_cpu_flags branch from f8fa688 to 269c415 Compare January 12, 2021 08:29

openshift-ci-robot added dco-signoff: no Indicates the PR's author has not DCO signed all their commits. and removed dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Jan 12, 2021

cynepco3hahue force-pushed the provide_infra_cpu_flags branch from a2ceff4 to b7e7ff7 Compare January 12, 2021 09:35

openshift-ci-robot added dco-signoff: yes Indicates the PR's author has DCO signed all their commits. and removed dco-signoff: no Indicates the PR's author has not DCO signed all their commits. labels Jan 12, 2021

TomSweeneyRedHat reviewed Jan 12, 2021

View reviewed changes

Artyom Lukianov added 4 commits January 12, 2021 17:48

Add new infra-containers-cpus to the CLI and config file

e0f7ccc

The new flag should provide a way to specify desired set of CPUs used to run infra containers. Signed-off-by: Artyom Lukianov <alukiano@redhat.com>

Add shell completion for infra-containers-cpu flag

b3fb25b

Signed-off-by: Artyom Lukianov <alukiano@redhat.com>

Set CPUs for the infra containers during the creation

5f9dbb1

Signed-off-by: Artyom Lukianov <alukiano@redhat.com>

Provide integration test for infra-ctr-cpuset feature

10c9e49

Signed-off-by: Artyom Lukianov <alukiano@redhat.com>

cynepco3hahue force-pushed the provide_infra_cpu_flags branch from b7e7ff7 to 10c9e49 Compare January 12, 2021 15:48

openshift-ci-robot assigned haircommander Jan 12, 2021

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jan 12, 2021

openshift-merge-robot merged commit 3a90a17 into cri-o:master Jan 12, 2021

haircommander mentioned this pull request Jan 12, 2021

[1.20] Provide functionality to start infra containers on the specified set of CPUs #4469

Merged

cynepco3hahue mentioned this pull request Jun 24, 2021

Guaranteed PODs CPUs are shared by other process running on the same host kubernetes/kubernetes#99895

Closed

james-masson mentioned this pull request Apr 30, 2024

Sandbox/Pause containers should run on a configurable cpuset to avoid interfering with low-jitter workloads containerd/containerd#10155

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Provide functionality to start infra containers on the specified set of CPUs #4459

Provide functionality to start infra containers on the specified set of CPUs #4459

cynepco3hahue commented Jan 6, 2021

openshift-ci-robot commented Jan 6, 2021

haircommander Jan 6, 2021

haircommander Jan 6, 2021

cynepco3hahue Jan 7, 2021

mrunalp Jan 7, 2021 •

edited

Loading

mrunalp Jan 7, 2021

cynepco3hahue Jan 7, 2021

haircommander Jan 6, 2021

cynepco3hahue Jan 7, 2021

haircommander Jan 6, 2021

cynepco3hahue Jan 7, 2021

haircommander Jan 6, 2021

haircommander Jan 6, 2021

cynepco3hahue Jan 7, 2021

codecov bot commented Jan 6, 2021 •

edited

Loading

haircommander commented Jan 6, 2021

TomSweeneyRedHat Jan 7, 2021

cynepco3hahue Jan 7, 2021

cynepco3hahue commented Jan 7, 2021

cynepco3hahue commented Jan 7, 2021

haircommander commented Jan 11, 2021

haircommander commented Jan 11, 2021

cynepco3hahue commented Jan 12, 2021

cynepco3hahue commented Jan 12, 2021

TomSweeneyRedHat Jan 12, 2021

cynepco3hahue Jan 12, 2021

TomSweeneyRedHat Jan 12, 2021

cynepco3hahue Jan 12, 2021

haircommander commented Jan 12, 2021

haircommander commented Jan 12, 2021

haircommander commented Jan 12, 2021

haircommander commented Jan 12, 2021

openshift-cherrypick-robot commented Jan 12, 2021

		--infra-containers-cpus="": CPU set to run infra containers, if not specified the CRI-O will use all online CPUs to run infra containers (default: "").
		You can specify CPUs in the Linux CPU list format.

	Determines the CPU set to run infra containers, if not specified the CRI-O will use all online CPUs to run infra containers.
	Determines the CPU set to run infra containers. If not specified, the CRI-O will use all online CPUs to run infra containers.

	Usage: "CPU set to run infra containers, if not specified the CRI-O will use all online CPUs to run infra containers (default: '').",
	Usage: "CPU set to run infra containers, if not specified CRI-O will use all online CPUs to run infra containers (default: '').",

Provide functionality to start infra containers on the specified set of CPUs #4459

Provide functionality to start infra containers on the specified set of CPUs #4459

Conversation

cynepco3hahue commented Jan 6, 2021

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

openshift-ci-robot commented Jan 6, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrunalp Jan 7, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jan 6, 2021 • edited Loading

Codecov Report

haircommander commented Jan 6, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cynepco3hahue commented Jan 7, 2021

cynepco3hahue commented Jan 7, 2021

haircommander commented Jan 11, 2021

haircommander commented Jan 11, 2021

cynepco3hahue commented Jan 12, 2021

cynepco3hahue commented Jan 12, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

haircommander commented Jan 12, 2021

haircommander commented Jan 12, 2021

haircommander commented Jan 12, 2021

haircommander commented Jan 12, 2021

openshift-cherrypick-robot commented Jan 12, 2021

mrunalp Jan 7, 2021 •

edited

Loading

codecov bot commented Jan 6, 2021 •

edited

Loading