cpu manager: handle reduced cpuset in root cgroup. #87522

ipuustin · 2020-01-24T14:47:35Z

If kubelet is started with a root cgroup which has a reduced cpuset (meaning it's not equal to the system CPU count), CPU manager may do CPU allocation to containers using CPUs which are not part of the available cpuset. This would lead to the container runtime cpuset write failing with a "permission denied" error and the container failing to launch. This PR enables CPU manage static policy to read the root cgroup cpuset value to prevent this from happening.

This PR can be tested by creating a new cgroup (foobar) and adjusting it's top-level cpuset to be smaller than the amount of available CPUs in the system. Kubelet can then be launched with the following options:

kubelet --cpu-manager-policy=static --system-reserved=cpu=1  --cgroup-root=foobar

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespace from that line:

/kind api-change

/kind bug

/kind cleanup
/kind deprecation
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:
This PR is missing tests. I was thinking of adding them once I get some green light to this fix.

Does this PR introduce a user-facing change?:

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot · 2020-01-24T14:47:44Z

Hi @ipuustin. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2020-01-24T14:48:54Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ipuustin
To complete the pull request process, please assign connordoyle
You can assign the PR to them by writing /assign @connordoyle in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

pkg/kubelet/cm/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ipuustin · 2020-01-27T16:17:48Z

/cc @ConnorDoyle

mattjmcnaughton · 2020-01-28T13:40:23Z

/ok-to-test

If kubelet is started with root cgroup which has a reduced cpuset (meaning it's not equal to the system CPU count), CPU manager may give out exclusive CPUs to the containers outside of the allowed cpuset. Read the root cgroup cpuset value to prevent this from happening.

ipuustin · 2020-01-29T09:19:51Z

Fixed a bunch of tests and added some new ones.

ipuustin · 2020-01-29T09:33:08Z

/retest

ipuustin · 2020-01-29T19:49:59Z

/retest

marquiz

Codewise looks good to me, just one minor comment

marquiz · 2020-02-06T09:23:43Z

pkg/kubelet/cm/cpumanager/cpu_manager.go

@@ -121,7 +121,7 @@ func (s *sourcesReadyStub) AddSource(source string) {}
 func (s *sourcesReadyStub) AllReady() bool          { return true }

 // NewManager creates new cpu manager based on provided policy
-func NewManager(cpuPolicyName string, reconcilePeriod time.Duration, machineInfo *cadvisorapi.MachineInfo, numaNodeInfo topology.NUMANodeInfo, specificCPUs cpuset.CPUSet, nodeAllocatableReservation v1.ResourceList, stateFileDirectory string, affinity topologymanager.Store) (Manager, error) {
+func NewManager(cpuPolicyName string, reconcilePeriod time.Duration, machineInfo *cadvisorapi.MachineInfo, numaNodeInfo topology.NUMANodeInfo, specificCPUs cpuset.CPUSet, nodeAllocatableReservation v1.ResourceList, stateFileDirectory string, affinity topologymanager.Store, cgroupPath string) (Manager, error) {


Just thinking would cpusetPath or similar be better name for the argument(?)

k8s-ci-robot · 2020-03-05T19:30:27Z

@ipuustin: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

fejta-bot · 2020-06-03T19:42:34Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot · 2020-07-03T20:24:35Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot · 2020-08-02T21:05:50Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

k8s-ci-robot · 2020-08-02T21:06:02Z

@fejta-bot: Closed this PR.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jan 24, 2020

k8s-ci-robot added area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jan 24, 2020

k8s-ci-robot requested review from klueska and sjenning January 24, 2020 14:49

k8s-ci-robot requested a review from ConnorDoyle January 27, 2020 16:17

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jan 28, 2020

ipuustin added 2 commits January 29, 2020 11:16

cpu manager: tests for reduced cgroup cpuset.

f8deca2

ipuustin force-pushed the cpuset-cgroup-for-cpu-manager branch from 62c4d17 to f8deca2 Compare January 29, 2020 09:19

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jan 29, 2020

ipuustin mentioned this pull request Jan 29, 2020

staticCPUs kubelet option to specify a restricted list of CPUs for exclusive assignment #87532

Closed

ipuustin mentioned this pull request Feb 5, 2020

REQUEST: New membership for ipuustin kubernetes/org#1615

Closed

6 tasks

marquiz reviewed Feb 6, 2020

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 5, 2020

marquiz mentioned this pull request May 15, 2020

REQUEST: New membership for marquiz kubernetes/org#1880

Closed

6 tasks

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 3, 2020

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jul 3, 2020

k8s-ci-robot closed this Aug 2, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cpu manager: handle reduced cpuset in root cgroup. #87522

cpu manager: handle reduced cpuset in root cgroup. #87522

ipuustin commented Jan 24, 2020

k8s-ci-robot commented Jan 24, 2020

k8s-ci-robot commented Jan 24, 2020

ipuustin commented Jan 27, 2020

mattjmcnaughton commented Jan 28, 2020

ipuustin commented Jan 29, 2020

ipuustin commented Jan 29, 2020

ipuustin commented Jan 29, 2020

marquiz left a comment

marquiz Feb 6, 2020

k8s-ci-robot commented Mar 5, 2020

fejta-bot commented Jun 3, 2020

fejta-bot commented Jul 3, 2020

fejta-bot commented Aug 2, 2020

k8s-ci-robot commented Aug 2, 2020

cpu manager: handle reduced cpuset in root cgroup. #87522

cpu manager: handle reduced cpuset in root cgroup. #87522

Conversation

ipuustin commented Jan 24, 2020

k8s-ci-robot commented Jan 24, 2020

k8s-ci-robot commented Jan 24, 2020

ipuustin commented Jan 27, 2020

mattjmcnaughton commented Jan 28, 2020

ipuustin commented Jan 29, 2020

ipuustin commented Jan 29, 2020

ipuustin commented Jan 29, 2020

marquiz left a comment

Choose a reason for hiding this comment

marquiz Feb 6, 2020

Choose a reason for hiding this comment

k8s-ci-robot commented Mar 5, 2020

fejta-bot commented Jun 3, 2020

fejta-bot commented Jul 3, 2020

fejta-bot commented Aug 2, 2020

k8s-ci-robot commented Aug 2, 2020