Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

processes from /system.slice using cores assigned to guaranteed Pods when --cpu-manager-policy=static is set #85764

Open
subrnath opened this issue Nov 30, 2019 · 0 comments

Comments

@subrnath
Copy link

@subrnath subrnath commented Nov 30, 2019

What happened:

Following is the configuration for kubelet to enable the --cpu-manager-policy=static feature.

--cpu-manager-policy=static --system-reserved=cpu=10,memory=10Gi --system-reserved-cgroup=/system.slice.

There is no separate cgroup for the kubelet and hence --kube-reserved is not used.

./user.slice
./system.slice
./system.slice/system-getty.slice

PSR field of "ps -eLF " command on the host shows that processes from /system.slice are using cores assigned to the Application Pods which are guaranteed and integer cores assigned.

Here is /var/lib/kubelet/cpu_manager_state which shows that this Pod is assigned 8-11 and 32 - 35

{"policyName":"static","defaultCpuSet":"0-7,12-31,36-47","entries":{"f7cc858318f3a84d9cbd340e058126379292f09578ca362680e3a350c8edad63":"8-11,32-35"},"checksum":462128154}

on the Host,
ps -eLF | awk '{print $9 " " $13 " " $14 " " $2 " " $3}' | sort | grep -i glusterfs
8 /usr/sbin/glusterfs --log-level=ERROR 8870 1

Here 8 is the core ,PID 8870 and PPID is 1. Following command shows that cgroup of the PID 8870 is /system.slice

$ cat /proc/8870/cgroup
11:hugetlb:/
10:freezer:/
9:blkio:/system.slice/run-rf63a79fe920945f497cdb8ed37874e36.scope
8:memory:/system.slice/run-rf63a79fe920945f497cdb8ed37874e36.scope
7:pids:/system.slice/run-rf63a79fe920945f497cdb8ed37874e36.scope
6:cpuset:/
5:net_cls,net_prio:/
4:perf_event:/
3:cpu,cpuacct:/system.slice/run-rf63a79fe920945f497cdb8ed37874e36.scope
2:devices:/system.slice/run-rf63a79fe920945f497cdb8ed37874e36.scope
1:name=systemd:/system.slice/run-rf63a79fe920945f497cdb8ed37874e36.scope

Similarly there are other processes from the /system.slice like docker daemon etc also using the cores assigned to this Pod.

What you expected to happen:

No system process from /system.slice should be scheduled on the cores assigned to this guaranteed Pod having integer CPU

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version): 1.15.3
  • Cloud provider or hardware configuration:
  • OS (e.g: cat /etc/os-release):

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.4 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.4 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

  • Kernel (e.g. uname -a):

4.10.0-27-generic

  • Install tools:
  • Network plugin and version (if this is a network-related bug):
  • Others:

/area kubelet
/sig node

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.