Race condition in Kubelet service

[PauloASilva]

@kubernetes/sig-api-machinery-bugs

What happened:

Race condition in Kubelet causes Kubernetes cluster to become unstable due to high CPU load when two nodes share the same hostname or --hostname-override value.

What you expected to happen:

Since node hostname or --hostname-override value is used to compute the etcd key name (/registry/minions/[NODE-HOSTNAME]), the system should either:

1. prevent nodes with a duplicate hostname or --hostname-override value to join the cluster,
2. add a prefix/suffix to the etcd key name (e.g. /registry/minions/[NODE-HOSTNAME]-[SUFFIX])

How to reproduce it (as minimally and precisely as possible):

1. Spawn 3 new servers

2. Setup Kubernetes on all 3 servers, according to the following table:

   Server ID	Hostname	Cluster Role
   0	k8s-master	Master
   1	k8s-node-1	Worker
   2	k8s-node-2	Worker

3. Start all 3 servers and initialize the cluster

4. Access the server w/ ID 1 and change its hostname to k8s-master

5. Run the kubeadm join with appropriate flags and tokens.

   You should see on stdout the following messages

   This node has joined the cluster:
   * Certificate signing request was sent to apiserver and a response was received.
   * The Kubelet was informed of the new secure connection details.
   
   Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
   

6. Access the server w/ ID 0 and run $ kubectl get nodes. The output should have a single entry.

7. Access the server w/ ID 2 and change its hostname to k8s-master

8. Repeat steps 5 and 6

9. Keep monitoring load average on server w/ ID 0.

Anything else we need to know?:

Running kubeadm join on nodes sharing the same hostname exits with a success exit code but kubectl get nodes on the control-pane fails to list them (only one is reported). This is also valid while changing the hostname of a node that already belongs to the cluster (e.g. sudo hostnamectl set-hostname DUPLICATE_HOSTNAME).

In such scenario, we can observe a severe CPU load increase on k8s master node, caused by concurrent updates to the etcd key /registry/minions/[NODE-HOSTNAME], which, in turn, generate several etcd events to be handled by k8s components.

Environment:

• Kubernetes version (use kubectl version):

  Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"clean", BuildDate:"2019-06-06T01:44:30Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"clean", BuildDate:"2019-06-06T01:36:19Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
  

• Cloud provider or hardware configuration:

  # lscpu
  Architecture:          x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Byte Order:            Little Endian
  CPU(s):                2
  On-line CPU(s) list:   0,1
  Thread(s) per core:    1
  Core(s) per socket:    1
  Socket(s):             2
  NUMA node(s):          1
  Vendor ID:             GenuineIntel
  CPU family:            6
  Model:                 6
  Model name:            QEMU Virtual CPU version 2.5+
  Stepping:              3
  CPU MHz:               2394.454
  BogoMIPS:              4788.90
  Hypervisor vendor:     KVM
  Virtualization type:   full
  L1d cache:             32K
  L1i cache:             32K
  L2 cache:              4096K
  L3 cache:              16384K
  NUMA node0 CPU(s):     0,1
  Flags:                 fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl xtopology eagerfpu pni cx16 x2apic hypervisor lahf_lm
  

  # free -h
                total        used        free      shared  buff/cache   available
  Mem:           990M        552M         68M         14M        369M        255M
  Swap:            0B          0B          0B
  

  # lshw -class network
  *-network
       description: Ethernet controller
       product: Virtio network device
       vendor: Red Hat, Inc.
       physical id: 3
       bus info: pci@0000:00:03.0
       version: 00
       width: 64 bits
       clock: 33MHz
       capabilities: msix bus_master cap_list rom
       configuration: driver=virtio-pci latency=0
       resources: irq:10 ioport:c0a0(size=32) memory:febd1000-febd1fff memory:fe000000-fe003fff memory:feb80000-febbffff
     *-virtio0
          description: Ethernet interface
          physical id: 0
          bus info: virtio@0
          logical name: eth0
          serial: 9e:a3:f5:c1:6e:36
          capabilities: ethernet physical
          configuration: broadcast=yes driver=virtio_net driverversion=1.0.0 ip=192.168.122.81 link=yes multicast=yes
  

• OS (e.g: cat /etc/os-release):

  NAME="CentOS Linux"
  VERSION="7 (Core)"
  ID="centos"
  ID_LIKE="rhel fedora"
  VERSION_ID="7"
  PRETTY_NAME="CentOS Linux 7 (Core)"
  ANSI_COLOR="0;31"
  CPE_NAME="cpe:/o:centos:centos:7"
  HOME_URL="https://www.centos.org/"
  BUG_REPORT_URL="https://bugs.centos.org/"
  CENTOS_MANTISBT_PROJECT="CentOS-7"
  CENTOS_MANTISBT_PROJECT_VERSION="7"
  REDHAT_SUPPORT_PRODUCT="centos"
  REDHAT_SUPPORT_PRODUCT_VERSION="7"
  

• Kernel (e.g. uname -a):

  Linux k8s-master 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
  

• Install tools:

  cat <<EOF > /etc/yum.repos.d/kubernetes.repo
  [kubernetes]
  name=Kubernetes
  baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
  enabled=1
  gpgcheck=1
  repo_gpgcheck=1
  gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
         https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
  EOF
  
  yum install kubeadm --nogpgcheck -y && \
    systemctl restart kubelet && systemctl enable kubelet
  

Cheers,
Paulo A. Silva

[PauloASilva]

@kubernetes/sig-api-machinery-bugs

[k8s-ci-robot]

@PauloASilva: Reiterating the mentions to trigger a notification:
@kubernetes/sig-api-machinery-bugs

In response to this:

@kubernetes/sig-api-machinery-bugs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[yue9944882]

/sig node

[mattjmcnaughton]

Thanks for reporting this issue @PauloASilva and thanks for the detailed bug report!

Could you share a little more about why multiple nodes would have the same hostname? I'm trying to figure out if the proper fix here is fixing this issue or making the implicit assumption that nodes in the same cluster don't have the same hostname more explicit.

[PauloASilva]

Hi @mattjmcnaughton,
I agree with you that there's no reason for two different nodes share the same hostname but let's look at it from a different perspective: security.

When targeting containerized applications an attacker will certainly look to break out of the container environment and execute code on the host OS with superuser privileges.

Security misconfiguration and running containers with known vulnerabilities are quite common, making this scenario a real thing.

In such a scenario, where the attacker is able to break out of the container and execute code on the host OS with superuser privileges, he will not only control the k8s worker node but also take down the cluster: as reported, this issue may lead to a Denial of Service.

We can also think about another scenario, not security related: a simple automation mistake, where a second worker node is deployed with a duplicate hostname would compromise the cluster.

To be honest, it doesn't look right to me, from a security standpoint, to use client input as (part of) a database key. Injection and key overriding are the first two things coming up to my mind.

I think the strongest argument in favor of using the hostname directly in the key name is the O(1) on key access: wouldn't it be possible to accomplish the same with other value than the hostname or adding a deterministic prefix/suffix to it?

Cheers,
Paulo A. Silva

[mattjmcnaughton]

Hmm, that's quite interesting!

If you believe this is a security issue, please reach out to the security team following the process outlined here: https://kubernetes.io/docs/reference/issues-security/security/. Let's wait for their response, and then we can take it from there.

[neolit123]

prevent nodes with a duplicate hostname or --hostname-override value to join the cluster,

this seems like the correct course. the "newer" kubelet should exit with an error.

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"clean", BuildDate:"2019-06-06T01:44:30Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"clean", BuildDate:"2019-06-06T01:36:19Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

what is the state in 1.15 and master?

[PauloASilva]

Hi @mattjmcnaughton,

Hmm, that's quite interesting!

If you believe this is a security issue, please reach out to the security team following the process outlined here: https://kubernetes.io/docs/reference/issues-security/security/. Let's wait for their response, and then we can take it from there.

Before opening this issue we have briefly discussed this with the security team and decided to make the discussion public to get more feedback from the community.

Let's put security aside for a second.
We know that with hostname collisions the database becomes inconsistent: it won't be able to provide accurate information about cluster nodes. Besides that, concurrent updates to the database key (/registry/minions/[NODE-HOSTNAME]) cause CPU load on k8s master node making the cluster unstable.

Isn't this enough to consider a different approach to compute the database key name?

If the answer to the previous question was "no", let's get back to the given exploitation scenario where an attacker, who was able to break out of a container and execute code on the host OS with superuser privileges, will be able to take down the cluster, using the controlled node to create a hostname collision (note that the node is already a member of the cluster).

@mattjmcnaughton May I ask what's your personal opinion on this?

Cheers,
Paulo A. Silva

[mattjmcnaughton]

Gotcha, thanks for the update :) My personal opinion is that I agree with you, but I'm not familiar with how the database key was originally decided. Could you ping whoever originally authored that code and see if they would be willing to weigh in? Thanks :)

[roycaihw]

cc @liggitt