Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cri + selinux: /etc/hosts from hostPath mount getting relabeled #6194

Closed
dweomer opened this issue Nov 2, 2021 · 3 comments
Closed

cri + selinux: /etc/hosts from hostPath mount getting relabeled #6194

dweomer opened this issue Nov 2, 2021 · 3 comments

Comments

@dweomer
Copy link
Contributor

dweomer commented Nov 2, 2021

Description

When running rke2/k3s pointing at our bundled (or stock) containerd, one can apply an unprivileged pod that relabels /etc/hosts on the host by mounting a hostPath /etc/hosts volume at the same location in the container:

apiVersion: v1
kind: Pod
metadata:
  name: test-agnhost
spec:
  containers:
  - image: k8s.gcr.io/e2e-test-images/agnhost:2.32
    name: agnhost-1
    args:
    - pause
    volumeMounts:
    - mountPath: /etc/hosts
      name: host-etc-hosts
  volumes:
  - name: host-etc-hosts
    hostPath:
      path: /etc/hosts
      type: FileOrCreate

Steps to reproduce the issue

  1. Install RKE2 on CentOS 8.x with SELinux enabled (the default when installed via https://get.rke2.io)
  2. Establish that your /etc/hosts is correctly labeled:
    [vagrant@node-1 ~]$ ls -alZ /etc/hosts
    -rw-r--r--. 1 root root system_u:object_r:net_conf_t:s0 182 Nov  2 00:11 /etc/hosts
    [vagrant@node-1 ~]$ 
    
  3. Apply this pod spec:
    apiVersion: v1
    kind: Pod
    metadata:
      name: test-agnhost
    spec:
      containers:
      - image: k8s.gcr.io/e2e-test-images/agnhost:2.32
        name: agnhost-1
        args:
        - pause
        volumeMounts:
        - mountPath: /etc/hosts
          name: host-etc-hosts
      volumes:
      - name: host-etc-hosts
        hostPath:
          path: /etc/hosts
          type: FileOrCreate
  4. Wiat for the pod to spin up then check your /etc/hosts:
    [vagrant@node-1 ~]$ ls -alZ /etc/hosts
    -rw-r--r--. 1 root root system_u:object_r:container_file_t:s0:c31,c726 182 Nov  2 00:11 /etc/hosts
    
    [vagrant@node-1 ~]$ ps auxZ | grep c31,c726 | grep -v grep
    system_u:system_r:container_t:s0:c31,c726 65535 31673 0.0  0.0 972    4 ?        Ss   00:27   0:00 /pause
    system_u:system_r:container_t:s0:c31,c726 root 31753 0.0  1.2 740120 26100 ?     Ssl  00:27   0:00 /agnhost pause
    

Describe the results you received and expected

Expected: /etc/hosts retained net_conf_t type label with no categories
Received: /etc/hosts relabeled to container_file_t with category labels specific to the container that bind-mounted it

What version of containerd are you using?

v1.5.7 (and v1.5.7-k3s1)

Any other relevant information

$ kubectl get node -o wide
NAME     STATUS   ROLES                       AGE   VERSION          INTERNAL-IP       EXTERNAL-IP   OS-IMAGE          KERNEL-VERSION          CONTAINER-RUNTIME
node-1   Ready    control-plane,etcd,master   15m   v1.22.3+rke2r1   192.168.121.104   <none>        CentOS Stream 8   4.18.0-277.el8.x86_64   containerd://1.5.7-k3s1

First seen while running k8s e2e conformance via sonobuoy against rke2 v1.22.2-rc1+rke2r1 (most recently against v1.22.3-rc1+rke2r1): rancher/rke2#2068

(Also seen with k3s 1.22.x that also ships with containerd v1.5.7)

Show configuration if it is related to CRI plugin.

# [vagrant@node-1 ~]$ sudo cat /var/lib/rancher/rke2/agent/etc/containerd/config.toml

[plugins.opt]
  path = "/var/lib/rancher/rke2/agent/containerd"

[plugins.cri]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = true
  sandbox_image = "index.docker.io/rancher/pause:3.5"

[plugins.cri.containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true

[plugins.cri.containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"
@dweomer
Copy link
Contributor Author

dweomer commented Nov 2, 2021

The Vagrantfile I used to reproduce:

  • usage: INSTALL_RKE2_CHANNEL=testing vagrant up --provision-with=shell,install-containerd,install-runc node-1
    (stock containerd and runc)
  • usage: INSTALL_RKE2_CHANNEL=testing vagrant up node-1
    (rke2/k3s containerd and runc)
Vagrant.configure("2") do |config|
  config.vm.box = "centos/stream8"

  config.vm.provider :virtualbox do |v|
    v.memory = 2048
    v.cpus = 2
  end
  config.vm.provider :libvirt do |v|
    v.memory = 2048
    v.cpus = 2
  end

  config.vm.define "node-1", primary: true do |node|
      node.vm.hostname = "node-1"
      node.vm.provision "install-rke2", type: "shell", run: "once" do |sh|
        sh.upload_path = "/tmp/vagrant-install-rke2"
        sh.env = ENV.select{|k,v| k.start_with?('RKE2_') || k.start_with?('INSTALL_RKE2_')}.merge({
            :INSTALL_RKE2_TYPE => 'server',
            :RKE2_KUBECONFIG_MODE => '0664',
            :RKE2_TOKEN => 'test',
        })
        sh.inline = <<~SHELL
            #!/usr/bin/env bash
            set -eux -o pipefail
            curl -fsSL https://get.rke2.io | sh -
            echo >> /etc/sysconfig/rke2-${INSTALL_RKE2_TYPE}
            bash +x -ec 'sh -c export | while read x v; do echo $v; done | grep -E "^(RKE2|CONTAINERD)_"' >> /etc/sysconfig/rke2-${INSTALL_RKE2_TYPE}
            restorecon -Frv /usr/local/bin
            systemctl enable --now rke2-${INSTALL_RKE2_TYPE}
        SHELL
      end
  end

  config.vm.define "node-2" do |node|
      node.vm.hostname = "node-2"
      node.vm.provision "install-rke2", type: "shell", run: "once" do |sh|
        sh.upload_path = "/tmp/vagrant-install-rke2"
        sh.env = ENV.select{|k,v| k.start_with?('RKE2_') || k.start_with?('INSTALL_RKE2_')}.merge({
            :INSTALL_RKE2_TYPE => 'agent',
            :RKE2_TOKEN => 'test',
            :RKE2_URL => 'https://node-1:9345',
        })
        sh.inline = <<~SHELL
            #!/usr/bin/env bash
            set -eux -o pipefail
            curl -fsSL https://get.rke2.io | sh -
            echo >> /etc/sysconfig/rke2-${INSTALL_RKE2_TYPE}
            bash +x -ec 'sh -c export | while read x v; do echo $v; done | grep -E "^(RKE2|CONTAINERD)_"' >> /etc/sysconfig/rke2-${INSTALL_RKE2_TYPE}
            restorecon -Frv /usr/local/bin
            systemctl enable --now rke2-${INSTALL_RKE2_TYPE}
        SHELL
      end
  end

  config.vm.define "node-3" do |node|
      node.vm.hostname = "node-3"
      node.vm.provision "install-rke2", type: "shell", run: "once" do |sh|
        sh.upload_path = "/tmp/vagrant-install-rke2"
        sh.env = ENV.select{|k,v| k.start_with?('RKE2_') || k.start_with?('INSTALL_RKE2_')}.merge({
            :INSTALL_RKE2_TYPE => 'agent',
            :RKE2_TOKEN => 'test',
            :RKE2_URL => 'https://node-1:9345',
        })
        sh.inline = <<~SHELL
            #!/usr/bin/env bash
            set -eux -o pipefail
            curl -fsSL https://get.rke2.io | sh -
            echo >> /etc/sysconfig/rke2-${INSTALL_RKE2_TYPE}
            bash +x -ec 'sh -c export | while read x v; do echo $v; done | grep -E "^(RKE2|CONTAINERD)_"' >> /etc/sysconfig/rke2-${INSTALL_RKE2_TYPE}
            restorecon -Frv /usr/local/bin
            systemctl enable --now rke2-${INSTALL_RKE2_TYPE}
        SHELL
      end
  end

  config.vm.provision "disable-swap", type: "shell", run: "once" do |sh|
    sh.upload_path = "/tmp/vagrant-disable-swap"
    sh.inline = <<~SHELL
        #!/usr/bin/env bash
        set -eux -o pipefail
        if [ -f /swapfile ]; then
            swapoff -a
            sed -e 's/.*swapfile.*//g' -i /etc/fstab
            rm -vf /swapfile
        fi
    SHELL
  end

  # Disabled by default. To run:
  #   vagrant up --provision-with=upgrade-packages
  # To upgrade only specific packages:
  #   UPGRADE_PACKAGES=selinux vagrant up --provision-with=upgrade-packages
  #
  config.vm.provision "upgrade-packages", type: "shell", run: "never" do |sh|
    sh.upload_path = "/tmp/vagrant-upgrade-packages"
    sh.env = {
        'UPGRADE_PACKAGES': ENV['UPGRADE_PACKAGES'],
    }
    sh.inline = <<~SHELL
        #!/usr/bin/env bash
        set -eux -o pipefail
        yum -y upgrade ${UPGRADE_PACKAGES}
    SHELL
  end

  # To re-run, installing CNI from RPM:
  #   INSTALL_PACKAGES="containernetworking-plugins" vagrant up --provision-with=install-packages
  #
  config.vm.provision "install-packages", type: "shell", run: "once" do |sh|
    sh.upload_path = "/tmp/vagrant-install-packages"
    sh.env = {
        'INSTALL_PACKAGES': ENV['INSTALL_PACKAGES'],
    }
    sh.inline = <<~SHELL
        #!/usr/bin/env bash
        set -eux -o pipefail
        yum -y install \
            curl \
            iptables \
            less \
            lsof \
            nc \
            socat \
            ${INSTALL_PACKAGES}
    SHELL
  end

  # SELinux is Enforcing by default.
  # To set SELinux as Disabled on a VM that has already been provisioned:
  #   SELINUX=Disabled vagrant up --provision-with=selinux
  # To set SELinux as Permissive on a VM that has already been provsioned
  #   SELINUX=Permissive vagrant up --provision-with=selinux
  config.vm.provision "selinux", type: "shell", run: "once" do |sh|
    sh.upload_path = "/tmp/vagrant-selinux"
    sh.env = {
        'SELINUX': ENV['SELINUX'] || "Enforcing"
    }
    sh.inline = <<~SHELL
        #!/usr/bin/env bash
        set -eux -o pipefail

        if ! type -p getenforce setenforce &>/dev/null; then
          echo SELinux is Disabled
          exit 0
        fi

        case "${SELINUX}" in
          Disabled)
            if mountpoint -q /sys/fs/selinux; then
              setenforce 0
              umount -v /sys/fs/selinux
            fi
            ;;
          Enforcing)
            mountpoint -q /sys/fs/selinux || mount -o rw,relatime -t selinuxfs selinuxfs /sys/fs/selinux
            setenforce 1
            ;;
          Permissive)
            mountpoint -q /sys/fs/selinux || mount -o rw,relatime -t selinuxfs selinuxfs /sys/fs/selinux
            setenforce 0
            ;;
          *)
            echo "SELinux mode not supported: ${SELINUX}" >&2
            exit 1
            ;;
        esac
        sestatus -v
    SHELL
  end

  config.vm.provision "install-rke2-common", type: "shell", run: "once" do |sh|
    sh.upload_path = "/tmp/vagrant-install-rke2-common"
    sh.inline = <<~SHELL
        #!/usr/bin/env bash
        cat <<-EOF > /etc/profile.d/rke2.sh
export KUBECONFIG=/etc/rancher/rke2/rke2.yaml PATH=/usr/local/bin:$PATH:/var/lib/rancher/rke2/bin
EOF
    SHELL
  end

  config.vm.provision "install-runc", type: "shell", run: "never" do |sh|
    sh.upload_path = "/tmp/vagrant-install-runc"
    sh.inline = <<~SHELL
        #!/usr/bin/env bash
        set -eux -o pipefail
        curl -fsSL --output /usr/local/bin/runc https://github.com/opencontainers/runc/releases/download/v1.0.2/runc.amd64
        chmod -v +x /usr/local/bin/runc
    SHELL
  end

  config.vm.provision "install-containerd", type: "shell", run: "never" do |sh|
    sh.upload_path = "/tmp/vagrant-install-containerd"
    sh.inline = <<~SHELL
        #!/usr/bin/env bash
        set -eux -o pipefail
        mkdir -vp /usr/local/lib/systemd/system/containerd.service.d
        cat <<-EOF > /usr/local/lib/systemd/system/containerd.service.d/00-local.conf
[Service]
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/sysctl -w net.bridge.bridge-nf-call-iptables=1
ExecStartPre=-/sbin/sysctl -w net.bridge.bridge-nf-call-ip6tables=1
ExecStartPre=-/sbin/sysctl -w net.ipv4.conf.all.forwarding=1
ExecStartPre=-/sbin/sysctl -w net.ipv6.conf.all.forwarding=1
EOF
        mkdir -vp /etc/containerd
        cat <<-EOF > /etc/containerd/config.toml
[plugins.opt]
  path = "/var/lib/rancher/rke2/agent/containerd"

[plugins.cri]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = true
  sandbox_image = "index.docker.io/rancher/pause:3.5"

[plugins.cri.containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true

[plugins.cri.containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"
EOF
        cat <<-EOF > /etc/crictl.yaml
runtime-endpoint: /run/containerd/containerd.sock
EOF
        mkdir -vp /etc/rancher/rke2
        cat <<-EOF > /etc/rancher/rke2/config.yaml
container-runtime-endpoint: /run/containerd/containerd.sock
EOF
        curl -fsSL https://github.com/containerd/containerd/releases/download/v1.5.7/containerd-1.5.7-linux-amd64.tar.gz | tar -xvz -C /usr/local
        curl -fsSL --output /usr/local/lib/systemd/system/containerd.service \
            https://raw.githubusercontent.com/containerd/containerd/v1.5.7/containerd.service
        /usr/local/bin/containerd --version
        systemctl enable --now containerd
    SHELL
  end

end

@dweomer
Copy link
Contributor Author

dweomer commented Nov 2, 2021

Hmm, suspecting that either the kubelet is incorrectly telling the CRI to relabel or that opencontainer/selinux has a bug

@rwhitworth
Copy link

For other's searching for this information, I believe this issue relates to:
CVE-2021-43816
GHSA-mvff-h3cj-wj9c

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants