New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cri + selinux: /etc/hosts from hostPath mount getting relabeled #6194
Comments
|
The
Vagrant.configure("2") do |config|
config.vm.box = "centos/stream8"
config.vm.provider :virtualbox do |v|
v.memory = 2048
v.cpus = 2
end
config.vm.provider :libvirt do |v|
v.memory = 2048
v.cpus = 2
end
config.vm.define "node-1", primary: true do |node|
node.vm.hostname = "node-1"
node.vm.provision "install-rke2", type: "shell", run: "once" do |sh|
sh.upload_path = "/tmp/vagrant-install-rke2"
sh.env = ENV.select{|k,v| k.start_with?('RKE2_') || k.start_with?('INSTALL_RKE2_')}.merge({
:INSTALL_RKE2_TYPE => 'server',
:RKE2_KUBECONFIG_MODE => '0664',
:RKE2_TOKEN => 'test',
})
sh.inline = <<~SHELL
#!/usr/bin/env bash
set -eux -o pipefail
curl -fsSL https://get.rke2.io | sh -
echo >> /etc/sysconfig/rke2-${INSTALL_RKE2_TYPE}
bash +x -ec 'sh -c export | while read x v; do echo $v; done | grep -E "^(RKE2|CONTAINERD)_"' >> /etc/sysconfig/rke2-${INSTALL_RKE2_TYPE}
restorecon -Frv /usr/local/bin
systemctl enable --now rke2-${INSTALL_RKE2_TYPE}
SHELL
end
end
config.vm.define "node-2" do |node|
node.vm.hostname = "node-2"
node.vm.provision "install-rke2", type: "shell", run: "once" do |sh|
sh.upload_path = "/tmp/vagrant-install-rke2"
sh.env = ENV.select{|k,v| k.start_with?('RKE2_') || k.start_with?('INSTALL_RKE2_')}.merge({
:INSTALL_RKE2_TYPE => 'agent',
:RKE2_TOKEN => 'test',
:RKE2_URL => 'https://node-1:9345',
})
sh.inline = <<~SHELL
#!/usr/bin/env bash
set -eux -o pipefail
curl -fsSL https://get.rke2.io | sh -
echo >> /etc/sysconfig/rke2-${INSTALL_RKE2_TYPE}
bash +x -ec 'sh -c export | while read x v; do echo $v; done | grep -E "^(RKE2|CONTAINERD)_"' >> /etc/sysconfig/rke2-${INSTALL_RKE2_TYPE}
restorecon -Frv /usr/local/bin
systemctl enable --now rke2-${INSTALL_RKE2_TYPE}
SHELL
end
end
config.vm.define "node-3" do |node|
node.vm.hostname = "node-3"
node.vm.provision "install-rke2", type: "shell", run: "once" do |sh|
sh.upload_path = "/tmp/vagrant-install-rke2"
sh.env = ENV.select{|k,v| k.start_with?('RKE2_') || k.start_with?('INSTALL_RKE2_')}.merge({
:INSTALL_RKE2_TYPE => 'agent',
:RKE2_TOKEN => 'test',
:RKE2_URL => 'https://node-1:9345',
})
sh.inline = <<~SHELL
#!/usr/bin/env bash
set -eux -o pipefail
curl -fsSL https://get.rke2.io | sh -
echo >> /etc/sysconfig/rke2-${INSTALL_RKE2_TYPE}
bash +x -ec 'sh -c export | while read x v; do echo $v; done | grep -E "^(RKE2|CONTAINERD)_"' >> /etc/sysconfig/rke2-${INSTALL_RKE2_TYPE}
restorecon -Frv /usr/local/bin
systemctl enable --now rke2-${INSTALL_RKE2_TYPE}
SHELL
end
end
config.vm.provision "disable-swap", type: "shell", run: "once" do |sh|
sh.upload_path = "/tmp/vagrant-disable-swap"
sh.inline = <<~SHELL
#!/usr/bin/env bash
set -eux -o pipefail
if [ -f /swapfile ]; then
swapoff -a
sed -e 's/.*swapfile.*//g' -i /etc/fstab
rm -vf /swapfile
fi
SHELL
end
# Disabled by default. To run:
# vagrant up --provision-with=upgrade-packages
# To upgrade only specific packages:
# UPGRADE_PACKAGES=selinux vagrant up --provision-with=upgrade-packages
#
config.vm.provision "upgrade-packages", type: "shell", run: "never" do |sh|
sh.upload_path = "/tmp/vagrant-upgrade-packages"
sh.env = {
'UPGRADE_PACKAGES': ENV['UPGRADE_PACKAGES'],
}
sh.inline = <<~SHELL
#!/usr/bin/env bash
set -eux -o pipefail
yum -y upgrade ${UPGRADE_PACKAGES}
SHELL
end
# To re-run, installing CNI from RPM:
# INSTALL_PACKAGES="containernetworking-plugins" vagrant up --provision-with=install-packages
#
config.vm.provision "install-packages", type: "shell", run: "once" do |sh|
sh.upload_path = "/tmp/vagrant-install-packages"
sh.env = {
'INSTALL_PACKAGES': ENV['INSTALL_PACKAGES'],
}
sh.inline = <<~SHELL
#!/usr/bin/env bash
set -eux -o pipefail
yum -y install \
curl \
iptables \
less \
lsof \
nc \
socat \
${INSTALL_PACKAGES}
SHELL
end
# SELinux is Enforcing by default.
# To set SELinux as Disabled on a VM that has already been provisioned:
# SELINUX=Disabled vagrant up --provision-with=selinux
# To set SELinux as Permissive on a VM that has already been provsioned
# SELINUX=Permissive vagrant up --provision-with=selinux
config.vm.provision "selinux", type: "shell", run: "once" do |sh|
sh.upload_path = "/tmp/vagrant-selinux"
sh.env = {
'SELINUX': ENV['SELINUX'] || "Enforcing"
}
sh.inline = <<~SHELL
#!/usr/bin/env bash
set -eux -o pipefail
if ! type -p getenforce setenforce &>/dev/null; then
echo SELinux is Disabled
exit 0
fi
case "${SELINUX}" in
Disabled)
if mountpoint -q /sys/fs/selinux; then
setenforce 0
umount -v /sys/fs/selinux
fi
;;
Enforcing)
mountpoint -q /sys/fs/selinux || mount -o rw,relatime -t selinuxfs selinuxfs /sys/fs/selinux
setenforce 1
;;
Permissive)
mountpoint -q /sys/fs/selinux || mount -o rw,relatime -t selinuxfs selinuxfs /sys/fs/selinux
setenforce 0
;;
*)
echo "SELinux mode not supported: ${SELINUX}" >&2
exit 1
;;
esac
sestatus -v
SHELL
end
config.vm.provision "install-rke2-common", type: "shell", run: "once" do |sh|
sh.upload_path = "/tmp/vagrant-install-rke2-common"
sh.inline = <<~SHELL
#!/usr/bin/env bash
cat <<-EOF > /etc/profile.d/rke2.sh
export KUBECONFIG=/etc/rancher/rke2/rke2.yaml PATH=/usr/local/bin:$PATH:/var/lib/rancher/rke2/bin
EOF
SHELL
end
config.vm.provision "install-runc", type: "shell", run: "never" do |sh|
sh.upload_path = "/tmp/vagrant-install-runc"
sh.inline = <<~SHELL
#!/usr/bin/env bash
set -eux -o pipefail
curl -fsSL --output /usr/local/bin/runc https://github.com/opencontainers/runc/releases/download/v1.0.2/runc.amd64
chmod -v +x /usr/local/bin/runc
SHELL
end
config.vm.provision "install-containerd", type: "shell", run: "never" do |sh|
sh.upload_path = "/tmp/vagrant-install-containerd"
sh.inline = <<~SHELL
#!/usr/bin/env bash
set -eux -o pipefail
mkdir -vp /usr/local/lib/systemd/system/containerd.service.d
cat <<-EOF > /usr/local/lib/systemd/system/containerd.service.d/00-local.conf
[Service]
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/sysctl -w net.bridge.bridge-nf-call-iptables=1
ExecStartPre=-/sbin/sysctl -w net.bridge.bridge-nf-call-ip6tables=1
ExecStartPre=-/sbin/sysctl -w net.ipv4.conf.all.forwarding=1
ExecStartPre=-/sbin/sysctl -w net.ipv6.conf.all.forwarding=1
EOF
mkdir -vp /etc/containerd
cat <<-EOF > /etc/containerd/config.toml
[plugins.opt]
path = "/var/lib/rancher/rke2/agent/containerd"
[plugins.cri]
stream_server_address = "127.0.0.1"
stream_server_port = "10010"
enable_selinux = true
sandbox_image = "index.docker.io/rancher/pause:3.5"
[plugins.cri.containerd]
snapshotter = "overlayfs"
disable_snapshot_annotations = true
[plugins.cri.containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"
EOF
cat <<-EOF > /etc/crictl.yaml
runtime-endpoint: /run/containerd/containerd.sock
EOF
mkdir -vp /etc/rancher/rke2
cat <<-EOF > /etc/rancher/rke2/config.yaml
container-runtime-endpoint: /run/containerd/containerd.sock
EOF
curl -fsSL https://github.com/containerd/containerd/releases/download/v1.5.7/containerd-1.5.7-linux-amd64.tar.gz | tar -xvz -C /usr/local
curl -fsSL --output /usr/local/lib/systemd/system/containerd.service \
https://raw.githubusercontent.com/containerd/containerd/v1.5.7/containerd.service
/usr/local/bin/containerd --version
systemctl enable --now containerd
SHELL
end
end |
|
Hmm, suspecting that either the kubelet is incorrectly telling the CRI to relabel or that opencontainer/selinux has a bug |
|
For other's searching for this information, I believe this issue relates to: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
When running rke2/k3s pointing at our bundled (or stock) containerd, one can apply an unprivileged pod that relabels
/etc/hostson the host by mounting a hostPath/etc/hostsvolume at the same location in the container:Steps to reproduce the issue
/etc/hostsis correctly labeled:/etc/hosts:Describe the results you received and expected
Expected:
/etc/hostsretainednet_conf_ttype label with no categoriesReceived:
/etc/hostsrelabeled tocontainer_file_twith category labels specific to the container that bind-mounted itWhat version of containerd are you using?
v1.5.7 (and v1.5.7-k3s1)
Any other relevant information
First seen while running k8s e2e conformance via sonobuoy against rke2 v1.22.2-rc1+rke2r1 (most recently against v1.22.3-rc1+rke2r1): rancher/rke2#2068
(Also seen with k3s 1.22.x that also ships with containerd v1.5.7)
Show configuration if it is related to CRI plugin.
The text was updated successfully, but these errors were encountered: