k3s-selinux testing rpm does not have a compatible version with Rhel 9+

[rancher-max]

Environmental Info:
K3s Version:

N/A

Node(s) CPU architecture, OS, and Version:

$ uname -a && cat /etc/os-release 
Linux ip-172-31-4-177.us-east-2.compute.internal 5.14.0-162.6.1.el9_1.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Sep 30 07:36:03 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux
NAME="Red Hat Enterprise Linux"
VERSION="9.1 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.1"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.1 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/9/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.1
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.1"

Cluster Configuration:

N/A

Describe the bug:

There is no RPM built yet in the new testing RPMs for EL9. Using latest: https://github.com/k3s-io/k3s-selinux/releases/tag/v1.3.testing.4. Therefore, the rpm fails to install on el9 machines due to incompatible container-selinux version.

Steps To Reproduce:

1. Install container-selinux rpm: sudo yum install -y container-selinux
2. Attempt to install k3s-selinux rpm: sudo rpm -i https://github.com/k3s-io/k3s-selinux/releases/download/v1.3.testing.4/k3s-selinux-1.3-4.el8.noarch.rpm

Expected behavior:

This should install the RPM successfully. OR there should be an el9 rpm that is available for el9 machines.

Actual behavior:

This fails to install with the following error:

warning: /var/tmp/rpm-tmp.fYXWX2: Header V3 RSA/SHA256 Signature, key ID d161f542: NOKEY
error: Failed dependencies:
	container-selinux < 2:2.191.0-1 is needed by k3s-selinux-1.3-4.el8.noarch

Additional context / logs:

$ rpm -qa | grep -i container-selinux
container-selinux-2.189.0-1.el9.noarch

$ sudo yum --showduplicates list container-selinux | expand
...
Installed Packages
container-selinux.noarch      3:2.189.0-1.el9        @rhel-9-appstream-rhui-rpms

It appears the epoch has increased from 2 to 3 and fails this check: Requires(post): container-selinux >= %{container_policy_epoch}:%{container_policyver}, container-selinux < %{container_policy_epoch}:%{container_policyver_max}

[galal-hussein]

@rancher-max @ShylajaDevadiga I think we should test this issue while the PR is open in master since its directly affecting all installations once its merged.

[mdrahman-suse]

@galal-hussein I am seeing the below with v1.3.testing.7/k3s-selinux-1.3-7.el9.noarch.rpm

$ sudo rpm -i https://github.com/k3s-io/k3s-selinux/releases/download/v1.3.testing.7/k3s-selinux-1.3-7.el9.noarch.rpm
warning: /var/tmp/rpm-tmp.GFYilo: Header V3 RSA/SHA256 Signature, key ID d161f542: NOKEY
Conflicting name type transition rules
Binary policy creation failed at /var/lib/selinux/targeted/tmp/modules/200/k3s/cil:135
Failed to generate binary
/usr/sbin/semodule:  Failed!
uavc:  op=load_policy lsm=selinrpm -qa | grep -i container-selinux-45-232 ~]

$ rpm -qa | grep -i container-selinux
container-selinux-2.205.0-1.el9_2.noarch

$ sudo yum --showduplicates list container-selinux | expand
...
Installed Packages
container-selinux.noarch      3:2.205.0-1.el9_2      @rhel-9-appstream-rhui-rpms

CC @rancher-max

[mdrahman-suse]

Verified using the commands below on Rhel 9.1

• Config.yaml

write-kubeconfig-mode: 644
cluster-init: true
node-name: server

$ sudo yum install -y container-selinux

$ curl -sfL https://raw.githubusercontent.com/galal-hussein/k3s/add_el9_selinux_rpm/install.sh | INSTALL_K3S_CHANNEL=testing sh -

$ kubectl get nodes
NAME     STATUS   ROLES                       AGE   VERSION
server   Ready    control-plane,etcd,master   5m   v1.27.1-rc3+k3s1

$ kubectl get pods -A
NAMESPACE     NAME                                     READY   STATUS      RESTARTS   AGE
kube-system   coredns-77ccd57875-8bbb2                 1/1     Running     0          8m
kube-system   helm-install-traefik-crd-npsqg           0/1     Completed   0          8m
kube-system   helm-install-traefik-w8zj4               0/1     Completed   1          8m
kube-system   local-path-provisioner-957fdf8bc-z68hn   1/1     Running     0          8m
kube-system   metrics-server-54dc485875-std7t          1/1     Running     0          8m
kube-system   svclb-traefik-7c672fc4-mrcqc             2/2     Running     0          8m
kube-system   traefik-84745cf649-ndh2p                 1/1     Running     0          8m

[rancher-max]

Validated in testing channel with latest install script from #7443

Multiple OSes were covered as part of this, including: RHEL 9.1, RHEL 8.7, Centos 7.8, Centos Stream 9, SLE Micro 5.4, SLES 15 SP4, Rocky Linux 8.6, Rocky Linux 9.1, and Fedora CoreOS 38. The solution involved creating RPMs for k3s-selinux where we didn't have them before (like for sle micro and coreos) and ensuring that some OSes us the latest container-selinux (>=191) and others pin a container-selinux that it was previously using (<191). Currently, these fixes are available only in the testing channel, but will be pushed out to latest and stable at the same time as the May patch releases (very soon).

Some of the el8 distros have updated container-selinux, but not all, so across the board that container-selinux version is being required to be <191. Same with "*sle*" distros. EL9 and coreos will use >=191.

The scenarios covered were:

1. Fresh install using new install script and pointing to testing channel
2. Upgrade by:
   a. Install with older install script and stable channel
   b. Upgrade using new install script and testing channel
3. For all cases, validate sudo ls -laZ /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/ results in either container_file_t for container-selinux >= 191, and container_ro_file_t for container-selinux < 191.

In all scenarios, the following config file was used:

# /etc/rancher/k3s/config.yaml
write-kubeconfig-mode: 644
selinux: true

Please be advised: for Fedora CoreOS and SLE Micro, a reboot is required after install. This is a limitation of the OSes themselves. The install script gives a warning indicating this is required, but it's easy to miss when using existing scripting to install k3s.

I'd also like to add that as and when the OSes currently using container-selinux < 191 update to the latest version, they might break. We will be keeping a pulse on the supported operating systems versions and update accordingly when it's appropriate.