Cannot run rancher on hardened system

[rancher-max]

Environmental Info:
RKE2 Version:

v1.22.12+rke2r1, v1.23.9+rke2r1, and v1.24.3+rke2r1

Node(s) CPU architecture, OS, and Version:

Any

Cluster Configuration:

3 servers. Also repro'ed on just 1 server.

Describe the bug:

After bringing up a hardened rke2 cluster and running rancher, I cannot access the rancher UI. The rancher pods appear to all be running, and the UI appears to be accessible via ClusterIP, but it fails to open when using the specified hostname.
I am using AWS in this testing and see that the TargetGroups setup to my LB are showing Unhealthy on 443 and 80 (though 9345 and 6443 are both showing Healthy).

Steps To Reproduce:

1. Install rke2 using profile: cis-1.6 in the config. Also include a tls-san using a valid hostname.
2. Ensure the nodes are all setup to recognize the hostname (for example, with AWS I register the nodes to four TargetGroups with ports: 6443, 9345, 80, and 443. Then attach those to a LoadBalancer. Then I create a route53 record pointing to the LoadBalancer DNS. I use the route53 record as the hostname in my tls-san in step 1 above).
3. Install rancher:

# Update helm, setup namespaces, and install cert-manager
helm repo add rancher-latest https://releases.rancher.com/server-charts/latest && \
helm repo add jetstack https://charts.jetstack.io && \
helm repo update && \
kubectl create namespace cattle-system && \
kubectl create namespace cert-manager && \
kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v1.8.0/cert-manager.crds.yaml && \
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --version v1.8.0

# Ensure cert-manager pods are running:
kubectl get pods --namespace cert-manager

# Install rancher. I've confirmed this happens using 2.6.7-rc5 and 2.6.6 both:
helm install rancher rancher-latest/rancher \
  --namespace cattle-system \
  --set hostname=my.redacted.hostname \
  --set rancherImageTag=v2.6.7-rc5 \
  --version=v2.6.7-rc5

4. The above steps will finish with an output that contains information to access the UI. Wait until all the pods are available, then access that link:

kubectl -n cattle-system rollout status deploy/rancher

Expected behavior:

The Rancher UI should be accessible.

Actual behavior:

The Rancher UI is not accessible at all.

Additional context / logs:

I believe this is related to the changes from #2206 probably mixed with the PSPs and NetworkPolicies we have in a hardened setup. I think the fix should include updating those netpols or psps to account for this hostnetwork change. The rancher pods do NOT deploy with any specific hostnetwork setting as far as I can tell, and I can't see any errors in the logs:
rancher-pod-logs.log
ingress-nginx.log

I'm happy to reproduce and gather any more information necessary.

[rajha-korithrien]

I can confirm this occurs on RKE2 v1.23.9 with the setting: profile: cis-1.6
The issue is that the rke2-ingress-nginx-controller Pods do not actually bind to the host ports 80 and 443

I have 3 nodes:

kubectl --kubeconfig rke2.yaml get nodes
NAME                   STATUS   ROLES                       AGE   VERSION
rancher-0   Ready    control-plane,etcd,master   17h   v1.23.9+rke2r1
rancher-1   Ready    control-plane,etcd,master   16h   v1.23.9+rke2r1
rancher-2   Ready    control-plane,etcd,master   15h   v1.23.9+rke2r1

And I have 3 ingress controller pods:

kubectl --kubeconfig rke2.yaml -n kube-system get pods
...
rke2-ingress-nginx-controller-h7b89                     1/1     Running     0          16h
rke2-ingress-nginx-controller-mpl7j                     1/1     Running     0          15h
rke2-ingress-nginx-controller-wxvdg                     1/1     Running     0          17h

I can verify the host ports are set and get the host for one of the Ingress controllers:

kubectl --kubeconfig rke2.yaml -n kube-system describe pod rke2-ingress-nginx-controller-h7b89
...
Node:         rancher-1/192.168.20.41
...
    Ports:         80/TCP, 443/TCP, 8443/TCP
    Host Ports:    80/TCP, 443/TCP, 0/TCP
...

When I ssh into that host, a netstat shows that nothing is listening on port 80 nor port 443:

[root@rancher-1 ~]# netstat -nlp | grep 80
tcp        0      0 192.168.20.41:2380      0.0.0.0:*               LISTEN      1659015/etcd
tcp        0      0 127.0.0.1:2380          0.0.0.0:*               LISTEN      1659015/etcd
tcp        0      0 127.0.0.1:10257         0.0.0.0:*               LISTEN      1659280/kube-contro
unix  2      [ ACC ]     STREAM     LISTENING     15406414 1880333/containerd-  /run/containerd/s/ccf8270fc5a7e0310a0b43f7a0842067755a49c5129a017ca9d4e88e531f6fcb
unix  2      [ ACC ]     STREAM     LISTENING     13580682 1659801/containerd-  /run/containerd/s/25dcbecccd7aa73a1c3467c7afb7acbac28f552bee53d4c8a772e2a69fc965fe
unix  2      [ ACC ]     STREAM     LISTENING     15373978 1878167/containerd-  /run/containerd/s/c8254eafebf386e47958d39fa84ed358b115258d7c3eeb0280c82fbf11107846
unix  2      [ ACC ]     STREAM     LISTENING     15405507 1880338/containerd-  /run/containerd/s/8bc8ae10ec1187c6895b3d21e109fed2364a52b058ba8043da648823ad823cd3
unix  2      [ ACC ]     STREAM     LISTENING     13577674 1659128/containerd-  /run/containerd/s/fdf9eef375d7ef47a06abe3f916948f300d5ca280eb5a2c46cd30e4a5c633b13
unix  2      [ ACC ]     STREAM     LISTENING     13577698 1659154/containerd-  /run/containerd/s/de51fe32ffe7b364f5b9749ed1d9f99643efa2f85724e23d4c0fe0ba4280a390
[root@prometheus-rancher-1 ~]# netstat -nlp | grep 443
tcp6       0      0 :::6443                 :::*                    LISTEN      1659091/kube-apiser
[root@rancher-1 ~]#

I believe that the nginx process in the container should be bound to the host port, and that doesn't appear to be the case.

[brandond]

As Max noted, the PR he linked disabled binding the pod to the host network. This shouldn't be necessary however; it works fine without host net when not hardened, and works fine without host net on RKE1. Needs further investigation.

[brandond]

As a temporary workaround for those that don't want to downgrade, you can deploy the following to revert to the chart values from previous releases:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-ingress-nginx
  namespace: kube-system
spec:
  valuesContent: |-
    controller:
      hostNetwork: true

If someone is able, I am curious if the following might also work:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-ingress-nginx
  namespace: kube-system
spec:
  valuesContent: |-
    controller:
      dnsPolicy: ClusterFirst

EDIT: this didn't work

[galal-hussein]

Applying a different dnsPolicy didnt appear to have any effect on the ingress nginx, but I think I found the root cause, since we were using hostnNetwork mode for nginx before, it was bypassing the Networkpolicy rules we add in CIS mode, so it was working fine up until we switched to hostPorts, which prevents traffic from outside the kube-system namespace to go to the ingress nginx pod on ports 443 and 80, to fix that we need to add an extra policy in CIS mode in addition to:

default-network-dns-policy    
default-network-policy        

The policy will look something like that:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  annotations:
  name: default-network-ingress-policy
  namespace: kube-system
spec:
  ingress:
  - ports:
    - port: 80
      protocol: TCP
    - port: 443
      protocol: TCP
  podSelector:
    matchLabels:
      app.kubernetes.io/name: rke2-ingress-nginx
  policyTypes:
  - Ingress

I have tested this fix and it works as expected.