Deviation from upstream restricted PSP

[SheilaghM]

SURE-3199

Rancher Server Setup

• Rancher version: Rancher v2.5.8

Describe the bug
runAsUser set to RunAsAny in the restricted PSP which is a deviation from upstream example PSP

To Reproduce
Checked the documentation and found below

This policy is based on the Kubernetes example restricted policy.
The upstream example restricted PSP shows below.

runAsUser:
# Require the container to run without root privileges.
rule: 'MustRunAsNonRoot'
But the Rancher restricted PSP is not the same.

Result
Default restricted policy allows the container to run processes with root user.

Expected Result
Default restricted policy should not allow the container to run processes with root user.
Or there should be a note in documentation if this PSP deviation from the upstream example is intentional.

[stale]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

[nobbs]

Still interested in a solution to this issue, as this behavior also occurs in versions 2.6.x, as far as I can tell.

[macedogm]

PR #36367

[macedogm]

Docs PR rancher/docs#3849

[macedogm]

Apologies, the issue was closed when the PR got merged. I'm reopening, so QA can properly validate it.

[macedogm]

Root Cause

The restricted PSP policy provided by Rancher has a deviation from the upstream restricted PSP provided by Kubernetes. The major deviation is that the Rancher provided PSP has:

  runAsUser:
    rule: 'runAsAny'

While the upstream has:

  runAsUser:
    rule: 'MustRunAsNonRoot'

In our docs we also wrongly state that our restricted PSP:

Prevents pods from running as a privileged user...

See this comment for more information.

What was fixed?

A new restricted-noroot PSP was created to more closely match k8s upstream restricted PSP, which now does prevents pods from running as a privileged user.

This new policy was created, instead of fixing the current provided policy, in order to avoid breaking customers' deployments that are using the restricted PSP and that might be running as a privileged user.

What should be tested?

1. A downstream cluster must be provisioned and configured to use restricted-noroot as its default PSP.
2. Rancher pods will use the default-psp PSP, that it's equivalent to unrestricted PSP provided by k8s upstream.
3. Rancher pods will work as expected. I.e.: must not be blocked by the new restricted-noroot PSP.
4. A new project and a new namespace must be created in the downstream cluster and configured to use the cluster default PSP, from step 1. Which in this case will be restricted-noroot.
5. A pod running as root must be created in the namespace from step 4.
6. The pod, from step 5, must fail to be created with an error message like Error: container has runAsNonRoot and image will run as root...

What areas could experience regressions?

Rancher pods might fail to run in case they bind to the restricted-noroot PSP instead of default-psp.

[anupama2501]

Reproduced on v2.6.3 Rancher helm installation

• Installed rancher HA on an RKE1 cluster with PSP set to true and Restricted
• Verified from the local cluster the rule in the default-psp is RunAsAny

• Created a downstream cluster with the psp set to restricted and RunasAny
• Create a workload deployment and verified that the pods came up active
• From cluster management > pod security policies > restricted > changed the runAsUser rule to MustRunAsNonRoot
• Created a downstream cluster with psp set to restricted
• Created a workload deployment and the pods fail to come up active with the error

Error: container has runAsNonRoot and image will run as root (pod: "test1-65cb547bd5-h7mbc_default(d7f44bee-771d-4564-94fe-bdf7966e681b)", container: container-0)

[anupama2501]

Validations on 2.6-head 9ae9

local k8s 1.21 and PSP set to restricted

1. Rancher helm installation with local on k8s 1.21 and psp set to restricted
2. Verified kubectl get psp has default-psp and restricted-psp listed
3. Create a downstream cluster with PSP restricted-noroot
4. Verified the PSPs created from kubectl get psp
5. Created a Project with default PSP - P1
6. Created a deployment d1 in project P1
7. Verified deployment doesn't come active
8. Created a project with unrestricted PSP - P2
9. Verified the psp created - kubectl get psp
10. Created a deployment in this project p2
11. Deployment came up active

[anupama2501]

Verified test cases on 2.6-head 978e165

1. local k8s v1.22.6-rancher1-2 and PSP set to restricted [local picks default-psp] - created 1 custom rke1 cluster with restricted-no-root, 1 rke1 node driver and and rke2 node driver cluster. Repeated the steps from the above comment - PASS
2. local psp with no restricted and a downstream restricted-no-root psp.Repeated the steps from the above comment - PASS
3. Local cluster rke1 with restricted PSP - default psp is picked, rancher pods deployed. Updated the default-psp runAsUSer rule to MustRunAsNoRoot. Deleted a rancher pod and verified the pods couldn't reconcile

Failed (12) | 52 mins ago | Error: container has runAsNonRoot and image will run as root (pod: "rancher-6988c76755-fq57r_cattle-system(2c02d913-d45c-498c-b492-56364f8aa5c5)", container: rancher)

4. local cluster set to restricted psp, and created a downstream cluster on restricted-psp. Changed the restricted psp to restricted-no-root. Deployed some workload pods with no security context added. Verified the pods errored out.
5. Enable downstream restricted psp and deleted the cluster - verified the clusters are deleted.

Upgrade test case:

• Created a local cluster on 2.6.3 with PSP set to downstream on psp restricted upgrade rancher server, updated the psp on the downstream cluster from restricted to restricted-no-root
• Ran the same validations from the above step - PASS