allow running buildah on newer nodes

[ibotty]

Changes

This will allow running buildah on nodes with kernel 5.12 or later.

Buildah requires the SETFCAP capability to work. This pull requests updates OpenShift SCC to allow adding this capability and the included buildah clustertask to request it.

I do not have access to clusters with older kernels so I don't know whether it will work on these nodes.

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

• Includes tests (if functionality changed/added)
• Includes docs (if user facing)
• Commit messages follow commit message best practices

See the contribution guide for more details.

Release Notes

buildah ClusterTask is now allowed SETFCAP capability

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: ibotty / name: Tobias Florek (025b581)

[tekton-robot]

Hi @ibotty. Thanks for your PR.

I'm waiting for a tektoncd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ibotty]

@ibotty: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

No release notes are necessary (afaict), so what should I do?

[nikhil-thomas]

/ok-to-test

[nikhil-thomas]

@gabemontero what are your thoughts 🧑‍💻

[nikhil-thomas]

@ibotty: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

No release notes are necessary (afaict), so what should I do?

i think a 1 line release will help our users to be aware of this change.

but to get rid of the block without adding a release-note add

```release-note
NONE
\```

at the end of our pr description.

[concaf]

a release note like this will help our users:

buildah ClusterTask is now allowed SETFCAP capability

[ibotty]

I added the release notes as suggested. You are certainly right.

BTW: I plan to do a pull request in tektoncd's catalog when the review here is done.

[concaf]

/assign @piyush-garg

[nikhil-thomas]

@piyush-garg do we need to add similar securityContext blocks to all the s2i tasks in our addons here

[piyush-garg]

@ibotty is this PR tested on any OpenShift cluster? I am not able to run the tasks as a normal user

[ibotty]

Yes, after manually changing the scc and copying the buildah-task to add the changes I can run it on OKD 4.8. I did not built a different operator and let it change the crds though.

[gabemontero]

ok @ibotty @nikhil-thomas @piyush-garg

• first, yes cap_setfcap was the additional permission that @nalind said was needed with the latest coreos changes on 4.11 and so on the surface this looks like the approach, though yeah, as was mentioned in slack, any of the s2i items in the catalog that are using buildah would need this too
• for background, here are what the man pages at https://man7.org/linux/man-pages/man7/capabilities.7.html say about it:

CAP_SETFCAP (since Linux 2.6.24)
              Set arbitrary capabilities on a file.

              Since Linux 5.12, this capability is also needed to map
              user ID 0 in a new user namespace; see [user_namespaces(7)](https://man7.org/linux/man-pages/man7/user_namespaces.7.html)
              for details.

for @nikhil-thomas 's question to myself, @adambkaplan, and @sbose78 about the security implications, I'm not sure I feel qualified to respond with confidence on how good or bad that additional change is, but I believe @nalind did tell me this was not a temporary state of affairs. That it was what would be needed moving forward.

@nalind - do you have any input on the relative safety or lack there of around CAP_SETFCAP

• lastly, when @nalind and I last talked, he mentioned he had buildah changes he needed to make to get things to work even with CAP_SETFCAP. @nalind - were you in fact able to make those changes yet? @nikhil-thomas said they are now using registry.redhat.io/rhel8/buildah@sha256:23fb7971ea6ac4aaaaa1139473a602df0df19222a3b5a76b551b2b9ddd92e927 and that it is working. Just wanted to make sure that lines up with your expectations.

that's all I got - thanks

[gabemontero]

note: as I understand it, "the problem" and need for this new permission only applies to running on OCP 4.11

[nikhil-thomas]

/hold

let us get some more 👀 on this.
🧑‍💻

[piyush-garg]

@ibotty is this PR tested on any OpenShift cluster? I am not able to run the tasks as a normal user

It did not work for me as a normal user, but worked as a kubeadmin

[ibotty]

@piyush-garg: Are you sure you the pipelines-scc got updated? Mine looks like

oc get scc pipelines-scc
NAME            PRIV    CAPS          SELINUX     RUNASUSER   FSGROUP     SUPGROUP   PRIORITY   READONLYROOTFS   VOLUMES
pipelines-scc   false   ["SETFCAP"]   MustRunAs   RunAsAny    MustRunAs   RunAsAny   10         false            ["configMap","downwardAPI","emptyDir","persistentVolumeClaim","projected","secret"]

[ibotty]

note: as I understand it, "the problem" and need for this new permission only applies to running on OCP 4.11

And on OKD since, I guess, forever.

[nalind]

for @nikhil-thomas 's question to myself, @adambkaplan, and @sbose78 about the security implications, I'm not sure I feel qualified to respond with confidence on how good or bad that additional change is, but I believe @nalind did tell me this was not a temporary state of affairs. That it was what would be needed moving forward.

That's right, I have no reason to expect the kernel to revert this.

@nalind - do you have any input on the relative safety or lack there of around CAP_SETFCAP

I'm not an expert on that.

* lastly, when @nalind and I last talked, he mentioned he had buildah changes he needed to make to get things to work even with CAP_SETFCAP.  @nalind - were you in fact able to make those changes yet?  @nikhil-thomas said they are now using `registry.redhat.io/rhel8/buildah@sha256:23fb7971ea6ac4aaaaa1139473a602df0df19222a3b5a76b551b2b9ddd92e927` and that it is working.  Just wanted to make sure that lines up with your expectations.

I must have misspoken. The most we can do there is check for that capability (near the existing check for CAP_SYS_ADMIN) and suggest it as a possible cause when we fail to set up the ID mappings.

[ibotty]

Argh! I forgot to mention, I am using the task with BUILDER_IMAGE=quay.io/buildah/stable:latest, because of missing redhat registry credentials.

[adambkaplan]

It should be noted that CAP_SETFCAP is permitted under the Baseline pod security standard:

https://kubernetes.io/docs/concepts/security/pod-security-standards/#baseline

[gabemontero]

It should be noted that CAP_SETFCAP is permitted under the Baseline pod security standard:

https://kubernetes.io/docs/concepts/security/pod-security-standards/#baseline

good catch @adambkaplan forgot to look there

yes that is probably the best answer to say it is "OK" for pipeline SA to pick this up

[piyush-garg]

@piyush-garg: Are you sure you the pipelines-scc got updated? Mine looks like

oc get scc pipelines-scc
NAME            PRIV    CAPS          SELINUX     RUNASUSER   FSGROUP     SUPGROUP   PRIORITY   READONLYROOTFS   VOLUMES
pipelines-scc   false   ["SETFCAP"]   MustRunAs   RunAsAny    MustRunAs   RunAsAny   10         false            ["configMap","downwardAPI","emptyDir","persistentVolumeClaim","projected","secret"]

Yes, i see it like this and not able to create pod. taskrun not able to convert to pod

[piyush-garg]

@ibotty This PR #813 worked for me

[sm43]

I tried this pr on OCP 4.11
by creating a non admin user too
got error as below for both users (admin and nonadmin users)

failed to create task run pod "buildah-run-kqc8j-run-buildah":
              pods "buildah-run-kqc8j-run-buildah-pod" is forbidden: unable to validate
              against any security context constraint: [provider "anyuid": Forbidden:
              not usable by user or serviceaccount, spec.containers[0].securityContext.capabilities.add:
              Invalid value: "CAP_SETFCAP": capability may not be added, spec.containers[1].securityContext.capabilities.add:
              Invalid value: "CAP_SETFCAP": capability may not be added, spec.containers[2].securityContext.capabilities.add:
              Invalid value: "CAP_SETFCAP": capability may not be added, provider
              "restricted": Forbidden: not usable by user or serviceaccount, provider
              "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider
              "hostmount-anyuid": Forbidden: not usable by user or serviceaccount,
              provider "machine-api-termination-handler": Forbidden: not usable by
              user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable
              by user or serviceaccount, provider "hostnetwork": Forbidden: not usable
              by user or serviceaccount, provider "hostaccess": Forbidden: not usable
              by user or serviceaccount, provider "node-exporter": Forbidden: not
              usable by user or serviceaccount, provider "privileged": Forbidden:
              not usable by user or serviceaccount]. Maybe missing or invalid Task
              test/buildah

steps performed

• OCP 4.11
• Created a user following https://docs.openshift.com/container-platform/4.10/authentication/identity_providers/configuring-htpasswd-identity-provider.html#identity-provider-overview_configuring-htpasswd-identity-provider
• Changed user to non admin one
• Created a Pipelinerun using buildah

---

pods created are using pipeline-scc

[ibotty]

Argh. While trying to debug the problem I noticed, that I did make a copy-paste error: the capabilities are not prepended with CAP_ in kubernetes. My "fixed" task had addCapabilities: [SETFCAP] set.
🤦

Should I update the PR to add (the right) capability to all other s2i tasks as well?

[piyush-garg]

We dont need to add the capability for this step

[piyush-garg]

Argh. While trying to debug the problem I noticed, that I did make a copy-paste error: the capabilities are not prepended with CAP_ in kubernetes. My "fixed" task had addCapabilities: [SETFCAP] set. facepalm

Should I update the PR to add (the right) capability to all other s2i tasks as well?

Aah thanks, i also misunderstood that we need to add the capabilities as like in linux but it is different for containers.

[tekton-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vdemeester

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [vdemeester]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[piyush-garg]

Should I update the PR to add (the right) capability to all other s2i tasks as well?

Yes we need to do the change for all s2i tasks too, but only for the build and push steps, not all

[sbose78]

Followed up, looks good 👍

[vdemeester]

/lgtm
I think we can lift the hold

[sm43]

/lgtm
/hold cancel