Make helm chart compliant with the Pod Security Standard "restricted"

[ecerulm]

Official Helm Chart version

1.7.0 (latest released)

Apache Airflow version

2.4.2

Kubernetes Version

1.23

Helm Chart configuration

securityContext:
  runAsNonRoot: true
  seccompProfile:
    type: RuntimeDefault

Docker Image customisations

No response

What happened

I have a namespace configured with the PSS restricted (pod security standard restricted) and airflow helm chart is not deployable in that namespace because the pods/containers are not compliant with the "restricted" policy (and I don't think is there any reason why they couldn't be made compliant)

Error creating: pods "airflow-run-airflow-migrations-2frnr" is forbidden: violates PodSecurity "restricted:v1.23": allowPrivilegeEscalation != false (container "run-airflow-migrations" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "run-airflow-migrations" must set securityContext.capabilities.drop=["ALL"])

What you think should happen instead

I think the helm chart should provide

• default container's securityContext SecurityContext in addition to the currently provided pod's securityContext PodSecurityContext. Things like the capabilities, and allowPrivilegeEscalation: false can only be set at the container's securityContext and not on the PodSecurityContext
  • the default container securityContext in the helm chart should be already compliant with the restricted PSS / PSA (pod security admission)
• the helm chart should allow to override the containers securityContext (currently only allows to override the pod's securityContext)

In other words, it should be deployable into a namespace with PSA (pod security admissiont) set to "restricted" without having to make any modification to the helm values (values.yaml)

How to reproduce

Create a kubenetes Namespace

kind: Namespace
apiVersion: v1
metadata:
  name: airflow
  creationTimestamp: '2022-10-27T11:26:18Z'
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.23

Deploy there,

You will see that the Job airflow-run-airflow-migrations would be stuck/ forbidden to run with

Error creating: pods "airflow-run-airflow-migrations-2frnr" is forbidden: violates PodSecurity "restricted:v1.23": allowPrivilegeEscalation != false (container "run-airflow-migrations" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "run-airflow-migrations" must set securityContext.capabilities.drop=["ALL"])

Anything else

I guess I should make a case why is important that airflow complies with the PSS restricted.

Running all the pods/container under the PSS restricted improves security in the case that somebody "hacks" that airflow. So even if they get a shell to the container, they can't become root in the container, and also they can't exploit any capabilities given to the container since the container must have drop all capabilities under the PSS restricted.

As a security-concerned system administrator I want to run all kubernetes workloads under the least privilege possible (hardening).

I do believe that there is no reason to for airflow to run with allowPrivilegeEscalation: true , or to run with linux any capabilities.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[ecerulm]

Related to #24588

That will allow to override both the PodSecurityContext (which is possible already today) and the container's SecurityContext (which is not possible today).

[ecerulm]

I try to get closer to the PSS restricted by specifying the following PodSecurityContext in the helm values

securityContext:
  runAsNonRoot: true
  seccompProfile:
    type: RuntimeDefault

and noticed that at least the StatefulSet airflow-worker does not work properly under that securityContext. The issue is that the airflow-worker-0 pod will mount the PVC on /opt/airflow/logs with ownership root:root and the running user airflow won't be able to write on that /opt/airflow/logs under that securityContext.

airflow@airflow-worker-0:/opt/airflow$ id
uid=50000(airflow) gid=0(root) groups=0(root)

I suspect the problem could be fixed by having the airflow user to belong to a group airflow and make the /opt/airflow/logs be owned by that group.

[potiuk]

Airlfow user in the reference docker image should belong to group root - which is actually to make it OpenShift compatible because it allows arbitrary user to be used in the container https://airflow.apache.org/docs/docker-stack/entrypoint.html#allowing-arbitrary-user-to-run-the-container

Actually airlfow shoudl be able to write to "root" group - and most of the writeable folders inside the image are "root" group owned.

[potiuk]

Airlfow user in the reference docker image should belong to group root - which is actually to make it OpenShift compatible because it allows arbitrary user to be used in the container https://airflow.apache.org/docs/docker-stack/entrypoint.html#allowing-arbitrary-user-to-run-the-container

Actually airlfow shoudl be able to write to "root" group - and most of the writeable folders inside the image are "root" group owned.

Just to add to it - using fsGroup: 0 in the security context should solve the problem.

[ctrought]

@potiuk If it is possible to support an arbitrary UID for airflow, then I guess it should be possible to run this under the restricted (<= OCP 4.11) restricted-v2 SCC in OpenShift (>= OCP 4.12) and the helm options for leveraging the anyuid would no longer be necessary which is generally frowned upon, is that right?

https://airflow.apache.org/docs/helm-chart/stable/production-guide.html#security-context-constraints

$ id
uid=1002920000 gid=0(root) groups=0(root),1002920000
$ ls -l /usr/local/airflow/entrypoint.sh
-rwxrwxr--. 1 airflow airflow 5026 Sep 17  2020 /usr/local/airflow/entrypoint.sh
$ id airflow
uid=1000(airflow) gid=1000(airflow) groups=1000(airflow)

$  ls -ld /usr/local/airflow/logs
drwxrwsrwx. 21 root 1002920000 19 Apr  7  2022 /usr/local/airflow/logs

Although there are airflow directories with the GID set to 0, the entrypoint is not. Our goal is to do the same as @ecerulm and satisfy the restricted PSA, but also satisfy the restricted-v2 SCC in OpenShift which requires arbitrary UIDs.

    - name: worker
      state:
        waiting:
          reason: CreateContainerError
          message: >
            container create failed: time="2023-08-11T14:56:29Z" level=error
            msg="runc create failed: unable to start container process: eaccess
            /usr/local/airflow/entrypoint.sh: permission denied"

I supposed we can try and customize the image and modify the ownership to support this but it would be nice if it were set with the permissions/ownership required for arbitrary UID support out of box :) 🤞
https://airflow.apache.org/docs/docker-stack/build.html

[potiuk]

@potiuk If it is possible to support an arbitrary UID for airflow, then I guess it should be possible to run this under the restricted (<= OCP 4.11) restricted-v2 SCC in OpenShift (>= OCP 4.12) and the helm options for leveraging the anyuid would no longer be necessary which is generally frowned upon, is that right?

Not sure if this is question or statement :D . What do you mean by entrypoint is not ? Entrypoint does not need to be executable or writeable by GID to be run by any user if that's what you mean (it is run via bash interpreter). And it happily works with arbitrary user, no problems with it:

[jarek:~/code/airflow] [airflow-3.9] update-breeze-documentation-about-updating-dependencies+* 36s ± docker run -it -u 234:0 apache/airflow bash

default@b4158874ff17:/opt/airflow$ id -u 
234
default@b4158874ff17:/opt/airflow$ id -g
0
default@b4158874ff17:/opt/airflow$ 

I am not sure If I know what do you propose? Can you elaborate and show some examples of what does not work and how you propose to fix it ?

I supposed we can try and customize the image and modify the ownership to support this but it would be nice if it were set with the permissions/ownership required for arbitrary UID support out of box :) crossed_fingers

I think it does, currently.

[potiuk]

Also see here: https://airflow.apache.org/docs/docker-stack/entrypoint.html#allowing-arbitrary-user-to-run-the-container where we explain how entrypoint works and what is done to support arbitrary user in the contaier to support OpenShift.

But maybe I am missing something @ctrought ?