securityContext of init containers contradicts new PodSecurity requirements

[RomanTheLegend]

Expected Behavior

Global problem: can't create new pods for tasks after update to 0.41 because of PodSecurity requirements. While there's a workaround to lower the requirements before each run ( kubectl label namespaces tekton-pipelines pod-security.kubernetes.io/enforce=baseline --overwrite ) we wanted to fix the root of the problem and harden the task pods by setting the correct securityContext to task pods/containers.
It worked for task container, but failed for Init containers as they do not inherit these settings.
After setting up stepTemplate in task definition, new pod should be created on task submission, with all init containers mirroring securityContext settings, just like task container does, but they don't.

Actual Behavior

Scheduled task doesn't get pod allocated because of namespace's securityPolicy restrictions:

failed to create task run pod "dummy-task-5p9bv": pods "dummy-task-5p9bv-pod" is forbidden: 
violates PodSecurity "restricted:latest": 
allowPrivilegeEscalation != false (containers "prepare", "place-scripts" must set securityContext.allowPrivilegeEscalation=false), 
unrestricted capabilities (containers "prepare", "place-scripts" must set securityContext.capabilities.drop=["ALL"]), 
runAsNonRoot != true (pod or containers "prepare", "place-scripts" must set securityContext.runAsNonRoot=true), 
seccompProfile (pod or containers "prepare", "place-scripts" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost").

Steps to Reproduce the Problem

1. Update to latest Tekton version (0.41.0) and GKE above 1.23 ( in this case 1.23.12-gke.1600)

2. Add stepTemplate to Task definition

spec:
  stepTemplate:
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      runAsNonRoot: true
      seccompProfile:
        type: RuntimeDefault

3. Run the task - the pod will not be created, in dashboard you'll find it was not scheduled because of pod-security.kubernetes.io/enforce=restricted policy applied to tekton-pipelines namespace

Additional Info

Related to #2933

• Kubernetes version: v1.23.12-gke.1600

• Tekton Pipeline version: v0.41.0

[dakky]

Seems to happen in 0.42.0 too 😢

[dibyom]

@RomanTheLegend Are you running your pipelineruns/taskruns in the tekton-pipelines namespace?

[RomanTheLegend]

@RomanTheLegend Are you running your pipelineruns/taskruns in the tekton-pipelines namespace?

We created a separate namespace for tekton tasks and that solved the problem with tasks not being executed (as it was suggested somewhere here, keep tekton-pipelines namespace only for controllers).
But since then I haven't checked if stepTemplate issue was resolved too because whole dev cluster was reinstalled as it usually happens and I need to set things up accordingly. Will check in the evening.

[NunoMaga]

+1
k8s v1.25.4
installed:

• https://storage.googleapis.com/tekton-releases/pipeline/previous/v0.43.1/release.yaml
• https://storage.googleapis.com/tekton-releases/triggers/previous/v0.22.0/release.yaml

│ Message: failed to create task run pod "install-foundry-cluster-services-k86d7-install-cluster-services": pods "install-foundry-cluster-ser0824dfbe6827b24cad162c0d0c801c7b-pod" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (containers "prepare", "place-scripts" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "prepare", "place-scripts" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "prepare", "place-scripts" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "prepare", "place-scripts" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost"). Maybe missing or invalid Task tekton-pipelines/install-cluster-services

[RomanTheLegend]

@NunoMaga These restrictions are per-namespace. The tekton-pipelines name is reserved for controllers, and restrictions are injected by-design.

Create new namespace for tasks i.e. tekton-tasks and ensure that all of your Pipelines, PipelineRuns and Tasks objects are using it in their declaration, for example:

---
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: workloads
  namespace: tekton-tasks
spec:
  pipelineRef:
    name: my-pipeline
  workspaces:
    - name: ws
      emptyDir: {}

This 'namespace' trick should be explicitly added to the README

[iainsproat]

I encountered this issue (but for the 'prepare' pod), despite having a separate namespace for the task.

As the original poster noted, this is due to 'pod-security.kubernetes.io/enforce': 'restricted' label on the namespace in which the tasks were being run.

Is there any reason the 'prepare' pod doesn't conform to a 'restricted' pod security?

[liuxuzxx]

@NunoMaga These restrictions are per-namespace. The tekton-pipelines name is reserved for controllers, and restrictions are injected by-design.

Create new namespace for tasks i.e. tekton-tasks and ensure that all of your Pipelines, PipelineRuns and Tasks objects are using it in their declaration, for example:

---
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: workloads
  namespace: tekton-tasks
spec:
  pipelineRef:
    name: my-pipeline
  workspaces:
    - name: ws
      emptyDir: {}

This 'namespace' trick should be explicitly added to the README

Yes It is right!