[Feature] Allow specifying a container within a Pod in Policy Exceptions

[MariamFahmy98]

Problem Statement

If we have a policy that disallows certain capabilities, and we have a deployment with two containers:

1. A sidecar container (e.g., Istio) with capabilities NET_ADMIN and NET_BIND_SERVICE.
2. An application container with no capabilities.

We need the policy exception to be applied only on the sidecar container.

Solution Description

We need to find a way to let the user specify a container within a pod in policy exceptions.

Alternatives

No response

Additional Context

No response

Slack discussion

No response

Research

• I have read and followed the documentation AND the troubleshooting guide.
• I have searched other issues in this repository and mine is not recorded.

[mark-winter-cko]

I've run in to this same issue with the isito-init container. For istio at least, one way around it is to use the Istio CNI plugin so that the istio-init container is not needed. However it would still be nice to be able to exclude particular containers in a kyverno policy.

[epacke]

Leaving this here in case it helps someone else. This policy adds an exception for kube-system for PSA and also for Istio sidecars Seccomp configs.

piVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: psa
spec:
  background: true
  validationFailureAction: Audit
  rules:
  - name: restricted
    match:
      any:
      - resources:
          kinds:
          - Pod
    exclude:
      any:
      - resources:
          namespaces:
          - kube-system
          - filebeat
    validate:
      podSecurity:
        level: restricted
        version: latest
        exclude:
        - controlName: Seccomp
          images:
          - repo.org.com/docker-hub/istio/proxyv2:*

Kudos to the wonderful people at Kyvernos Slack Channel for prodding me in the right direction. ❤️

[ron1]

@MariamFahmy98 Am I correct that the goal here is to allow Policy Exceptions for psa validation subrules in general?

[MariamFahmy98]

There is another issue for allowing pod security exemption in exceptions. Here's the PR: #8580

This issue is for allowing the selection of a specific container name within the deployment to be excluded.

[ishworgurung]

We raise various sysctl knobs in our initContainers and would benefit from this feature: the ability to target specific container.

thanks

[realshuting]

Part of #8663?

[chipzoller]

Seems related but not the same.

[jseiser]

I am very new to all of this, but I think its essentially the issue we are having

#9358

We use linkerd, and the baseline policy shows every one of our pods failing the disallow capabilites rule because of the linkerd-init container

        add:
        - NET_ADMIN
        - NET_RAW

[ranjit-se7en]

I am very new to all of this, but I think its essentially the issue we are having

#9358

We use linkerd, and the baseline policy shows every one of our pods failing the disallow capabilites rule because of the linkerd-init container

        add:
        - NET_ADMIN
        - NET_RAW

We have the same issue in our clusters with linkerd enabled and the baseline policy disallow capabilities shows violations for all pods with linkerd enabled.

[chipzoller]

https://kyverno.io/blog/2024/02/04/securing-services-meshes-easier-with-kyverno/

[rucciva]

hi @chipzoller , thanks for sharing the blog, i've tried creating similar policy

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: podsecurity-subrule-baseline
  annotations:
    policies.kyverno.io/title: Baseline Pod Security Standards
    policies.kyverno.io/category: Pod Security, EKS Best Practices
    policies.kyverno.io/severity: high
    kyverno.io/kyverno-version: 1.8.0
    policies.kyverno.io/minversion: 1.8.0
    kyverno.io/kubernetes-version: "1.24"
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/description: >-
      The baseline profile of the Pod Security Standards is a collection of the
      most basic and important steps that can be taken to secure Pods. Beginning
      with Kyverno 1.8, an entire profile may be assigned to the cluster through a
      single rule. This policy configures the baseline profile through the latest
      version of the Pod Security Standards cluster wide.
spec:
  background: true
  validationFailureAction: enforce
  rules:
    - name: baseline
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        podSecurity:
          level: baseline
          version: latest
          exclude:
            - controlName: Capabilities
              images:
                  - "*/linkerd/proxy-init*"

but the creation of this following pod was not denied. What could be the reason?

apiVersion: v1
kind: Pod
metadata:
  name: pod-with-net-admin
spec:
  containers:
    - name: your-container-name
      image: your-image
      securityContext:
        capabilities:
          add:
            - NET_ADMIN

my kyverno version is 1.11.4

[chipzoller]

Works for me:

k create -f temp1.yaml 
Error from server: error when creating "temp1.yaml": admission webhook "validate.kyverno.svc-fail" denied the request: 

resource Pod/default/pod-with-net-admin was blocked due to the following policies 

podsecurity-subrule-baseline:
  baseline: |
    Validation rule 'baseline' failed. It violates PodSecurity "baseline:latest": ({Allowed:false ForbiddenReason:non-default capabilities ForbiddenDetail:container "your-container-name" must not include "NET_ADMIN" in securityContext.capabilities.add})

[rucciva]

thanks for validating @chipzoller .
just want to confirm, do you have linkerd injection on your pod?

me and my team guest that the policy will exempts the pod when exclusion are found, which is weird and does not align with the description here

For example, the below policy enforces the restricted profile but exempts containers running either the nginx or redis image from following the Capabilities control.

[chipzoller]

No service mesh when I tested.

[rucciva]

No service mesh when I tested.

i see, can i ask you if it's not too much to retest with the following linkerd-injected definition of the same pod

apiVersion: v1
kind: Pod
metadata:
  name: pod-with-net-admin
spec:
  containers:
    - name: your-container-name
      image: your-image
      securityContext:
        capabilities:
          add:
            - NET_ADMIN
  initContainers:
  - args:
    - --incoming-proxy-port
    - "4143"
    - --outgoing-proxy-port
    - "4140"
    - --proxy-uid
    - "2102"
    - --inbound-ports-to-ignore
    - 4190,4191,4567,4568
    - --outbound-ports-to-ignore
    - 4567,4568
    image: cr.l5d.io/linkerd/proxy-init:v2.2.3
    imagePullPolicy: IfNotPresent
    name: linkerd-init
    resources:
      limits:
        cpu: 100m
        memory: 20Mi
      requests:
        cpu: 100m
        memory: 20Mi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        add:
        - NET_ADMIN
        - NET_RAW
        - NET_ADMIN
      privileged: false
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 65534
      seccompProfile:
        type: RuntimeDefault
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: FallbackToLogsOnError
    volumeMounts:
    - mountPath: /run
      name: linkerd-proxy-init-xtables-lock
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-lmn62
      readOnly: true

[chipzoller]

I see, yes in this case it was allowed in Kyverno 1.11.4 but blocked (correct) in 1.12.0-rc.5 when I tested. I was looking for where this was addressed but maybe it was part of some other change. In any case, the policy with your Pod works as intended on 1.12.

[rucciva]

thanks a lot @chipzoller