Add ability to run Mutating Webhook BEFORE OpenShift updates pod defenition

[mlbiam]

When I configure a mutating webhook the pod definition that comes into the webhook has already had its security information added. It makes transforming the pod to change the service account, sec, etc difficult and is leading to odd results. For instance once the runAsUser and serviceAccountName and openshift.io/scc I can't rsh into the container anymore.

Version

oc v3.9.0
kubernetes v1.9.1+cbc5b49
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://os.server.lan:443
openshift v3.9.0+ba7faec-1
kubernetes v1.9.1+a0ce1bc657

Steps To Reproduce

1. Enable Mutating webhooks
2. deploy mutating webhook for pods CREATE and UPDATE
3. Deploy the below pod definition:

apiVersion: v1
kind: Pod
metadata:
  name: krbtest
  labels:
    name: krbtest
  annotations:
    com.tremolosecurity.openshift: krb5_sidecar
spec:
  containers:
    - resources:
      image: mlbiam/krbdonothing
      name: motorcycle

Current Result

this is what is being submitted to my webhook (yaml version)

---
request:
  uid: 62d87962-9819-11e8-95dd-525400887c40
  userInfo:
    injectedIdentity: eyJraWQiOiJDTj1qd3Qtc2lnLCBPVT1kZXYsIE89ZGV2LCBMPWRldiwgU1Q9ZGV2LCBDPWRldi1DTj1qd3Qtc2lnLCBPVT1kZXYsIE89ZGV2LCBMPWRldiwgU1Q9ZGV2LCBDPWRldi0xNTI4Njg0NDk0MTI5IiwiYWxnIjoiUlMyNTYifQ.eyJpc3MiOiJodHRwczovL3VuaXNvbi1vcGEudW5pc29uLnN2Yy5jbHVzdGVyLmxvY2FsIiwiYXVkIjoiaHR0cHM6Ly9vcGEudW5pc29uLnN2YyIsImV4cCI6MTUzMzQwOTYwNywianRpIjoiYmdyUExnbmxXNE1rajhOR2VYVnVRZyIsImlhdCI6MTUzMzQwOTU0NywibmJmIjoxNTMzNDA5NDg3LCJub25jZSI6ImYwNTI1MTk5LTE3YTgtNGFmYi1iN2RkLTZkNjg0M2U0ZDUzZSIsInN1YiI6ImZyZWVpcGEzQGVudDJrMTIuZG9tYWluLmNvbSIsInJvbGVzIjpbIk9wZW5TaGlmdCAtIGFkbWluaXN0cmF0b3JzLWZyZWVpcGEzLXByb2plY3QiLCJPcGVuU2hpZnQgLSBlZGl0b3ItZnJlZWlwYTMtcHJvamVjdCIsIkZyZWVJUEEgLSBhcHBsaWNhdGlvbnMtb3BlbnNoaWZ0IiwiRnJlZUlQQSAtIGFwcGxpY2F0aW9ucy1vcGVuc2hpZnQta2V5dGFicyJdLCJ1aWRudW1iZXIiOiIxNjA4MTI0NTAifQ.kwaCS65MKcJ07M_cx4nixIALeDz9xtY52xoSEb1j7ojDa_ZydyQlgCBOqhHPedttvT6d-9plNd-vhXHMs_NOxhHV8NY6cZQ3FRdhYJyitedSWZhfDSSTPZ0KlazxRWBhgpNZ5mrdbWurxC6vm6X2uWTk61RWegBsWHG9NBxR7fg_RwNcWKMQaOvW0QUwNXexcN96jBPZ5vF6pnvH8MLwu5AQDaR6jISykuCnhJ2tkdk-W3jfVE8Q-qpEzaZ64gKwbbY1Vw_lbsv3NTxyztUhqJZD72P11UUFmFgyP5KdFGnZLaqmzhivIVDpyeLLgvAv0RKi_CIDbnxFMC4wEkcLRg
    uid: 8e804b18-7e5f-11e8-acd7-525400887c40
    extra:
      scopes.authorization.openshift.io:
      - user:full
    groups:
    - administrators-freeipa3-project
    - editor-freeipa3-project
    - system:authenticated:oauth
    - system:authenticated
    username: freeipa3@ent2k12.domain.com
  resource:
    resource: pods
    version: v1
    group: ''
  kind:
    kind: Pod
    version: v1
    group: ''
  namespace: freeipa3-project
  oldObject: 
  operation: CREATE
  object:
    metadata:
      name: krbtest
      namespace: freeipa3-project
      creationTimestamp: 
      annotations:
        com.tremolosecurity.openshift: krb5_sidecar
        openshift.io/scc: restricted
      labels:
        name: krbtest
    spec:
      dnsPolicy: ClusterFirst
      terminationGracePeriodSeconds: 30
      serviceAccountName: default
      imagePullSecrets:
      - name: default-dockercfg-6zz7j
      volumes:
      - name: default-token-s6d2l
        secret:
          secretName: default-token-s6d2l
      containers:
      - image: mlbiam/krbdonothing
        imagePullPolicy: Always
        terminationMessagePolicy: File
        terminationMessagePath: "/dev/termination-log"
        name: motorcycle
        resources: {}
        securityContext:
          runAsUser: 1000300000
          capabilities:
            drop:
            - KILL
            - MKNOD
            - SETGID
            - SETUID
        volumeMounts:
        - mountPath: "/var/run/secrets/kubernetes.io/serviceaccount"
          name: default-token-s6d2l
          readOnly: true
      serviceAccount: default
      securityContext:
        seLinuxOptions:
          level: s0:c17,c14
        fsGroup: 1000300000
      restartPolicy: Always
      schedulerName: default-scheduler
      nodeSelector:
        node-role.kubernetes.io/compute: 'true'
    status: {}
apiVersion: admission.k8s.io/v1beta1
kind: AdmissionReview

Expected Result

apiVersion: v1
kind: Pod
metadata:
  name: krbtest
  labels:
    name: krbtest
  annotations:
    com.tremolosecurity.openshift: krb5_sidecar
spec:
  containers:
    - resources:
      image: mlbiam/krbdonothing
      name: motorcycle

Additional Information

[try to run $ oc adm diagnostics (or oadm diagnostics) command if possible]
[if you are reporting issue related to builds, provide build logs with BUILD_LOGLEVEL=5]
[consider attaching output of the $ oc get all -o json -n <namespace> command to the issue]
[visit https://docs.openshift.org/latest/welcome/index.html]

[mfojtik]

@deads2k @sttts sounds related to our past discussion of mutating hook chains :)

[sttts]

This works as intended: webhooks come last in the chain (with very few exceptions, compare https://github.com/kubernetes/kubernetes/blob/f1c202d392332af53541f59651702ad717f3cbf0/pkg/kubeapiserver/options/plugins.go#L65). We don't have the concept of desired order for webhooks today.

[mlbiam]

I haven't tried this on my upstream k8s clusters to see the difference, i can do that. I would think that if openshift is going to drive additional security contexts based on the pod submission wouldn't you want that step to be last?

[sttts]

In upstream it will be the same. Basically only quota and initializers are done after the webhook.

[deads2k]

Webhooks come last in the chain because a webhook could know about the compiled chain, but the compiled chain cannot know about the webhooks.

SCC and PSP (the upstream equivalent) have two stages: mutation and validation. The mutation step is done early, but the validation step is done after all mutators. This ensures that webhooks cannot violate the security constraints imposed.

[skonto]

@sttts hi, I am facing the same issue where I am trying to patch a pod at the webhook side linked to a specific scc.
So here is my printed pod before I patch at the webhook side:

 PRE-PATCH pod spec:&PodSpec{Volumes:[{spark-local-dir-1 {nil EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {spark-conf-volume {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:spark-pi-1562246028552-driver-conf-map,},Items:[],DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil}} {spark-operator-ns-fdp-sparkoperator-token-2b7c8 {nil nil nil nil nil &SecretVolumeSource{SecretName:spark-operator-ns-fdp-sparkoperator-token-2b7c8,Items:[],DefaultMode:nil,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}],Containers:[{spark-kubernetes-driver lightbend/spark:2.1.1-OpenShift-2.4.3-rh-2.12 [] [driver --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.examples.SparkPi spark-internal 100000000]  [{driver-rpc-port 0 7078 TCP } {blockmanager 0 7079 TCP } {spark-ui 0 4040 TCP }] [] [{SPARK_DRIVER_BIND_ADDRESS  EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {SPARK_LOCAL_DIRS /var/data/spark-daf902f0-e51c-4277-ad5e-5535104884bc nil} {SPARK_CONF_DIR /opt/spark/conf nil}] {map[memory:{{926941184 0} {<nil>} 884Mi BinarySI}] map[cpu:{{1 0} {<nil>} 1 DecimalSI} memory:{{926941184 0} {<nil>} 884Mi BinarySI}]} [{spark-local-dir-1 false /var/data/spark-daf902f0-e51c-4277-ad5e-5535104884bc  <nil>} {spark-conf-volume false /opt/spark/conf  <nil>} {spark-operator-ns-fdp-sparkoperator-token-2b7c8 true /var/run/secrets/kubernetes.io/serviceaccount  <nil>}] [] nil nil nil /dev/termination-log File Always &SecurityContext{Capabilities:&Capabilities{Add:[],Drop:[KILL],},Privileged:nil,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,} false false false}],RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{node-role.kubernetes.io/compute: true,},ServiceAccountName:spark-operator-ns-fdp-sparkoperator,DeprecatedServiceAccount:spark-operator-ns-fdp-sparkoperator,NodeName:,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:&SELinuxOptions{User:,Role:,Type:,Level:s0:c20,c5,},RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:*1000390000,RunAsGroup:nil,Sysctls:[],},ImagePullSecrets:[{spark-operator-ns-fdp-sparkoperator-dockercfg-m5gph}],Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[],AutomountServiceAccountToken:nil,Tolerations:[],HostAliases:[],PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[],RuntimeClassName:nil,EnableServiceLinks:nil,}

FSGroup:*1000390000 is correct as it will get the default one, initially the pod is submitted without a security context.

my scc is applied correctly:

ObjectMeta:&ObjectMeta{Name:spark-pi-driver,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{spark-app-selector: spark-b55ab4ceac894df1b8c9fcf8b42ddbc6,spark-role: driver,sparkoperator.k8s.io/app-name: spark-pi,sparkoperator.k8s.io/launched-by-spark-operator: true,sparkoperator.k8s.io/submission-id: 25e394db-80e5-45c7-bc17-6b173004c6dc,version: 2.4.3,},Annotations:map[string]string{openshift.io/scc: scc-test,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,}

my patch ops submitted as response to k8s is:
I0704 13:13:49.882479      12 webhook.go:307] PATCH OPS Object:[{"op":"add","path":"/metadata/ownerReferences","value":[{"apiVersion":"sparkoperator.k8s.io/v1beta1","kind":"SparkApplication","name":"spark-pi","uid":"8e068eba-9e5d-11e9-9719-065625d6fbaa","controller":true}]},{"op":"replace","path":"/spec/securityContext","value":{"fsGroup":1234}},{"op":"replace","path":"/spec/containers","value":[{"name":"spark-kubernetes-driver","image":"lightbend/spark:2.1.1-OpenShift-2.4.3-rh-2.12","args":["driver","--properties-file","/opt/spark/conf/spark.properties","--class","org.apache.spark.examples.SparkPi","spark-internal","100000000"],"ports":[{"name":"driver-rpc-port","containerPort":7078,"protocol":"TCP"},{"name":"blockmanager","containerPort":7079,"protocol":"TCP"},{"name":"spark-ui","containerPort":4040,"protocol":"TCP"}],"env":[{"name":"SPARK_DRIVER_BIND_ADDRESS","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}},{"name":"SPARK_LOCAL_DIRS","value":"/var/data/spark-daf902f0-e51c-4277-ad5e-5535104884bc"},{"name":"SPARK_CONF_DIR","value":"/opt/spark/conf"}],"resources":{"limits":{"memory":"884Mi"},"requests":{"cpu":"1","memory":"884Mi"}},"volumeMounts":[{"name":"spark-local-dir-1","mountPath":"/var/data/spark-daf902f0-e51c-4277-ad5e-5535104884bc"},{"name":"spark-conf-volume","mountPath":"/opt/spark/conf"},{"name":"spark-operator-ns-fdp-sparkoperator-token-2b7c8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always","securityContext":{"capabilities":{"drop":["KILL"]}}}]},{"op":"replace","path":"/metadata/annotations","value":{"openshift.io/scc":"scc-test"}}]

My custom scc only sets:
  FSGroup Strategy: MustRunAs			
    Ranges:					1000-6000

I got this error:

Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://172.30.0.1/api/v1/namespaces/spark/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "spark-pi-driver" is forbidden: unable to validate against any pod security policy: [fsGroup: Invalid value: []int64{1234}: 1234 is not an allowed group].
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(Operatio

My question is, is a webhook allowed to do any patch that complies with a specific scc? What is the role of RunAsUser when scc is validated?

[sttts]

@skonto the validating SCC admission run after your webhook will try to find a matching SCC, but doesn't find it. Can that be?

About the topic of this issue: 1.15 will bring reinvocation of webhooks. So in OpenShift 4.3 native admission steps will be run after your webhook has be mutated an object.

[skonto]

Thanks @sttts, it does not make sense so far, because I can match my scc if I create the pod within a running pod via kubectl (spark-operator-ns-fdp-sparkoperator service account above has been added to the scc). I am running within a pod because this is how the spark operator creates the Spark pods by using its pod service account namely spark-operator-ns-fdp-sparkoperator and the token under /var/run/.... So I mimicked that with kubectl in order to make a comparison and it worked (no mutation).

Now the problem with the Spark operator (which does mutation) is for some reason the webhook cannot apply the patch . When the response is returned to the K8s api server what does the validation do? How it picks up the appropriate scc? Would the validation accept an arbitrary patched pod spec? If you check the pod submitted it uses as sa the spark-operator-ns-fdp-sparkoperator one which has been added to my custom scc, why this is not enough?

[sttts]

Here is the validation logic for SCCs: https://github.com/openshift/origin/blob/master/vendor/k8s.io/kubernetes/openshift-kube-apiserver/admission/security/sccadmission/admission.go#L105