Kubernetes Mutating Admission Webhook for environment injection

This repo hosts a MutatingAdmissionWebhook that injects environment variables, dns options and node affinity into pod containers prior to persistence of the object. Node affinity is currently limited to RequiredDuringSchedulingIgnoredDuringExecution selector terms.

The image can be used along with several Kubernetes resources to update specific settings on your pods based on a label applied to the namespace of the pod i.e.

• Apply the label to the namespace and all pods within will have the same additional configuration applied at scheduling time.

The following options are available for configuration (if existing configuration exists then the new configuration you supply will be appended, it does not replace existing configuration).

• Environment Variables
• DNS Options
• Required Node Affinity terms
• Preferred Node Affinity terms
• Tolerations
• Topology Spread Constraints

Each configuration type is optional so your configmap or values file will only include those that you want to change.

Example config map:

apiVersion: v1
kind: ConfigMap
metadata:
  name: env-injector-webhook-configmap
data:
  envconfig.yaml: |
    tolerations:
      - key: kubernetes.azure.com/scalesetpriority
        effect: NoSchedule
        operator: Equal
        value: spot
    topologyConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
        nodeAffinityPolicy: Honor
        nodeTaintsPolicy: Honor
        labelSelector:
          matchLabels:
            app.kubernetes.io/name: test-app
        matchLabelKeys:
          - pod-template-hash

example values.yaml file (Helm):

environment: {}
dnsOptions: {}
requiredNodeAffinityTerms: {}
preferredNodeAffinityTerms: {}
tolerations:
  - key: kubernetes.azure.com/scalesetpriority
    effect: NoSchedule
    operator: Equal
    value: spot
topologyConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    nodeAffinityPolicy: Honor
    nodeTaintsPolicy: Honor
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: test-app
    matchLabelKeys:
      - pod-template-hash

Prerequisites

Kubernetes 1.22.0 or above with the admissionregistration.k8s.io/v1 API enabled. Verify that by the following command:

kubectl api-versions | grep admissionregistration.k8s.io/v1

The result should be:

admissionregistration.k8s.io/v1

In addition, the MutatingAdmissionWebhook and ValidatingAdmissionWebhook admission controllers should be added and listed in the correct order in the admission-control flag of kube-apiserver.

Build

Within this repository is a Dockerfile, this should be used when there are changes made to the Golang code.

Build and push docker image

docker build --tag k8s-env-injector:latest .

docker tag k8s-env-injector <your container repository>/k8s-env-injector:<tag>

docker push <your container repository>/k8s-env-injector:<tag>

Deploy

Create a signed cert/key pair and store it in a Kubernetes secret that will be consumed by env-injector deployment

./deployment/webhook-create-signed-cert.sh \
    --service env-injector-webhook-svc \
    --secret env-injector-webhook-certs \
    --namespace default

NOTE: This creates a secret within your namespace so you need to use this namespace for the rest of the deployment steps

Patch the MutatingWebhookConfiguration by set caBundle with correct value from Kubernetes cluster

cat deployment/mutatingwebhook.yaml | \
    deployment/webhook-patch-ca-bundle.sh > \
    deployment/mutatingwebhook-ca-bundle.yaml

This will update the local deployment/mutatingwebhook-ca-bundle.yaml file with a new CA bundle string, make sure to check that it also has the matching namespace file before you deploy.

Deploy resources

kubectl create -f deployment/configmap.yaml
kubectl create -f deployment/deployment.yaml
kubectl create -f deployment/service.yaml
kubectl create -f deployment/mutatingwebhook-ca-bundle.yaml

Verify

The environment inject webhook should be running now in your namespace, you can verify by:

kubectl get pods
NAME                                                  READY     STATUS    RESTARTS   AGE
env-injector-webhook-deployment-bbb689d69-882dd   1/1       Running   0          5m

kubectl get deployment
NAME                                  DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
env-injector-webhook-deployment   1         1         1            1           5m

Add a label to the namespace that you want env-injector to make changes to with: env-injector=enabled

kubectl label namespace default hmcts.github.com/envInjector=enabled
kubectl get namespace -L hmcts.github.com/envInjector

Output:

NAME              STATUS   AGE    ENVINJECTOR
default           Active   4d3h   enabled
kube-node-lease   Active   4d3h   
kube-public       Active   4d3h   
kube-system       Active   4d3h   

Create a test deployment in your namespace, the following is an example that can be used for quick results:

cat <<EOF | kubectl create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      containers:
      - name: sleep
        image: hmctspublic.azurecr.io/docker-curl
        command: ["sleep","1d"]
        imagePullPolicy: Always
EOF

To verify that the changes have been applied you can check the output of the pod. The following example is based on using the deployment above:

kubectl get pod -l app=sleep -o json 

You should be able to see your additional config listed as part of the pod spec. If you have JQ installed you can narrow down the results to what you want to see:

kubectl get pod -l app=sleep -o json | jq '.spec.dnsOptions'

Helm chart

A Helm chart is also available, see env-injector-webhook. This can be installed in a single step using helm 2 or 3, e.g.

$ helm upgrade env-injector-webhook env-injector-webhook --install --namespace <your namespace>

Note: As the pods and service need to have:

• a secret containing a signed certificate and key
• a mutating webhook patched with the CA Bundle the script executed from pre-install-job.yaml takes care of creating them executing as a helm pre-install + pre-upgrade hook. This allows the installation/upgrade steps to execute in the right order, but has the (unfortunate) side effect of leaving around the secret and mutating webhook when the chart is deleted. For that reason a pre-upgrade + post-delete helm hook takes care of deleting secret and admission webhook.

Updates

If you wish to update or increase the coverage of this webhook you can use the following API Guide for Kubernetes and Golang:

• https://pkg.go.dev/k8s.io/api/core/v1

Notes

This repo is based on the excellent tutorial available at: morvencao/kube-mutating-webhook-tutorial