[Sample] rewrite image registry

[JimBugwadia]

Problem Statement

rewrite the image registry, to force all pulls throug a proxy cache e.g. with harbor

Solution Description

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: replace-image-registry
spec:
  background: false
  rules:
    - name: replace-image-registry
      match:
        any:
        - resources:
            kinds:
            - Pod
      mutate:
        foreach:
        - list: "request.object.spec.containers"
          context: 
          - name: imageData
            imageRegistry: 
              reference: "{{ element.image }}"
          preconditions:
            all:
            - key: "{{imageData.image}}"
              operator: Equals
              value: "docker.io/*"
          patchStrategicMerge:
            spec:
              containers:
              - name: "{{ element.name }}"
                image: "harbor2.zoller.com/dockerhub-proxy/{{imageData.repository}}:{{imageData.identifier}}"

Example "Good" Resource

No response

Example "Bad" Resource

No response

Other Comments

No response

Slack discussion

https://kubernetes.slack.com/archives/CLGR9BJU9/p1704740196988469

Troubleshooting

• I have read and followed the documentation AND the troubleshooting guide.
• I have verified the policy does not exist in the samples library.

[siddhikhapare]

/assign

[siddhikhapare]

I would like to work on this

[jseiser]

We tested the above, and it didnt work with the bare registry, so we ended up with this.

    spec:
      background: false
      rules:
      - name: prepend-registry-containers
        match:
          any:
          - resources:
              kinds:
              - Pod
        preconditions:
          all:
          - key: "{{request.operation || 'BACKGROUND'}}"
            operator: AnyIn
            value:
            - CREATE
            - UPDATE
        mutate:
          foreach:
          - list: "request.object.spec.containers"
            patchStrategicMerge:
              spec:
                containers:
                - name: "{{ element.name }}"           
                  image: "harbor.tld/{{ images.containers.\"{{element.name}}\".registry}}/{{ images.containers.\"{{element.name}}\".path}}:{{images.containers.\"{{element.name}}\".tag}}"
      - name: prepend-registry-initcontainers
        match:
          any:
          - resources:
              kinds:
              - Pod
        preconditions:
          all:
          - key: "{{request.operation || 'BACKGROUND'}}"
            operator: AnyIn
            value:
            - CREATE
            - UPDATE
          - key: "{{ request.object.spec.initContainers[] || '' | length(@) }}"
            operator: GreaterThanOrEquals
            value: 1
        mutate:
          foreach:
          - list: "request.object.spec.initContainers"
            patchStrategicMerge:
              spec:
                initContainers:
                - name: "{{ element.name }}"           
                  image: "harbor.tld/{{ images.initContainers.\"{{element.name}}\".registry}}/{{ images.initContainers.\"{{element.name}}\".path}}:{{images.initContainers.\"{{element.name}}\".tag}}"

This assumes your harbor proxy caches, have the same exact name as the registry you are replacing e.g. harbor.tld/quay.io and harbor.tld/dockerhub.io Also, we generate this with TF, which is why you see those extra \ escapes

[siddhikhapare]

@JimBugwadia I have also tested this policy -

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: replace-image-registry
  annotations:
    policies.kyverno.io/title: Replace Image Registry
    kyverno.io/kubernetes-version: "1.24"
spec:
  background: false
  rules:
    - name: replace-image-registry-pod-containers
      match:
        any:
          - resources:
            kinds:
              - Pod
      mutate:
        foreach:
          - list: "request.object.spec.containers"
            context:
              - name: imageData
                imageRegistry:
                  reference: "{{ element.image }}"
            preconditions:
              any:
                - key: "{{imageData.image}}"
                  operator: Equals
                  value: "docker.io/*"
            patchStrategicMerge:
              spec:
                containers:
                  - name: "{{ element.name }}"
                    image: "harbor.example.com/{{imageData.repository}}:{{imageData.identifier}}"
    - name: replace-image-registry-pod-initcontainers
      match:
        any:
        - resources:
            kinds:
            - Pod
      preconditions:
        all:
          - key: "{{ request.object.spec.initContainers[] || `[]` | length(@) }}"
            operator: GreaterThanOrEquals
            value: 1
      mutate:
        foreach:
          - list: "request.object.spec.initContainers"
            context:
              - name: imageData
                imageRegistry:
                  reference: "{{ element.image }}"
            preconditions:
              any:
                - key: "{{imageData.image}}"
                  operator: Equals
                  value: "docker.io/*"
            patchStrategicMerge:
              spec:
                initContainers:
                  - name: "{{ element.name }}"
                    image: "harbor.example.com/{{imageData.repository}}:{{imageData.identifier}}"

I got this output -
It should not fail for this patchedresource but it failed

apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod1
spec:
  containers:
    - image: harbor.example.com/nginx:latest
      name: docker-with-registry

harbor is running -

[jseiser]

Wanted to note, what we used above, doesnt work well either. It appears to work fine, and then you will randomly get pods that have the registry doubled up.

Successfully pulled image "harbor.tld/cr.l5d.io/linkerd/proxy-init:v2.2.3" in 909ms (909ms including waiting)
Successfully pulled image "harbor.tld/cr.l5d.io/linkerd/proxy:stable-2.14.8" in 794ms (794ms including waiting)
Back-off pulling image "harbor.tld/harbor.tld/registry.k8s.io/ingress-nginx/controller:v1.9.5"

I cant figure out why, when i test it locally

apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  namespace: policy-test
  labels:
    app: myapp
spec:
  containers:
    - name: docker-io
      image: docker.io/velero/velero:v1.6.2
      imagePullPolicy: Always

    - name: docker-library-no-reg
      image: nginx:1.25.3-perl
      imagePullPolicy: Always

    - name: docker-library-partial-reg
      image: jenkins/jenkins:2.426.2-jdk17
      imagePullPolicy: Always

    - name: quay
      image: quay.io/jetstack/cert-manager-controller:v1.12.7
      imagePullPolicy: Always

    - name: quay2
      image: quay.io/jetstack/cert-manager-controller:v1.12.7
      imagePullPolicy: Always

  initContainers:
    - name: init-myservice
      image: busybox:1.28
      command: ["sh", "-c", "sleep 2; done"]

I have no problems, but ive noticed with my helm charts we have deployed, it will randomly happen like the above.

[JimBugwadia]

See: kyverno/kyverno#9381

[siddhikhapare]

See: kyverno/kyverno#9381

Thanks @JimBugwadia @chipzoller

[chipzoller]

@jseiser, the reason you were probably seeing the doubled-up name prefix was due to Pod UPDATE ops. To avoid that, it's best to fire on CREATE requests only. The point may be moot if you still haven't found anything that'll work currently.