Fix kyverno on k8s-infra-ppc64le-prow-build cluster

[mkumatag]

Rightnow kyverno is failing in the k8s-infra-ppc64le-prow-build cluster

Following pods are in ImagePullBackOff state:

kgpall| grep ImagePullBackOff
kyverno            kyverno-cleanup-admission-reports-29034490-htbzn           0/1     ImagePullBackOff   0            26d     172.20.83.193    compute-4         <none>           <none>
kyverno            kyverno-cleanup-cluster-admission-reports-29034490-nzjtb   0/1     ImagePullBackOff   0            26d     172.20.154.1     compute-0         <none>           <none>

events:

3m50s       Normal    BackOff              pod/kyverno-cleanup-admission-reports-29034490-htbzn           Back-off pulling image "bitnami/kubectl:1.26.10"
3m50s       Warning   Failed               pod/kyverno-cleanup-admission-reports-29034490-htbzn           Error: ImagePullBackOff
8m49s       Warning   Failed               pod/kyverno-cleanup-cluster-admission-reports-29034490-nzjtb   Failed to pull image "bitnami/kubectl:1.26.4": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/bitnami/kubectl:1.26.4": no match for platform in manifest: not found
3m54s       Normal    BackOff              pod/kyverno-cleanup-cluster-admission-reports-29034490-nzjtb   Back-off pulling image "bitnami/kubectl:1.26.4"
3m54s       Warning   Failed               pod/kyverno-cleanup-cluster-admission-reports-29034490-nzjtb   Error: ImagePullBackOff

This is happening because this image doesn't contain the manifest for the ppc64le platform:

% podman manifest inspect bitnami/kubectl:1.26.4
{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
    "manifests": [
        {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "size": 429,
            "digest": "sha256:d96977749ea9374977a5c673c8ca73a008028c98cf1929642154e1c3d16b46de",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "size": 429,
            "digest": "sha256:29b1e667f2a3bbe3c11ecea9583cb6dc5a0024ac7abbd85add12ac67dd2858f9",
            "platform": {
                "architecture": "arm64",
                "os": "linux"
            }
        }
    ]
}

[mkumatag]

/sig k8s-infra

[dims]

@mkumatag https://explore.ggcr.dev/?image=registry.k8s.io%2Fkubectl:v1.32.3 we have one too!

[dims]

@mkumatag looks like their helm charts have enough support to inject our own image, can we please try that?

I also opened a PR just to test the theory (assuming they test their helm chart in presubmits!) - https://github.com/kyverno/kyverno/pull/12729/files

[mkumatag]

@mkumatag looks like their helm charts have enough support to inject our own image, can we please try that?

I also opened a PR just to test the theory (assuming they test their helm chart in presubmits!) - https://github.com/kyverno/kyverno/pull/12729/files

ah, I can try this.

[mkumatag]

Something is fishy, these cronjobs are no more present in the chart which is getting deployed via k8s apps

[mkumatag]

and the one deployed on the cluster shows v1.11.1 version

kubectl get deployment -n kyverno -o=yaml | grep image
          image: ghcr.io/kyverno/kyverno:v1.11.1
          imagePullPolicy: IfNotPresent
          image: ghcr.io/kyverno/kyvernopre:v1.11.1
          imagePullPolicy: IfNotPresent
          image: ghcr.io/kyverno/background-controller:v1.11.1
          imagePullPolicy: IfNotPresent
          image: ghcr.io/kyverno/cleanup-controller:v1.11.1
          imagePullPolicy: IfNotPresent
          image: ghcr.io/kyverno/reports-controller:v1.11.1
          imagePullPolicy: IfNotPresent

and the version in the chart says this:

appVersion: v1.13.4

[Prajyot-Parab]

Update -

Deployments on the cluster show correct v1.13.4 version now, as per the chart

# kubectl get deployment -n kyverno -o=yaml | grep image
          - --enableReporting=validate,mutate,mutateExisting,imageVerify,generate
          image: ghcr.io/kyverno/kyverno:v1.13.4
          imagePullPolicy: IfNotPresent
          image: ghcr.io/kyverno/kyvernopre:v1.13.4
          imagePullPolicy: IfNotPresent
          - --enableReporting=validate,mutate,mutateExisting,imageVerify,generate
          image: ghcr.io/kyverno/background-controller:v1.13.4
          imagePullPolicy: IfNotPresent
          image: ghcr.io/kyverno/cleanup-controller:v1.13.4
          imagePullPolicy: IfNotPresent
          - --enableReporting=validate,mutate,mutateExisting,imageVerify,generate
          image: ghcr.io/kyverno/reports-controller:v1.13.4
          imagePullPolicy: IfNotPresent

However the issue of missing ppc64le platform support in underlying manifest images still exists

# kubectl get pods -A | grep ImagePullBackOff
kyverno            kyverno-admission-controller-657b7bf875-v96zd              0/1     Init:ImagePullBackOff   0                2d1h
kyverno            kyverno-background-controller-77774cd959-dh4bf             0/1     ImagePullBackOff        0                2d1h
kyverno            kyverno-cleanup-admission-reports-29034490-jrzfz           0/1     ImagePullBackOff        0                23d
kyverno            kyverno-cleanup-cluster-admission-reports-29034490-825l5   0/1     ImagePullBackOff        0                23d
kyverno            kyverno-reports-controller-57676fff6d-6pxct                0/1     ImagePullBackOff        0                2d1h