fix ibm build cluster secrets

[upodroid]

Fixing error from #7904

7904 didn't work because you didn't add the objects to the files being loaded by kustomize. I also fixed the external-secrets deployment code.

/cc @Rajalakshmi-Girish @mkumatag

[mkumatag]

I see this empty file kubernetes/ibm-ppc64le/helm/kyverno.yaml is this intentional?

[mkumatag]

and I also see that some of the pods in the kyverno namespace are ImagePullBackOff state

% kubectl get pods -n kyverno
NAME                                                       READY   STATUS             RESTARTS   AGE
kyverno-admission-controller-6468f4bbd4-bx8lw              1/1     Running            0          6d17h
kyverno-background-controller-86d48fc7db-jnfc2             1/1     Running            0          6d17h
kyverno-cleanup-admission-reports-29034490-htbzn           0/1     ImagePullBackOff   0          6d17h
kyverno-cleanup-cluster-admission-reports-29034490-nzjtb   0/1     ImagePullBackOff   0          6d17h
kyverno-cleanup-controller-7766d9df55-vdh4z                1/1     Running            0          6d17h
kyverno-reports-controller-8c4667558-k4vwl                 1/1     Running            0          6d17h

events:

 % kubectl get events -n kyverno
LAST SEEN   TYPE      REASON               OBJECT                                                         MESSAGE
26m         Normal    Pulling              pod/kyverno-cleanup-admission-reports-29034490-htbzn           Pulling image "bitnami/kubectl:1.26.10"
26m         Warning   Failed               pod/kyverno-cleanup-admission-reports-29034490-htbzn           Failed to pull image "bitnami/kubectl:1.26.10": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/bitnami/kubectl:1.26.10": no match for platform in manifest: not found
109s        Normal    BackOff              pod/kyverno-cleanup-admission-reports-29034490-htbzn           Back-off pulling image "bitnami/kubectl:1.26.10"
94s         Warning   Failed               pod/kyverno-cleanup-admission-reports-29034490-htbzn           Error: ImagePullBackOff
6m49s       Normal    JobAlreadyActive     cronjob/kyverno-cleanup-admission-reports                      Not starting job because prior execution is running and concurrency policy is Forbid
6m49s       Warning   TooManyMissedTimes   cronjob/kyverno-cleanup-admission-reports                      too many missed start times. Set or decrease .spec.startingDeadlineSeconds or check clock skew
36m         Normal    Pulling              pod/kyverno-cleanup-cluster-admission-reports-29034490-nzjtb   Pulling image "bitnami/kubectl:1.26.4"
36m         Warning   Failed               pod/kyverno-cleanup-cluster-admission-reports-29034490-nzjtb   Failed to pull image "bitnami/kubectl:1.26.4": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/bitnami/kubectl:1.26.4": no match for platform in manifest: not found
105s        Normal    BackOff              pod/kyverno-cleanup-cluster-admission-reports-29034490-nzjtb   Back-off pulling image "bitnami/kubectl:1.26.4"
105s        Warning   Failed               pod/kyverno-cleanup-cluster-admission-reports-29034490-nzjtb   Error: ImagePullBackOff
6m49s       Normal    JobAlreadyActive     cronjob/kyverno-cleanup-cluster-admission-reports              Not starting job because prior execution is running and concurrency policy is Forbid
6m49s       Warning   TooManyMissedTimes   cronjob/kyverno-cleanup-cluster-admission-reports              too many missed start times. Set or decrease .spec.startingDeadlineSeconds or check clock skew

Not sure if this will have any impact on the cluster.!

[upodroid]

I see this empty file kubernetes/ibm-ppc64le/helm/kyverno.yaml is this intentional?

Yes, ArgoCD will read that file, we are currently using the default helm values.

As for the ImagePullBackOff errs, might have been docker rate limits.

[mkumatag]

As for the ImagePullBackOff errs, might have been docker rate limits.

na, the problem is that manifest doens't image for linux/ppc64le platform:

Need to find some way to override this image to point to the right one.

[mkumatag]

lets merge this, will look at the failing piece in a different issue
/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mkumatag, upodroid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• kubernetes/OWNERS [upodroid]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment