Application catalog is missing icons after upgrading from KKP v2.24.4

[embik]

What happened?

In v2.25.0, we are adding some nice icons to the default application catalog. Unfortunately, it looks like on an environment upgraded from KKP v2.24.4, the catalog looks like this:

I can also see that the ApplicationDefinitions don't have the base64-encoded logo:

apiVersion: v1
items:
- apiVersion: apps.kubermatic.k8c.io/v1
  kind: ApplicationDefinition
  metadata:
    creationTimestamp: "2024-03-01T09:02:45Z"
    finalizers:
    - kubermatic.k8c.io/cleanup-seed-application-definition
    generation: 2
    name: argocd
    resourceVersion: "79508"
    uid: d031b583-096d-41c7-834c-976d22671c99
  spec:
    defaultValues:
      server:
        service:
          type: LoadBalancer
    description: Argo CD - Declarative, GitOps Continuous Delivery Tool for Kubernetes.
    method: helm
    versions:
    - template:
        source:
          helm:
            chartName: argo-cd
            chartVersion: 6.0.0
            url: https://argoproj.github.io/argo-helm
      version: v2.10.0
- apiVersion: apps.kubermatic.k8c.io/v1
  kind: ApplicationDefinition
  metadata:
    creationTimestamp: "2024-03-01T09:02:45Z"
    finalizers:
    - kubermatic.k8c.io/cleanup-seed-application-definition
    generation: 2
    name: cert-manager
    resourceVersion: "79510"
    uid: e88b57c2-e1d5-43db-9d40-c5c100d0165e
  spec:
    defaultValues:
      installCRDs: true
    description: cert-manager is a Kubernetes addon to automate the management and
      issuance of TLS certificates from various issuing sources.
    method: helm
    versions:
    - template:
        source:
          helm:
            chartName: cert-manager
            chartVersion: v1.14.1
            url: https://charts.jetstack.io
      version: v1.14.1
- apiVersion: apps.kubermatic.k8c.io/v1
  kind: ApplicationDefinition
  metadata:
    creationTimestamp: "2024-03-01T08:29:47Z"
    finalizers:
    - kubermatic.k8c.io/cleanup-seed-application-definition
    generation: 1
    labels:
      apps.kubermatic.k8c.io/managed-by: kkp
      apps.kubermatic.k8c.io/type: cni
    name: cilium
    resourceVersion: "7724"
    uid: b6c36247-ee05-4dff-9984-8f772b89fdd5
  spec:
    defaultValues:
      hubble:
        relay:
          enabled: true
        tls:
          auto:
            method: cronJob
        ui:
          enabled: true
      operator:
        replicas: 1
    description: Cilium CNI - eBPF-based Networking, Security, and Observability
    method: helm
    versions:
    - template:
        source:
          helm:
            chartName: cilium
            chartVersion: 1.13.0
            url: oci://quay.io/kubermatic/helm-charts
      version: 1.13.0
    - template:
        source:
          helm:
            chartName: cilium
            chartVersion: 1.13.3
            url: oci://quay.io/kubermatic/helm-charts
      version: 1.13.3
    - template:
        source:
          helm:
            chartName: cilium
            chartVersion: 1.13.4
            url: oci://quay.io/kubermatic/helm-charts
      version: 1.13.4
    - template:
        source:
          helm:
            chartName: cilium
            chartVersion: 1.13.6
            url: oci://quay.io/kubermatic/helm-charts
      version: 1.13.6
    - template:
        source:
          helm:
            chartName: cilium
            chartVersion: 1.13.7
            url: oci://quay.io/kubermatic/helm-charts
      version: 1.13.7
    - template:
        source:
          helm:
            chartName: cilium
            chartVersion: 1.13.8
            url: oci://quay.io/kubermatic/helm-charts
      version: 1.13.8
    - template:
        source:
          helm:
            chartName: cilium
            chartVersion: 1.14.1
            url: oci://quay.io/kubermatic/helm-charts
      version: 1.14.1
    - template:
        source:
          helm:
            chartName: cilium
            chartVersion: 1.14.2
            url: oci://quay.io/kubermatic/helm-charts
      version: 1.14.2
    - template:
        source:
          helm:
            chartName: cilium
            chartVersion: 1.14.3
            url: oci://quay.io/kubermatic/helm-charts
      version: 1.14.3
- apiVersion: apps.kubermatic.k8c.io/v1
  kind: ApplicationDefinition
  metadata:
    creationTimestamp: "2024-03-01T09:02:46Z"
    finalizers:
    - kubermatic.k8c.io/cleanup-seed-application-definition
    generation: 2
    name: falco
    resourceVersion: "79512"
    uid: 9b3dde33-2a84-43a4-a1b4-3034c72ade63
  spec:
    defaultValues: {}
    description: Falco is a cloud native runtime security tool for Linux operating
      systems.
    method: helm
    versions:
    - template:
        source:
          helm:
            chartName: falco
            chartVersion: 4.1.1
            url: https://falcosecurity.github.io/charts
      version: 0.37.0
- apiVersion: apps.kubermatic.k8c.io/v1
  kind: ApplicationDefinition
  metadata:
    creationTimestamp: "2024-03-01T09:02:46Z"
    finalizers:
    - kubermatic.k8c.io/cleanup-seed-application-definition
    generation: 2
    name: flux2
    resourceVersion: "79513"
    uid: 437e3b78-7aa9-4a72-b2c6-332a114f15fc
  spec:
    defaultValues: {}
    description: Flux is a tool for keeping Kubernetes clusters in sync with sources
      of configuration (like Git repositories), and automating updates to configuration
      when there is new code to deploy.
    method: helm
    versions:
    - template:
        source:
          helm:
            chartName: flux2
            chartVersion: 2.12.2
            url: https://fluxcd-community.github.io/helm-charts
      version: 2.2.2
- apiVersion: apps.kubermatic.k8c.io/v1
  kind: ApplicationDefinition
  metadata:
    creationTimestamp: "2024-03-01T10:52:58Z"
    finalizers:
    - kubermatic.k8c.io/cleanup-seed-application-definition
    generation: 1
    name: k8sgpt
    resourceVersion: "79519"
    uid: 5dcd9200-18c6-4a06-9102-961b17abc9be
  spec:
    defaultValues:
      grafanaDashboard:
        enabled: false
        folder:
          annotation: grafana_folder
          name: ai
        label:
          key: grafana_dashboard
          value: "1"
      serviceMonitor:
        additionalLabels: {}
        enabled: false
    description: K8sGPT is a tool for scanning your kubernetes clusters, diagnosing
      and triaging issues in simple english.
    method: helm
    versions:
    - template:
        source:
          helm:
            chartName: k8sgpt-operator
            chartVersion: 0.1.0
            url: https://charts.k8sgpt.ai/
      version: v0.3.26
- apiVersion: apps.kubermatic.k8c.io/v1
  kind: ApplicationDefinition
  metadata:
    creationTimestamp: "2024-03-01T09:02:46Z"
    finalizers:
    - kubermatic.k8c.io/cleanup-seed-application-definition
    generation: 1
    name: kube-vip
    resourceVersion: "79521"
    uid: 3ef0a89a-9fd8-4966-8911-f317551429bb
  spec:
    defaultValues: {}
    description: kube-vip provides Kubernetes clusters with a virtual IP and load
      balancer for both the control plane (for building a highly-available cluster)
      and Kubernetes Services of type LoadBalancer without relying on any external
      hardware or software.
    method: helm
    versions:
    - template:
        source:
          helm:
            chartName: kube-vip
            chartVersion: 0.4.4
            url: https://kube-vip.github.io/helm-charts
      version: v0.4.1
- apiVersion: apps.kubermatic.k8c.io/v1
  kind: ApplicationDefinition
  metadata:
    creationTimestamp: "2024-03-01T10:52:59Z"
    finalizers:
    - kubermatic.k8c.io/cleanup-seed-application-definition
    generation: 1
    name: kubevirt
    resourceVersion: "79524"
    uid: 40ee3f6e-e74e-4cf0-89e4-02e49b6200a0
  spec:
    defaultValues: {}
    description: KubeVirt with Containerized Data Importer
    method: helm
    versions:
    - template:
        source:
          helm:
            chartName: kubevirt
            chartVersion: v1.1.0
            url: oci://quay.io/kubermatic/helm-charts
      version: v1.1.0
- apiVersion: apps.kubermatic.k8c.io/v1
  kind: ApplicationDefinition
  metadata:
    creationTimestamp: "2024-03-01T09:02:46Z"
    finalizers:
    - kubermatic.k8c.io/cleanup-seed-application-definition
    generation: 2
    name: metallb
    resourceVersion: "79525"
    uid: 1d6191c7-31d5-491c-9aec-cdb37595f184
  spec:
    defaultValues: {}
    description: MetalLB is a load-balancer implementation for bare metal Kubernetes
      clusters, using standard routing protocols.
    method: helm
    versions:
    - template:
        source:
          helm:
            chartName: metallb
            chartVersion: 0.14.3
            url: https://metallb.github.io/metallb
      version: v0.14.3
- apiVersion: apps.kubermatic.k8c.io/v1
  kind: ApplicationDefinition
  metadata:
    creationTimestamp: "2024-03-01T09:02:46Z"
    finalizers:
    - kubermatic.k8c.io/cleanup-seed-application-definition
    generation: 2
    name: nginx
    resourceVersion: "79526"
    uid: 22778f12-dda8-40f9-bd51-6d058803ff04
  spec:
    defaultValues: {}
    description: Ingress controller for Kubernetes using NGINX as a reverse proxy
      and load balancer.
    method: helm
    versions:
    - template:
        source:
          helm:
            chartName: ingress-nginx
            chartVersion: 4.9.1
            url: https://kubernetes.github.io/ingress-nginx
      version: 1.9.6
- apiVersion: apps.kubermatic.k8c.io/v1
  kind: ApplicationDefinition
  metadata:
    creationTimestamp: "2024-03-01T09:02:46Z"
    finalizers:
    - kubermatic.k8c.io/cleanup-seed-application-definition
    generation: 1
    name: trivy
    resourceVersion: "79529"
    uid: a8ea61dd-9431-4e54-b2d0-3d89cad3634a
  spec:
    defaultValues: {}
    description: Trivy is a simple and comprehensive vulnerability/misconfiguration/secret
      scanner for containers and other artifacts.
    method: helm
    versions:
    - template:
        source:
          helm:
            chartName: trivy
            chartVersion: 0.7.0
            url: https://aquasecurity.github.io/helm-charts/
      version: 0.37.2
- apiVersion: apps.kubermatic.k8c.io/v1
  kind: ApplicationDefinition
  metadata:
    creationTimestamp: "2024-03-01T09:02:46Z"
    finalizers:
    - kubermatic.k8c.io/cleanup-seed-application-definition
    generation: 2
    name: trivy-operator
    resourceVersion: "79530"
    uid: d820a7aa-6f83-4e85-88a5-4a72ec45800f
  spec:
    defaultValues:
      trivy:
        ignoreUnfixed: true
    description: Trivy-Operator is Kubernetes-native security toolkit.
    method: helm
    versions:
    - template:
        source:
          helm:
            chartName: trivy-operator
            chartVersion: 0.20.5
            url: https://aquasecurity.github.io/helm-charts/
      version: 0.18.4
kind: List
metadata:
  resourceVersion: ""

This even affects new applications like k8sgpt.

Expected behavior

On QA (which was installed from 2.25.0-beta.0), it looks fine:

How to reproduce the issue?

• Set up KKP v2.24.4 with --deploy-default-app-catalog.
• Upgrade KKP to v2.25.0-beta.2 with --deploy-default-app-catalog.
• Observe that all apps do not have an icon in the dashboard wizards to set up apps.

How is your environment configured?

• KKP version: v2.25.0-beta.2
• Shared or separate master/seed clusters?: shared

Provide your KKP manifest here (if applicable)

# paste manifest here

What cloud provider are you running on?

N/A

What operating system are you running in your user cluster?

N/A

Additional information

#13143 could be related. Not sure.

[embik]

/milestone KKP 2.25

[csengerszabo]

/assign @archups

[xrstf]

This is caused by the old kubermatic-webhook not yet knowing about the new fields in the AppDef CRD. The installer needs to wait for the KKP operator to roll out the new webhook and importantly, wait for the new webhook to be up and running before trying to update the AppDefs.

[xrstf]

Step 1 to solve this is #13176