BUG: Fail to deploy helm chart from plain HTTP registry

[wahabmk]

Problem Description

The following error is seen in the ClusterSummary status with the a local https://kcm-local-registry:5000 OCI Helm registry:

Name:         p--alarkov-aws-standalone-storage-capi-alarkov-aws-standalone-storage
Namespace:    kcm-system
Labels:       k0rdent.mirantis.com/managed=true
              projectsveltos.io/cluster-name=alarkov-aws-standalone-storage
              projectsveltos.io/cluster-type=Capi
              projectsveltos.io/profile-name=alarkov-aws-standalone-storage
Annotations:  <none>
API Version:  config.projectsveltos.io/v1beta1
Kind:         ClusterSummary
Metadata:
  Creation Timestamp:  2025-02-05T12:41:15Z
  Finalizers:
    clustersummaryfinalizer.projectsveltos.io
  Generation:  1
  Owner References:
    API Version:     config.projectsveltos.io/v1beta1
    Kind:            Profile
    Name:            alarkov-aws-standalone-storage
    UID:             edb925d7-00b3-4f68-a9db-337e7e1384e8
  Resource Version:  20369
  UID:               49a5b58c-e268-48a2-831a-5bb34ac37ee6
Spec:
  Cluster Name:       alarkov-aws-standalone-storage
  Cluster Namespace:  kcm-system
  Cluster Profile Spec:
    Cluster Selector:
      Match Labels:
        helm.toolkit.fluxcd.io/name:       alarkov-aws-standalone-storage
        helm.toolkit.fluxcd.io/namespace:  kcm-system
    Continue On Conflict:                  true
    Continue On Error:                     false
    Helm Charts:
      Chart Name:         ingress-nginx
      Chart Version:      4.11.3
      Helm Chart Action:  Install
      Registry Credentials Config:
      Release Name:       ingress-nginx
      Release Namespace:  ingress-nginx
      Repository Name:    ingress-nginx
      Repository URL:     oci://ghcr.io/k0rdent/catalog/charts
      Chart Name:         cert-manager
      Chart Version:      1.16.2
      Helm Chart Action:  Install
      Registry Credentials Config:
      Release Name:       cert-manager
      Release Namespace:  cert-manager
      Repository Name:    cert-manager
      Repository URL:     oci://ghcr.io/k0rdent/catalog/charts
      Values:             cert-manager:
  crds:
    enabled: true
      Chart Name:         kof-storage
      Chart Version:      0.1.0
      Helm Chart Action:  Install
      Registry Credentials Config:
        Plain HTTP:       true
      Release Name:       kof-storage
      Release Namespace:  kof
      Repository Name:    kof-storage
      Repository URL:     oci://kcm-local-registry:5000/charts
      Values:             victoriametrics:
  vmauth:
    ingress:
      host: vmauth.alarkov-aws-standalone-storage.project2a-dev.mirantis.com
  security:
    username_key: username
    password_key: password
    credentials_secret_name: storage-vmuser-credentials
grafana:
  ingress:
    host: grafana.alarkov-aws-standalone-storage.project2a-dev.mirantis.com
  security:
    credentials_secret_name: grafana-admin-credentials
cert-manager:
  email: alarkov@mirantis.com
external-dns:
  enabled: true
    Policy Refs:
      Deployment Type:       Remote
      Kind:                  ConfigMap
      Name:                  aws-cluster-identity-resource-template
      Namespace:             kcm-system
    Reloader:                false
    Stop Matching Behavior:  WithdrawPolicies
    Sync Mode:               Continuous
    Template Resource Refs:
      Identifier:  InfrastructureProviderIdentity
      Resource:
        API Version:  infrastructure.cluster.x-k8s.io/v1beta2
        Kind:         AWSClusterStaticIdentity
        Name:         aws-cluster-identity
        Namespace:    kcm-system
      Identifier:     InfrastructureProviderIdentitySecret
      Resource:
        API Version:  v1
        Kind:         Secret
        Name:         aws-cluster-identity-secret
        Namespace:    kcm-system
    Tier:             2147483547
  Cluster Type:       Capi
Status:
  Dependencies:  no dependencies
  Feature Summaries:
    Feature ID:         Resources
    Hash:               2GYI7BOL2wEoQaiQn4U3hZUINvTAGo6Urk5UQp0lxGk=
    Last Applied Time:  2025-02-05T12:58:18Z
    Status:             Provisioning
    Failure Message:    failed to do request: Head "https://kcm-local-registry:5000/v2/charts/kof-storage/manifests/0.1.0": http: server gave HTTP response to HTTPS client
    Feature ID:         Helm
    Hash:               oDBPSp7kAH3ey53U1YqH7/trr0AwumNBNoPTBH6CH0s=
    Last Applied Time:  2025-02-05T12:58:18Z
    Status:             Provisioning
  Helm Release Summaries:
    Release Name:       ingress-nginx
    Release Namespace:  ingress-nginx
    Status:             Managing
    Values Hash:        Eq4yyx7ALQHto1gbEnwf7jsNxTVy7WuvI5choD2C4SY=
    Release Name:       cert-manager
    Release Namespace:  cert-manager
    Status:             Managing
    Values Hash:        5aWdA2n8mzuNcUm4jFmOOSWZcvabnN1Z0wyEJm4Bg1U=
    Release Name:       kof-storage
    Release Namespace:  kof
    Status:             Managing
Events:                 <none>

This issue was resolved when downgrading Sveltos from 0.46.0 --> 0.45.0. So it was introduced sometime after 0.45.0 probably in https://github.com/projectsveltos/addon-controller/pull/980/files.

The helm repository being used is Flux HelmRepository type:

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  labels:
    k0rdent.mirantis.com/managed: "true"
  name: kcm-templates
  namespace: kcm-system
spec:
  insecure: true
  interval: 10m0s
  provider: generic
  type: oci
  url: oci://kcm-local-registry:5000/charts

Whereas the actual regsitry application is being run as a docker container on the kind network:

➜  ~ docker ps
CONTAINER ID   IMAGE                  COMMAND                  CREATED        STATUS        PORTS                       NAMES
8ec58d66f964   registry:2             "/entrypoint.sh /etc…"   22 hours ago   Up 22 hours   127.0.0.1:5001->5000/tcp    kcm-local-registry
29ca9bb48f88   kindest/node:v1.32.0   "/usr/local/bin/entr…"   22 hours ago   Up 22 hours   127.0.0.1:51496->6443/tcp   kcm-dev-control-plane

System Information

CLUSTERAPI VERSION: v1.9.6
SVELTOS VERSION: v0.50.0
KUBERNETES VERSION:

Client Version: v1.30.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.32.0
WARNING: version difference between client (1.30) and server (1.32) exceeds the supported minor version skew of +/-1