Helm charts repository index.yaml retention policy

[carrodher]

As reported in this issue (#8433), in the last few times we are facing some issues with the index.yaml associated with the Bitnami Helm charts repository.

Current situation

After some investigation, it seems the root cause is related to CloudFront reaching some limits due to the volume of traffic when serving the index.yaml.
This index.yaml contains all the Bitnami Helm charts history (around 15300 entries), producing a pretty fat 14MB file. Given the size of the file and the volume of traffic, thousands of terabytes of download traffic per month are being generated.

One of the alternatives considered was the use of compression at CloudFront, in that case, this solution doesn’t work since compression is not used by the Helm client (helm) itself (see helm/helm#8070) so it doesn’t solve the reported issue.

Mitigation

As the first line of action, we will reduce the size of the index.yaml by removing some old versions and keeping all versions for a period of time (6 months).

⚠️ Please note this action is not removing/deleting any Helm chart, packaged tarballs (.tgz) won't be removed, this action is only affecting index.yaml used to list the Helm charts. Previous versions of the index.yaml can be used to install old Helm charts.

Please note Helm charts tarballs (.tgz) won't be removed, this action is only affecting index.yaml.

Result

Applying this approach (#10530), we obtained the following results:

Total chart versions
* Before: 15260
* Removed: 12138
* After: 3122

Producing a reduced 3.5MB index.yaml.

🔧 Workaround for previous versions

The index.yaml is stored in this repository under the index branch, users should be able to use any commit in that branch to add a previous version of the index.yaml.

• Manually using helm repo add

$ helm repo add bitnami-full-index https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami
"bitnami-full-index" has been added to your repositories

• When used as a dependency in Chart.yaml:

  - name: postgresql
    version: 8.1.0
-   repository: https://charts.bitnami.com/bitnami
+   repository: https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami
    condition: postgresql.enabled

[DmytroDrachov]

Thank you very much

[sisrael-dn]

Hi,
https://charts.bitnami.com/bitnami seems to return 403 AccessDenied error at the moment.
Is this related to this issue?
My guess is that if you work with S3 bucket as CloudFront origin, the new index.yaml got the wrong ACL now (not publicly readable)

[potiuk]

This is TERRIBLY wrong. For example, we released Helm Chart to our users https://airflow.apache.org/docs/helm-chart/stable/index.html with bitnami's Postgres 10.5.3 and all the charts our users are using are useless now if they use embedded Postgres.

How is it possible you do such a breaking change in such a way ? This must be some kind of a joke.

This totally undermines any trust in bitnami as a source of charts - You basically rendered all historical charts released by likely huge number of projects useless.

[bschurig]

As reported in this issue (#8433), in the last few times we are facing some issues with the index.yaml associated with the Bitnami Helm charts repository.

Current situation

After some investigation, it seems the root cause is related to CloudFront reaching some limits due to the volume of traffic when serving the index.yaml. This index.yaml contains all the Bitnami Helm charts history (around 15300 entries), producing a pretty fat 14MB file. Given the size of the file and the volume of traffic, thousands of terabytes of download traffic per month are being generated.

Mitigation

As the first line of action, we will reduce the size of the index.yaml by removing some old versions and keeping all versions for a period of time (6 months).

Please note Helm charts tarballs (.tgz) won't be removed, this action is only affecting index.yaml.

Result

Applying this approach (#10530), we obtained the following results:

Total chart versions
* Before: 15260
* Removed: 12138
* After: 3122

Producing a reduced 3.5MB index.yaml.

Workaround for previous versions

The index.yaml is stored in this repository under the index branch, users should be able to use any commit in that branch to add a previous version of the index.yaml, for example:

$ helm repo add bitnami-pre-2022 https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
"bitnami-pre-2022" has been added to your repositories

It would be fantastic if someone could get this added to the readme.

[GMZwinge]

Instead of removing all versions older than 6 months, would it be possible to keep the last major/minor versions older than 6 months, eg: mongodb-9.2.6.

[carrodher]

Are old chart versions not available through the OCI repo? I'm seeing this:

Downloading solr from repo oci://registry-1.docker.io/bitnamicharts
Save error occurred:  could not download oci://registry-1.docker.io/bitnamicharts/solr: registry-1.docker.io/bitnamicharts/solr:6.1.4: not found
Error: could not download oci://registry-1.docker.io/bitnamicharts/solr: registry-1.docker.io/bitnamicharts/solr:6.1.4: not found

Right, old charts were not retroactively published as OCI packages. In the case of Solr, the oldest one is 6.2.2, see https://hub.docker.com/r/bitnamicharts/solr/tags

[dunn]

@carrodher will the OCI repository retain all packages going forward? having access to old charts was the primary attraction of OCI for me (and probably most others) watching this issue

[carrodher]

@carrodher will the OCI repository retain all packages going forward? having access to old charts was the primary attraction of OCI for me (and probably most others) watching this issue

Yes, all the Helm charts versions will be retained as OCI packages in DockerHub from when we start pushing the packages there. What is not in DockerHub as OCI packages are Helm charts versions previous to the implementation of OCI as a new distribution channel.

[carrodher]

We just merged this PR (#15998) modifying the Chart.yaml and Chart.lock in order to use the dependencies from the DockerHub OCI registry instead of the Bitnami index.
In the same way, the README files were modified showing the new method to install the Bitnami Helm charts.

[fmonpelat]

hi, i'm getting not found with this link: https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami is there another one? I need an old version and reading, it will not be in OCI...

[agomezmoron]

Hi @fmonpelat, did you run?

$ helm repo add bitnami-full-index https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami
"bitnami-full-index" has been added to your repositories

Remember you have to use that to add the new repo. The URL https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami will give you always a 404 error code because when you add the repo, what it's really using is the index.yaml located at https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami/index.yaml.

Please check that and let us know if you keep having that issue.

Cheers

[sushmithan2]

Not able to find mongo chart version 10.20.0 in bitnami repo

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.