Helm 2 and 3 not able to deal with large releases

[vsliouniaev]

• Output of helm version: 3.2.1
• Output of kubectl version: 1.18.2
• Cloud Provider/Platform: kind

Helm install fails with

Error: create: failed to create: Request entity too large: limit is 3145728

I don't have junk in my directory, I simply have a large chart where multiple people are managing their dashboards and this has grown beyond helm's current ability to manage them. This issue has existed since helm version 2.

Previously reported in:

• Error: create: failed to create: Request entity too large: limit is 3145728 #8062
• Error: create: failed to create: Request entity too large: limit is 3145728 #7264
• Helm 3: Generation of Helm secrets is failing #6711
• helm cannot find releases #5107
• Helm deployed my chart resources, but fails to record the release so it cannot be managed #4574
• helm not warning before adding large files to deployment and failing in inconsistent state #4215
• release: not found when I do helm install on a chart with some weird properties #1996
• ConfigMap size limit prevents storing release #1413

[mattfarina]

You may have seen, from previous issues, that this is because a version of the manifests is stored on the release object. Release objects are limited in size by etcd, the underlying datastore used by Kubernetes.

I thought there was work being done to try and help with this issue but I cannot find it off hand. I added this issue as a topic for the next developer call to see if I can track down that work.

[mattfarina]

I want to add a little context...

1. The manifests are stored on the release object to handle situations like rollback and the 3-way merge on upgrade. The 3-way diff is needed so that upgrades don't wipe away changes from systems like service meshes.
2. There is the helm get manifest command that enables one to get the manifests at any time.
3. We have been working to improve on the use of storage space. For example, certain things are now gzipped to reduce space on the manifest.
4. There is an alpha SQL storage back-end for releases. You can store them in PostegreSQL instead of secrets/etcd. This enables the storage of much larger charts via a back-end that supports it.

I can understand any frustration over the size issues. This is an issue of tradeoffs in features.

I want to look at ways to improve the store to the largest releases we can. Ideas and PRs are welcome for improving things.

[vsliouniaev]

Could this information be split between multiple etcd records?

[bacongobbler]

The number of cases where one has a chart that goes above the 1MB limit has been incredibly low. This might be the first genuine case where a chart requires more than 1MB of storage.

Could this information be split between multiple etcd records?

We have talked about partitioning across multiple configmaps in the past. That seems like a dirty hack to get around the 1MB limit. We also cannot introduce storage partitioning in Helm 3 without also supporting the existing non-partitioned backend, as we'd have to handle the upgrade path from non-partitioning systems to partitioning systems.

One other alternative is to look into whether you can tune etcd's backend to store values larger than 1MB in size.

From https://etcd.io/docs/v3.2.17/dev-guide/limit/:

etcd is designed to handle small key value pairs typical for metadata. Larger requests will work, but may increase the latency of other requests. For the time being, etcd guarantees to support RPC requests with up to 1MB of data. In the future, the size limit may be loosened or made configurable.

Perhaps we could look into supporting another alternative backend like partitioned-configmap as an experimental feature to handle such use cases. It might be a larger hassle writing one than it would be to set up a database and use the postgresql backend though.

I like @mattfarina's suggestion to find out what is necessary and what can be jettisoned from the storage backend to handle Helm's current functionality. That could give us a little more room to work with for the time being. Depending on how far above your chart is over the 1MB limit that may be a more reasonable option to pursue than to re-write the storage backend.

Let us know your thoughts.

[vsliouniaev]

In my opinion, Helm's entire attraction is the simplicity of use. Since its tied to Kubernetes on the back-end, which is currently tied to etcd, working within the available limitations seems like the only sensible approach to me, which I why I would pursue storing a release across multiple keys as a reasonable solution.

Running a stateful workload on Kubernetes just to support large helm releases is not something that I would be encouraging my users to do. Similarly, bumping etcd storage to 10x or 100x would likely come with unreasonable down-sides for general cluster stabilty.

As it stands, I'm going to be dropping my use of helm in favour of an alternative release mechanism because that's an out that I do have.

I'd be happy if there was some documentation provided that outlined the limitation of release sizes.

[bacongobbler]

I'd be happy if there was some documentation provided that outlined the limitation of release sizes.

helm/helm-www#558 documents the differences between some of the backend systems including the limitations of the configmap/secret backends and the reasons for introducing the postgresql backend, but it's still in progress. Once that is merged it will be available on the helm website.

[tculp]

What about adding an annotation to the release secret, such as "continued-in: sh.helm.release.v1.release.v1.part2". A solution like this seems like it would be reverse-compatible, requiring no changes for releases that fit within a single object.

[diepes]

We also managed to hit the limit. Probably due the use of secrets plugin, and adding multiple certificates in secrets.yaml etc.

In our case we dont require all the advanced features of helm (maybe in future), as we use helm as a templating tool and a release pipeline to deploy and rollback(redeploy old version)

For us adding a way to just disable the history, or just keep a simple summery in the history (e.g. command line used) would probably be a solution.

As to losing 3way merge, this gives us regular deployment failures, with immutable fields etc, and we now just delete objects (sometimes namespace) and re-deploy to fix helm deployment failures, so disabling/losing this could also be a benefit.

[github-actions]

This issue has been marked as stale because it has been open for 90 days with no activity. This thread will be automatically closed in 30 days if no further activity occurs.

[HamzaZo]

+1 to re-open this issue.

cc @mattfarina any update about this issue ? Thanks

[hickeyma]

Could it be that the cluster cannot deploy any more secrets when it hits a limit? In other words, the error limit is not necessarily for the Helm release secret but for the cluster limit?

Some context with a similar issue in helm/helm-2to3#171 (comment).