Helm3: Use registry as dependency URL

[phoenix-bjoern]

Helm 3 has a new chart command to push/pull charts to a OCI registry. This seems to be an alternative to repositories to maintain dependencies. But ATM you have to retrieve each of the charts manually:

helm3 chart pull registry.local/application/helm-chart:latest
helm3 chart export registry.local/application/helm-chart:latest

I would expect that helm could manage the charts retrieved from OCI registries in the same way it pulls dependencies from chart repositories. Therefore chart dependencies in OCI registries should be declared in Charts.yaml in the same way:

dependencies: 
  - name: my-application
    version: 1.0.0
    condition: my-application.enabled
    repository: registry.local/application/helm-chart:latest

However this currently just causes an error.

Do I misunderstand the concept of OCI image/registry support for charts? Or is the additional support for OCI registries as dependency just a missing feature?

[bacongobbler]

Thanks for raising this.

OCI support as a getter (a dependency) is currently a missing feature. OCI registry support is experimental and isn't fully integrated with the rest of the CLI similar to chart repositories.

[bcg62]

this is a blocker for adoption, would much prefer to use registry storage and break free of repo managers.

this is core functionality needed for common use cases

[bacongobbler]

@bcg62 feel like taking this one on as a contribution?

[phroggyy]

@bacongobbler this is something I'm currently needing to proceed with a much simplified implementation of an umbrella chart. Hence, I'd love to have a look at contributing this functionality (perhaps alongside resolving #6982 which I believe is effectively two sides of the same coin). Do you, as a core maintainer, have a preference for how to annotate registry retrieval? Done implicitly, either through parsing the URL (checking for a tag by splitting on :) or trying the source in both "modes", or explicitly, e.g oci: true, or oci@regist.ry/some/chart (latest implicit, unless specified).

To summarise the options:

dependencies: 
  - name: my-application
    version: 1.0.0
    repository: https://eu.gcr.io/project/helm-chart:latest # implicit OCI by specifying a tag

  - name: my-application
    version: 1.0.0
    repository: https://eu.gcr.io/project/helm-chart # implicit: check if it works as a regular chart first, otherwise try OCI with `:latest`

  - name: my-application
    version: 1.0.0
    repository: oci@eu.gcr.io/project/helm-chart # explicit OCI by specifying the mode in the URL

  - name: my-application
    version: 1.0.0
    oci: true # explicit boolean annotation
    repository: eu.gcr.io/project/helm-chart # explicit OCI by specifying the mode in the URL

Alternatively we could use something like https+oci as the protocol, or even just oci

[bacongobbler]

the current convention is scheme://URL, so the third option as oci://eu.gcr.io/project/helm-chart would be the best option. The tag is unnecessary; that can be picked up with the version field.

[phroggyy]

My one concern with doing that is that we currently validate the Version field to be SemVer, a constraint we'd probably want to remove if we want to support e.g git commit refs and latest as tags (common for Docker image tags in CI, and I'd like to use it for application charts).

I'd almost say we should make the version field optional instead, since one image tag could only ever reference one chart version, but one chart version could be referenced by several image tags. This would allow us to keep the chart version semver, while allowing image tags to be arbitrary.

As for the format, having looked at the code, I'd agree on an oci protocol, with a new RegistryGetter.

[bacongobbler]

I think it's still worth to keep the semver comparison as a core value. Docker's "latest" tag is just a convention championed by Docker, but it is not a constraint on the OCI protocol.

By removing the semantic version comparison, we can no longer guarantee that helm upgrade myrelease oci://eu.gcr.io/project/helm-chart will fetch the latest version of the chart, which is used commonly in CI/CD pipelines.

For example, many organizations release versions in a non-deterministic order to previous versions, i.e. v1.0.0, v1.1.0, then v1.0.1 in a non-sequential order. Helm does that often as well: we release new versions of Helm 3 (Helm 3.1.0) and Helm 2 (Helm 2.16.3).

To support a "latest" tag, we would need to write code in our CI/CD pipelines to determine what release should be "latest". in our case, we'd need to codify that "latest" does not mean "most recently deployed", but "the highest version as per semantic versioning". By allowing Helm to perform a semantic version comparison out of the gate, there's no need for organizations to track "latest", and helm install mychart will always install the "latest" version available.

A great example of this in practice can be found in #7597, where github's "latest" release URL points to whichever release was last tagged... Not exactly a great experience for our users.

Hope that helps provide some context on the design decisions around keeping the semantic version comparison kicking around.

[phroggyy]

Yep, I definitely don't mind keeping SemVer around for the chart, but I'm a bit concerned about using it to infer the image tag – what do you think about, when using the OCI protocol for the source, we only enforce either version or an image tag? I feel like that would allow both workflows without inhibiting either. The only question left is: what do we do if the downloaded chart is not the specified version (applies in all cases where version is specified) – error?

[TBBle]

Carrying back from #7613, since it's not directly related to that PR.

Not having SemVer matching for OCI registries seems like a significant impediment (it's 100% a blocker for one of my use-cases, but not for the primary use-case).

Are there any thoughts on how to alleviate this? Do we still end up having to host an index somewhere that lists all published versions of a chart, so that we can get the correct tag to use?

The benefits I was looking for in an OCI registry were that I could push charts up to, e.g., ECR in parallel to the Docker images, rather than having to host my own Helm Chart registry instance somewhere on the cloud. And we can use literal versions easily enough as we'd only publish release versions.

I'd however hoped to also pull our CI system across to use an OCI registry and replace the Helm chart repo we use now, but our dependency charts depend on SemVer matching to allow painless updating to pre-release sub-charts, and also locking down to release subcharts for parent chart releases.

[jdolitsky]

@TBBle what you are requesting likely requires expansion on the OCI distribution spec. In the past the spec contained a /v2/_catalog endpoint to provide search which has since been removed. Some registries provide their own search mechanism which is not part of the spec.

The latest conversation on this topic can be found here: opencontainers/distribution-spec#71

[TBBle]

If you know the repository (chart name, I assume) then the list tags API would be sufficient to enumerate the available versions, and then perform a SemVer match against, wouldn't it?

So really only sufficient for helm install, but not helm search repo, as the latter expects to be able to match chart names by sub-string. (Hence, good enough for my specific use-cases)

Because tags can't hold SemVer completely, either a mangling would need to be defined, or the search/install would have to pull each chart's top-level data to get the actual SemVer. The latter sounds painful, I'm not sure how bad it would be in practice? The tag list doesn't give the digest, so you can't deduplicate from just that list for example.

It would combine well with helm search hub though, as that could return an OCI registry entry as the URL, which would combine nicely with #7419 to let you helm search hub and then helm install from the hub directly, including possibly a different version than Hub advertised.

[TBBle]

Give the last few comments on opencontainers/distribution-spec#71, perhaps to fully address this, we'd need to define a 'Helm' extension for the registry API to extract Helm charts and their SemVers in a single API call? This would allow searching for charts in the absence of a general 'list repositories' API, and also could make SemVer matching more efficient by implementing it on the server side. It would be optional for SemVer matching: when the extension does not exist, helm install can still walk the list of tags as described in my previous comment.

[phroggyy]

@TBBle I think the mangling approach for incompatible tags would be the way to go. You could quite easily just replace the + with a . I reckon. It does make the tag look like an invalid SemVer tag (1.2.3.buildtag) but it might be the most legible, and won't clash with actual valid SemVer

[TBBle]

Another option might be to support 'chart lists' similar to 'manifest lists' which adds an extra step to publishing, if you want to support version lookup. Then you can push your charts with whatever tags (or no tags) you want, and the 'chart list' is a list of versions and their chart content digest.

It would be an extra step after pushing the manifest, but could be automated by the tag-list+pull-each-chart approach above, pushing that load to the upload-time rather than the download-time.

The problem is, what tag do you put on a 'chart list'? It'd have to be a well-known tag, or you're back in the same boat... Or it could be that what you advertise on Helm Hub for example is the full repository path for the chart-list, including a tag for the chart-list.

So in theory, you could have variant chart lists for the same chart by tag, e.g.,

• stable-charts.example.com/mongodb:default
• stable-charts.example.com/mongodb:replicaset

where right now those are two distinct charts. (This is a bad example, that's not actually the difference in those charts).

To be clear, I'm not convinced this is a great idea; I'm struggling to identify a real use-case for multiple chart lists in a repository and it seems like more work than I'm willing to put into it right now anyway. ^_^

[phroggyy]

@TBBle could you open a new issue about this. I believe we should close this issue once #7613 is merged. We can then tackle this discussion separately, and discuss it from that new existing behaviour. Just to make it easier for the maintainers to track and triage :)

[phroggyy]

Another thing to discuss with regards to OCI is provenance files, which we don't have a concept of with OCI. We might be able to discuss these two concepts together, as they both relate to something that isn't directly the chart content. I'm thinking we might be able to utilise the solution for one to guide the other. Not sure if there'll be any overlap, but figured I'd mention it.

[github-actions]

This issue has been marked as stale because it has been open for 90 days with no activity. This thread will be automatically closed in 30 days if no further activity occurs.

[bacongobbler]

I believe this can now be closed after merging #8843. Let me know if I missed something.