[Discussion/Feature] Supporting SemVer and signed releases with OCI Registries

[phroggyy]

There are two important aspects of Helm repositories that I haven't been able to really visualise an implementation of for OCI Registries. These are SemVer matching (requires indexing) and signature verification (provenance files in standard repositories).

I'm opening this ticket to open a discussion on how we could resolve these matters. I'm not too familiar with all different OCI specs, but I'm hoping someone else might be!

For the SemVer matching, one idea would be to use the OCI Image Index. I'm not entirely sure if that fits the needs of Helm, but at first glance it looks like it might.

Secure distribution through signature verification is a bit harder as it seems there's no OCI standard for it yet. Red Hat and rkt use Simple Signing, while Docker has Docker Content Trust. It's also worthwhile mentioning that Simple Signing has been built into https://github.com/containers/image.

Would love to hear some ideas on how these issues could be resolved, which could bring us closer to feature parity between the two.

[hickeyma]

/cc @jdolitsky

[mattfarina]

@phroggyy These are great questions...

As for signing... Helm has provenance and signing. It uses PGP. Helm supports that in repositories today but how we handle things in OCI registries is, as of yet, unclear.

Red Hat simple signing is PGP based which is similar to this. Singularity uses PGP as well. This is another image type. Docker uses Notary which is based on TUF. The first version of Notary has an issue where an image in one location copied to another location can have a different digest which means it needs a different signature. It's not portable all the time.

This is something we should totally figure out.

Since Helm requires SemVer, I was thinking about using tags. Those can be queried using the distribution API. Thoughts?

[phroggyy]

@mattfarina using tags looks like a good idea – I was not aware that tag listing was part of the spec, so thanks for pointing that out!

One thing that further complicates matters is that the spec and #7613 implementation support arbitrary (non-SemVer) image tags. Do we want to explicitly disallow that and limit Helm images to SemVer-compatible tags, or keep support for non-SemVer tags?

The way it is currently implemented is the following:

repository: oci://registry/group/image:arbitrary-tag
version: "1.0" # not used because a tag was specified in repository
---
repository: oci://registry/group/image
version: "1.0" #this gets appended to the repository as no tag was specified there, for oci://registry/group/image:1.0

[mattfarina]

The tagging and versioning are complicated. For instance...

• tags in an OCI registry are really just a pointer to digest id. They are mutable. It's conceptually similar to tags in git.
• what happens when the tagged version is different from the one in the Chart.yaml file?
• Helm assumes SemVer all over the place for charts. This is very useful for dependencies. If we break from it for OCI registries a bunch of other things in the workflows are going to break or be strange. We should be consistent for functionality (e.g., linting) and for documentation.

I also think we should have the principle of least astonishment. We should provide helpful hints and direction.

@phroggyy what is the use case for...

repository: oci://registry/group/image:arbitrary-tag
version: "1.0" # not used because a tag was specified in repository

Where the tag and the version are different.

[bacongobbler]

I believe this is because helm chart save allows you to package a chart under another tag than what is described in your Chart.yaml.

From https://helm.sh/docs/topics/registries/#the-chart-subcommand:

$ helm chart save mychart/ localhost:5000/myrepo/mychart:2.7.0

Notice that the tag can be set by the user, even though the package version is already known in Chart.yaml. This seems like a design flaw.

I briefly explained earlier in #6593 (comment) that this would not be a great design choice. @TBBle also laid out some great arguments against allowing non-semver package tags. For that reason it does not makes sense to allow OCI repositories to specify the tag in the repository URL. It should be inferred by the version field, or it should use the repository tags API to find the latest semver-compatible version as we do with chart repositories.

[phroggyy]

@mattfarina e.g automated releases w/ git tag (or latest). For now, it is mainly for compatibility with helm chart commands. Not something we have to support, but if we won't support it, that needs to be documented (and we should update the helm chart commands for it)

[TBBle]

I'm not sure about latest (we've removed it from our Docker use-cases locally where possible), but for automated builds with git, it's a challenge because the OCI spec doesn't let you use + in your tag, so you can't use the SemVer build tag (the place to record a git hash) in the Registry tag.

This means you actually cannot (without perhaps an agreed/known escaping) use full SemVer versions as OCI tags.

That said, if you use "SemVer without the build tag", as an OCI Registry tag, that shouldn't conflict because you probably shouldn't be publishing charts with all the same versions except the build tag, as SemVer does not order on the build-tag so if that's all you're distinguishing by, you need to name it explicitly every time. And it's easy enough to strip the build tag when talking to the OCI Registry to retrieve files.

"SemVer without the build tag" also works with the Listing Image Tags API, for the use-cases of SemVer matching in helm install --version (and related) and helm dependency update. This is not sufficient for helm list, but for that you need to retrieve more chart metadata than the verison anyway, so it was never going to be.

This means using the pre-release tag for autobuilds, and so (for git) you need some other total ordering of your CI builds as the git hash is not sortable. We use GitLab CI Job Number, because it also lets us trace an image back to its build-log trivially by substituting a URL; hopefully most CI systems have something like this that can be leveraged in the build. Even when using SVN or Perforce (which has globally-incrementing commit references), you want a rebuild of the same commit to be newer, so you still need some other total ordering of builds.

Overall, I'd strongly lean towards SemVer-expectation (or SemVer-reqirement) in OCI tags, because it's the least-surprise path from how the rest of Helm works. It might be overly surprising for people who expect OCI Registry support in Helm to work like OCI Registry support in Docker though.

And the people who want Helm to be less SemVer-tied, of course. They probably would like OCI Registry tags to be appVersion... which I think would be a bad idea for similar reasons to making --version use appVersion.

[jdolitsky]

I've just put together this proposal with how I think we should probably move forward: #8387

[github-actions]

This issue has been marked as stale because it has been open for 90 days with no activity. This thread will be automatically closed in 30 days if no further activity occurs.