Proposal to converge helm repositories with docker registries

[SteveLasker]

@jdolitsky discussion on convergence of helm repos and container registries for Helm 3

[jdolitsky]

@SteveLasker lots of interesting stuff here. Need to take some time to comment on the various topics.

The biggest thing here that seems to conflict with the current approach is that this might prevent somebody from hosting a static Helm chart repository. How can somebody host a static repository with this model? What happens to the repository index? (index.yaml/index.json)

cc @technosophos @jdolitsky @ankushchadha @elioengcomp @ljnelson @bacongobbler @jainishshah17 @saracen @DarthFennec @mattfarina

[elioengcomp]

At first glance it sounds challenging, given the differences in the package composition, indexing mechanism, API specification and native client supported features.

I went through the proposal and I think they are positive from the client side perspective in terms of establishing a standard way to handle both docker images and helm charts. However, I'm struggling to see why one would like to merge them on the server side. It seems to me that this would make things more complicated to manage.

@SteveLasker and @jdolitsky can you elaborate more on the benefits that an initiative like this could provide?

[mattfarina]

@SteveLasker thanks for the PR and thoughts. As @jdolitsky noted, it's going to take me a few days to work through this.

My first five thoughts, without reading it through and processing it are:

• Many applications, as deployed, aren't a 1:1 chart to container image. Those are simple cases. Many cases include multiple different images and containers doing different things. How does that work here?
• I wonder how this compares to the app registry work CoreOS/Quay previously worked on?
• We are going to have a push for distributed registries. That means many registries and we want those to be easy to host. As easy as we can get so something like Homebrew Taps. Static repos via GitHub would be similar. How would this change?
• What happens to static helm repos? Many people already use those. Even those who don't host container image repos.
• Search provides an interesting angle in all of this. Container image repos do search server side. This is good for the provider as they get search metadata they can use. Search UX is more painful though. You need to specify the repo to search. You need this so you don't accidentally leak internal details to external services. The current Helm setup allows helm to pull down the details and perform the search client side. The UX is easier and metadata on the search isn't leaked. How would a change in repo style impact this and the philosophies that underpin it?

I'll give the proposals a good read through.

[SteveLasker]

Hey @mattfarina @elioengcomp;
Let me try to hit them one by one. But, the purpose was to start the discussion, so it's incomplete assuming there are additional complexities to account for.

• 1:1 - the proposal isn't actually meant to imply 1:1 of images and charts. The assumption is most scenarios would have a chart that deploys a collection of images. The images it references would be in the same registry, leveraging a common auth flow. The same auth used for the chart should be enabled to the images it references. It so happens that ACR supports deep namespacing. A chart might be stored at: demo42.azurecr.io/marketing/campaigns/q1-2019, while the images the chart references might be at: demo42.azurecr.io/marketing/campaigns/q1-2019/web, demo42.azurecr.io/marketing/campaigns/q1-2019/email, ... However, that's just an interesting optimization. The proposal is really focused on the fact that a chart references images. How can we minimize the differences between them?
• app registry: I hadn't seen this, but it does look very consistent.

helm registry push --namespace $USERNAME quay.io
helm registry install quay.io/$USERNAME/nginx

• Static repos: great question. Helm 3 will have breaking changes. Since Charts are based on images, if we can make it super easy to enable Helm Charts within registries as a common model, do we need another solution? Do they have to be compatible? Enabling helm charts over storage was a great way to seed the community. Is this still needed going forward? Or, is the impedance mismatch, concept count really needed?

• Search is a great scenario. And something we all (Azure, Google, AWS, Docker, Quay) need improvements. All container artifacts have various elements of meta-data. While it's possible to bring all this meta-data locally, does that scale to the types of queries someone is attempting to accomplish? If we can evolve the search capabilities of registries, would that investment pay off across all registry artifacts? The current model requires me to pull charts to do a search. But, until I've done the search, do i know what charts to pull, ...to do a search?

• indexing, API, client supported features When we released ACR Helm support, we attempted to incorporate these changes. But, to your point, it would be significant. Since Helm 3 is a significant change, is this the right time to make these changes to make fewer breaking changes, with more value?

• Server side merging Each cloud & vendor that's implemented docker/distribution has extended it to support additional media types. From OCI, Helm Charts, and other things. The underlying storage model is quite powerful. We've all implemented a means to handle high volume, concurrency, and even geo-replication scenarios. Adding Helm is relatively easy to the storage. The proposal is more focused on how we can make the experience better, with the assumption we've done the work to support Charts.

I should also add, this is my first time attempting to propose, discuss and make changes to an existing OSS project. So, I'm navigating this the best way I know. If there are better ways, please let me know. I know there's discussions on a Helm Summit next quarter. I'm hoping to continue this discussion, framing a goal and figuring out what we need to achieve to meet those goals. Then review again at the Helm Summit.
Steve

[jzelinskie]

I wonder how this compares to the app registry work CoreOS/Quay previously worked on?

App Registry was pretty much a prototype of this idea. Our goal was that distribution of Helm charts would simply become another mimetype specified in an OCI-like JSON manifest.

App Registry didn't really pick up traction because the lack of backwards compatibility with the existing Helm repository model. We had discussions about what the scope of work on the registry side of Helm v3 would be and it seemed like that the ultimate decision was that making a change like this is far too radical.

I'm gladly in favor of anything similar minded to this, but I also understand the desired to maintain backwards compatibility and easily be able to serve static repositories.

[ant31]

helm repos and container registries

that was the intent of the AppRegistry, it does not only mimic the 'docker' push/pull ux but also aim to follow the OCI spec, so that any existing container-registry can easily implement it to store helm chart.

Quay.io integrated it
https://quay.io/application/ant31/cookieapp?tab=channels

helm registry push quay.io/ant31/cookieapp:v1.4.0
helm registry pull quay.io/ant31/cookieapp@sha256:$SHA
helm registry install quay.io/ant31/cookieapp:latest

I could detail more or demo it, if it helps to discuss from a concrete example.

[SteveLasker]

Hi Jimmy, Antoine,
These both sound very similar. The helm registry push url/image:version is what we'd like to see simplify to simply be helm push|install url/image:version but recognize core changes are needed to make this work, such the proposal for Helm3 as a major change opportunity.
The presumption is the storage of container related assets in registries is a commodity. Each cloud/vendor has implemented a robust implementation under docker/distribution. If we can up the commodity a bit further, aligning helm and registries, we can focus on newer value for customers above. (effectively letting us compete on the values above), while we can all agree to a standards below to build upon.

Are either of you planning to be at DockerConEU or KubeCon NA? Perhaps we can meet, or is there another channel to discuss further?

[jzelinskie]

I'll be at KubeCon NA in December. I'm also always on the Kubernetes slack.

[mattfarina]

@SteveLasker @jzelinskie I wanted to pass on a few thoughts as things are continuing in the background:

1. I kinda like the idea of charts as other objects in container repos
2. We need people who can work on this if it's something we move on. Do either of you have people willing to write code to make this happen?
3. If this takes time, I think we can layer it in later without a major helm version bump. I think there are a couple strategies we could look at. Because this is a major change I just want it to be non-blocking for helm v3 if we move forward on it.

[ant31]

I've digged dicussions and presentations from the past two years:

• App-Registry (cnr) presentation in the ML
• Intro App-registrty slides (for sig-apps)
• Feedback from App-registry user
• Brandon Philips Keynote @kubecon
• Quay.io integration blog post

The idea is to reuse as much knowledge from the container ecosystem as possible in building an API for hosting applications.
Some key points:

• it reuses the OCI’s data structures (like descriptors) and its content addressable manifest pattern for security, signing, and non-trusted mirroring.
• It is close to the docker registry data model to make it easy for existing systems like gcr.io, quay.io, or any other registry to implement the service API and let user push and manage their applications like they are used to do with images.
• It converges with ‘images’ data structure and returns a single manifest-list (oci-spec), that could contains both applications’ manifests in different formats and the container-images’ manifests. In the future this could be the basis for a single distributable bundle of Chart/Resource/etc + required Images

[jdolitsky]

I think that this topic is valid regardless of which direction we go as it relates to Docker convergence.

We should remove this and open it as its own proposal. People would benefit from this.

Please also see related issue #54

[SteveLasker]

there are several items here. Happy to split up and converge with other proposals, like #54 as it makes sense. Perhaps to discuss on the call today to figure out specific next steps.

[eldada]

Typo? ...two tags equate tot the same... -> ...two tags equate to the same

[ant31]

cc @jzelinskie
We made some changes around that. For production usually it's not recommended to use mutable tags,
In example, v1.1.0 could refer to different sha overtime. Usually the recommendation is to directly use the sha256, to guaranty which software-version is getting pulled.

To simplify this process, and push best practices,
We tried a slightly different UX than docker:

1. We introduced the concept of channels, a channel is a pointer to a sha and it can move over time: e.g beta, stable, latest, master, dev .....
   Since a channel is 'nearly' a current docker tag, the command line is the same, using :

[**login url**] / [**namespace**] / [**chart**] : [**channel**]

e.g: https://quay.io/application/charts/ghost?tab=releases, the channel 'stable' is currently pointing to v1.0.1

helm registry pull quay.io/charts/ghost:stable 
Pull package: charts/ghost... 
./charts_ghost_1.0.1

2. A tag/version is immutable and can be seen as an alias to the sha256. It's more convinent to remember but as 'safe' as using the sha.
   The user refer to an immutable version/tag with @:

helm registry pull quay.io/charts/ghost@1.0.1 

This command is equivalent to:

helm registry pull quay.io/charts/ghost@sha256:35d3ec4fd61914383be8e94d3fe8a74fc24b2e2baedf87042b13d033e674290c

wdyt ?

[SteveLasker]

Channels are a very interesting concept, specifically as chart owners wish to flight updates to a specific version, and they provide a preview for what's coming.
The value I see for mutable tags(versions) is the ability to get security updates to a specific version without direct intervention:

• I deploy version 1 of a thing
• Version 2 of the thing is made available. By calling it version 2, it's presumed to be breaking changes, and I (as the developer/ops person) should opt into version 2
• A security flaw is found in sub-particle-a that both Version 1 and 2 reference in their charts.
• The security fix is flighted to the preview channel for both version 1 & 2
• After approval, thing: version 1 and thing:2 are updated.
• My company is watching changes to version 1, and quickly takes the update, tests it, and deploys the updated "stable" version.

In the above example, the thing owners were able to flight a security fix to version 1 & 2. The users of thing:1 were able to get updates, without having to involve development teams to make specific changes.

The great thing about additional channels is an org can also run automated tests on the preview channel. If any errors occur, they can automate reporting of issues.

If i reference a sha directly, I'm always bound to that very specific version, with no ability to move forward for automated patching. I wrote about this here: https://stevelasker.blog/2018/03/01/docker-tagging-best-practices-for-tagging-and-versioning-docker-images/

I think channels and stable tags are better together, but not mutual exclusive.

[ant31]

The value I see for mutable tags(versions) is the ability to get security updates to a specific version without direct intervention:

That exactly the purpose of the channel, and was automatic upgrade was the main focus this feature. You can of course use channels like version/tags, or 'allow' mutable tags (explicitly allow force push) but default/recommended usage would be:

channel: mutable
tag: non-mutable

Automatic upgrades should be opt-in, not a side effect.

• I deploy channel 1-stable of a thing (points to the version/tag 1.2.0)

• Version 2 of the thing is made available. By calling it version/tag 2.0.0, it's presumed to be breaking changes, and I (as the developer/ops person) should opt into channel 2-stable

• A security flaw is found in sub-particle-a that both Version 1.2.0 and 2.0.0 reference in their charts.
  The security fix is flighted to both channels 1-beta & 2-beta,

• After approval, version/tags 1.2.1 and 2.0.1 are pushed, channels 1-stable is updated to points to 1.2.1, and channel 2-beta is updated to 2.0.1

• My company is watching changes to version 1-stable, and quickly takes the update, tests it, and deploys the updated "stable" version.

In term of use experience
The chart maintainer it is doing:

1. Release the security fix in the beta channels

# Push 1.2.1 to the beta channel
$ helm registry push quay.io/myorg  --channel 1-beta
>>> Release '1.2.1' added to the '1-beta' channel

# Push 2.0.1 to the beta channel
$ helm registry push quay.io/myorg  --channel 2-beta
>>> Release '2.0.1' added to the '2-beta' channel

2. Review channels:

$ helm registry channel quay.io/myorg/thing
channel    release
---------  ---------
1-stable     1.2.0
1-beta        1.2.1
2-stable      2.0.0
2-beta        2.0.1

2. Once confirmed / approved, promote from beta to stable

$ helm registry push quay.io/myorg  --channel 1-stable
>>> Release '1.2.1' added to the '1-stable' channel
$ helm registry push quay.io/myorg  --channel 2-stable
>>> Release '2.0.1' added to the '2-stable' channel

For the application use the experience is either:

• manually pull versions

helm registry install quay.io/org/thing@v1.2.1

• Trust the channels and get automatic latest versions

helm registry install quay.io/org/thing:1-stable

At least, we should be able to 'protect' a tag from being 'force' pushed.

[SteveLasker]

I see where your going. It's basically a more explicit way to opt into updates for a stable tag.
The thing that strikes me with this is:

• customers are already struggling with stable and unique tags
• they don't seem to be understanding the wave of problems ahead of them for patching containers
• when I tell them about it, they either think containers aren't as great as they hoped (looking backwards), or they get it and love the ability to patch, test, then deploy a container
• channels adds yet another concept they are struggling to keep up with
• how can we get this into distribution across all registries so it's a common thing. Unless it's common, I struggle with it getting great adoption

I do like the expectations. Just struggle with getting it to common usage.

Is this something we can layer in, as customers feel the pain?

[ant31]

channels adds yet another concept they are struggling to keep up with

We can completely layer that, and not even talk about channels to begin with.
Channels are not a key feature, nor required to use a registry, then it should not be a big deal if vendor don't adopt it at the same time.

I'm bringing that early, because it could change the cli UX.
With app-registry, after several iteration we landed with:

helm pull domain/org/app:channel
helm pull domain/org/app@tag
helm pull domain/org/app@sha256

If we delay channels for let's say12 months, and everyone is using the syntax:

helm pull domain/org/app:tag

it will be impossible to switch to :channel / @tag.

So the only question to answer early is:

• how users will pull or deploy from a channel via the CLI ?

helm pull domain/org/app --channel beta
helm pull domain/org/app+beta
helm pull domain/org/app:beta  # lookup for the tag beta first , then for the channel 
....

ps:
About customers, we presented it saying: "you can subscribe to stable/beta/alpha channels and get automatic upgrades or... continue to manually manage versions". This looked much more like a step forward than backward in term of maintenance and operations.

[ant31]

See the comment about 'channels'. We enforced the semver for 'tags/version', and let a free-form for channels'

[ankushchadha]

Hi @SteveLasker

The proposal is very interesting, a common auth flow simplifies UX but there are 4 broad categories that need to be addressed -

A) There are valid use cases where the helm chart doesn't belong in a docker registry -
i. If a helm chart references only other sub-charts (uber chart) to deploy complete applications does it belong in a Docker registry?
ii. If a chart only applies security configurations such as secrets it does not belong in a Docker registry.

B) Release Cadence
i. Storing ever-changing charts in a docker registry of stable images will pollute the Docker registry. Both docker images and helm charts usually have a different release cadence.
ii. Docker images are usually re-tagged to indicate the maturity of the binary and then pushed to different registries. Example: Once rt-stage.jfrog.io/my-image:1.0 is verified, it gets promoted to prod and in this process, it gets retagged as rt-prod.jfrog.io/my-image:1.0 & pushed to a prod Docker registry. Does it mean that helm charts are copied over to a prod Docker registry each time a docker image gets promoted?

C) Chart Neutrality - This proposal can potentially encourage charts to be written in a way that it is cloud provider dependent. It will be hard for users to expect the same chart to run on all cloud providers and even on-prem. Hybrid becomes complicated.

D) Version - It is not a good idea to use dynamic version 'latest' as a helm chart version. Users are very selective in terms of what they want to deploy in production.

[mortent]

I like the idea of making Helm repositories feel more like Docker repositories, but I don't think there is any reason to require Helm charts and Docker images to be hosted in the same repo. If a provider wants to use one registry for both thats fine, but that shouldn't be an expectation.

If we want to transition completely to a new repo format for Helm 3, we would need a pretty detailed spec pretty soon. We would like to get input from the major cloud providers on the spec and then allow time for all of them to provide support for this by the time Helm 3 is released. That sounds like a very challenging timeline.
We should also make sure there is a solution for people who wants to host their helm repository on-prem.

[SteveLasker]

Hi Morten,
Of course there's no requirement to host them in the same registry. It's more a matter of today,, there's no standard and the Helm CLI is pretty inconsistent in many places making the experience harder. As part of this effort, I did reach out to Ahmet to find some folks at google for gcr that could collaborate, but none yet. I've also reached out to AWS as well, and just realized I owe a response back.
The goal here is to do this in such a way that anyone using an instance of docker/distribution could use this new thing. Particularly, including on-prem registry providers. My hope/goal is we would implement something atop the docker/distribution spec, in standard way so the opposite is true. Anyone already running docker/distribution can easily host helm chart repos. We've implemented helm chart repos on ACR without changes to distribution, so it's most likely a matter of providing a helm chart repo API that can interact with the distribution APIs.
But, yes, we do feel the time pressure and would want to get going soon. We had a good call today to start down this path.
Steve

[kfox1111]

Its not clear to me from this description if the docker registry needs to be enhanced to support charts. Is that a requirement, or are the charts stored in OCI format in the registry so that any normal registry will work unmodified? It would be kind of useful to be able to use docker pull/push / skopeo to copy around the charts too...

[SteveLasker]

Incorporated a bunch of the recent conversations, including KubeCon and the OCI proposals. https://github.com/helm/community/blob/3689b3202e35361274241dc4ec188e1e6f1a2e53/proposals/helm-repo-container-registry-convergence/readme.md

[bacongobbler]

Hi @SteveLasker!

Helm 3 has since been released. I don't imagine we'll be merging this into the "Helm 3 proposals" any time soon because Helm 3 has already been released with experimental OCI support. Therefore I'm going ahead and closing this. Please feel free to carry on conversation over in the helm ticket queue.

If there's documentation in here that could prove useful, please feel free to propose it over to the helm-www repo as supporting documentation.

https://helm.sh/docs/topics/registries/

Thanks!