Kustomize Packaging

[stealthybox]

Referenced in #14, we would like installers to be able to load kustomize packages.
We also expect that addon-operators can benefit from this ComponentConfig work and packaging.

This implies improvements for packaging and distributing these kustomize bases and overlays.

git support with tags/refs and nested dirs already seems to be built into kustomize which is a great starting point.

Some good things to work on:

• OCI image package format
  • OCI registries are a pre-req to running a cluster
  • allows signing and verification
  • use https://github.com/deislabs/oras?
• Cluster Bundle based package
  • no deps other than CRD's in k8s API
  • can use typed refs to load related bundles inside the cluster from installer ComponentConfig (interesting MIME-type questions for the URI here)

[jzelinskie]

+1 to using ORAS. I'm an OCI maintainer and trying to ensure that there is a cohesive strategy for how everyone is storing non-container artifacts before we cut a stable OCI-distribution release.

[stealthybox]

@jzelinskie do you have an opinion on using https://github.com/containers/skopeo + https://github.com/openSUSE/umoci as libraries to pull and unpack?

I was able to get something working with the commandline tools and some canonical file layout.
The creation UX was just to docker build a Dockerfile where the resulting kustomize layer(s) were in an image under /addon/

[jzelinskie]

It depends on what you'd like to accomplish. If you want to simply store YAML inside of container layers and pull them out, you could use skopeo/umoci. If you want to actually differentiate between regular container images and Kustomize YAML at the registry level, then you want to use ORAS and configure a custom mimetype. ORAS is internally using libraries from containerd -- they are just flexible enough to configure the mimetype whereas I'm not sure if skopeo or umoci have a public API that allows that level of configuration.

[stealthybox]

If we use a different mime-type -- we'll probably need to use something buildkit based to assemble the OCI image instead of using canonical folders in a Dockerfile. This is definitely more formal.
One thing I found interesting about using the docker layers was that it provided another means of extending a package (since you can overlay the files).

umoci appears to only have UX for operating on files, but you can unpack and repack the image and make raw edits:

  umoci insert --image oci:foo mybinary /usr/bin/mybinary
  umoci insert --image oci:foo myconfigdir /etc/myconfigdir
  umoci insert --image oci:foo --opaque myoptdir /opt
  umoci insert --image oci:foo --whiteout /some/old/dir

I'm not sure about modifying the mime-types when using it as a library /cc @cyphar

WRT implementation:
What feels most appropriate in my opinion is for the unpack functionality whether it's ORAS based or using skopeo/umoci/something-else be integrated into the kustomize ref parser /w URI's and the execution occurring within kustomize libs (as opposed to us first unpacking and passing the resulting dir into kustomize).

[stealthybox]

IIRC Helm3 is using ORAS with dedicated MIME-types so that's a consideration for parity.

[stealthybox]

Some initial POC work on this is posted:
https://github.com/ecordell/kpg

Thanks for getting started @ecordell

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.