feat!: Introduce OCI multi-image index manifests #873

nderjung · 2023-10-11T17:35:25Z

Prerequisite checklist

Read the contribution guidelines regarding submitting new changes to the project;
Tested your changes against relevant architectures and platforms;
Ran make fmt on your commit series before opening this PR;
Updated relevant documentation.

Description of changes

Warning
This PR contains breaking changes to the OCI package manager. Upon release, all packages built on the OCI package manager will need to be re-packaged!

This large refactor reworks the OCI package manager to assume the use of the OCI image specification's index such that we can represent multiple manifests within a single canonical referencable name. This means that we can build multiple unikernels with the same name. This makes sense for multi-target projects which build the same application but target different hardware/platform vendors.

The structure of the index can be found here and in this implementation we itemized each manifest in the index in the same way. When we use the Catalog method now, we are able to query a remote registry based on the selected OS (platform) and architecture as well as any embedded KConfig options (if they are embedded).

When packaging multiple targets of a single application, the resulting package will be listed for all unique targets:

TYPE  NAME        VERSION  FORMAT  DIGEST   PLAT
app   helloworld  latest   oci     e8be0a9  qemu/x86_64
app   helloworld  latest   oci     a6802f4  qemu/arm64

Use cases include selecting appropriately based on the host's architecture as well as feature selection via embedded KConfig options.

Notable additional improvements which are incorporated in this pull request include:

Better offline mode in each package manager: all internal package managers were optimized to reduce instantiation time which makes a noticeable impact when using kraft in offline mode;
Better outputs in kraft pkg ls which now has dynamic column data information;
Better prompts in kraft pkg for selecting and packaging individual targets from an application project;
kraft pkg prune has been renamed to kraft pkg rm and now works for all underlying implementations (per-manifest type for libraries and components, and directory and containerd for OCI packages);
Better handling of insecure authenticated OCI registries; and,
Natively selecting the host platform and architecture in kraft pkg pull when no additional flags are provided.

GitHub-Closes: #704
GitHub-Supersedes: #719
GitHub-Depends-On: #874
GitHub-Depends-On: #878

cmd/kraft/build/build.go

oci/manifest.go

oci/handler/directory.go

manifest/manager.go

oci/handler/containerd.go

oci/handler/directory.go

oci/pack.go

cmd/kraft/pkg/pkg.go

packmanager/strategy.go

cmd/kraft/pkg/pull/pull.go

oci/README.md

craciunoiuc

I left all comments inline. Also check the results from gofumpt.

nderjung · 2023-10-13T14:25:30Z

Git format check is failing because there are commit messages with URL citations that are longer than 74 characters.

craciunoiuc · 2023-10-13T14:48:39Z

yup, noticed

I will now do a pass and try most of the build/run/pkg/pull/push arguments

Signed-off-by: Cezar Craciunoiu <cezar.craciunoiu@unikraft.io>

This commit removes the field method "format" from the target specification. The usage was intended to be part of the Kraftfile specification, wherein a user would specify the format of the target via: ```yaml targets: - name: example architecture: x86_64 platform: qemu format: oci ``` However, this was never realised and introduces some complications for now that are not resolved, specifically to do with the name of the target which conflicts with internal usage (by both the `unikraft/` package and the relevant package manager). In this commit, refactor the `kraft pkg` subcommand to ignore this attribute Signed-off-by: Alexander Jung <alex@unikraft.io>

Signed-off-by: Alexander Jung <alex@unikraft.io>

Use internal KraftKit knowledge of the fact that the config often lists the well-known path of the Manifest package manager's remote index. This is obviously not an OCI image registry so we can safely skip it. Doing this speeds up the kraft CLI and the instantiation of the OCI Package Manager in general by a noticable amount, especially with limited internet connectivity (as none is subsequently required). Signed-off-by: Alexander Jung <alex@unikraft.io>

This commit reworks the `kraft pkg push` subcommand to a). better handle selection via the `--format` flag by iterating over package managers and b). outputs relevant information package `Columns` to give a better picture of what is being pushed (especially artifacts that have identical names, i.e. OCI images). Signed-off-by: Alexander Jung <alex@unikraft.io>

This commit fixes an issue where `kraft pkg pull` was a). not passing the architecture and platform when positional arguments were set, such that form: ```console kraft pkg pull --arch x86_64 unikraft.org/nginx:latest ``` Would pull all artifacts instead of by architecture; and b). additionally setting KConfig options via the new `--kconfig|-k` option(s) such that packages can be queried by individual configuration parameters. Signed-off-by: Alexander Jung <alex@unikraft.io>

The user can use the shorthand `--plat qemu/x86_64` instead of setting `--plat qemu --arch x86_64`. Signed-off-by: Alexander Jung <alex@unikraft.io>

Without requesting an update from the package manager, the compatibility check can fail for sources which are external to the host. Signed-off-by: Alexander Jung <alex@unikraft.io>

Signed-off-by: Alexander Jung <alex@unikraft.io>

To prevent any out-of-order package writes, resort to non-parallel builds. It is also not necessarily more efficient handle packaging in parallel since files are always locked. In turn, this also results in the output of the packaging to read better. Signed-off-by: Alexander Jung <alex@unikraft.io>

This commit first names the flag `--all-versions` to `--all`. This doesn't affect existing functionality since this flag was unused. Instead, this flag is now used to determine whether to pull "everything" from the selected catalog based on the provided name. When unset, and additionally when `--arch` and `--plat` are unset, the host is queried for its architecture and platform, as a usability improvement. Signed-off-by: Alexander Jung <alex@unikraft.io>

Previously, the `NewManifestFromDigest` did not set the list of OS features from the Image config which contains the list of KConfig options for the image. As a result, information was lost if the image was re-serialized. This commit fixes this by replacing the incorrectly set `OSVariant` with `OSFeatures`. Signed-off-by: Alexander Jung <alex@unikraft.io>

Default to setting the KConfig for the image in the packaging process. Ultimately, this information enriches the resulting artifact and helps in determing the composition of the embedded unikernel, this should be default functionality. Signed-off-by: Alexander Jung <alex@unikraft.io>

This method abstract the `unikraft.Nameable` interface to an even simpler method. Signed-off-by: Alexander Jung <alex@unikraft.io>

The `multiselect` package enables quick TUI-based prompts to narrow a larger selection into a narrower selection. Signed-off-by: Alexander Jung <alex@unikraft.io>

This commit reworks `kraft pkg push` to use an interactive prompt to select what to package before packing. In addition, it uses the native `target.Filter` method determine packages based on the provided CLI flags for `--plat`, `--arch` and `--target`. Signed-off-by: Alexander Jung <alex@unikraft.io>

Signed-off-by: Alexander Jung <alex@unikraft.io>

In the scenario where a user has already pushed the same manifest to a remote registry and still has the manifest locally, the `Catalog` method would return both entries. Instead, use the platform checksum as a unique index in a map to prevent duplicates. The local will be preferred since this is queried last. Signed-off-by: Alexander Jung <alex@unikraft.io>

Previously, the `WithAuth` would have always been called with a nil `authn.AuthConfig`. This commit ensures that option method is only used when authentication details are found. Signed-off-by: Alexander Jung <alex@unikraft.io>

This commit incorporates two changes to the `defaultAuths` method. The first is to skip unset entries from `$HOME/.docker/config.json` and to not populate those in the return set of auths. The second is to prefer KraftKit's authentication details over Dockers. Signed-off-by: Alexander Jung <alex@unikraft.io>

craciunoiuc

What can go ~~wrong~~ right?

Reviewed-by: Cezar Craciunoiu cezar.craciunoiu@unikraft.io
Approved-by: Cezar Craciunoiu cezar.craciunoiu@unikraft.io

The target's attribute `Format` was deprecated in unikraft#873. Signed-off-by: Alexander Jung <alex@unikraft.io>

nderjung requested a review from craciunoiuc October 11, 2023 17:35