CFE-783: A variety of changes needed for correct operation with multi architecture catalogs #611

jchunkins · 2023-04-19T23:37:25Z

Description

This change brings support for processing multi and single architecture catalog images within oc mirror.

TL;DR: The first item in the manifest list has its FBC catalog extracted, which is then processed as needed and then “reinserted” (i.e. added as a new layer) into each image referenced in the manifest list. In other words, the images are modified with the same updated FBC content. The binaries within each architecture catalog are unmodified.

To understand the changes in this PR, we need to discuss OCI layouts, which is a on disk representation of image references. The OCI layout specification shares similar concepts with the docker image v2s2 specification in that they both define manifests (i.e. a JSON document) that describe a single image or a list of images (i.e. a fat manifest). The primary difference between the two standards is the mediaType which indicates the standard being used (you can see the compatibility matrix for details). Essentially, OCI attempts to keep backwards compatibility with docker, and adds its own enhancements (e.g. annotations). These manifests can be pushed/pulled to/from a remote docker registry in either format, and can be converted from one type to another if desired. In the diagram above, you can see that manifest lists and image manifests are the entities that can be moved into and out of a OCI layout on a file system.

An OCI layout consists of an index.json file that acts as the entry point for discovering what images are stored within the layout on the disk. This index.json file can reference any number of single or multi architecture images, however oc mirror only references a single image at a time in a given OCI layout. The image stored in the OCI layout can be either a single architecture image (i.e. an image manifest) or a multi architecture image (i.e. a manifest list). A manifest list image is a logical grouping of an architecture specific image manifests stored in an array. This arrangement allows a single digest reference to represent all of the architecture specific images, and tools such as skopeo or podman can automatically pull an image that matches the platform that the tool is being executed on. An individual architecture specific image is referred to as an image manifest. A image manifest has references to a config JSON file that describes detailed layer history, the platform the image runs on and runtime information necessary to start an image. A image manifest, also has references to one or more layers, which are just tar files, which could be compressed or uncompressed. As you can see in the above diagram, all of these internal references are by digest, which is a hash computed by taking a sha256sum of the manifest or tar file. Note that the digest of the config file itself is referred to as the "image id" whereas an image digest is hash of the image manifest itself.

An example of a OCI layout on disk looks like this:

/path/to/oci/layout
├── blobs
│   └── sha256
│       ├── 01c6d5dcde3e9f2de758d794a77974600fe5e0b7a8c2ce2833eede2e8b25e7e5
│       └── ...
├── index.json
└── oci-layout

The index.json will have references to manifests and tar files that reside in the blobs/sha256 directory. Technically the index.json file is its own form of manifest list, but it is important to note that the index.json file itself has no representation in a docker registry. Tools such as skopeo will generate a OCI layout using what I call "manifest list indirection" where the manifest list resides in the blobs directory and the index.json references this manifest list by its digest. However some tools will set the media type of the index.json to indicate that its a manifest list and not use this "indirection" (i.e. the index.json directly refers to the image manifests and includes the platform information directly). This PR handles both of these scenarios, and ensures that the appropriate images are processed.

OCI layouts are stored in the oc mirror workspace under <oc-mirror-workspace>/src/catalogs/<repoPath>/layout. A diagram of this workspace looks like this:

	/tmp/cwd/oc-mirror-workspace
	├── results-1675904745
	└── src
		├── catalogs
		│   └── icr.io                                                                              ─┐
		│       └── cpopen                                                                           ├─ catalog path (one per catalog)
		│           └── ibm-zcon-zosconnect-catalog                                                  │
		│               └── sha256:6f02ecef46020bcd21bdd24a01f435023d5fc3943972ef0d9769d5276e178e76 ─┘
		│                   ├── include-config.gob         <—— this represents v1alpha2.IncludeConfig for associated with index/index.json
		│                   ├── index                      <—— this represents a decl config for "single architecture image" image
		│                   │   │                              which was selected (note all images store the same content)
		│                   │   └── index.json             <—— Declarative Config
		│                   └── layout                     <—— OCI layout is capable of holding “manifest list” and “single architecture" images
		│                       ├── blobs
		│                       │   └── sha256
		│                       │       ├── 01c6d5dcde3e9f2de758d794a77974600fe5e0b7a8c2ce2833eede2e8b25e7e5
		│                       │       ├── 1cd0595314a53d179ddaf68761c9f40c4d9d1bcd3f692d1c005938dac2993db6
		│                       │       ├── 1ff4ea896d6b958aa060e04eb090f70c563ad0650e6b362c1a1d67582acb3b8e
		│                       │       ├── 25d123725cf91c20b497ca9dae8e0a6e8dedd8fe64f83757f3b41f6ac447eac0
		│                       │       ├── 2d55550cefe39606cc933a2ed1d242b3fd9a72d85e92a2c6b52b9623a6f4fe6a
		│                       │       ├── 34e12aa195bcd8366fbab95a242e8214ae959259bd0a1119c28d124f5799f502
		│                       │       ├── 49a32e2e950732d5a638d5486968dcc3096f940a94811cfec99bd5a4f9e1ad49
		│                       │       ├── 561bc8bee264b124ba5673e73ad36589abbecf0b15fb845ed9aab4e640989fbc
		│                       │       ├── 6672e188b9c3f7274b7ebf4498b34f951bc20ea86a8d72367eab363f1722d2ed
		│                       │       ├── 7062267a99d0149e4129843a9e3257b882920fb8554ec2068a264b37539768bc
		│                       │       ├── ae475359a3fb8fe5e3dff0626f0c788b94340416eb5c453339abc884ac86b671
		│                       │       ├── b2dd6105dc025aa5c5e8b75e0b2d8a390951369801d049fc0de2586917a42772
		│                       │       ├── c03c8f94bb495320bbe862bc69349dbdf9f2a29b83d5b344b3930890aaf89d7d
		│                       │       ├── dc1b9846d7994450e74b9cde2e621f8c1d98cdf1debd591db291785dd3fc6446
		│                       │       └── f89d6e2463fc5fff8bba9e568ec28a6030076dbc412bd52dff6dbf2b5897a59d
		│                       ├── index.json
		│                       └── oci-layout
		├── charts
		├── publish
		│   └── .metadata.json
		├── release-signatures
		└── v2

At a high level, this PR allows oc mirror traverse the OCI layout to pick an image for processing depending on what's stored in the layout:

for a single image manifest, we can use this image directly
for a manifest list, we choose the first image is the list. We can do this because the catalog content is identical in all images.

Once an image is chosen, the declarative config (i.e. the JSON/YAML files that represent the OLM catalog) is extracted from the image for processing. The existing code which performed this extraction was done incorrectly, and this PR addresses this issue. When an OCI image is referenced in the ImageSetConfiguration, and the --use-oci-feature is set, this declarative config is placed into <current working directory>/olm_artifacts/<repo>/<config folder> which acts as a temporary location to store the extracted data. After the catalog is processed, the content is written to <oc-mirror-workspace>/src/catalogs/<repoPath>/index. This catalog content is then added to ALL image manifests that are stored within the OCI layout by adding new layers that represent the updated catalog content. This is handled within the ImageBuilder, which has been modified to handle image updates for both single and multi arch images. Other changes within the PR relate to properly handling file paths and docker reference parsing.

Also, to be clear, additional images defined in the ImageSetConfiguration, and related images found within the catalog itself are already capable of mirroring single and multi platform images already because the "oc image mirror" codebase knows how to do this already. This PR is primarily focused on making catalog image handling multi architecture "aware".

User Stories

As a user I want to perform a "mirror to mirror" operation using manifest list and single image docker references in the operators portion of the ImageSetConfiguration
As a user I want to perform a "mirror to disk" operation using manifest list and single image docker references in the operators portion of the ImageSetConfiguration
As a user I want to perform a "oci layout to mirror" operation using an oci reference in the operator portion of the ImageSetConfiguration. The content of the OCI layout can contain either a manifest list or a single image manifest

Acceptance Criteria

All mirror operations following the user stories should complete successfully, resulting in the catalog being pushed to its remote repository along with any related images or additional images (if any are supplied in the ImageSetConfiguration).

Functional Changes

pkg/cli/mirror/fbc_operators.go

remove findFBCConfig & replace with extractDeclarativeConfigFromImage This is necessary because the layer processing did not handle whiteout files. Switched to go-containerregistry which handles this correctly.
modify getManifest so that it is fat manifest aware, and returns the first manifest in the list, or for single images, returns that manifest
Removed functions that are no longer necessary: GetCatalogConfigPath, getConfigPathFromConfigLayer, UntarLayers

pkg/cli/mirror/create.go

make createOlmArtifactsForOCI manifest list aware and discovers the first image in the list, or for single images, uses that image directly
calls replacement function extractDeclarativeConfigFromImage
stores full path to the artifact in a map so this path does not need to be recalculated later
strip protocol from ctlg.OCIFBCPath when constructing a layoutPath

pkg/cli/mirror/options.go

add operatorCatalogToFullArtifactPath map[string]string which stores temporary paths to declarative config directory
- key: OCI URI (e.g. oci://foo which originates with v1alpha2.Operator.Catalog)
- value: /olm_artifacts//

pkg/cli/mirror/operator.go

initialize operatorCatalogToFullArtifactPath
make use of operatorCatalogToFullArtifactPath map to obtain the full path to the olm_artifacts directory and its contents and remove getOperatorCatalogRef() since it is no longer necessary
make writeLayout() use the image config file to discover the platform when adding images, otherwise fall back to linux/amd64

pkg/image/image.go

replace getManifest with getFirstDigestFromPath which now gets the first digest for manifest lists, or the image digest for single image
change logging behavior in ParseReference() when path does not exist and invalidate the ID in that case.
in ParseReference, make sure that the docker reference uses lower case values for "name" and "namespace" which is required to make valid docker references. This covers the case where a OCI filepath uses uppercase letters in its path.

pkg/cli/mirror/catalog_images.go

in rebuildCatalogs, make sure that the catalog source docker reference uses lower case values for "name" and namespace" which is required to make valid docker references. This covers the case where a OCI filepath uses uppercase letters in its path.

pkg/image/builder/image_builder.go

make image builder handle single arch and multi arch images

Non-functional Changes

pkg/cli/mirror/mirror.go

removed redundant AsRespoitory() call since RepositoryName() already does this
use constant for olm_artifacts directory
operatorCatalogToFullArtifactPath

pkg/cli/options.go

use constant for workspace

all files

add documentation and change formatting for long function signatures
if any file used deprecated io/ioutil, replace with io or os package as necessary
update unit tests as necessary (the bulk of the files are related to new test cases)

Test Changes

internal/testutils/testutils.go

update and create new functions for creating single/multiarch images
created *WithURL function variants to allow pre-existing registry in tests because the defaults server won't allow external calls while debugging

pkg/image/builder/image_builder_test.go

update test cases for image builder

Fixes # (CFE-783)

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

Test With Docker (mirror to mirror)

Use a ImageSetConfiguration which points to a "fat manifest" catalog:

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
storageConfig:
  local:
    path: /tmp/localstorage
mirror:
  operators:
  - catalog: icr.io/cpopen/ibm-zcon-zosconnect-catalog@sha256:6f02ecef46020bcd21bdd24a01f435023d5fc3943972ef0d9769d5276e178e76

Start a local docker registry to act as the destination

docker run -d -p 5000:5000 --name registry registry:2

Run oc mirror like this (no errors should occur):

oc mirror -c "/path/to/ImageSetConfiguration.yml" --dest-skip-tls --dest-use-http docker://localhost:5000

Throw away your local docker registry

docker stop registry
docker rm registry

Test With Docker (mirror to disk)

Use a ImageSetConfiguration which points to a "fat manifest" catalog:

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
storageConfig:
  local:
    path: /tmp/localstorage
mirror:
  operators:
  - catalog: icr.io/cpopen/ibm-zcon-zosconnect-catalog@sha256:6f02ecef46020bcd21bdd24a01f435023d5fc3943972ef0d9769d5276e178e76

Run oc mirror like this (no errors should occur):

oc mirror -c "/path/to/ImageSetConfiguration.yml" file://tmp/mirrortodisktest

Test With OCI

Pull a docker "fat manifest" image into a location on the file system:

skopeo copy --all --format v2s2 docker://icr.io/cpopen/ibm-zcon-zosconnect-catalog@sha256:6f02ecef46020bcd21bdd24a01f435023d5fc3943972ef0d9769d5276e178e76 oci:///Users/jhunkins/temp/ocmirror

Use a ImageSetConfiguration which points to this newly downloaded image like this:

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
storageConfig:
  local:
    path: /tmp/localstorage
mirror:
  operators:
  - catalog: oci:///Users/jhunkins/temp/ocmirror

Start a local docker registry to act as the destination

docker run -d -p 5000:5000 --name registry registry:2

Run oc mirror like this (no errors should occur):

oc mirror -c "/path/to/ImageSetConfiguration.yml" --use-oci-feature --oci-insecure-signature-policy --dest-skip-tls --dest-use-http docker://localhost:5000

Throw away your local docker registry

docker stop registry
docker rm registry

Test Configuration:

Firmware version:
Hardware:
Toolchain:
SDK:

Checklist:

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
Any dependent changes have been merged and published in downstream modules

openshift-ci-robot · 2023-04-20T16:08:36Z