From a641574fb8c2dfeffffa5b4a5454ce265f272f04 Mon Sep 17 00:00:00 2001 From: Joe Lanford Date: Thu, 29 Jul 2021 10:09:22 -0400 Subject: [PATCH 01/13] wip: initial docs for declarative config Signed-off-by: Joe Lanford --- .../en/docs/Reference/declarative-config.md | 341 ++++++++++++++++++ 1 file changed, 341 insertions(+) create mode 100644 content/en/docs/Reference/declarative-config.md diff --git a/content/en/docs/Reference/declarative-config.md b/content/en/docs/Reference/declarative-config.md new file mode 100644 index 00000000..62856a96 --- /dev/null +++ b/content/en/docs/Reference/declarative-config.md @@ -0,0 +1,341 @@ +--- +title: "Declarative Config" +linkTitle: "Declarative Config" +weight: 4 +date: 2021-07-29 +--- + +Declarative Config (DC) is the latest iteration of OLM's index format. It is a fully plaintext-based (JSON or YAML) evolution of the previous sqlite database format that is fully backwards compatible. + +## Motivation +The primary motivations for this new format are to enable index editting, composability, and extensibility. + +### Editting + +With DC, users interacting with the contents of an index are able to make direct changes to the index format and verify that their changes are valid. + +Because the index is now stored in plaintext rather than an sqlite database, index maintainers can update an index without the use of a custom index-aware tool, like `opm`. + +This editability opens the door for new features and extensions that were otherwise difficult to implement and maintain in a single tool. For example: +- Promoting an existing bundle to a new channel +- Changing the default channel of a package +- Adding, updating, or removing upgrade edges + +### Composability + +Declarative Configs are stored in an arbitrary directory hierarchy, which enables index composition. If I have two separate DC directories, `indexA` and `indexB`, I can make a new combined index by making a new directory `indexC` and copying `indexA` and `indexB` into it. + +This composability enables decentralized indexes. The format permits operator authors to maintain operator-specific indexes and index maintainers to trivially build an index composed of individual operator indexes. + +One of the major benefits is that those who are most familiar with an operator, its dependencies, and its upgrade compatibility (i.e. the operator authors) are able to maintain their own operator-specific index and have direct control over its contents. DC moves the task of building and maintaining indexes more towards operator authors, thus giving composite index maintainers more time to build value around their compositions. + +Another benefit is that the format enables composite index maintainers to build their index without the knowledge or coordination of the maintainers of the sub-indexes. Indexes like this can be composed by combining multiple other indexes or by extracting only necessary subsets of one index, or a combination of both of these. + +### Extensibility + +Another motivation is to enable more extensibility around indexes. DC is a low-level representation of an index. While it can be maintained directly in its low-level form, we expect many index maintainers to build interesting extensions on top that can be used by their own custom tooling to make all sorts of mutations. For example, one could imagine a tool that translates a high-level API like (mode=semver) down to the low-level DC format for upgrade edges. Or perhaps an index maintainer needs to customize all of the bundle metadata by adding a new property to bundles that meet a certain criteria. + +The OLM developer community will be making use of this extensibility to build more official tooling on top of the low-level APIs, but the major benefit is that index maintainers have this capability as well. + +## Specification + +### Structure + +Declarative config can be stored and loaded from directory-based filesystems. + +`opm` loads declarative config by walking the root directory and recursing into subdirectories. It attempts to load every file it finds as declarative config and fails if any errors occur. + +Non-DC files can be ignored using `.indexignore` files, which behave identically to `.gitignore` files. That is, they have the same rules for [patterns](https://git-scm.com/docs/gitignore#_pattern_format) and precedence. + +> #### Example `.gitignore` file +> ```gitignore +> # Ignore everything except non-object .json and .yaml files +> **/* +> !*.json +> !*.yaml +> **/objects/*.json +> **/objects/*.yaml +> ``` + + +Index maintainers have flexibility to chose their desired layout, but the OLM team recommends storing each package's DC blobs in separate sub-directories. Each individual DC file can be either JSON or YAML -- it is not necessary for every file in an index to use the same format. + +This layout has the property that each sub-directoriy in the directory hierarchy is a self-contained index, which makes index composition, discovery, and navigation as simple as trivial filesystem operations. + +> #### Basic recommended structure +> ``` +> index +> ├── pkgA +> │ └── index.yaml +> ├── pkgB +> │ ├── .indexignore +> │ ├── index.yaml +> │ └── objects +> │ └── pkgB.v0.1.0.clusterserviceversion.yaml +> └── pkgC +> └── index.json +> ``` + +This `index` could also be trivially included in a parent index by somply copying it into the parent index's root directory. + +### Schema + +At its core, declarative config is a simple format that can be extended with arbitrary schemas. The format that all DC blobs must adhere to is the `Meta` schema. The below [cue][cuelang-spec] `_Meta` schema defines all DC blobs. + +> **NOTE**: No cue schemas listed in this specification should be considered exhaustive. The `opm validate` command has additional validations that are difficult/impossible to express concisely in cue. + +```cue +_Meta: { + // schema is required and must be a non-empty string + schema: string & !="" + + // package is optional, but if it's defined, it must be a non-empty string + package?: string & !="" + + // properties is optional, but if it's defined, it must be a list of 0 or more properties + properties?: [... #Property] +} + +#Property: { + // type is required + type: string & !="" + + // value is required, and it must not be null + value: !=null +} +``` + +### OLM-defined schemas + +An OLM index currently uses two schemas: `olm.package` and `olm.bundle`, which correspond to OLM's existing package and bundle concepts. + +Each operator package in an index requires exactly one `olm.package` blob and one or more `olm.bundle` blobs. + +> **NOTE**: All `olm.*` schemas are reserved for OLM-defined schemas. Custom schemas must use a unique prefix (e.g. a domain that you own). + +#### `olm.package` + +An `olm.package` defines package-level metadata for an operator. This includes its name, description, default channel and icon. + +The `olm.package` [cue][cuelang-spec] schema is: +```cue +#Package: { + schema: "olm.package" + + // Package name + name: string & !="" + + // A description of the package + description?: string + + // The package's default channel + defaultChannel: string & !="" + + // An optional icon + icon?: { + base64data: string + mediatype: string + } +} +``` + +#### `olm.bundle` + + + +The `olm.bundle` cue schema is: +```cue +#Bundle: { + schema: "olm.bundle" + package: string & !="" + name: string & !="" + image: string & !="" + properties: [...#Property] + relatedImages?: [...#RelatedImage] +} + +#Property: { + // type is required + type: string & !="" + + // value is required, and it must not be null + value: !=null +} + +#RelatedImage: { + // image is the image reference + image: string & !="" + + // name is an optional descriptive name for an image that + // helps identify its purpose in the context of the bundle + name?: string & !="" +} +``` + +### Properties + +Properties are arbitrary pieces of metadata that can be attached to DC schemas. The type field is a string that effectively specifies the semantic and syntactic meaning of the value field. The value can be any arbitrary JSON/YAML. + + +OLM defines a handful of property types, again using the reserved `olm.*` prefix. + +#### `olm.package` + +An `olm.package` property defines the package name and version. This is a required property on bundles, and there must be exactly one of these properties. The `packageName` must match the bundle's first-class `package` field, and the `version` must be a valid [semantic version][semver] + +The `olm.package` property [cue][cuelang-spec] schema is: +```cue +#PropertyPackage: { + type: "olm.package" + value: { + packageName: string & !="" + version: string & !="" + } +} +``` + +#### `olm.gvk` + +An `olm.gvk` property defines the group, version, and kind (GVK) of a Kubernetes API that is provided by this bundle. This property is used by OLM to resolve a bundle with this property as a dependency for other bundles that list the same GVK as a required API. The GVK must adhere to Kubernetes GVK validations. + +The `olm.gvk` property [cue][cuelang-spec] schema is: +```cue +#PropertyGVK: { + type: "olm.gvk" + value: { + group: string & !="" + version: string & !="" + kind: string & !="" + } +} +``` + +#### `olm.channel` + +An `olm.channel` property defines a channel that a bundle is in, and optionally, the name of another bundle that it replaces in that channel. + +A bundle can include multiple `olm.channel` properties, but it is invalid to define multiple `olm.channel` properties for the same channel name. + +Lastly, it is valid for an `olm.channel`'s replaces value to reference another bundle that cannot be found in this index (or even another index) as long as other channel invariants still hold (e.g. a channel cannot have multiple heads). + +The `olm.channel` property [cue][cuelang-spec] schema is: +```cue +#PropertyChannel: { + type: "olm.channel" + value: { + name: string & !="" + replaces?: string & !="" + } +} +``` + +#### `olm.skips` + +An `olm.skips` property defines another bundle that this bundle skips. This property applies to all channels. + +Any number of skips properties can be set on a bundle. + +The `olm.skips` property [cue][cuelang-spec] schema is: +```cue +#PropertySkips: { + type: "olm.skips" + value: string & !="" +} +``` + +#### `olm.skipRange` + +An `olm.skipRange` property defines a [range of semver versions][semver-range] of other bundles that this bundle skips. This property applies to all channels. + +It is invalid to include multiple `olm.skipRange` properties on a bundle. + +The `olm.skipRange` property [cue][cuelang-spec] schema is: +```cue +#PropertySkipRange: { + type: "olm.skipRange" + value: string & !="" +} +``` + +#### `olm.package.required` + +An `olm.package.required` property defines the package name and version range of another package that this bundle requires. For every required package property a bundle lists, OLM will ensure there is an operator installed on the cluster for the listed package and in the required version range. The `versionRange` field must be a valid [semver range][semver-range]. + +The `olm.package.required` property [cue][cuelang-spec] schema is: +```cue +#PropertyPackageRequired: { + type: "olm.package.required" + value: { + packageName: string & !="" + versionRange: string & !="" + } +} +``` + + +#### `olm.gvk.required` + +An `olm.gvk.required` property defines the group, version, and kind (GVK) of a Kubernetes API that this bundle requires. For every required GVK property a bundle lists, OLM will ensure there is an operator installed on the cluster that provides it. The GVK must adhere to Kubernetes GVK validations. + +The `olm.gvk.required` property [cue][cuelang-spec] schema is: +```cue +#PropertyGVKRequired: { + type: "olm.gvk.required" + value: { + group: string & !="" + version: string & !="" + kind: string & !="" + } +} +``` + +#### `olm.bundle.object` (alpha) + +`olm.bundle.object` properties are used to inline (or reference) a bundle's manifests directly in the index. + +> **NOTE**: Core OLM does not require `olm.bundle.object` properties to be included on bundles. However, the OLM Package Server (used by tooling such as the kubectl operator plugin and the OpenShift console) does require these properties to be able to serve metadata about the packages in an index. +> +> This property is in _alpha_ because it will likely be rendered obsolete when updates can be made to the OLM Package Server to no longer require manifests in the index. + +A bundle object property can contain inlined data using the `value.data` field, which must the base64-encoded string of that manifest + +Alternately, a bundle object property can be a reference to a file relative to the location of file in which the bundle is declared. Any referenced files must be within the declarative config root. + +The `olm.bundle.object` property [cue][cuelang-spec] schema is: +```cue + +#PropertyBundleObject: { + type: "olm.bundle.object" + value: #propertyBundleObjectRef | #propertyBundleObjectData +} + +#propertyBundleObjectRef: { + ref: string & !="" +} + +#propertyBundleObjectData: { + data: string & !="" +} +``` + +[cuelang-spec]: https://cuelang.org/docs/references/spec/ +[semver]: https://semver.org/spec/v2.0.0.html +[semver-range]: https://github.com/blang/semver/blob/master/README.md#ranges + +## CLI + +### `opm init` +### `opm render` +### `opm validate` +### `opm serve` +### `opm alpha diff` +### `opm alpha generate dockerfile` + +## Workflows + +### Operator authors & package maintainers +### Index maintainers +### Cluster administrators + From 86535e78342ec02bbb3a4900cc50d8e8399761c9 Mon Sep 17 00:00:00 2001 From: Joe Lanford Date: Wed, 11 Aug 2021 23:11:24 -0400 Subject: [PATCH 02/13] apply suggestions; add CLI and Workflows sections Signed-off-by: Joe Lanford --- .../en/docs/Reference/declarative-config.md | 303 +++++++++++++++++- 1 file changed, 299 insertions(+), 4 deletions(-) diff --git a/content/en/docs/Reference/declarative-config.md b/content/en/docs/Reference/declarative-config.md index 62856a96..e5811566 100644 --- a/content/en/docs/Reference/declarative-config.md +++ b/content/en/docs/Reference/declarative-config.md @@ -8,13 +8,13 @@ date: 2021-07-29 Declarative Config (DC) is the latest iteration of OLM's index format. It is a fully plaintext-based (JSON or YAML) evolution of the previous sqlite database format that is fully backwards compatible. ## Motivation -The primary motivations for this new format are to enable index editting, composability, and extensibility. +The primary motivations for this new format are to enable index editing, composability, and extensibility. ### Editting With DC, users interacting with the contents of an index are able to make direct changes to the index format and verify that their changes are valid. -Because the index is now stored in plaintext rather than an sqlite database, index maintainers can update an index without the use of a custom index-aware tool, like `opm`. +Because the index is now stored in plaintext rather than a sqlite database, index maintainers can update an index without the use of a custom index-aware tool, like `opm`. This editability opens the door for new features and extensions that were otherwise difficult to implement and maintain in a single tool. For example: - Promoting an existing bundle to a new channel @@ -27,6 +27,8 @@ Declarative Configs are stored in an arbitrary directory hierarchy, which enable This composability enables decentralized indexes. The format permits operator authors to maintain operator-specific indexes and index maintainers to trivially build an index composed of individual operator indexes. +> NOTE: Duplicate packages and duplicate bundles within a package are not permitted. The `opm validate` command will return an error if any duplicates are found. + One of the major benefits is that those who are most familiar with an operator, its dependencies, and its upgrade compatibility (i.e. the operator authors) are able to maintain their own operator-specific index and have direct control over its contents. DC moves the task of building and maintaining indexes more towards operator authors, thus giving composite index maintainers more time to build value around their compositions. Another benefit is that the format enables composite index maintainers to build their index without the knowledge or coordination of the maintainers of the sub-indexes. Indexes like this can be composed by combining multiple other indexes or by extracting only necessary subsets of one index, or a combination of both of these. @@ -76,7 +78,7 @@ This layout has the property that each sub-directoriy in the directory hierarchy > └── index.json > ``` -This `index` could also be trivially included in a parent index by somply copying it into the parent index's root directory. +This `index` could also be trivially included in a parent index by simply copying it into the parent index's root directory. ### Schema @@ -327,15 +329,308 @@ The `olm.bundle.object` property [cue][cuelang-spec] schema is: ## CLI ### `opm init` + +``` +Generate an olm.package declarative config blob + +Usage: + opm init [flags] + +Flags: + -c, --default-channel string The channel that subscriptions will default to if unspecified + -d, --description string Path to the operator's README.md (or other documentation) + -h, --help help for init + -i, --icon string Path to package's icon + -o, --output string Output format (json|yaml) (default "json") + +Global Flags: + --skip-tls skip TLS certificate verification for container image registries while pulling bundles or index +``` + ### `opm render` + +``` +Generate a declarative config blob from the provided index images, bundle images, and sqlite database files + +Usage: + opm render [index-image | bundle-image | sqlite-file]... [flags] + +Flags: + -h, --help help for render + -o, --output string Output format (json|yaml) (default "json") + +Global Flags: + --skip-tls skip TLS certificate verification for container image registries while pulling bundles or index +``` + ### `opm validate` + +``` +Validate the declarative config JSON file(s) in a given directory + +Usage: + opm validate [flags] + +Flags: + -h, --help help for validate + +Global Flags: + --skip-tls skip TLS certificate verification for container image registries while pulling bundles or index +``` + ### `opm serve` + +``` +serve declarative configs via grpc + +Usage: + opm serve [flags] + +Flags: + --debug enable debug logging + -h, --help help for serve + -p, --port string port number to serve on (default "50051") + -t, --termination-log string path to a container termination log file (default "/dev/termination-log") + +Global Flags: + --skip-tls skip TLS certificate verification for container image registries while pulling bundles or index +``` + ### `opm alpha diff` + +``` +Diff a set of old and new catalog references ("refs") to produce a declarative config containing only packages channels, and versions not present in the old set, and versions that differ between the old and new sets. This is known as "latest" mode. These references are passed through 'opm render' to produce a single declarative config. + + This command has special behavior when old-refs are omitted, called "heads-only" mode: instead of the output being that of 'opm render refs...' (which would be the case given the preceding behavior description), only the channel heads of all channels in all packages are included in the output, and dependencies. Dependencies are assumed to be provided by either an old ref, in which case they are not included in the diff, or a new ref, in which case they are included. Dependencies provided by some catalog unknown to 'opm alpha diff' will not cause the command to error, but an error will occur if that catalog is not serving these dependencies at runtime. + +Usage: + opm alpha diff [old-refs]... new-refs... [flags] + +Examples: + # Diff a catalog at some old state and latest state into a declarative config index. + mkdir -p catalog-index + opm alpha diff registry.org/my-catalog:abc123 registry.org/my-catalog:def456 -o yaml > ./my-catalog-index/index.yaml + + # Build and push this index into an index image. + opm alpha generate dockerfile ./my-catalog-index + docker build -t registry.org/my-catalog:latest-abc123-def456 -f index.Dockerfile . + docker push registry.org/my-catalog:latest-abc123-def456 + + # Create a new catalog from the heads of an existing catalog, then build and push the image like above. + opm alpha diff registry.org/my-catalog:def456 -o yaml > my-catalog-index/index.yaml + docker build -t registry.org/my-catalog:headsonly-def456 -f index.Dockerfile . + docker push registry.org/my-catalog:headsonly-def456 + +Flags: + --ca-file string the root Certificates to use with this command + --debug enable debug logging + -h, --help help for diff + -o, --output string Output format (json|yaml) (default "yaml") + +Global Flags: + --skip-tls skip TLS certificate verification for container image registries while pulling bundles or index +``` + ### `opm alpha generate dockerfile` +``` +Generate a Dockerfile for a declarative config index. + +This command creates a Dockerfile in the same directory as the +(named .Dockerfile) that can be used to build the index. If a +Dockerfile with the same name already exists, this command will fail. + +When specifying extra labels, note that if duplicate keys exist, only the last +value of each duplicate key will be added to the generated Dockerfile. + +Usage: + opm alpha generate dockerfile [flags] + +Flags: + -i, --binary-image string Image in which to build catalog. (default "quay.io/operator-framework/upstream-opm-builder") + -l, --extra-labels strings Extra labels to include in the generated Dockerfile. Labels should be of the form 'key=value'. + -h, --help help for dockerfile + +Global Flags: + --skip-tls skip TLS certificate verification for container image registries while pulling bundles or index +``` + ## Workflows ### Operator authors & package maintainers -### Index maintainers + +Declarative config moves ownership and maintenance of the index into the hands of individual operator authors and package maintainers, giving them much more control over the contents of their index. +They no longer have to rely on the specific APIs available via an `opm` command that modifies an opaque database. +Instead, they can use whatever tools fit their workflows to create and manipulate their declarative configs. + +The general workflow for operator authors is: +1. Initialize Package (once) + ``` + mkdir index + opm init my-package --default-channel=alpha -o yaml > index/index.yaml + ``` +2. Generate dockerfile (once) + ``` + opm alpha generate dockerfile index + ``` +3. Add bundle to index (as new bundles are released) + ``` + opm render my-org/my-bundle: >> index/index.yaml + ./idempotent-post-process.sh + docker build -t my-org/my-index: . -f index.Dockerfile + docker push my-org/my-index: + ``` +4. Custom edits to the `index/index.yaml` only as necessary to resolve issues in index metadata of already released bundles. + +#### A few notes + +**In step 3**, `./idempotent-post-process.sh` could be a collection of post-processing steps that, for example: + - Ensures the added bundle is correctly added into the existing upgrade graph(s) of your package channels. + - Ensures channel heads have `olm.bundle.object` properties. + - Runs `opm validate` to verify the index can be served. + - Produces a visualization of the upgrade graphs for each channel. + +Operator authors can use any tooling that works for their process and produces an index that validates with `opm validate`. Some examples of post-processing tools include: + - `jq` or `yq` for arbitrary JSON or YAML manipulations. + - `declcfg` for similar functionality provided by `opm index add`. + - `vim` -- hand-editing DC is completely acceptable if that fits your workflow + +**For step 4**, OLM's general advice is that bundle images and their metadata should be treated as immutable. +If a broken bundle has been pushed to an index, you must assume that at least one of your users has upgraded to that bundle. +Based on that assumption, you must release another bundle with an upgrade edge from the broken bundle to ensure users with the broken bundle installed receive an upgrade. +OLM will not upgrade an installed bundle if the contents of that bundle are updated in the index. + +However, there are some cases where a change in the index metadata is preferred. For example: +- Channel promotion - if you already released a bundle and later decide that you'd like to add it to another channel, simply add an `olm.channel` property to the `olm.bundle` +- New upgrade edges - if you release a new 1.2.z (e.g. 1.2.4), but 1.3.0 is already released, you can update the index metadata for 1.3.0 to skip 1.2.4. + +OLM highly recommends storing index metadata in source control and treating the source-controlled metadata as the source of truth. Updates to index images should: +1. Update the source-controlled index directory with a new commit. +2. Build and push the index image. OLM suggests using a consistent tagging taxonomy (e.g. `:latest` or `:` so that users can receive updates to an index as they become available. + +### Catalog maintainers + +With declarative config, catalog maintainers can focus on operator curation and compatibility. +Since operator authors have already produced operator-specific indexes for their operators, catalog +maintainers can build their catalog simply by rendering each operator index into a subdirectory of the +catalog's root index directory. + +#### Example + +There are many possible ways to build a catalog, but an extremely simple approach would be to: + +1. Maintain a single configuration file containing image references for each operator in the catalog + ```yaml + name: community-operators + repo: quay.io/community-operators/catalog + tag: latest + references: + - name: etcd-operator + image: quay.io/etcd-operator/index@sha256:5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03 + - name: prometheus-operator + image: quay.io/prometheus-operator/index@sha256:e258d248fda94c63753607f7c4494ee0fcbe92f1a76bfdac795c9d84101eb317 + ``` + +2. Run a simple script that parses this file and creates a new catalog from its references + ```bash + name=$(yq eval '.name' catalog.yaml) + mkdir "$name" + yq eval '.name + "/" + .references[].name' catalog.yaml | xargs mkdir + for l in $(yq e '.name as $catalog | .references[] | .image + "|" + $catalog + "/" + .name + "/index.yaml"' catalog.yaml); do + image=$(echo $l | cut -d'|' -f1) + file=$(echo $l | cut -d'|' -f2) + opm render "$image" > "$file" + done + opm alpha generate dockerfile "$name" + indexImage=$(yq eval '.repo + ":" + .tag' catalog.yaml) + docker build -t "$indexImage" -f "$name.Dockerfile" . + docker push "$indexImage" + ``` + ### Cluster administrators +For cluster administrators, declarative config-based indexes are largely the same as sqlite-based indexes. +Both index types serve the exact same GRPC interface required by OLM's on-cluster components. + +One benefit of declarative config is that it is possible to build an index entirely with Kubernetes APIs: +1. Populate a `ConfigMap` with a declarative config index file: + + ```sh + cat < Date: Mon, 16 Aug 2021 10:54:17 -0400 Subject: [PATCH 03/13] address feedback from PR Signed-off-by: Joe Lanford --- .../en/docs/Reference/declarative-config.md | 94 ++----------------- 1 file changed, 9 insertions(+), 85 deletions(-) diff --git a/content/en/docs/Reference/declarative-config.md b/content/en/docs/Reference/declarative-config.md index e5811566..7723b402 100644 --- a/content/en/docs/Reference/declarative-config.md +++ b/content/en/docs/Reference/declarative-config.md @@ -10,7 +10,7 @@ Declarative Config (DC) is the latest iteration of OLM's index format. It is a f ## Motivation The primary motivations for this new format are to enable index editing, composability, and extensibility. -### Editting +### Editing With DC, users interacting with the contents of an index are able to make direct changes to the index format and verify that their changes are valid. @@ -381,7 +381,11 @@ Global Flags: ### `opm serve` ``` -serve declarative configs via grpc +This command serves declarative configs via a GRPC server. + +NOTE: The declarative config directory is loaded by the serve command at +startup. Changes made to the declarative config after the this command starts +will not be reflected in the served content. Usage: opm serve [flags] @@ -498,7 +502,7 @@ Operator authors can use any tooling that works for their process and produces a **For step 4**, OLM's general advice is that bundle images and their metadata should be treated as immutable. If a broken bundle has been pushed to an index, you must assume that at least one of your users has upgraded to that bundle. Based on that assumption, you must release another bundle with an upgrade edge from the broken bundle to ensure users with the broken bundle installed receive an upgrade. -OLM will not upgrade an installed bundle if the contents of that bundle are updated in the index. +OLM will not reinstall an installed bundle if the contents of that bundle are updated in the index. However, there are some cases where a change in the index metadata is preferred. For example: - Channel promotion - if you already released a bundle and later decide that you'd like to add it to another channel, simply add an `olm.channel` property to the `olm.bundle` @@ -552,85 +556,5 @@ There are many possible ways to build a catalog, but an extremely simple approac For cluster administrators, declarative config-based indexes are largely the same as sqlite-based indexes. Both index types serve the exact same GRPC interface required by OLM's on-cluster components. -One benefit of declarative config is that it is possible to build an index entirely with Kubernetes APIs: -1. Populate a `ConfigMap` with a declarative config index file: - - ```sh - cat < Date: Thu, 19 Aug 2021 15:55:02 -0400 Subject: [PATCH 04/13] rename to file-based-catalogs.md Signed-off-by: Joe Lanford --- ...ative-config.md => file-based-catalogs.md} | 41 +++++++++---------- 1 file changed, 20 insertions(+), 21 deletions(-) rename content/en/docs/Reference/{declarative-config.md => file-based-catalogs.md} (86%) diff --git a/content/en/docs/Reference/declarative-config.md b/content/en/docs/Reference/file-based-catalogs.md similarity index 86% rename from content/en/docs/Reference/declarative-config.md rename to content/en/docs/Reference/file-based-catalogs.md index 7723b402..779239ad 100644 --- a/content/en/docs/Reference/declarative-config.md +++ b/content/en/docs/Reference/file-based-catalogs.md @@ -1,18 +1,18 @@ --- -title: "Declarative Config" -linkTitle: "Declarative Config" +title: "File-based Catalogs" +linkTitle: "File-based Catalogs" weight: 4 date: 2021-07-29 --- -Declarative Config (DC) is the latest iteration of OLM's index format. It is a fully plaintext-based (JSON or YAML) evolution of the previous sqlite database format that is fully backwards compatible. +File-based catalogs are the latest iteration of OLM's index format. It is a fully plaintext-based (JSON or YAML) evolution of the previous sqlite database format that is fully backwards compatible. ## Motivation The primary motivations for this new format are to enable index editing, composability, and extensibility. ### Editing -With DC, users interacting with the contents of an index are able to make direct changes to the index format and verify that their changes are valid. +With file-based catalogs, users interacting with the contents of an index are able to make direct changes to the index format and verify that their changes are valid. Because the index is now stored in plaintext rather than a sqlite database, index maintainers can update an index without the use of a custom index-aware tool, like `opm`. @@ -23,19 +23,19 @@ This editability opens the door for new features and extensions that were otherw ### Composability -Declarative Configs are stored in an arbitrary directory hierarchy, which enables index composition. If I have two separate DC directories, `indexA` and `indexB`, I can make a new combined index by making a new directory `indexC` and copying `indexA` and `indexB` into it. +File-based catalogs are stored in an arbitrary directory hierarchy, which enables index composition. If I have two separate file-based catalog directories, `indexA` and `indexB`, I can make a new combined index by making a new directory `indexC` and copying `indexA` and `indexB` into it. This composability enables decentralized indexes. The format permits operator authors to maintain operator-specific indexes and index maintainers to trivially build an index composed of individual operator indexes. > NOTE: Duplicate packages and duplicate bundles within a package are not permitted. The `opm validate` command will return an error if any duplicates are found. -One of the major benefits is that those who are most familiar with an operator, its dependencies, and its upgrade compatibility (i.e. the operator authors) are able to maintain their own operator-specific index and have direct control over its contents. DC moves the task of building and maintaining indexes more towards operator authors, thus giving composite index maintainers more time to build value around their compositions. +One of the major benefits is that those who are most familiar with an operator, its dependencies, and its upgrade compatibility (i.e. the operator authors) are able to maintain their own operator-specific index and have direct control over its contents. File-based catalogs move the task of building and maintaining indexes more towards operator authors, thus giving composite index maintainers more time to build value around their compositions. Another benefit is that the format enables composite index maintainers to build their index without the knowledge or coordination of the maintainers of the sub-indexes. Indexes like this can be composed by combining multiple other indexes or by extracting only necessary subsets of one index, or a combination of both of these. ### Extensibility -Another motivation is to enable more extensibility around indexes. DC is a low-level representation of an index. While it can be maintained directly in its low-level form, we expect many index maintainers to build interesting extensions on top that can be used by their own custom tooling to make all sorts of mutations. For example, one could imagine a tool that translates a high-level API like (mode=semver) down to the low-level DC format for upgrade edges. Or perhaps an index maintainer needs to customize all of the bundle metadata by adding a new property to bundles that meet a certain criteria. +Another motivation is to enable more extensibility around indexes. The file-based catalog spec is a low-level representation of an index. While it can be maintained directly in its low-level form, we expect many index maintainers to build interesting extensions on top that can be used by their own custom tooling to make all sorts of mutations. For example, one could imagine a tool that translates a high-level API like (mode=semver) down to the low-level file-based catalog format for upgrade edges. Or perhaps an index maintainer needs to customize all of the bundle metadata by adding a new property to bundles that meet a certain criteria. The OLM developer community will be making use of this extensibility to build more official tooling on top of the low-level APIs, but the major benefit is that index maintainers have this capability as well. @@ -43,11 +43,11 @@ The OLM developer community will be making use of this extensibility to build mo ### Structure -Declarative config can be stored and loaded from directory-based filesystems. +File-based catalogs can be stored and loaded from directory-based filesystems. -`opm` loads declarative config by walking the root directory and recursing into subdirectories. It attempts to load every file it finds as declarative config and fails if any errors occur. +`opm` loads the catalog by walking the root directory and recursing into subdirectories. It attempts to load every file it finds and fails if any errors occur. -Non-DC files can be ignored using `.indexignore` files, which behave identically to `.gitignore` files. That is, they have the same rules for [patterns](https://git-scm.com/docs/gitignore#_pattern_format) and precedence. +Non-catalog files can be ignored using `.indexignore` files, which behave identically to `.gitignore` files. That is, they have the same rules for [patterns](https://git-scm.com/docs/gitignore#_pattern_format) and precedence. > #### Example `.gitignore` file > ```gitignore @@ -60,7 +60,7 @@ Non-DC files can be ignored using `.indexignore` files, which behave identically > ``` -Index maintainers have flexibility to chose their desired layout, but the OLM team recommends storing each package's DC blobs in separate sub-directories. Each individual DC file can be either JSON or YAML -- it is not necessary for every file in an index to use the same format. +Index maintainers have flexibility to chose their desired layout, but the OLM team recommends storing each package's file-based catalog blobs in separate sub-directories. Each individual file can be either JSON or YAML -- it is not necessary for every file in an index to use the same format. This layout has the property that each sub-directoriy in the directory hierarchy is a self-contained index, which makes index composition, discovery, and navigation as simple as trivial filesystem operations. @@ -82,7 +82,7 @@ This `index` could also be trivially included in a parent index by simply copyin ### Schema -At its core, declarative config is a simple format that can be extended with arbitrary schemas. The format that all DC blobs must adhere to is the `Meta` schema. The below [cue][cuelang-spec] `_Meta` schema defines all DC blobs. +At its core, file-based catalogs use a simple format that can be extended with arbitrary schemas. The format that all file-based catalog blobs must adhere to is the `Meta` schema. The below [cue][cuelang-spec] `_Meta` schema defines all file-based catalog blobs. > **NOTE**: No cue schemas listed in this specification should be considered exhaustive. The `opm validate` command has additional validations that are difficult/impossible to express concisely in cue. @@ -178,7 +178,7 @@ The `olm.bundle` cue schema is: ### Properties -Properties are arbitrary pieces of metadata that can be attached to DC schemas. The type field is a string that effectively specifies the semantic and syntactic meaning of the value field. The value can be any arbitrary JSON/YAML. +Properties are arbitrary pieces of metadata that can be attached to file-based catalog schemas. The type field is a string that effectively specifies the semantic and syntactic meaning of the value field. The value can be any arbitrary JSON/YAML. OLM defines a handful of property types, again using the reserved `olm.*` prefix. @@ -303,7 +303,7 @@ The `olm.gvk.required` property [cue][cuelang-spec] schema is: A bundle object property can contain inlined data using the `value.data` field, which must the base64-encoded string of that manifest -Alternately, a bundle object property can be a reference to a file relative to the location of file in which the bundle is declared. Any referenced files must be within the declarative config root. +Alternately, a bundle object property can be a reference to a file relative to the location of file in which the bundle is declared. Any referenced files must be within the catalog root. The `olm.bundle.object` property [cue][cuelang-spec] schema is: ```cue @@ -463,9 +463,9 @@ Global Flags: ### Operator authors & package maintainers -Declarative config moves ownership and maintenance of the index into the hands of individual operator authors and package maintainers, giving them much more control over the contents of their index. +File-based catalogs move ownership and maintenance of the index into the hands of individual operator authors and package maintainers, giving them much more control over the contents of their index. They no longer have to rely on the specific APIs available via an `opm` command that modifies an opaque database. -Instead, they can use whatever tools fit their workflows to create and manipulate their declarative configs. +Instead, they can use whatever tools fit their workflows to create and manipulate their catalogs. The general workflow for operator authors is: 1. Initialize Package (once) @@ -497,7 +497,7 @@ The general workflow for operator authors is: Operator authors can use any tooling that works for their process and produces an index that validates with `opm validate`. Some examples of post-processing tools include: - `jq` or `yq` for arbitrary JSON or YAML manipulations. - `declcfg` for similar functionality provided by `opm index add`. - - `vim` -- hand-editing DC is completely acceptable if that fits your workflow + - `vim` -- hand-editing file-based catalogs is completely acceptable if that fits your workflow **For step 4**, OLM's general advice is that bundle images and their metadata should be treated as immutable. If a broken bundle has been pushed to an index, you must assume that at least one of your users has upgraded to that bundle. @@ -514,7 +514,7 @@ OLM highly recommends storing index metadata in source control and treating the ### Catalog maintainers -With declarative config, catalog maintainers can focus on operator curation and compatibility. +With file-based catalogs, catalog maintainers can focus on operator curation and compatibility. Since operator authors have already produced operator-specific indexes for their operators, catalog maintainers can build their catalog simply by rendering each operator index into a subdirectory of the catalog's root index directory. @@ -553,8 +553,7 @@ There are many possible ways to build a catalog, but an extremely simple approac ### Cluster administrators -For cluster administrators, declarative config-based indexes are largely the same as sqlite-based indexes. +For cluster administrators, file-based catalogs are largely the same as sqlite-based catalogs. Both index types serve the exact same GRPC interface required by OLM's on-cluster components. -Cluster administrators are not required to make any changes to their clusters to support declarative -config-based indexes. +Cluster administrators are not required to make any changes to their clusters to support file-based catalogs. From c455d59100e4d13df6017fd5250444a4756a05f1 Mon Sep 17 00:00:00 2001 From: Joe Lanford Date: Tue, 31 Aug 2021 17:10:51 -0400 Subject: [PATCH 05/13] Update "Creating an Index" page to focus on file-based catalogs, other tweaks to FBC reference page Signed-off-by: Joe Lanford --- .../en/docs/Reference/file-based-catalogs.md | 209 ++++++++++-------- content/en/docs/Tasks/creating-an-index.md | 97 ++++++-- 2 files changed, 198 insertions(+), 108 deletions(-) diff --git a/content/en/docs/Reference/file-based-catalogs.md b/content/en/docs/Reference/file-based-catalogs.md index 779239ad..a27eb3a6 100644 --- a/content/en/docs/Reference/file-based-catalogs.md +++ b/content/en/docs/Reference/file-based-catalogs.md @@ -5,39 +5,57 @@ weight: 4 date: 2021-07-29 --- -File-based catalogs are the latest iteration of OLM's index format. It is a fully plaintext-based (JSON or YAML) evolution of the previous sqlite database format that is fully backwards compatible. +File-based catalogs are the latest iteration of OLM's index format. It is a fully plaintext-based (JSON or YAML) +evolution of the previous sqlite database format that is fully backwards compatible. -## Motivation -The primary motivations for this new format are to enable index editing, composability, and extensibility. +## Design +The primary design goal for this format is to enable index editing, composability, and extensibility. ### Editing -With file-based catalogs, users interacting with the contents of an index are able to make direct changes to the index format and verify that their changes are valid. +With file-based catalogs, users interacting with the contents of an index are able to make direct changes to the index +format and verify that their changes are valid. -Because the index is now stored in plaintext rather than a sqlite database, index maintainers can update an index without the use of a custom index-aware tool, like `opm`. +Because this format is plaintext JSON or YAML, index maintainers can easily manipulate index metadata by hand or with +widely known and supported JSON or YAML tooling (e.g. `jq`). -This editability opens the door for new features and extensions that were otherwise difficult to implement and maintain in a single tool. For example: +This editability enables features and user-defined extensions, such as: - Promoting an existing bundle to a new channel - Changing the default channel of a package -- Adding, updating, or removing upgrade edges +- Custom algorithms for adding, updating, adding removing upgrade edges. ### Composability -File-based catalogs are stored in an arbitrary directory hierarchy, which enables index composition. If I have two separate file-based catalog directories, `indexA` and `indexB`, I can make a new combined index by making a new directory `indexC` and copying `indexA` and `indexB` into it. +File-based catalogs are stored in an arbitrary directory hierarchy, which enables index composition. If I have two +separate file-based catalog directories, `indexA` and `indexB`, I can make a new combined index by making a new +directory `indexC` and copying `indexA` and `indexB` into it. -This composability enables decentralized indexes. The format permits operator authors to maintain operator-specific indexes and index maintainers to trivially build an index composed of individual operator indexes. +This composability enables decentralized indexes. The format permits operator authors to maintain operator-specific +indexes and index maintainers to trivially build an index composed of individual operator indexes. -> NOTE: Duplicate packages and duplicate bundles within a package are not permitted. The `opm validate` command will return an error if any duplicates are found. +> NOTE: Duplicate packages and duplicate bundles within a package are not permitted. The `opm validate` command will +> return an error if any duplicates are found. -One of the major benefits is that those who are most familiar with an operator, its dependencies, and its upgrade compatibility (i.e. the operator authors) are able to maintain their own operator-specific index and have direct control over its contents. File-based catalogs move the task of building and maintaining indexes more towards operator authors, thus giving composite index maintainers more time to build value around their compositions. +Since operator authors are most familiar with their operator, its dependencies, and its upgrade compatibility, they are +able to maintain their own operator-specific index and have direct control over its contents. With file-based catalogs, +operator authors own the task of building and maintaining their packages in an index. Composite index maintainers treat +packages as a black box; they own the task of curating the packages in their catalog and publishing the catalog to +users. -Another benefit is that the format enables composite index maintainers to build their index without the knowledge or coordination of the maintainers of the sub-indexes. Indexes like this can be composed by combining multiple other indexes or by extracting only necessary subsets of one index, or a combination of both of these. +File-based catalogs can be composed by combining multiple other catalogs or by extracting subsets of one catalog, or a +combination of both of these. ### Extensibility -Another motivation is to enable more extensibility around indexes. The file-based catalog spec is a low-level representation of an index. While it can be maintained directly in its low-level form, we expect many index maintainers to build interesting extensions on top that can be used by their own custom tooling to make all sorts of mutations. For example, one could imagine a tool that translates a high-level API like (mode=semver) down to the low-level file-based catalog format for upgrade edges. Or perhaps an index maintainer needs to customize all of the bundle metadata by adding a new property to bundles that meet a certain criteria. +The final design goal is to provide extensibility around indexes. The file-based catalog spec is a low-level +representation of an index. While it can be maintained directly in its low-level form, we expect many index maintainers +to build interesting extensions on top that can be used by their own custom tooling to make all sorts of mutations. For +example, one could imagine a tool that translates a high-level API like (mode=semver) down to the low-level file-based +catalog format for upgrade edges. Or perhaps an index maintainer needs to customize all of the bundle metadata by adding +a new property to bundles that meet a certain criteria. -The OLM developer community will be making use of this extensibility to build more official tooling on top of the low-level APIs, but the major benefit is that index maintainers have this capability as well. +The OLM developer community will be making use of this extensibility to build more official tooling on top of the +low-level APIs, but the major benefit is that index maintainers have this capability as well. ## Specification @@ -45,9 +63,11 @@ The OLM developer community will be making use of this extensibility to build mo File-based catalogs can be stored and loaded from directory-based filesystems. -`opm` loads the catalog by walking the root directory and recursing into subdirectories. It attempts to load every file it finds and fails if any errors occur. +`opm` loads the catalog by walking the root directory and recursing into subdirectories. It attempts to load every file +it finds and fails if any errors occur. -Non-catalog files can be ignored using `.indexignore` files, which behave identically to `.gitignore` files. That is, they have the same rules for [patterns](https://git-scm.com/docs/gitignore#_pattern_format) and precedence. +Non-catalog files can be ignored using `.indexignore` files, which behave identically to `.gitignore` files. That is, +they have the same rules for [patterns](https://git-scm.com/docs/gitignore#_pattern_format) and precedence. > #### Example `.gitignore` file > ```gitignore @@ -60,9 +80,12 @@ Non-catalog files can be ignored using `.indexignore` files, which behave identi > ``` -Index maintainers have flexibility to chose their desired layout, but the OLM team recommends storing each package's file-based catalog blobs in separate sub-directories. Each individual file can be either JSON or YAML -- it is not necessary for every file in an index to use the same format. +Index maintainers have flexibility to chose their desired layout, but the OLM team recommends storing each package's +file-based catalog blobs in separate sub-directories. Each individual file can be either JSON or YAML -- it is not +necessary for every file in an index to use the same format. -This layout has the property that each sub-directoriy in the directory hierarchy is a self-contained index, which makes index composition, discovery, and navigation as simple as trivial filesystem operations. +This layout has the property that each sub-directoriy in the directory hierarchy is a self-contained index, which makes +index composition, discovery, and navigation as simple as trivial filesystem operations. > #### Basic recommended structure > ``` @@ -78,13 +101,17 @@ This layout has the property that each sub-directoriy in the directory hierarchy > └── index.json > ``` -This `index` could also be trivially included in a parent index by simply copying it into the parent index's root directory. +This `index` could also be trivially included in a parent index by simply copying it into the parent index's root +directory. ### Schema -At its core, file-based catalogs use a simple format that can be extended with arbitrary schemas. The format that all file-based catalog blobs must adhere to is the `Meta` schema. The below [cue][cuelang-spec] `_Meta` schema defines all file-based catalog blobs. +At its core, file-based catalogs use a simple format that can be extended with arbitrary schemas. The format that all +file-based catalog blobs must adhere to is the `Meta` schema. The below [cue][cuelang-spec] `_Meta` schema defines all +file-based catalog blobs. -> **NOTE**: No cue schemas listed in this specification should be considered exhaustive. The `opm validate` command has additional validations that are difficult/impossible to express concisely in cue. +> **NOTE**: No cue schemas listed in this specification should be considered exhaustive. The `opm validate` command has +> additional validations that are difficult/impossible to express concisely in cue. ```cue _Meta: { @@ -109,15 +136,18 @@ _Meta: { ### OLM-defined schemas -An OLM index currently uses two schemas: `olm.package` and `olm.bundle`, which correspond to OLM's existing package and bundle concepts. +An OLM index currently uses two schemas: `olm.package` and `olm.bundle`, which correspond to OLM's existing package and +bundle concepts. Each operator package in an index requires exactly one `olm.package` blob and one or more `olm.bundle` blobs. -> **NOTE**: All `olm.*` schemas are reserved for OLM-defined schemas. Custom schemas must use a unique prefix (e.g. a domain that you own). +> **NOTE**: All `olm.*` schemas are reserved for OLM-defined schemas. Custom schemas must use a unique prefix (e.g. a +> domain that you own). #### `olm.package` -An `olm.package` defines package-level metadata for an operator. This includes its name, description, default channel and icon. +An `olm.package` defines package-level metadata for an operator. This includes its name, description, default channel +and icon. The `olm.package` [cue][cuelang-spec] schema is: ```cue @@ -178,14 +208,18 @@ The `olm.bundle` cue schema is: ### Properties -Properties are arbitrary pieces of metadata that can be attached to file-based catalog schemas. The type field is a string that effectively specifies the semantic and syntactic meaning of the value field. The value can be any arbitrary JSON/YAML. +Properties are arbitrary pieces of metadata that can be attached to file-based catalog schemas. The type field is a +string that effectively specifies the semantic and syntactic meaning of the value field. The value can be any arbitrary +JSON/YAML. OLM defines a handful of property types, again using the reserved `olm.*` prefix. #### `olm.package` -An `olm.package` property defines the package name and version. This is a required property on bundles, and there must be exactly one of these properties. The `packageName` must match the bundle's first-class `package` field, and the `version` must be a valid [semantic version][semver] +An `olm.package` property defines the package name and version. This is a required property on bundles, and there must +be exactly one of these properties. The `packageName` must match the bundle's first-class `package` field, and the +`version` must be a valid [semantic version][semver] The `olm.package` property [cue][cuelang-spec] schema is: ```cue @@ -200,7 +234,9 @@ The `olm.package` property [cue][cuelang-spec] schema is: #### `olm.gvk` -An `olm.gvk` property defines the group, version, and kind (GVK) of a Kubernetes API that is provided by this bundle. This property is used by OLM to resolve a bundle with this property as a dependency for other bundles that list the same GVK as a required API. The GVK must adhere to Kubernetes GVK validations. +An `olm.gvk` property defines the group, version, and kind (GVK) of a Kubernetes API that is provided by this bundle. +This property is used by OLM to resolve a bundle with this property as a dependency for other bundles that list the same +GVK as a required API. The GVK must adhere to Kubernetes GVK validations. The `olm.gvk` property [cue][cuelang-spec] schema is: ```cue @@ -216,11 +252,14 @@ The `olm.gvk` property [cue][cuelang-spec] schema is: #### `olm.channel` -An `olm.channel` property defines a channel that a bundle is in, and optionally, the name of another bundle that it replaces in that channel. +An `olm.channel` property defines a channel that a bundle is in, and optionally, the name of another bundle that it +replaces in that channel. -A bundle can include multiple `olm.channel` properties, but it is invalid to define multiple `olm.channel` properties for the same channel name. +A bundle can include multiple `olm.channel` properties, but it is invalid to define multiple `olm.channel` properties +for the same channel name. -Lastly, it is valid for an `olm.channel`'s replaces value to reference another bundle that cannot be found in this index (or even another index) as long as other channel invariants still hold (e.g. a channel cannot have multiple heads). +Lastly, it is valid for an `olm.channel`'s replaces value to reference another bundle that cannot be found in this index +(or even another index) as long as other channel invariants still hold (e.g. a channel cannot have multiple heads). The `olm.channel` property [cue][cuelang-spec] schema is: ```cue @@ -249,7 +288,8 @@ The `olm.skips` property [cue][cuelang-spec] schema is: #### `olm.skipRange` -An `olm.skipRange` property defines a [range of semver versions][semver-range] of other bundles that this bundle skips. This property applies to all channels. +An `olm.skipRange` property defines a [range of semver versions][semver-range] of other bundles that this bundle skips. +This property applies to all channels. It is invalid to include multiple `olm.skipRange` properties on a bundle. @@ -263,7 +303,10 @@ The `olm.skipRange` property [cue][cuelang-spec] schema is: #### `olm.package.required` -An `olm.package.required` property defines the package name and version range of another package that this bundle requires. For every required package property a bundle lists, OLM will ensure there is an operator installed on the cluster for the listed package and in the required version range. The `versionRange` field must be a valid [semver range][semver-range]. +An `olm.package.required` property defines the package name and version range of another package that this bundle +requires. For every required package property a bundle lists, OLM will ensure there is an operator installed on the +cluster for the listed package and in the required version range. The `versionRange` field must be a valid +[semver range][semver-range]. The `olm.package.required` property [cue][cuelang-spec] schema is: ```cue @@ -279,7 +322,9 @@ The `olm.package.required` property [cue][cuelang-spec] schema is: #### `olm.gvk.required` -An `olm.gvk.required` property defines the group, version, and kind (GVK) of a Kubernetes API that this bundle requires. For every required GVK property a bundle lists, OLM will ensure there is an operator installed on the cluster that provides it. The GVK must adhere to Kubernetes GVK validations. +An `olm.gvk.required` property defines the group, version, and kind (GVK) of a Kubernetes API that this bundle requires. +For every required GVK property a bundle lists, OLM will ensure there is an operator installed on the cluster that +provides it. The GVK must adhere to Kubernetes GVK validations. The `olm.gvk.required` property [cue][cuelang-spec] schema is: ```cue @@ -297,13 +342,18 @@ The `olm.gvk.required` property [cue][cuelang-spec] schema is: `olm.bundle.object` properties are used to inline (or reference) a bundle's manifests directly in the index. -> **NOTE**: Core OLM does not require `olm.bundle.object` properties to be included on bundles. However, the OLM Package Server (used by tooling such as the kubectl operator plugin and the OpenShift console) does require these properties to be able to serve metadata about the packages in an index. +> **NOTE**: Core OLM does not require `olm.bundle.object` properties to be included on bundles. However, the OLM Package +> Server (used by tooling such as the kubectl operator plugin and the OpenShift console) does require these properties +> to be able to serve metadata about the packages in an index. > -> This property is in _alpha_ because it will likely be rendered obsolete when updates can be made to the OLM Package Server to no longer require manifests in the index. +> This property is in _alpha_ because it will likely be rendered obsolete when updates can be made to the OLM Package +> Server to no longer require manifests in the index. -A bundle object property can contain inlined data using the `value.data` field, which must the base64-encoded string of that manifest +A bundle object property can contain inlined data using the `value.data` field, which must the base64-encoded string of +that manifest. -Alternately, a bundle object property can be a reference to a file relative to the location of file in which the bundle is declared. Any referenced files must be within the catalog root. +Alternately, a bundle object property can be a reference to a file relative to the location of file in which the bundle +is declared. Any referenced files must be within the catalog root. The `olm.bundle.object` property [cue][cuelang-spec] schema is: ```cue @@ -328,6 +378,10 @@ The `olm.bundle.object` property [cue][cuelang-spec] schema is: ## CLI + + ### `opm init` ``` @@ -459,68 +513,37 @@ Global Flags: --skip-tls skip TLS certificate verification for container image registries while pulling bundles or index ``` -## Workflows +## Guidelines -### Operator authors & package maintainers +### Immutable bundles -File-based catalogs move ownership and maintenance of the index into the hands of individual operator authors and package maintainers, giving them much more control over the contents of their index. -They no longer have to rely on the specific APIs available via an `opm` command that modifies an opaque database. -Instead, they can use whatever tools fit their workflows to create and manipulate their catalogs. - -The general workflow for operator authors is: -1. Initialize Package (once) - ``` - mkdir index - opm init my-package --default-channel=alpha -o yaml > index/index.yaml - ``` -2. Generate dockerfile (once) - ``` - opm alpha generate dockerfile index - ``` -3. Add bundle to index (as new bundles are released) - ``` - opm render my-org/my-bundle: >> index/index.yaml - ./idempotent-post-process.sh - docker build -t my-org/my-index: . -f index.Dockerfile - docker push my-org/my-index: - ``` -4. Custom edits to the `index/index.yaml` only as necessary to resolve issues in index metadata of already released bundles. - -#### A few notes - -**In step 3**, `./idempotent-post-process.sh` could be a collection of post-processing steps that, for example: - - Ensures the added bundle is correctly added into the existing upgrade graph(s) of your package channels. - - Ensures channel heads have `olm.bundle.object` properties. - - Runs `opm validate` to verify the index can be served. - - Produces a visualization of the upgrade graphs for each channel. - -Operator authors can use any tooling that works for their process and produces an index that validates with `opm validate`. Some examples of post-processing tools include: - - `jq` or `yq` for arbitrary JSON or YAML manipulations. - - `declcfg` for similar functionality provided by `opm index add`. - - `vim` -- hand-editing file-based catalogs is completely acceptable if that fits your workflow - -**For step 4**, OLM's general advice is that bundle images and their metadata should be treated as immutable. -If a broken bundle has been pushed to an index, you must assume that at least one of your users has upgraded to that bundle. -Based on that assumption, you must release another bundle with an upgrade edge from the broken bundle to ensure users with the broken bundle installed receive an upgrade. -OLM will not reinstall an installed bundle if the contents of that bundle are updated in the index. +OLM's general advice is that bundle images and their metadata should be treated as immutable. If a broken bundle has +been pushed to an index, you must assume that at least one of your users has upgraded to that bundle. Based on that +assumption, you must release another bundle with an upgrade edge from the broken bundle to ensure users with the broken +bundle installed receive an upgrade. OLM will not reinstall an installed bundle if the contents of that bundle are +updated in the index. However, there are some cases where a change in the index metadata is preferred. For example: -- Channel promotion - if you already released a bundle and later decide that you'd like to add it to another channel, simply add an `olm.channel` property to the `olm.bundle` -- New upgrade edges - if you release a new 1.2.z (e.g. 1.2.4), but 1.3.0 is already released, you can update the index metadata for 1.3.0 to skip 1.2.4. +- Channel promotion - if you already released a bundle and later decide that you'd like to add it to another channel, + simply add an `olm.channel` property to the `olm.bundle` +- New upgrade edges - if you release a new 1.2.z (e.g. 1.2.4), but 1.3.0 is already released, you can update the index + metadata for 1.3.0 to skip 1.2.4. + +### Use of source control -OLM highly recommends storing index metadata in source control and treating the source-controlled metadata as the source of truth. Updates to index images should: +OLM highly recommends storing index metadata in source control and treating the source-controlled metadata as the source +of truth. Updates to index images should: 1. Update the source-controlled index directory with a new commit. -2. Build and push the index image. OLM suggests using a consistent tagging taxonomy (e.g. `:latest` or `:` so that users can receive updates to an index as they become available. +2. Build and push the index image. OLM suggests using a consistent tagging taxonomy (e.g. `:latest` or + `:` so that users can receive updates to an index as they become available. -### Catalog maintainers +## Example: Building a composite catalog With file-based catalogs, catalog maintainers can focus on operator curation and compatibility. Since operator authors have already produced operator-specific indexes for their operators, catalog maintainers can build their catalog simply by rendering each operator index into a subdirectory of the catalog's root index directory. -#### Example - There are many possible ways to build a catalog, but an extremely simple approach would be to: 1. Maintain a single configuration file containing image references for each operator in the catalog @@ -551,9 +574,13 @@ There are many possible ways to build a catalog, but an extremely simple approac docker push "$indexImage" ``` -### Cluster administrators - -For cluster administrators, file-based catalogs are largely the same as sqlite-based catalogs. -Both index types serve the exact same GRPC interface required by OLM's on-cluster components. +## Automation -Cluster administrators are not required to make any changes to their clusters to support file-based catalogs. +Operator authors and catalog maintainers are encouraged to automate their index maintenance with CI/CD workflows. +Catalog maintainers could further improve on this by building Git-ops automation that: +1. Checks that PR authors are permitted to make the requested changes (e.g. updating their package's image reference) +2. Checks that the index updates pass `opm validate` +3. Checks that the updated bundle and/or index image reference(s) exist, the index images run successfully in a cluster, + and operators from that package can be successfully installed. +4. Automatically merges PRs that pass these checks. +5. Automatically rebuilds and republishes the index image. diff --git a/content/en/docs/Tasks/creating-an-index.md b/content/en/docs/Tasks/creating-an-index.md index f8f3aff5..8af58eb8 100644 --- a/content/en/docs/Tasks/creating-an-index.md +++ b/content/en/docs/Tasks/creating-an-index.md @@ -5,37 +5,100 @@ description: > Add/Remove a collection of bundles to/from an Index --- -## Prerequisites +## Prerequisites -- [opm](https://github.com/operator-framework/operator-registry/releases) `v1.14.0+` -- [docker](https://docs.docker.com/install/) version `17.03`+ or [podman](https://github.com/containers/libpod/blob/master/install.md) `v1.2.0+` or [buildah](https://github.com/containers/buildah/blob/master/install.md) `v1.7+`. +- [opm](https://github.com/operator-framework/operator-registry/releases) `v1.18.0+` +## Creating an Index -# Creating an Index +`OLM`'s `CatalogSource` [CRD][catalogsource-crd] defines a reference to a catalog of operators that are available to +install onto a cluster. To make your operator bundle available, you can add the bundle to a container image which the +`CatalogSource` points to. This image contains a record of bundle images that OLM can pull and extract the manifests +from in order to install an operator. -`OLM`'s `CatalogSource` [CRD][catalogsource-crd] define a reference to a catalog of operators that are available to install onto a cluster. To make your operator bundle available, you can add the bundle to a container image which the `CatalogSource` points to. This image contains a record of bundle images that OLM can pull and extract the manifests from in order to install an operator. +OLM uses a plaintext [file-based catalog][file-based-catalog-spec] format (JSON or YAML) to store these records in an index, and `opm` has tooling +that helps initialize an index, render new records into it, and then validate that the index is valid. Let's walk +through a simple example. ->Note: The container image also contains information that represents the upgrade graphs between different operator versions, an operator's dependencies etc graphically. To learn more about the upgrade graph of an operator, checkout the [creating an upgrade graph doc][upgrade-graph-doc] +First, we need to initialize our index, so we'll make a directory for it, generate a Dockerfile that can build an index +image, and then populate our index with our package definition. -So, to make your operator available to OLM, you can generate an index image via opm with your bundle reference included: +### Initialize the index ```sh -$ opm index add --bundles quay.io/my-container-registry-namespace/my-manifest-bundle:0.0.1 --tag quay.io/my-container-registry-namespace/my-index:1.0.0 -$ podman push quay.io/my-container-registry-namespace/my-index:1.0.0 +$ mkdir example-operator-index +$ opm alpha generate dockerfile example-operator-index +$ opm init example-operator \ + --default-channel=preview \ + --description=./README.md \ + --icon=./example-operator.svg \ + -output yaml > example-operator-index/index.yaml ``` -The resulting image is referred to as an `Index`. Now that image is available for clusters to use and reference with `CatalogSources` on their cluster. +Let's validate our index to see if we're ready to ship! +```sh +$ opm validate example-operator-index +FATA[0000] invalid index: +└── invalid package "example-operator": + └── invalid channel "preview": + └── channel must contain at least one bundle +``` + +Alright, so it's not valid. It looks like we need to add a bundle, so let's do +that next... -`Index` images are additive, so you can add a new version of your operator bundle when you publish a new version of your operator: +### Add a bundle to the index -```bash -$ opm index add --bundles quay.io/my-container-registry-namespace/my-manifest-bundle:0.0.2 --from-index quay.io/my-container-registry-namespace/my-index:1.0.0 --tag quay.io/my-container-registry-namespace/my-index:1.0.1 +```sh +$ opm render quay.io/example-inc/example-operator-bundle:v0.1.0 \ + --output=yaml > example-operator-index/index.yaml ``` -### Other operations on an Index using `opm` +Let's validate again: +```sh +$ opm validate example-operator-index +``` + +Success! There were no errors and we got a `0` error code. + +In the general case, adding a bundle involves three discreet steps: +1. Render the bundle into the index using `opm render ` +2. Add the bundle into desired channels and update the channels' upgrade edges + to stitch the bundle into the correct place. +3. Validate the resulting index. + +> NOTE: Index metadata should be stored in a version control system (e.g. `git`) and index images should be rebuilt from source +whenever updates are made to ensure that all index changes are auditable. + +**Step 1** is just a simple `opm render` command. + +**Step 2** has no defined standards other than that the result must pass validation in step 3. Some operator authors may +decide to hand edit channels and upgrade edges. Others may decide to implement automation (e.g. to idempotently +build semver-based channels and upgrade graphs based solely on the versions of the operators in the package). There is +no right or wrong answer for implementing this step as long as `opm validate` is successful. + +There are some guidelines to keep in mind though: +1. Once a bundle is present in an index, you should assume that one of your users has installed it. With that in mind, + you should take care to avoid stranding users that have that version installed. Put another way, make sure that + all previously published bundles in an index have a path to the current/new channel head. +2. Keep the semantics of the upgrade edges you use in mind. `opm validate` is not able to tell you if you have a sane + upgrade graph. To learn more about the upgrade graph of an operator, checkout the + [creating an upgrade graph doc][upgrade-graph-doc] + +### Build and push the index image + +Lastly, we can build and push our index: + +```sh +$ docker build . \ + -f example-operator-index.Dockerfile \ + -t quay.io/example-inc/example-operator-index:latest +$ docker push quay.io/example-inc/example-operator-index:latest +``` -The `opm index` command contains additional sub-commands that can be used to perform different operations like remove an operator from an index, prune an index of all but specified operators etc. Please checkout the documentation under `opm index -h` for more information. - +The resulting image is referred to as an `Index`. Now that image is available for clusters to use and reference with +`CatalogSources` on their cluster. [catalogsource-crd]: /docs/concepts/crds/catalogsource -[upgrade-graph-doc]: /docs/concepts/olm-architecture/operator-catalog/creating-an-update-graph \ No newline at end of file +[file-based-catalog-spec]: /docs/reference/file-based-catalogs +[upgrade-graph-doc]: /docs/concepts/olm-architecture/operator-catalog/creating-an-update-graph From f9804d509149b18c623e9ebaf79ebec316a6e00b Mon Sep 17 00:00:00 2001 From: Joe Lanford Date: Fri, 3 Sep 2021 08:42:46 -0400 Subject: [PATCH 06/13] updates for new olm.channel schema, remove end-of-line whitespace Signed-off-by: Joe Lanford --- .../en/docs/Reference/file-based-catalogs.md | 126 ++++++++---------- content/en/docs/Tasks/creating-an-index.md | 4 +- 2 files changed, 61 insertions(+), 69 deletions(-) diff --git a/content/en/docs/Reference/file-based-catalogs.md b/content/en/docs/Reference/file-based-catalogs.md index a27eb3a6..29bb3012 100644 --- a/content/en/docs/Reference/file-based-catalogs.md +++ b/content/en/docs/Reference/file-based-catalogs.md @@ -14,7 +14,7 @@ The primary design goal for this format is to enable index editing, composabilit ### Editing With file-based catalogs, users interacting with the contents of an index are able to make direct changes to the index -format and verify that their changes are valid. +format and verify that their changes are valid. Because this format is plaintext JSON or YAML, index maintainers can easily manipulate index metadata by hand or with widely known and supported JSON or YAML tooling (e.g. `jq`). @@ -40,7 +40,7 @@ Since operator authors are most familiar with their operator, its dependencies, able to maintain their own operator-specific index and have direct control over its contents. With file-based catalogs, operator authors own the task of building and maintaining their packages in an index. Composite index maintainers treat packages as a black box; they own the task of curating the packages in their catalog and publishing the catalog to -users. +users. File-based catalogs can be composed by combining multiple other catalogs or by extracting subsets of one catalog, or a combination of both of these. @@ -82,7 +82,7 @@ they have the same rules for [patterns](https://git-scm.com/docs/gitignore#_patt Index maintainers have flexibility to chose their desired layout, but the OLM team recommends storing each package's file-based catalog blobs in separate sub-directories. Each individual file can be either JSON or YAML -- it is not -necessary for every file in an index to use the same format. +necessary for every file in an index to use the same format. This layout has the property that each sub-directoriy in the directory hierarchy is a self-contained index, which makes index composition, discovery, and navigation as simple as trivial filesystem operations. @@ -153,7 +153,7 @@ The `olm.package` [cue][cuelang-spec] schema is: ```cue #Package: { schema: "olm.package" - + // Package name name: string & !="" @@ -162,7 +162,7 @@ The `olm.package` [cue][cuelang-spec] schema is: // The package's default channel defaultChannel: string & !="" - + // An optional icon icon?: { base64data: string @@ -171,10 +171,53 @@ The `olm.package` [cue][cuelang-spec] schema is: } ``` +#### `olm.channel` + +An `olm.channel` defines a channel within a package, the bundle entries that are members +of the channel, and the upgrade edges for those bundles. + +A bundle can included as an entry in multiple `olm.channel` blobs, but it can have only one entry per channel. + +Also, it is valid for an entry's replaces value to reference another bundle name that cannot be found in this index +(or even another index) as long as other channel invariants still hold (e.g. a channel cannot have multiple heads). + +The `olm.channel` [cue][cuelang-spec] schema is: +```cue +#Channel: { + schema: "olm.channel" + package: string & !="" + name: string & !="" + entries: [...#ChannelEntry] +} + +#ChannelEntry: { + // name is required. It is the name of an `olm.bundle` that + // is present in the channel. + name: string & !="" + + // replaces is optional. It is the name of bundle that is replaced + // by this entry. It does not have to be present in the entry list. + replaces?: string & !="" + + // skips is optional. It is a list of bundle names that are skipped by + // this entry. The skipped bundles do not have to be present in the + // entry list. + skips?: [...string & !=""] + + // skipRange is optional. It is the semver range of bundle versions + // that are skipped by this entry. + skipRange?: string & !="" +} +``` + +For more information about defining upgrade edges, see the [upgrade graph reference documentation][upgrade-graph-doc]. + +[upgrade-graph-doc]: /docs/concepts/olm-architecture/operator-catalog/creating-an-update-graph + #### `olm.bundle` The `olm.bundle` cue schema is: @@ -199,7 +242,7 @@ The `olm.bundle` cue schema is: #RelatedImage: { // image is the image reference image: string & !="" - + // name is an optional descriptive name for an image that // helps identify its purpose in the context of the bundle name?: string & !="" @@ -250,57 +293,6 @@ The `olm.gvk` property [cue][cuelang-spec] schema is: } ``` -#### `olm.channel` - -An `olm.channel` property defines a channel that a bundle is in, and optionally, the name of another bundle that it -replaces in that channel. - -A bundle can include multiple `olm.channel` properties, but it is invalid to define multiple `olm.channel` properties -for the same channel name. - -Lastly, it is valid for an `olm.channel`'s replaces value to reference another bundle that cannot be found in this index -(or even another index) as long as other channel invariants still hold (e.g. a channel cannot have multiple heads). - -The `olm.channel` property [cue][cuelang-spec] schema is: -```cue -#PropertyChannel: { - type: "olm.channel" - value: { - name: string & !="" - replaces?: string & !="" - } -} -``` - -#### `olm.skips` - -An `olm.skips` property defines another bundle that this bundle skips. This property applies to all channels. - -Any number of skips properties can be set on a bundle. - -The `olm.skips` property [cue][cuelang-spec] schema is: -```cue -#PropertySkips: { - type: "olm.skips" - value: string & !="" -} -``` - -#### `olm.skipRange` - -An `olm.skipRange` property defines a [range of semver versions][semver-range] of other bundles that this bundle skips. -This property applies to all channels. - -It is invalid to include multiple `olm.skipRange` properties on a bundle. - -The `olm.skipRange` property [cue][cuelang-spec] schema is: -```cue -#PropertySkipRange: { - type: "olm.skipRange" - value: string & !="" -} -``` - #### `olm.package.required` An `olm.package.required` property defines the package name and version range of another package that this bundle @@ -345,7 +337,7 @@ The `olm.gvk.required` property [cue][cuelang-spec] schema is: > **NOTE**: Core OLM does not require `olm.bundle.object` properties to be included on bundles. However, the OLM Package > Server (used by tooling such as the kubectl operator plugin and the OpenShift console) does require these properties > to be able to serve metadata about the packages in an index. -> +> > This property is in _alpha_ because it will likely be rendered obsolete when updates can be made to the OLM Package > Server to no longer require manifests in the index. @@ -361,7 +353,7 @@ The `olm.bundle.object` property [cue][cuelang-spec] schema is: #PropertyBundleObject: { type: "olm.bundle.object" value: #propertyBundleObjectRef | #propertyBundleObjectData -} +} #propertyBundleObjectRef: { ref: string & !="" @@ -468,12 +460,12 @@ Examples: # Diff a catalog at some old state and latest state into a declarative config index. mkdir -p catalog-index opm alpha diff registry.org/my-catalog:abc123 registry.org/my-catalog:def456 -o yaml > ./my-catalog-index/index.yaml - + # Build and push this index into an index image. opm alpha generate dockerfile ./my-catalog-index docker build -t registry.org/my-catalog:latest-abc123-def456 -f index.Dockerfile . docker push registry.org/my-catalog:latest-abc123-def456 - + # Create a new catalog from the heads of an existing catalog, then build and push the image like above. opm alpha diff registry.org/my-catalog:def456 -o yaml > my-catalog-index/index.yaml docker build -t registry.org/my-catalog:headsonly-def456 -f index.Dockerfile . @@ -517,7 +509,7 @@ Global Flags: ### Immutable bundles -OLM's general advice is that bundle images and their metadata should be treated as immutable. If a broken bundle has +OLM's general advice is that bundle images and their metadata should be treated as immutable. If a broken bundle has been pushed to an index, you must assume that at least one of your users has upgraded to that bundle. Based on that assumption, you must release another bundle with an upgrade edge from the broken bundle to ensure users with the broken bundle installed receive an upgrade. OLM will not reinstall an installed bundle if the contents of that bundle are @@ -525,16 +517,16 @@ updated in the index. However, there are some cases where a change in the index metadata is preferred. For example: - Channel promotion - if you already released a bundle and later decide that you'd like to add it to another channel, - simply add an `olm.channel` property to the `olm.bundle` + simply add an entry for your bundle in another `olm.channel` blob. - New upgrade edges - if you release a new 1.2.z (e.g. 1.2.4), but 1.3.0 is already released, you can update the index metadata for 1.3.0 to skip 1.2.4. - + ### Use of source control OLM highly recommends storing index metadata in source control and treating the source-controlled metadata as the source of truth. Updates to index images should: 1. Update the source-controlled index directory with a new commit. -2. Build and push the index image. OLM suggests using a consistent tagging taxonomy (e.g. `:latest` or +2. Build and push the index image. OLM suggests using a consistent tagging taxonomy (e.g. `:latest` or `:` so that users can receive updates to an index as they become available. ## Example: Building a composite catalog @@ -561,7 +553,7 @@ There are many possible ways to build a catalog, but an extremely simple approac 2. Run a simple script that parses this file and creates a new catalog from its references ```bash name=$(yq eval '.name' catalog.yaml) - mkdir "$name" + mkdir "$name" yq eval '.name + "/" + .references[].name' catalog.yaml | xargs mkdir for l in $(yq e '.name as $catalog | .references[] | .image + "|" + $catalog + "/" + .name + "/index.yaml"' catalog.yaml); do image=$(echo $l | cut -d'|' -f1) diff --git a/content/en/docs/Tasks/creating-an-index.md b/content/en/docs/Tasks/creating-an-index.md index 8af58eb8..c9dc7933 100644 --- a/content/en/docs/Tasks/creating-an-index.md +++ b/content/en/docs/Tasks/creating-an-index.md @@ -92,7 +92,7 @@ Lastly, we can build and push our index: ```sh $ docker build . \ -f example-operator-index.Dockerfile \ - -t quay.io/example-inc/example-operator-index:latest + -t quay.io/example-inc/example-operator-index:latest $ docker push quay.io/example-inc/example-operator-index:latest ``` @@ -101,4 +101,4 @@ The resulting image is referred to as an `Index`. Now that image is available fo [catalogsource-crd]: /docs/concepts/crds/catalogsource [file-based-catalog-spec]: /docs/reference/file-based-catalogs -[upgrade-graph-doc]: /docs/concepts/olm-architecture/operator-catalog/creating-an-update-graph +[upgrade-graph-doc]: /docs/concepts/olm-architecture/operator-catalog/creating-an-update-graph From 3b66bde1b2c6345808e2ae84b2ae331de315d6c2 Mon Sep 17 00:00:00 2001 From: Joe Lanford Date: Fri, 3 Sep 2021 08:56:25 -0400 Subject: [PATCH 07/13] addressing some of the PR feedback Signed-off-by: Joe Lanford --- .../en/docs/Reference/file-based-catalogs.md | 30 ++++++++++++------- content/en/docs/Tasks/creating-an-index.md | 2 +- 2 files changed, 20 insertions(+), 12 deletions(-) diff --git a/content/en/docs/Reference/file-based-catalogs.md b/content/en/docs/Reference/file-based-catalogs.md index 29bb3012..31527c52 100644 --- a/content/en/docs/Reference/file-based-catalogs.md +++ b/content/en/docs/Reference/file-based-catalogs.md @@ -22,7 +22,7 @@ widely known and supported JSON or YAML tooling (e.g. `jq`). This editability enables features and user-defined extensions, such as: - Promoting an existing bundle to a new channel - Changing the default channel of a package -- Custom algorithms for adding, updating, adding removing upgrade edges. +- Custom algorithms for adding, updating, adding removing upgrade edges ### Composability @@ -45,6 +45,8 @@ users. File-based catalogs can be composed by combining multiple other catalogs or by extracting subsets of one catalog, or a combination of both of these. +See the [Building a composite catalog](#building-a-composite-catalog) section for a simple example. + ### Extensibility The final design goal is to provide extensibility around indexes. The file-based catalog spec is a low-level @@ -80,11 +82,11 @@ they have the same rules for [patterns](https://git-scm.com/docs/gitignore#_patt > ``` -Index maintainers have flexibility to chose their desired layout, but the OLM team recommends storing each package's +Index maintainers have the flexibility to chose their desired layout, but the OLM team recommends storing each package's file-based catalog blobs in separate sub-directories. Each individual file can be either JSON or YAML -- it is not necessary for every file in an index to use the same format. -This layout has the property that each sub-directoriy in the directory hierarchy is a self-contained index, which makes +This layout has the property that each sub-directory in the directory hierarchy is a self-contained index, which makes index composition, discovery, and navigation as simple as trivial filesystem operations. > #### Basic recommended structure @@ -136,10 +138,11 @@ _Meta: { ### OLM-defined schemas -An OLM index currently uses two schemas: `olm.package` and `olm.bundle`, which correspond to OLM's existing package and -bundle concepts. +An OLM index currently uses three schemas: `olm.package`, `olm.channel`, and `olm.bundle`, which correspond to OLM's +existing package and bundle concepts. -Each operator package in an index requires exactly one `olm.package` blob and one or more `olm.bundle` blobs. +Each operator package in an index requires exactly one `olm.package` blob, at least one `olm.channel` blob, and one or +more `olm.bundle` blobs. > **NOTE**: All `olm.*` schemas are reserved for OLM-defined schemas. Custom schemas must use a unique prefix (e.g. a > domain that you own). @@ -341,11 +344,11 @@ The `olm.gvk.required` property [cue][cuelang-spec] schema is: > This property is in _alpha_ because it will likely be rendered obsolete when updates can be made to the OLM Package > Server to no longer require manifests in the index. -A bundle object property can contain inlined data using the `value.data` field, which must the base64-encoded string of -that manifest. +A bundle object property can contain inlined data using the `value.data` field, which must be the base64-encoded string +of that manifest. -Alternately, a bundle object property can be a reference to a file relative to the location of file in which the bundle -is declared. Any referenced files must be within the catalog root. +Alternately, a bundle object property can be a reference to a file relative to the location of the file in which the +bundle is declared. Any referenced files must be within the catalog root. The `olm.bundle.object` property [cue][cuelang-spec] schema is: ```cue @@ -529,7 +532,12 @@ of truth. Updates to index images should: 2. Build and push the index image. OLM suggests using a consistent tagging taxonomy (e.g. `:latest` or `:` so that users can receive updates to an index as they become available. -## Example: Building a composite catalog + + +## Examples +### Building a composite catalog With file-based catalogs, catalog maintainers can focus on operator curation and compatibility. Since operator authors have already produced operator-specific indexes for their operators, catalog diff --git a/content/en/docs/Tasks/creating-an-index.md b/content/en/docs/Tasks/creating-an-index.md index c9dc7933..685b8105 100644 --- a/content/en/docs/Tasks/creating-an-index.md +++ b/content/en/docs/Tasks/creating-an-index.md @@ -7,7 +7,7 @@ description: > ## Prerequisites -- [opm](https://github.com/operator-framework/operator-registry/releases) `v1.18.0+` +- [opm](https://github.com/operator-framework/operator-registry/releases) `v1.19.0+` ## Creating an Index From 56a9f780f105073066d8cda87b02ccdd275aa5c3 Mon Sep 17 00:00:00 2001 From: Joe Lanford Date: Tue, 7 Sep 2021 21:18:28 -0400 Subject: [PATCH 08/13] fix issue with rendering titled code blocks Signed-off-by: Joe Lanford --- content/en/docs/Reference/file-based-catalogs.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/docs/Reference/file-based-catalogs.md b/content/en/docs/Reference/file-based-catalogs.md index 31527c52..5efccb13 100644 --- a/content/en/docs/Reference/file-based-catalogs.md +++ b/content/en/docs/Reference/file-based-catalogs.md @@ -71,7 +71,7 @@ it finds and fails if any errors occur. Non-catalog files can be ignored using `.indexignore` files, which behave identically to `.gitignore` files. That is, they have the same rules for [patterns](https://git-scm.com/docs/gitignore#_pattern_format) and precedence. -> #### Example `.gitignore` file +> **Example `.gitignore` file** > ```gitignore > # Ignore everything except non-object .json and .yaml files > **/* @@ -89,7 +89,7 @@ necessary for every file in an index to use the same format. This layout has the property that each sub-directory in the directory hierarchy is a self-contained index, which makes index composition, discovery, and navigation as simple as trivial filesystem operations. -> #### Basic recommended structure +> **Basic recommended structure** > ``` > index > ├── pkgA From 4a656b15e4f506669adee99e49c58046b2b451bb Mon Sep 17 00:00:00 2001 From: Joe Lanford Date: Tue, 7 Sep 2021 22:29:05 -0400 Subject: [PATCH 09/13] further updates to update upgrade graph topics to use rather than CSV fields/bundle metadata Signed-off-by: Joe Lanford --- .../creating-an-update-graph.md | 219 +++++++++++++----- ...ommunicating-operator-conditions-to-olm.md | 2 +- .../contribution-guidelines/upgrade-graphs.md | 8 +- 3 files changed, 171 insertions(+), 58 deletions(-) diff --git a/content/en/docs/Concepts/olm-architecture/operator-catalog/creating-an-update-graph.md b/content/en/docs/Concepts/olm-architecture/operator-catalog/creating-an-update-graph.md index 88237d71..4610b87f 100644 --- a/content/en/docs/Concepts/olm-architecture/operator-catalog/creating-an-update-graph.md +++ b/content/en/docs/Concepts/olm-architecture/operator-catalog/creating-an-update-graph.md @@ -8,26 +8,31 @@ description: > # Creating an Update Graph -OLM provides a variety of ways to specify updates between operator versions as well as different add -modes (requires bundle format) to control how versions fit into the catalog. +OLM provides a variety of ways to specify updates between operator versions. ## Methods for Specifying Updates +All update graphs are defined in file-based catalogs via `olm.channel` blobs. Each `olm.channel` defines the set of +bundles present in the channel and the update graph edges between each entry in the channel. + ### Replaces -For explicit updates from one CSV to another, you can specify the CSV name to replace in your CSV as +For explicit updates from one CSV to another, you can specify the CSV name to replace in your channel entry as such: ```yaml -metadata: - name: myoperator.v1.0.1 -spec: - replaces: myoperator.v1.0.0 +--- +schema: olm.channel +package: myoperator +channel: stable +entries: + - name: myoperator.v1.0.1 + replaces: myoperator.v1.0.0 ``` -In order for `myoperator.v1.0.1` to be added to the catalog successfully, `myoperator.v1.0.0` needs to -be included in your manifests or have already been added to that catalog (in the case where packaging -is done in the bundle format). +Note that it is not required for there to be an entry for `myoperator.v1.0.0` in the catalog as long as +other channel invariants (verified by `opm validate`) still hold. Generally, this means that the tail of the channel's +`replaces` chain can replace a bundle that is not present in the catalog. An update sequence of bundles created via `replaces` will have updates step through each version in the chain. For example, given @@ -40,7 +45,7 @@ A subscription on `myoperator.v1.0.0` will update to `myoperator.v1.0.2` through Installing from the UI today will always install the latest of a given channel. However, installing specific versions is possible with this update graph by modifying the `startingCSV` field -of the subscription to point to the desired CSV name. Note that, in this case, the subscription will +of the subscription to point to the desired bundle name. Note that, in this case, the subscription will need its `approval` field set to `Manual` to ensure that the subscription does not auto-update and instead stays pinned to the specified version. @@ -49,13 +54,16 @@ instead stays pinned to the specified version. In order to skip through certain updates you can specify a list of CSV names to be skipped as such: ```yaml -metadata: - name: myoperator.v1.0.3 -spec: - replaces: myoperator.v1.0.0 - skips: - - myoperator.v1.0.1 - - myoperator.v1.0.2 +--- +schema: olm.channel +package: myoperator +channel: stable +entries: + - name: myoperator.v1.0.3 + replaces: myoperator.v1.0.0 + skips: + - myoperator.v1.0.1 + - myoperator.v1.0.2 ``` Using the above graph, this will mean subscriptions on `myoperator.v1.0.0` can update directly to @@ -70,45 +78,25 @@ Skipped CSVs do not need to be present in a catalog or set of manifests prior to ### SkipRange -OLM also allows you to specify updates through version ranges in your CSV. This requires your CSVs +OLM also allows you to specify updates through version ranges in your channel entry. This requires your CSVs to define a version in their version field which must follow the [semver spec](https://semver.org/). Internally, OLM uses the [blang/semver](https://github.com/blang/semver) go library. ```yaml -metadata: - name: myoperator.v1.0.3 - annotations: - olm.skipRange: ">=1.0.0 <1.0.3" +--- +schema: olm.channel +package: myoperator +channel: stable +entries: + - name: myoperator.v1.0.3 + skipRange: ">=1.0.0 <1.0.3" ``` -The version specifying the `olm.skipRange` will be presented as a direct (one hop) update to +The entry specifying the `skipRange` will be presented as a direct (one hop) update to any version from that package within that range. The versions in this range do not need to be in the index in order for bundle addition to be successful. We recommend avoiding using unbound ranges such as `<1.0.3`. -**Warning:** Adding a bundle that only specifies skipRange to a given channel will wipe out all -the previous content in that channel. This means directly installing past versions by editing -the `startingCSV` field of the subscription is not possible when using skiprange only. In order -for past versions to be installable by `startingCSV` while also benefitting from the `skipRange` -feature, you will need to also connect past edges by setting a `replaces` field in addition to -the `olm.skipRange`. For example, assuming the above update graph: - -```txt -myoperator.v1.0.0 -> myoperator.v1.0.1 -> myoperator.v1.0.2 -``` - -In order to keep these versions installable by `startingCSV` when `myoperator.v1.0.3` is added, -the CSV for `myoperator.v1.0.3` needs to have the following: - -```yaml -metadata: - name: myoperator.v1.0.3 - annotations: - olm.skipRange: ">=1.0.0 <1.0.3" -spec: - replaces: myoperator.v1.0.2 -``` - SkipRange by itself is useful for teams who are not interested in supporting directly installing versions within a given range or for whom consumers of the operator are always on the latest version. @@ -118,7 +106,7 @@ version. ### Cross channel updates Cross channel updates are not possible today in an automated way. In order for your subscription -to switch channels, the cluster admin must manually change the `channel` field in the subcription +to switch channels, the cluster admin must manually change the `channel` field in the subscription object. Changing this field does not always result in receiving updates from that channel. For that to @@ -126,16 +114,141 @@ occur, updates paths must be available from the given version to versions in the #### If using replaces -The CSV name currently installed must be in the `replaces` field of a CSV in the new channel. +The CSV name currently installed must be in the `replaces` field of an entry in the new channel. #### If using skips -The CSV name currently installed must be in the `skips` field of a CSV in the new channel. +The CSV name currently installed must be in the `skips` field of an entry in the new channel. #### If using skipRange -The version currently installed must be in the `olm.skipRange` field of a CSV in the new channel. +The version currently installed must be in the `skipRange` field of an entry in the new channel. + +## Channel Promotion + +A popular channel strategy is to use `alpha`, `beta`, and `stable` channels, where every new bundle is added to `alpha`, +a subset of bundles from `alpha` are promoted to `beta`, and a further subset of bundles from `beta` are promoted to +`stable`. + +An operator author wishing to employ the `alpha`/`beta`/`stable` channel strategy may have created and updated their +`olm.channel` blobs following the below scenario. + +First, an operator author releases a few versions into the `alpha` channel, with a simple linear replaces chain. +```yaml +--- +schema: olm.channel +package: myoperator +channel: alpha +entries: + - name: myoperator.v0.1.0 + - name: myoperator.v0.2.0 + replaces: myoperator.v0.1.0 + - name: myoperator.v0.3.0 + replaces: myoperator.v0.2.0 +``` -## Add Modes +Next, an operator author decides that `myoperator.v0.2.0` is a good candidate to promote to `beta`, so they add a new +`olm.channel` schema and include `myoperator.v0.2.0` as the first entry, taking care to preserve any upgrade edges from +`alpha` to enable users to switch from the `alpha` to the `beta` channel if they have `myoperator.v0.1.0` installed. -TODO +> NOTE: This bundle already exists in the index (the image was already built and pushed, and the index already contains +> an `olm.bundle` blob and an entry for it in the `alpha` channel). The channel promotion step for `myoperator.v0.2.0` +> is to simply create a new `olm.channel` for the `beta` channel and include an entry for `myoperator.v0.2.0`. + +```yaml +--- +schema: olm.channel +package: myoperator +channel: alpha +entries: + - name: myoperator.v0.1.0 + - name: myoperator.v0.2.0 + replaces: myoperator.v0.1.0 + - name: myoperator.v0.3.0 + replaces: myoperator.v0.2.0 +--- +schema: olm.channel +package: myoperator +channel: beta +entries: + - name: myoperator.v0.2.0 + replaces: myoperator.v0.1.0 +``` + +The operator continues releasing bundles to `alpha` and promotes `myoperator.v0.4.0` to `beta`. + +> NOTE: In channel `alpha`, `myoperator.v0.4.0` replaces `myoperator.v0.3.0`, but in channel `beta`, `myoperator.v0.4.0` +> replaces `myoperator.v0.2.0` because the `beta` channel does not include every entry from alpha. + +```yaml +--- +schema: olm.channel +package: myoperator +channel: alpha +entries: + - name: myoperator.v0.1.0 + - name: myoperator.v0.2.0 + replaces: myoperator.v0.1.0 + - name: myoperator.v0.3.0 + replaces: myoperator.v0.2.0 + - name: myoperator.v0.4.0 + replaces: myoperator.v0.3.0 + - name: myoperator.v0.5.0 + replaces: myoperator.v0.4.0 +--- +schema: olm.channel +package: myoperator +channel: beta +entries: + - name: myoperator.v0.2.0 + replaces: myoperator.v0.1.0 + - name: myoperator.v0.4.0 + replaces: myoperator.v0.2.0 +``` + +Finally, the operator author releases several more bundles and makes several more promotions, finally deciding to +promote `myoperator.v0.4.0` to the stable channel. With `myoperator.v0.4.0` being the first entry in the `stable` +channel, the operator author has decided to add `replaces` and `skips` edges for all previously released bundles to +assist users in moving to the `stable` channel directly from the `alpha` and `beta` channels. However, it would have +been equally valid to require that a user have either `myoperator.v0.2.0` or `myoperator.v0.3.0` installed (e.g. if the +direct upgrade from `myoperator.v0.1.0` has not been tested or is known to be unsupported). + +```yaml +--- +schema: olm.channel +package: myoperator +channel: alpha +entries: + - name: myoperator.v0.1.0 + - name: myoperator.v0.2.0 + replaces: myoperator.v0.1.0 + - name: myoperator.v0.3.0 + replaces: myoperator.v0.2.0 + - name: myoperator.v0.4.0 + replaces: myoperator.v0.3.0 + - name: myoperator.v0.5.0 + replaces: myoperator.v0.4.0 + - name: myoperator.v0.6.0 + replaces: myoperator.v0.5.0 +--- +schema: olm.channel +package: myoperator +channel: beta +entries: + - name: myoperator.v0.2.0 + replaces: myoperator.v0.1.0 + - name: myoperator.v0.4.0 + replaces: myoperator.v0.2.0 + - name: myoperator.v0.6.0 + replaces: myoperator.v0.4.0 +--- +schema: olm.channel +package: myoperator +channel: stable +entries: + - name: myoperator.v0.4.0 + replaces: myoperator.v0.1.0 + skips: + - myoperator.v0.2.0 + - myoperator.v0.3.0 +``` diff --git a/content/en/docs/advanced-tasks/communicating-operator-conditions-to-olm.md b/content/en/docs/advanced-tasks/communicating-operator-conditions-to-olm.md index 3600c007..d03cddf7 100644 --- a/content/en/docs/advanced-tasks/communicating-operator-conditions-to-olm.md +++ b/content/en/docs/advanced-tasks/communicating-operator-conditions-to-olm.md @@ -22,7 +22,7 @@ OLM introduced a new [CustomResourceDefinition](https://kubernetes.io/docs/conce The `Upgradeable` "OLM Supported Condition" prevents the existing CSV from being replaced by a newer version of the CSV. When the `Upgradeable` condition is set to `False`, OLM will: -* Prevent a CSV that replaces the operator's existing CSV from leaving the PendingPhase. +* Prevent a channel entry in a subscribed package that replaces the operator's existing CSV from leaving the PendingPhase. The `Upgradeable` condition might be useful when: diff --git a/content/en/docs/contribution-guidelines/upgrade-graphs.md b/content/en/docs/contribution-guidelines/upgrade-graphs.md index 02dd81d3..fb92de29 100644 --- a/content/en/docs/contribution-guidelines/upgrade-graphs.md +++ b/content/en/docs/contribution-guidelines/upgrade-graphs.md @@ -56,21 +56,21 @@ flowchart TB classDef installed fill:#34ebba; A(v0.0.1) --> B(v0.0.4) -{{}} | An upgrade path to replace one operator bundle version for another using the [CSV][csv-definition] `replaces spec. (eg.`spec.replaces: exampleoperator.v0.0.1`). More info: [here][upgrade-path-replaces]. +{{}} | An upgrade path to replace one operator bundle version for another using the [`olm.channel`][olm-channel] `replaces` field. More info: [here][upgrade-path-replaces]. | {{}} flowchart TB classDef head fill:#ffbfcf; classDef installed fill:#34ebba; A(v0.0.1) x--x |v0.0.2,v0.0.3| B(v0.0.4) -{{}} | An upgrade path to skip versions in the upgrade path using the [CSV][csv-definition] `skips` spec. (eg.`spec.skips: exampleoperator.v0.0.2, exampleoperator.v0.0.3`). More info: [here][upgrade-path-skips]. | +{{}} | An upgrade path to skip versions in the upgrade path using the [`olm.channel`][olm-channel] `skips` field. More info: [here][upgrade-path-skips]. | | {{}} flowchart TB classDef head fill:#ffbfcf; classDef installed fill:#34ebba; C(v2.0.3) o--o |>= 2.0.4 < 3.0.0| D(v3.0.1) -{{}} | An upgrade path to skip a range of operator bundle versions using the [CSV][csv-definition] `olm.skipRange` annotation. (E.g `annotations.olm.skipRange: '>= 2.0.4 < 3.0.0'`) More info: [here][upgrade-path-skiprange]. | +{{}} | An upgrade path to skip a range of operator bundle versions using the [`olm.channel`][olm-channel] `skipRange` field. More info: [here][upgrade-path-skiprange]. | | {{}} flowchart TB classDef head fill:#ffbfcf; @@ -235,7 +235,7 @@ flowchart TB E(v0.0.2 \n fa:fa-tag label=value) {{}} | -[csv-definition]:/docs/concepts/crds/clusterserviceversion/ +[olm-channel]:/docs/reference/file-based-catalogs/#olmchannel [upgrade-path-replaces]:/docs/concepts/olm-architecture/operator-catalog/creating-an-update-graph/#replaces [upgrade-path-skips]:/docs/concepts/olm-architecture/operator-catalog/creating-an-update-graph/#skips [upgrade-path-skiprange]:/docs/concepts/olm-architecture/operator-catalog/creating-an-update-graph/#skiprange \ No newline at end of file From 9ff57ed22efa956be79d2cc786b6d256c5ac66be Mon Sep 17 00:00:00 2001 From: Joe Lanford Date: Tue, 7 Sep 2021 22:49:28 -0400 Subject: [PATCH 10/13] avoid using ordered lists (our docsy theme mods don't render the numbers) Signed-off-by: Joe Lanford --- .../en/docs/Reference/file-based-catalogs.md | 21 ++++++++++--------- content/en/docs/Tasks/creating-an-index.md | 18 ++++++++-------- 2 files changed, 20 insertions(+), 19 deletions(-) diff --git a/content/en/docs/Reference/file-based-catalogs.md b/content/en/docs/Reference/file-based-catalogs.md index 5efccb13..fd02350a 100644 --- a/content/en/docs/Reference/file-based-catalogs.md +++ b/content/en/docs/Reference/file-based-catalogs.md @@ -528,8 +528,8 @@ However, there are some cases where a change in the index metadata is preferred. OLM highly recommends storing index metadata in source control and treating the source-controlled metadata as the source of truth. Updates to index images should: -1. Update the source-controlled index directory with a new commit. -2. Build and push the index image. OLM suggests using a consistent tagging taxonomy (e.g. `:latest` or +- Update the source-controlled index directory with a new commit. +- Build and push the index image. OLM suggests using a consistent tagging taxonomy (e.g. `:latest` or `:` so that users can receive updates to an index as they become available. ## Examples + ### Building a composite catalog With file-based catalogs, catalog maintainers can focus on operator curation and compatibility. @@ -546,7 +547,7 @@ catalog's root index directory. There are many possible ways to build a catalog, but an extremely simple approach would be to: -1. Maintain a single configuration file containing image references for each operator in the catalog +- Maintain a single configuration file containing image references for each operator in the catalog ```yaml name: community-operators repo: quay.io/community-operators/catalog @@ -558,7 +559,7 @@ There are many possible ways to build a catalog, but an extremely simple approac image: quay.io/prometheus-operator/index@sha256:e258d248fda94c63753607f7c4494ee0fcbe92f1a76bfdac795c9d84101eb317 ``` -2. Run a simple script that parses this file and creates a new catalog from its references +- Run a simple script that parses this file and creates a new catalog from its references ```bash name=$(yq eval '.name' catalog.yaml) mkdir "$name" @@ -578,9 +579,9 @@ There are many possible ways to build a catalog, but an extremely simple approac Operator authors and catalog maintainers are encouraged to automate their index maintenance with CI/CD workflows. Catalog maintainers could further improve on this by building Git-ops automation that: -1. Checks that PR authors are permitted to make the requested changes (e.g. updating their package's image reference) -2. Checks that the index updates pass `opm validate` -3. Checks that the updated bundle and/or index image reference(s) exist, the index images run successfully in a cluster, - and operators from that package can be successfully installed. -4. Automatically merges PRs that pass these checks. -5. Automatically rebuilds and republishes the index image. +- Checks that PR authors are permitted to make the requested changes (e.g. updating their package's image reference) +- Checks that the index updates pass `opm validate` +- Checks that the updated bundle and/or index image reference(s) exist, the index images run successfully in a cluster, + and operators from that package can be successfully installed. +- Automatically merges PRs that pass these checks. +- Automatically rebuilds and republishes the index image. diff --git a/content/en/docs/Tasks/creating-an-index.md b/content/en/docs/Tasks/creating-an-index.md index 685b8105..7b577bbb 100644 --- a/content/en/docs/Tasks/creating-an-index.md +++ b/content/en/docs/Tasks/creating-an-index.md @@ -62,10 +62,10 @@ $ opm validate example-operator-index Success! There were no errors and we got a `0` error code. In the general case, adding a bundle involves three discreet steps: -1. Render the bundle into the index using `opm render ` -2. Add the bundle into desired channels and update the channels' upgrade edges - to stitch the bundle into the correct place. -3. Validate the resulting index. +- Render the bundle into the index using `opm render ` +- Add the bundle into desired channels and update the channels' upgrade edges + to stitch the bundle into the correct place. +- Validate the resulting index. > NOTE: Index metadata should be stored in a version control system (e.g. `git`) and index images should be rebuilt from source whenever updates are made to ensure that all index changes are auditable. @@ -78,11 +78,11 @@ build semver-based channels and upgrade graphs based solely on the versions of t no right or wrong answer for implementing this step as long as `opm validate` is successful. There are some guidelines to keep in mind though: -1. Once a bundle is present in an index, you should assume that one of your users has installed it. With that in mind, - you should take care to avoid stranding users that have that version installed. Put another way, make sure that - all previously published bundles in an index have a path to the current/new channel head. -2. Keep the semantics of the upgrade edges you use in mind. `opm validate` is not able to tell you if you have a sane - upgrade graph. To learn more about the upgrade graph of an operator, checkout the +- Once a bundle is present in an index, you should assume that one of your users has installed it. With that in mind, + you should take care to avoid stranding users that have that version installed. Put another way, make sure that + all previously published bundles in an index have a path to the current/new channel head. +- Keep the semantics of the upgrade edges you use in mind. `opm validate` is not able to tell you if you have a sane + upgrade graph. To learn more about the upgrade graph of an operator, checkout the [creating an upgrade graph doc][upgrade-graph-doc] ### Build and push the index image From 22c2ec0f69c3da72da89ab453be8a3c23b950453 Mon Sep 17 00:00:00 2001 From: Joe Lanford Date: Thu, 9 Sep 2021 09:18:39 -0400 Subject: [PATCH 11/13] update creating-an-index.md steps with olm.channel blob Signed-off-by: Joe Lanford --- content/en/docs/Tasks/creating-an-index.md | 28 ++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/content/en/docs/Tasks/creating-an-index.md b/content/en/docs/Tasks/creating-an-index.md index 7b577bbb..c9022649 100644 --- a/content/en/docs/Tasks/creating-an-index.md +++ b/content/en/docs/Tasks/creating-an-index.md @@ -32,7 +32,7 @@ $ opm init example-operator \ --default-channel=preview \ --description=./README.md \ --icon=./example-operator.svg \ - -output yaml > example-operator-index/index.yaml + --output yaml > example-operator-index/index.yaml ``` Let's validate our index to see if we're ready to ship! @@ -51,12 +51,36 @@ that next... ```sh $ opm render quay.io/example-inc/example-operator-bundle:v0.1.0 \ - --output=yaml > example-operator-index/index.yaml + --output=yaml >> example-operator-index/index.yaml ``` Let's validate again: +``` +$ opm validate example-operator-index +FATA[0000] package "example-operator", bundle "example-operator.v0.1.0" not found in any channel entries +``` + +### Add a channel entry for the bundle + +We rendered the bundle, but we still haven't yet added it to any channels. +Let's initialize a channel: +```sh +cat << EOF >> example-operator-index/index.yaml +--- +schema: olm.channel +package: example-operator +name: preview +entries: + - name: example-operator.v0.1.0 +EOF +``` + +Is the third time the charm for `opm validate`? + ```sh $ opm validate example-operator-index +$ echo $? +0 ``` Success! There were no errors and we got a `0` error code. From 92059d33022d89b0acc9185260eba8bfb47f0d71 Mon Sep 17 00:00:00 2001 From: Joe Lanford Date: Thu, 9 Sep 2021 09:23:17 -0400 Subject: [PATCH 12/13] more updates from PR feedback Signed-off-by: Joe Lanford --- content/en/docs/Tasks/creating-an-index.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/content/en/docs/Tasks/creating-an-index.md b/content/en/docs/Tasks/creating-an-index.md index c9022649..ac49d9fc 100644 --- a/content/en/docs/Tasks/creating-an-index.md +++ b/content/en/docs/Tasks/creating-an-index.md @@ -13,7 +13,7 @@ description: > `OLM`'s `CatalogSource` [CRD][catalogsource-crd] defines a reference to a catalog of operators that are available to install onto a cluster. To make your operator bundle available, you can add the bundle to a container image which the -`CatalogSource` points to. This image contains a record of bundle images that OLM can pull and extract the manifests +`CatalogSource` references. This image contains a record of bundle images that OLM can pull and extract the manifests from in order to install an operator. OLM uses a plaintext [file-based catalog][file-based-catalog-spec] format (JSON or YAML) to store these records in an index, and `opm` has tooling @@ -86,7 +86,7 @@ $ echo $? Success! There were no errors and we got a `0` error code. In the general case, adding a bundle involves three discreet steps: -- Render the bundle into the index using `opm render ` +- Render the bundle into the index using `opm render `. - Add the bundle into desired channels and update the channels' upgrade edges to stitch the bundle into the correct place. - Validate the resulting index. @@ -107,7 +107,7 @@ There are some guidelines to keep in mind though: all previously published bundles in an index have a path to the current/new channel head. - Keep the semantics of the upgrade edges you use in mind. `opm validate` is not able to tell you if you have a sane upgrade graph. To learn more about the upgrade graph of an operator, checkout the - [creating an upgrade graph doc][upgrade-graph-doc] + [creating an upgrade graph doc][upgrade-graph-doc]. ### Build and push the index image @@ -120,8 +120,7 @@ $ docker build . \ $ docker push quay.io/example-inc/example-operator-index:latest ``` -The resulting image is referred to as an `Index`. Now that image is available for clusters to use and reference with -`CatalogSources` on their cluster. +Now the index image is available for clusters to use and reference with `CatalogSources` on their cluster. [catalogsource-crd]: /docs/concepts/crds/catalogsource [file-based-catalog-spec]: /docs/reference/file-based-catalogs From 1ba1fb05ce3fad51007eb79fc7c7a7793654cf47 Mon Sep 17 00:00:00 2001 From: Joe Lanford Date: Thu, 9 Sep 2021 17:21:35 -0400 Subject: [PATCH 13/13] more updates from PR feedback --- .../operator-catalog/creating-an-update-graph.md | 15 +++++++++------ content/en/docs/Tasks/creating-an-index.md | 8 ++++---- 2 files changed, 13 insertions(+), 10 deletions(-) diff --git a/content/en/docs/Concepts/olm-architecture/operator-catalog/creating-an-update-graph.md b/content/en/docs/Concepts/olm-architecture/operator-catalog/creating-an-update-graph.md index 4610b87f..cb92e031 100644 --- a/content/en/docs/Concepts/olm-architecture/operator-catalog/creating-an-update-graph.md +++ b/content/en/docs/Concepts/olm-architecture/operator-catalog/creating-an-update-graph.md @@ -12,12 +12,12 @@ OLM provides a variety of ways to specify updates between operator versions. ## Methods for Specifying Updates -All update graphs are defined in file-based catalogs via `olm.channel` blobs. Each `olm.channel` defines the set of +All update graphs are defined in [file-based catalogs][file-based-catalog-spec] via `olm.channel` blobs. Each `olm.channel` defines the set of bundles present in the channel and the update graph edges between each entry in the channel. ### Replaces -For explicit updates from one CSV to another, you can specify the CSV name to replace in your channel entry as +For explicit updates from one operator version to another, you can specify the operator name to replace in your channel entry as such: ```yaml @@ -31,7 +31,7 @@ entries: ``` Note that it is not required for there to be an entry for `myoperator.v1.0.0` in the catalog as long as -other channel invariants (verified by `opm validate`) still hold. Generally, this means that the tail of the channel's +other channel invariants (verified by [`opm validate`][opm-validate-cli]) still hold. Generally, this means that the tail of the channel's `replaces` chain can replace a bundle that is not present in the catalog. An update sequence of bundles created via `replaces` will have updates step through each version in @@ -45,13 +45,13 @@ A subscription on `myoperator.v1.0.0` will update to `myoperator.v1.0.2` through Installing from the UI today will always install the latest of a given channel. However, installing specific versions is possible with this update graph by modifying the `startingCSV` field -of the subscription to point to the desired bundle name. Note that, in this case, the subscription will +of the subscription to point to the desired operator name. Note that, in this case, the subscription will need its `approval` field set to `Manual` to ensure that the subscription does not auto-update and instead stays pinned to the specified version. ### Skips -In order to skip through certain updates you can specify a list of CSV names to be skipped as such: +In order to skip through certain updates, you can specify a list of operator names to be skipped. For example: ```yaml --- @@ -74,7 +74,7 @@ that are already on `myoperator.v1.0.1` or `myoperator.v1.0.2` will still be abl This is particularly useful if `myoperator.v1.0.1` and `myoperator.v1.0.2` are affected by a CVE or contain bugs. -Skipped CSVs do not need to be present in a catalog or set of manifests prior to adding to a catalog. +Skipped operators do not need to be present in a catalog or set of manifests prior to adding to a catalog. ### SkipRange @@ -252,3 +252,6 @@ entries: - myoperator.v0.2.0 - myoperator.v0.3.0 ``` + +[file-based-catalog-spec]: /docs/reference/file-based-catalogs +[opm-validate-cli]: /docs/reference/file-based-catalogs/#opm-validate diff --git a/content/en/docs/Tasks/creating-an-index.md b/content/en/docs/Tasks/creating-an-index.md index ac49d9fc..4d3670f7 100644 --- a/content/en/docs/Tasks/creating-an-index.md +++ b/content/en/docs/Tasks/creating-an-index.md @@ -11,10 +11,10 @@ description: > ## Creating an Index -`OLM`'s `CatalogSource` [CRD][catalogsource-crd] defines a reference to a catalog of operators that are available to -install onto a cluster. To make your operator bundle available, you can add the bundle to a container image which the -`CatalogSource` references. This image contains a record of bundle images that OLM can pull and extract the manifests -from in order to install an operator. +`OLM`'s `CatalogSource` [CRD][catalogsource-crd] accepts a container image reference to a catalog of operators that can +be made available to install in a cluster. You can make your operator bundle available to install in a cluster by adding +it to a catalog, packaging the catalog in a container image, and then using that image reference in the `CatalogSource`. +This image contains all of the metadata required for OLM to manage the lifecycles of all of the operators it contains. OLM uses a plaintext [file-based catalog][file-based-catalog-spec] format (JSON or YAML) to store these records in an index, and `opm` has tooling that helps initialize an index, render new records into it, and then validate that the index is valid. Let's walk