diff --git a/content/en/docs/Concepts/olm-architecture/operator-catalog/creating-an-update-graph.md b/content/en/docs/Concepts/olm-architecture/operator-catalog/creating-an-update-graph.md index 88237d71..cb92e031 100644 --- a/content/en/docs/Concepts/olm-architecture/operator-catalog/creating-an-update-graph.md +++ b/content/en/docs/Concepts/olm-architecture/operator-catalog/creating-an-update-graph.md @@ -8,26 +8,31 @@ description: > # Creating an Update Graph -OLM provides a variety of ways to specify updates between operator versions as well as different add -modes (requires bundle format) to control how versions fit into the catalog. +OLM provides a variety of ways to specify updates between operator versions. ## Methods for Specifying Updates +All update graphs are defined in [file-based catalogs][file-based-catalog-spec] via `olm.channel` blobs. Each `olm.channel` defines the set of +bundles present in the channel and the update graph edges between each entry in the channel. + ### Replaces -For explicit updates from one CSV to another, you can specify the CSV name to replace in your CSV as +For explicit updates from one operator version to another, you can specify the operator name to replace in your channel entry as such: ```yaml -metadata: - name: myoperator.v1.0.1 -spec: - replaces: myoperator.v1.0.0 +--- +schema: olm.channel +package: myoperator +channel: stable +entries: + - name: myoperator.v1.0.1 + replaces: myoperator.v1.0.0 ``` -In order for `myoperator.v1.0.1` to be added to the catalog successfully, `myoperator.v1.0.0` needs to -be included in your manifests or have already been added to that catalog (in the case where packaging -is done in the bundle format). +Note that it is not required for there to be an entry for `myoperator.v1.0.0` in the catalog as long as +other channel invariants (verified by [`opm validate`][opm-validate-cli]) still hold. Generally, this means that the tail of the channel's +`replaces` chain can replace a bundle that is not present in the catalog. An update sequence of bundles created via `replaces` will have updates step through each version in the chain. For example, given @@ -40,22 +45,25 @@ A subscription on `myoperator.v1.0.0` will update to `myoperator.v1.0.2` through Installing from the UI today will always install the latest of a given channel. However, installing specific versions is possible with this update graph by modifying the `startingCSV` field -of the subscription to point to the desired CSV name. Note that, in this case, the subscription will +of the subscription to point to the desired operator name. Note that, in this case, the subscription will need its `approval` field set to `Manual` to ensure that the subscription does not auto-update and instead stays pinned to the specified version. ### Skips -In order to skip through certain updates you can specify a list of CSV names to be skipped as such: +In order to skip through certain updates, you can specify a list of operator names to be skipped. For example: ```yaml -metadata: - name: myoperator.v1.0.3 -spec: - replaces: myoperator.v1.0.0 - skips: - - myoperator.v1.0.1 - - myoperator.v1.0.2 +--- +schema: olm.channel +package: myoperator +channel: stable +entries: + - name: myoperator.v1.0.3 + replaces: myoperator.v1.0.0 + skips: + - myoperator.v1.0.1 + - myoperator.v1.0.2 ``` Using the above graph, this will mean subscriptions on `myoperator.v1.0.0` can update directly to @@ -66,49 +74,29 @@ that are already on `myoperator.v1.0.1` or `myoperator.v1.0.2` will still be abl This is particularly useful if `myoperator.v1.0.1` and `myoperator.v1.0.2` are affected by a CVE or contain bugs. -Skipped CSVs do not need to be present in a catalog or set of manifests prior to adding to a catalog. +Skipped operators do not need to be present in a catalog or set of manifests prior to adding to a catalog. ### SkipRange -OLM also allows you to specify updates through version ranges in your CSV. This requires your CSVs +OLM also allows you to specify updates through version ranges in your channel entry. This requires your CSVs to define a version in their version field which must follow the [semver spec](https://semver.org/). Internally, OLM uses the [blang/semver](https://github.com/blang/semver) go library. ```yaml -metadata: - name: myoperator.v1.0.3 - annotations: - olm.skipRange: ">=1.0.0 <1.0.3" +--- +schema: olm.channel +package: myoperator +channel: stable +entries: + - name: myoperator.v1.0.3 + skipRange: ">=1.0.0 <1.0.3" ``` -The version specifying the `olm.skipRange` will be presented as a direct (one hop) update to +The entry specifying the `skipRange` will be presented as a direct (one hop) update to any version from that package within that range. The versions in this range do not need to be in the index in order for bundle addition to be successful. We recommend avoiding using unbound ranges such as `<1.0.3`. -**Warning:** Adding a bundle that only specifies skipRange to a given channel will wipe out all -the previous content in that channel. This means directly installing past versions by editing -the `startingCSV` field of the subscription is not possible when using skiprange only. In order -for past versions to be installable by `startingCSV` while also benefitting from the `skipRange` -feature, you will need to also connect past edges by setting a `replaces` field in addition to -the `olm.skipRange`. For example, assuming the above update graph: - -```txt -myoperator.v1.0.0 -> myoperator.v1.0.1 -> myoperator.v1.0.2 -``` - -In order to keep these versions installable by `startingCSV` when `myoperator.v1.0.3` is added, -the CSV for `myoperator.v1.0.3` needs to have the following: - -```yaml -metadata: - name: myoperator.v1.0.3 - annotations: - olm.skipRange: ">=1.0.0 <1.0.3" -spec: - replaces: myoperator.v1.0.2 -``` - SkipRange by itself is useful for teams who are not interested in supporting directly installing versions within a given range or for whom consumers of the operator are always on the latest version. @@ -118,7 +106,7 @@ version. ### Cross channel updates Cross channel updates are not possible today in an automated way. In order for your subscription -to switch channels, the cluster admin must manually change the `channel` field in the subcription +to switch channels, the cluster admin must manually change the `channel` field in the subscription object. Changing this field does not always result in receiving updates from that channel. For that to @@ -126,16 +114,144 @@ occur, updates paths must be available from the given version to versions in the #### If using replaces -The CSV name currently installed must be in the `replaces` field of a CSV in the new channel. +The CSV name currently installed must be in the `replaces` field of an entry in the new channel. #### If using skips -The CSV name currently installed must be in the `skips` field of a CSV in the new channel. +The CSV name currently installed must be in the `skips` field of an entry in the new channel. #### If using skipRange -The version currently installed must be in the `olm.skipRange` field of a CSV in the new channel. +The version currently installed must be in the `skipRange` field of an entry in the new channel. + +## Channel Promotion + +A popular channel strategy is to use `alpha`, `beta`, and `stable` channels, where every new bundle is added to `alpha`, +a subset of bundles from `alpha` are promoted to `beta`, and a further subset of bundles from `beta` are promoted to +`stable`. + +An operator author wishing to employ the `alpha`/`beta`/`stable` channel strategy may have created and updated their +`olm.channel` blobs following the below scenario. + +First, an operator author releases a few versions into the `alpha` channel, with a simple linear replaces chain. +```yaml +--- +schema: olm.channel +package: myoperator +channel: alpha +entries: + - name: myoperator.v0.1.0 + - name: myoperator.v0.2.0 + replaces: myoperator.v0.1.0 + - name: myoperator.v0.3.0 + replaces: myoperator.v0.2.0 +``` + +Next, an operator author decides that `myoperator.v0.2.0` is a good candidate to promote to `beta`, so they add a new +`olm.channel` schema and include `myoperator.v0.2.0` as the first entry, taking care to preserve any upgrade edges from +`alpha` to enable users to switch from the `alpha` to the `beta` channel if they have `myoperator.v0.1.0` installed. + +> NOTE: This bundle already exists in the index (the image was already built and pushed, and the index already contains +> an `olm.bundle` blob and an entry for it in the `alpha` channel). The channel promotion step for `myoperator.v0.2.0` +> is to simply create a new `olm.channel` for the `beta` channel and include an entry for `myoperator.v0.2.0`. + +```yaml +--- +schema: olm.channel +package: myoperator +channel: alpha +entries: + - name: myoperator.v0.1.0 + - name: myoperator.v0.2.0 + replaces: myoperator.v0.1.0 + - name: myoperator.v0.3.0 + replaces: myoperator.v0.2.0 +--- +schema: olm.channel +package: myoperator +channel: beta +entries: + - name: myoperator.v0.2.0 + replaces: myoperator.v0.1.0 +``` + +The operator continues releasing bundles to `alpha` and promotes `myoperator.v0.4.0` to `beta`. + +> NOTE: In channel `alpha`, `myoperator.v0.4.0` replaces `myoperator.v0.3.0`, but in channel `beta`, `myoperator.v0.4.0` +> replaces `myoperator.v0.2.0` because the `beta` channel does not include every entry from alpha. + +```yaml +--- +schema: olm.channel +package: myoperator +channel: alpha +entries: + - name: myoperator.v0.1.0 + - name: myoperator.v0.2.0 + replaces: myoperator.v0.1.0 + - name: myoperator.v0.3.0 + replaces: myoperator.v0.2.0 + - name: myoperator.v0.4.0 + replaces: myoperator.v0.3.0 + - name: myoperator.v0.5.0 + replaces: myoperator.v0.4.0 +--- +schema: olm.channel +package: myoperator +channel: beta +entries: + - name: myoperator.v0.2.0 + replaces: myoperator.v0.1.0 + - name: myoperator.v0.4.0 + replaces: myoperator.v0.2.0 +``` + +Finally, the operator author releases several more bundles and makes several more promotions, finally deciding to +promote `myoperator.v0.4.0` to the stable channel. With `myoperator.v0.4.0` being the first entry in the `stable` +channel, the operator author has decided to add `replaces` and `skips` edges for all previously released bundles to +assist users in moving to the `stable` channel directly from the `alpha` and `beta` channels. However, it would have +been equally valid to require that a user have either `myoperator.v0.2.0` or `myoperator.v0.3.0` installed (e.g. if the +direct upgrade from `myoperator.v0.1.0` has not been tested or is known to be unsupported). -## Add Modes +```yaml +--- +schema: olm.channel +package: myoperator +channel: alpha +entries: + - name: myoperator.v0.1.0 + - name: myoperator.v0.2.0 + replaces: myoperator.v0.1.0 + - name: myoperator.v0.3.0 + replaces: myoperator.v0.2.0 + - name: myoperator.v0.4.0 + replaces: myoperator.v0.3.0 + - name: myoperator.v0.5.0 + replaces: myoperator.v0.4.0 + - name: myoperator.v0.6.0 + replaces: myoperator.v0.5.0 +--- +schema: olm.channel +package: myoperator +channel: beta +entries: + - name: myoperator.v0.2.0 + replaces: myoperator.v0.1.0 + - name: myoperator.v0.4.0 + replaces: myoperator.v0.2.0 + - name: myoperator.v0.6.0 + replaces: myoperator.v0.4.0 +--- +schema: olm.channel +package: myoperator +channel: stable +entries: + - name: myoperator.v0.4.0 + replaces: myoperator.v0.1.0 + skips: + - myoperator.v0.2.0 + - myoperator.v0.3.0 +``` -TODO +[file-based-catalog-spec]: /docs/reference/file-based-catalogs +[opm-validate-cli]: /docs/reference/file-based-catalogs/#opm-validate diff --git a/content/en/docs/Reference/file-based-catalogs.md b/content/en/docs/Reference/file-based-catalogs.md new file mode 100644 index 00000000..fd02350a --- /dev/null +++ b/content/en/docs/Reference/file-based-catalogs.md @@ -0,0 +1,587 @@ +--- +title: "File-based Catalogs" +linkTitle: "File-based Catalogs" +weight: 4 +date: 2021-07-29 +--- + +File-based catalogs are the latest iteration of OLM's index format. It is a fully plaintext-based (JSON or YAML) +evolution of the previous sqlite database format that is fully backwards compatible. + +## Design +The primary design goal for this format is to enable index editing, composability, and extensibility. + +### Editing + +With file-based catalogs, users interacting with the contents of an index are able to make direct changes to the index +format and verify that their changes are valid. + +Because this format is plaintext JSON or YAML, index maintainers can easily manipulate index metadata by hand or with +widely known and supported JSON or YAML tooling (e.g. `jq`). + +This editability enables features and user-defined extensions, such as: +- Promoting an existing bundle to a new channel +- Changing the default channel of a package +- Custom algorithms for adding, updating, adding removing upgrade edges + +### Composability + +File-based catalogs are stored in an arbitrary directory hierarchy, which enables index composition. If I have two +separate file-based catalog directories, `indexA` and `indexB`, I can make a new combined index by making a new +directory `indexC` and copying `indexA` and `indexB` into it. + +This composability enables decentralized indexes. The format permits operator authors to maintain operator-specific +indexes and index maintainers to trivially build an index composed of individual operator indexes. + +> NOTE: Duplicate packages and duplicate bundles within a package are not permitted. The `opm validate` command will +> return an error if any duplicates are found. + +Since operator authors are most familiar with their operator, its dependencies, and its upgrade compatibility, they are +able to maintain their own operator-specific index and have direct control over its contents. With file-based catalogs, +operator authors own the task of building and maintaining their packages in an index. Composite index maintainers treat +packages as a black box; they own the task of curating the packages in their catalog and publishing the catalog to +users. + +File-based catalogs can be composed by combining multiple other catalogs or by extracting subsets of one catalog, or a +combination of both of these. + +See the [Building a composite catalog](#building-a-composite-catalog) section for a simple example. + +### Extensibility + +The final design goal is to provide extensibility around indexes. The file-based catalog spec is a low-level +representation of an index. While it can be maintained directly in its low-level form, we expect many index maintainers +to build interesting extensions on top that can be used by their own custom tooling to make all sorts of mutations. For +example, one could imagine a tool that translates a high-level API like (mode=semver) down to the low-level file-based +catalog format for upgrade edges. Or perhaps an index maintainer needs to customize all of the bundle metadata by adding +a new property to bundles that meet a certain criteria. + +The OLM developer community will be making use of this extensibility to build more official tooling on top of the +low-level APIs, but the major benefit is that index maintainers have this capability as well. + +## Specification + +### Structure + +File-based catalogs can be stored and loaded from directory-based filesystems. + +`opm` loads the catalog by walking the root directory and recursing into subdirectories. It attempts to load every file +it finds and fails if any errors occur. + +Non-catalog files can be ignored using `.indexignore` files, which behave identically to `.gitignore` files. That is, +they have the same rules for [patterns](https://git-scm.com/docs/gitignore#_pattern_format) and precedence. + +> **Example `.gitignore` file** +> ```gitignore +> # Ignore everything except non-object .json and .yaml files +> **/* +> !*.json +> !*.yaml +> **/objects/*.json +> **/objects/*.yaml +> ``` + + +Index maintainers have the flexibility to chose their desired layout, but the OLM team recommends storing each package's +file-based catalog blobs in separate sub-directories. Each individual file can be either JSON or YAML -- it is not +necessary for every file in an index to use the same format. + +This layout has the property that each sub-directory in the directory hierarchy is a self-contained index, which makes +index composition, discovery, and navigation as simple as trivial filesystem operations. + +> **Basic recommended structure** +> ``` +> index +> ├── pkgA +> │ └── index.yaml +> ├── pkgB +> │ ├── .indexignore +> │ ├── index.yaml +> │ └── objects +> │ └── pkgB.v0.1.0.clusterserviceversion.yaml +> └── pkgC +> └── index.json +> ``` + +This `index` could also be trivially included in a parent index by simply copying it into the parent index's root +directory. + +### Schema + +At its core, file-based catalogs use a simple format that can be extended with arbitrary schemas. The format that all +file-based catalog blobs must adhere to is the `Meta` schema. The below [cue][cuelang-spec] `_Meta` schema defines all +file-based catalog blobs. + +> **NOTE**: No cue schemas listed in this specification should be considered exhaustive. The `opm validate` command has +> additional validations that are difficult/impossible to express concisely in cue. + +```cue +_Meta: { + // schema is required and must be a non-empty string + schema: string & !="" + + // package is optional, but if it's defined, it must be a non-empty string + package?: string & !="" + + // properties is optional, but if it's defined, it must be a list of 0 or more properties + properties?: [... #Property] +} + +#Property: { + // type is required + type: string & !="" + + // value is required, and it must not be null + value: !=null +} +``` + +### OLM-defined schemas + +An OLM index currently uses three schemas: `olm.package`, `olm.channel`, and `olm.bundle`, which correspond to OLM's +existing package and bundle concepts. + +Each operator package in an index requires exactly one `olm.package` blob, at least one `olm.channel` blob, and one or +more `olm.bundle` blobs. + +> **NOTE**: All `olm.*` schemas are reserved for OLM-defined schemas. Custom schemas must use a unique prefix (e.g. a +> domain that you own). + +#### `olm.package` + +An `olm.package` defines package-level metadata for an operator. This includes its name, description, default channel +and icon. + +The `olm.package` [cue][cuelang-spec] schema is: +```cue +#Package: { + schema: "olm.package" + + // Package name + name: string & !="" + + // A description of the package + description?: string + + // The package's default channel + defaultChannel: string & !="" + + // An optional icon + icon?: { + base64data: string + mediatype: string + } +} +``` + +#### `olm.channel` + +An `olm.channel` defines a channel within a package, the bundle entries that are members +of the channel, and the upgrade edges for those bundles. + +A bundle can included as an entry in multiple `olm.channel` blobs, but it can have only one entry per channel. + +Also, it is valid for an entry's replaces value to reference another bundle name that cannot be found in this index +(or even another index) as long as other channel invariants still hold (e.g. a channel cannot have multiple heads). + +The `olm.channel` [cue][cuelang-spec] schema is: +```cue +#Channel: { + schema: "olm.channel" + package: string & !="" + name: string & !="" + entries: [...#ChannelEntry] +} + +#ChannelEntry: { + // name is required. It is the name of an `olm.bundle` that + // is present in the channel. + name: string & !="" + + // replaces is optional. It is the name of bundle that is replaced + // by this entry. It does not have to be present in the entry list. + replaces?: string & !="" + + // skips is optional. It is a list of bundle names that are skipped by + // this entry. The skipped bundles do not have to be present in the + // entry list. + skips?: [...string & !=""] + + // skipRange is optional. It is the semver range of bundle versions + // that are skipped by this entry. + skipRange?: string & !="" +} +``` + +For more information about defining upgrade edges, see the [upgrade graph reference documentation][upgrade-graph-doc]. + +[upgrade-graph-doc]: /docs/concepts/olm-architecture/operator-catalog/creating-an-update-graph + +#### `olm.bundle` + + + +The `olm.bundle` cue schema is: +```cue +#Bundle: { + schema: "olm.bundle" + package: string & !="" + name: string & !="" + image: string & !="" + properties: [...#Property] + relatedImages?: [...#RelatedImage] +} + +#Property: { + // type is required + type: string & !="" + + // value is required, and it must not be null + value: !=null +} + +#RelatedImage: { + // image is the image reference + image: string & !="" + + // name is an optional descriptive name for an image that + // helps identify its purpose in the context of the bundle + name?: string & !="" +} +``` + +### Properties + +Properties are arbitrary pieces of metadata that can be attached to file-based catalog schemas. The type field is a +string that effectively specifies the semantic and syntactic meaning of the value field. The value can be any arbitrary +JSON/YAML. + + +OLM defines a handful of property types, again using the reserved `olm.*` prefix. + +#### `olm.package` + +An `olm.package` property defines the package name and version. This is a required property on bundles, and there must +be exactly one of these properties. The `packageName` must match the bundle's first-class `package` field, and the +`version` must be a valid [semantic version][semver] + +The `olm.package` property [cue][cuelang-spec] schema is: +```cue +#PropertyPackage: { + type: "olm.package" + value: { + packageName: string & !="" + version: string & !="" + } +} +``` + +#### `olm.gvk` + +An `olm.gvk` property defines the group, version, and kind (GVK) of a Kubernetes API that is provided by this bundle. +This property is used by OLM to resolve a bundle with this property as a dependency for other bundles that list the same +GVK as a required API. The GVK must adhere to Kubernetes GVK validations. + +The `olm.gvk` property [cue][cuelang-spec] schema is: +```cue +#PropertyGVK: { + type: "olm.gvk" + value: { + group: string & !="" + version: string & !="" + kind: string & !="" + } +} +``` + +#### `olm.package.required` + +An `olm.package.required` property defines the package name and version range of another package that this bundle +requires. For every required package property a bundle lists, OLM will ensure there is an operator installed on the +cluster for the listed package and in the required version range. The `versionRange` field must be a valid +[semver range][semver-range]. + +The `olm.package.required` property [cue][cuelang-spec] schema is: +```cue +#PropertyPackageRequired: { + type: "olm.package.required" + value: { + packageName: string & !="" + versionRange: string & !="" + } +} +``` + + +#### `olm.gvk.required` + +An `olm.gvk.required` property defines the group, version, and kind (GVK) of a Kubernetes API that this bundle requires. +For every required GVK property a bundle lists, OLM will ensure there is an operator installed on the cluster that +provides it. The GVK must adhere to Kubernetes GVK validations. + +The `olm.gvk.required` property [cue][cuelang-spec] schema is: +```cue +#PropertyGVKRequired: { + type: "olm.gvk.required" + value: { + group: string & !="" + version: string & !="" + kind: string & !="" + } +} +``` + +#### `olm.bundle.object` (alpha) + +`olm.bundle.object` properties are used to inline (or reference) a bundle's manifests directly in the index. + +> **NOTE**: Core OLM does not require `olm.bundle.object` properties to be included on bundles. However, the OLM Package +> Server (used by tooling such as the kubectl operator plugin and the OpenShift console) does require these properties +> to be able to serve metadata about the packages in an index. +> +> This property is in _alpha_ because it will likely be rendered obsolete when updates can be made to the OLM Package +> Server to no longer require manifests in the index. + +A bundle object property can contain inlined data using the `value.data` field, which must be the base64-encoded string +of that manifest. + +Alternately, a bundle object property can be a reference to a file relative to the location of the file in which the +bundle is declared. Any referenced files must be within the catalog root. + +The `olm.bundle.object` property [cue][cuelang-spec] schema is: +```cue + +#PropertyBundleObject: { + type: "olm.bundle.object" + value: #propertyBundleObjectRef | #propertyBundleObjectData +} + +#propertyBundleObjectRef: { + ref: string & !="" +} + +#propertyBundleObjectData: { + data: string & !="" +} +``` + +[cuelang-spec]: https://cuelang.org/docs/references/spec/ +[semver]: https://semver.org/spec/v2.0.0.html +[semver-range]: https://github.com/blang/semver/blob/master/README.md#ranges + +## CLI + + + +### `opm init` + +``` +Generate an olm.package declarative config blob + +Usage: + opm init [flags] + +Flags: + -c, --default-channel string The channel that subscriptions will default to if unspecified + -d, --description string Path to the operator's README.md (or other documentation) + -h, --help help for init + -i, --icon string Path to package's icon + -o, --output string Output format (json|yaml) (default "json") + +Global Flags: + --skip-tls skip TLS certificate verification for container image registries while pulling bundles or index +``` + +### `opm render` + +``` +Generate a declarative config blob from the provided index images, bundle images, and sqlite database files + +Usage: + opm render [index-image | bundle-image | sqlite-file]... [flags] + +Flags: + -h, --help help for render + -o, --output string Output format (json|yaml) (default "json") + +Global Flags: + --skip-tls skip TLS certificate verification for container image registries while pulling bundles or index +``` + +### `opm validate` + +``` +Validate the declarative config JSON file(s) in a given directory + +Usage: + opm validate [flags] + +Flags: + -h, --help help for validate + +Global Flags: + --skip-tls skip TLS certificate verification for container image registries while pulling bundles or index +``` + +### `opm serve` + +``` +This command serves declarative configs via a GRPC server. + +NOTE: The declarative config directory is loaded by the serve command at +startup. Changes made to the declarative config after the this command starts +will not be reflected in the served content. + +Usage: + opm serve [flags] + +Flags: + --debug enable debug logging + -h, --help help for serve + -p, --port string port number to serve on (default "50051") + -t, --termination-log string path to a container termination log file (default "/dev/termination-log") + +Global Flags: + --skip-tls skip TLS certificate verification for container image registries while pulling bundles or index +``` + +### `opm alpha diff` + +``` +Diff a set of old and new catalog references ("refs") to produce a declarative config containing only packages channels, and versions not present in the old set, and versions that differ between the old and new sets. This is known as "latest" mode. These references are passed through 'opm render' to produce a single declarative config. + + This command has special behavior when old-refs are omitted, called "heads-only" mode: instead of the output being that of 'opm render refs...' (which would be the case given the preceding behavior description), only the channel heads of all channels in all packages are included in the output, and dependencies. Dependencies are assumed to be provided by either an old ref, in which case they are not included in the diff, or a new ref, in which case they are included. Dependencies provided by some catalog unknown to 'opm alpha diff' will not cause the command to error, but an error will occur if that catalog is not serving these dependencies at runtime. + +Usage: + opm alpha diff [old-refs]... new-refs... [flags] + +Examples: + # Diff a catalog at some old state and latest state into a declarative config index. + mkdir -p catalog-index + opm alpha diff registry.org/my-catalog:abc123 registry.org/my-catalog:def456 -o yaml > ./my-catalog-index/index.yaml + + # Build and push this index into an index image. + opm alpha generate dockerfile ./my-catalog-index + docker build -t registry.org/my-catalog:latest-abc123-def456 -f index.Dockerfile . + docker push registry.org/my-catalog:latest-abc123-def456 + + # Create a new catalog from the heads of an existing catalog, then build and push the image like above. + opm alpha diff registry.org/my-catalog:def456 -o yaml > my-catalog-index/index.yaml + docker build -t registry.org/my-catalog:headsonly-def456 -f index.Dockerfile . + docker push registry.org/my-catalog:headsonly-def456 + +Flags: + --ca-file string the root Certificates to use with this command + --debug enable debug logging + -h, --help help for diff + -o, --output string Output format (json|yaml) (default "yaml") + +Global Flags: + --skip-tls skip TLS certificate verification for container image registries while pulling bundles or index +``` + +### `opm alpha generate dockerfile` + +``` +Generate a Dockerfile for a declarative config index. + +This command creates a Dockerfile in the same directory as the +(named .Dockerfile) that can be used to build the index. If a +Dockerfile with the same name already exists, this command will fail. + +When specifying extra labels, note that if duplicate keys exist, only the last +value of each duplicate key will be added to the generated Dockerfile. + +Usage: + opm alpha generate dockerfile [flags] + +Flags: + -i, --binary-image string Image in which to build catalog. (default "quay.io/operator-framework/upstream-opm-builder") + -l, --extra-labels strings Extra labels to include in the generated Dockerfile. Labels should be of the form 'key=value'. + -h, --help help for dockerfile + +Global Flags: + --skip-tls skip TLS certificate verification for container image registries while pulling bundles or index +``` + +## Guidelines + +### Immutable bundles + +OLM's general advice is that bundle images and their metadata should be treated as immutable. If a broken bundle has +been pushed to an index, you must assume that at least one of your users has upgraded to that bundle. Based on that +assumption, you must release another bundle with an upgrade edge from the broken bundle to ensure users with the broken +bundle installed receive an upgrade. OLM will not reinstall an installed bundle if the contents of that bundle are +updated in the index. + +However, there are some cases where a change in the index metadata is preferred. For example: +- Channel promotion - if you already released a bundle and later decide that you'd like to add it to another channel, + simply add an entry for your bundle in another `olm.channel` blob. +- New upgrade edges - if you release a new 1.2.z (e.g. 1.2.4), but 1.3.0 is already released, you can update the index + metadata for 1.3.0 to skip 1.2.4. + +### Use of source control + +OLM highly recommends storing index metadata in source control and treating the source-controlled metadata as the source +of truth. Updates to index images should: +- Update the source-controlled index directory with a new commit. +- Build and push the index image. OLM suggests using a consistent tagging taxonomy (e.g. `:latest` or + `:` so that users can receive updates to an index as they become available. + + + +## Examples + +### Building a composite catalog + +With file-based catalogs, catalog maintainers can focus on operator curation and compatibility. +Since operator authors have already produced operator-specific indexes for their operators, catalog +maintainers can build their catalog simply by rendering each operator index into a subdirectory of the +catalog's root index directory. + +There are many possible ways to build a catalog, but an extremely simple approach would be to: + +- Maintain a single configuration file containing image references for each operator in the catalog + ```yaml + name: community-operators + repo: quay.io/community-operators/catalog + tag: latest + references: + - name: etcd-operator + image: quay.io/etcd-operator/index@sha256:5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03 + - name: prometheus-operator + image: quay.io/prometheus-operator/index@sha256:e258d248fda94c63753607f7c4494ee0fcbe92f1a76bfdac795c9d84101eb317 + ``` + +- Run a simple script that parses this file and creates a new catalog from its references + ```bash + name=$(yq eval '.name' catalog.yaml) + mkdir "$name" + yq eval '.name + "/" + .references[].name' catalog.yaml | xargs mkdir + for l in $(yq e '.name as $catalog | .references[] | .image + "|" + $catalog + "/" + .name + "/index.yaml"' catalog.yaml); do + image=$(echo $l | cut -d'|' -f1) + file=$(echo $l | cut -d'|' -f2) + opm render "$image" > "$file" + done + opm alpha generate dockerfile "$name" + indexImage=$(yq eval '.repo + ":" + .tag' catalog.yaml) + docker build -t "$indexImage" -f "$name.Dockerfile" . + docker push "$indexImage" + ``` + +## Automation + +Operator authors and catalog maintainers are encouraged to automate their index maintenance with CI/CD workflows. +Catalog maintainers could further improve on this by building Git-ops automation that: +- Checks that PR authors are permitted to make the requested changes (e.g. updating their package's image reference) +- Checks that the index updates pass `opm validate` +- Checks that the updated bundle and/or index image reference(s) exist, the index images run successfully in a cluster, + and operators from that package can be successfully installed. +- Automatically merges PRs that pass these checks. +- Automatically rebuilds and republishes the index image. diff --git a/content/en/docs/Tasks/creating-an-index.md b/content/en/docs/Tasks/creating-an-index.md index f8f3aff5..4d3670f7 100644 --- a/content/en/docs/Tasks/creating-an-index.md +++ b/content/en/docs/Tasks/creating-an-index.md @@ -5,37 +5,123 @@ description: > Add/Remove a collection of bundles to/from an Index --- -## Prerequisites +## Prerequisites -- [opm](https://github.com/operator-framework/operator-registry/releases) `v1.14.0+` -- [docker](https://docs.docker.com/install/) version `17.03`+ or [podman](https://github.com/containers/libpod/blob/master/install.md) `v1.2.0+` or [buildah](https://github.com/containers/buildah/blob/master/install.md) `v1.7+`. +- [opm](https://github.com/operator-framework/operator-registry/releases) `v1.19.0+` +## Creating an Index -# Creating an Index +`OLM`'s `CatalogSource` [CRD][catalogsource-crd] accepts a container image reference to a catalog of operators that can +be made available to install in a cluster. You can make your operator bundle available to install in a cluster by adding +it to a catalog, packaging the catalog in a container image, and then using that image reference in the `CatalogSource`. +This image contains all of the metadata required for OLM to manage the lifecycles of all of the operators it contains. -`OLM`'s `CatalogSource` [CRD][catalogsource-crd] define a reference to a catalog of operators that are available to install onto a cluster. To make your operator bundle available, you can add the bundle to a container image which the `CatalogSource` points to. This image contains a record of bundle images that OLM can pull and extract the manifests from in order to install an operator. +OLM uses a plaintext [file-based catalog][file-based-catalog-spec] format (JSON or YAML) to store these records in an index, and `opm` has tooling +that helps initialize an index, render new records into it, and then validate that the index is valid. Let's walk +through a simple example. ->Note: The container image also contains information that represents the upgrade graphs between different operator versions, an operator's dependencies etc graphically. To learn more about the upgrade graph of an operator, checkout the [creating an upgrade graph doc][upgrade-graph-doc] +First, we need to initialize our index, so we'll make a directory for it, generate a Dockerfile that can build an index +image, and then populate our index with our package definition. -So, to make your operator available to OLM, you can generate an index image via opm with your bundle reference included: +### Initialize the index ```sh -$ opm index add --bundles quay.io/my-container-registry-namespace/my-manifest-bundle:0.0.1 --tag quay.io/my-container-registry-namespace/my-index:1.0.0 -$ podman push quay.io/my-container-registry-namespace/my-index:1.0.0 +$ mkdir example-operator-index +$ opm alpha generate dockerfile example-operator-index +$ opm init example-operator \ + --default-channel=preview \ + --description=./README.md \ + --icon=./example-operator.svg \ + --output yaml > example-operator-index/index.yaml ``` -The resulting image is referred to as an `Index`. Now that image is available for clusters to use and reference with `CatalogSources` on their cluster. +Let's validate our index to see if we're ready to ship! +```sh +$ opm validate example-operator-index +FATA[0000] invalid index: +└── invalid package "example-operator": + └── invalid channel "preview": + └── channel must contain at least one bundle +``` + +Alright, so it's not valid. It looks like we need to add a bundle, so let's do +that next... + +### Add a bundle to the index + +```sh +$ opm render quay.io/example-inc/example-operator-bundle:v0.1.0 \ + --output=yaml >> example-operator-index/index.yaml +``` + +Let's validate again: +``` +$ opm validate example-operator-index +FATA[0000] package "example-operator", bundle "example-operator.v0.1.0" not found in any channel entries +``` + +### Add a channel entry for the bundle + +We rendered the bundle, but we still haven't yet added it to any channels. +Let's initialize a channel: +```sh +cat << EOF >> example-operator-index/index.yaml +--- +schema: olm.channel +package: example-operator +name: preview +entries: + - name: example-operator.v0.1.0 +EOF +``` -`Index` images are additive, so you can add a new version of your operator bundle when you publish a new version of your operator: +Is the third time the charm for `opm validate`? -```bash -$ opm index add --bundles quay.io/my-container-registry-namespace/my-manifest-bundle:0.0.2 --from-index quay.io/my-container-registry-namespace/my-index:1.0.0 --tag quay.io/my-container-registry-namespace/my-index:1.0.1 +```sh +$ opm validate example-operator-index +$ echo $? +0 ``` -### Other operations on an Index using `opm` +Success! There were no errors and we got a `0` error code. + +In the general case, adding a bundle involves three discreet steps: +- Render the bundle into the index using `opm render `. +- Add the bundle into desired channels and update the channels' upgrade edges + to stitch the bundle into the correct place. +- Validate the resulting index. + +> NOTE: Index metadata should be stored in a version control system (e.g. `git`) and index images should be rebuilt from source +whenever updates are made to ensure that all index changes are auditable. + +**Step 1** is just a simple `opm render` command. + +**Step 2** has no defined standards other than that the result must pass validation in step 3. Some operator authors may +decide to hand edit channels and upgrade edges. Others may decide to implement automation (e.g. to idempotently +build semver-based channels and upgrade graphs based solely on the versions of the operators in the package). There is +no right or wrong answer for implementing this step as long as `opm validate` is successful. + +There are some guidelines to keep in mind though: +- Once a bundle is present in an index, you should assume that one of your users has installed it. With that in mind, + you should take care to avoid stranding users that have that version installed. Put another way, make sure that + all previously published bundles in an index have a path to the current/new channel head. +- Keep the semantics of the upgrade edges you use in mind. `opm validate` is not able to tell you if you have a sane + upgrade graph. To learn more about the upgrade graph of an operator, checkout the + [creating an upgrade graph doc][upgrade-graph-doc]. + +### Build and push the index image + +Lastly, we can build and push our index: + +```sh +$ docker build . \ + -f example-operator-index.Dockerfile \ + -t quay.io/example-inc/example-operator-index:latest +$ docker push quay.io/example-inc/example-operator-index:latest +``` -The `opm index` command contains additional sub-commands that can be used to perform different operations like remove an operator from an index, prune an index of all but specified operators etc. Please checkout the documentation under `opm index -h` for more information. - +Now the index image is available for clusters to use and reference with `CatalogSources` on their cluster. [catalogsource-crd]: /docs/concepts/crds/catalogsource -[upgrade-graph-doc]: /docs/concepts/olm-architecture/operator-catalog/creating-an-update-graph \ No newline at end of file +[file-based-catalog-spec]: /docs/reference/file-based-catalogs +[upgrade-graph-doc]: /docs/concepts/olm-architecture/operator-catalog/creating-an-update-graph diff --git a/content/en/docs/advanced-tasks/communicating-operator-conditions-to-olm.md b/content/en/docs/advanced-tasks/communicating-operator-conditions-to-olm.md index 3600c007..d03cddf7 100644 --- a/content/en/docs/advanced-tasks/communicating-operator-conditions-to-olm.md +++ b/content/en/docs/advanced-tasks/communicating-operator-conditions-to-olm.md @@ -22,7 +22,7 @@ OLM introduced a new [CustomResourceDefinition](https://kubernetes.io/docs/conce The `Upgradeable` "OLM Supported Condition" prevents the existing CSV from being replaced by a newer version of the CSV. When the `Upgradeable` condition is set to `False`, OLM will: -* Prevent a CSV that replaces the operator's existing CSV from leaving the PendingPhase. +* Prevent a channel entry in a subscribed package that replaces the operator's existing CSV from leaving the PendingPhase. The `Upgradeable` condition might be useful when: diff --git a/content/en/docs/contribution-guidelines/upgrade-graphs.md b/content/en/docs/contribution-guidelines/upgrade-graphs.md index 02dd81d3..fb92de29 100644 --- a/content/en/docs/contribution-guidelines/upgrade-graphs.md +++ b/content/en/docs/contribution-guidelines/upgrade-graphs.md @@ -56,21 +56,21 @@ flowchart TB classDef installed fill:#34ebba; A(v0.0.1) --> B(v0.0.4) -{{}} | An upgrade path to replace one operator bundle version for another using the [CSV][csv-definition] `replaces spec. (eg.`spec.replaces: exampleoperator.v0.0.1`). More info: [here][upgrade-path-replaces]. +{{}} | An upgrade path to replace one operator bundle version for another using the [`olm.channel`][olm-channel] `replaces` field. More info: [here][upgrade-path-replaces]. | {{}} flowchart TB classDef head fill:#ffbfcf; classDef installed fill:#34ebba; A(v0.0.1) x--x |v0.0.2,v0.0.3| B(v0.0.4) -{{}} | An upgrade path to skip versions in the upgrade path using the [CSV][csv-definition] `skips` spec. (eg.`spec.skips: exampleoperator.v0.0.2, exampleoperator.v0.0.3`). More info: [here][upgrade-path-skips]. | +{{}} | An upgrade path to skip versions in the upgrade path using the [`olm.channel`][olm-channel] `skips` field. More info: [here][upgrade-path-skips]. | | {{}} flowchart TB classDef head fill:#ffbfcf; classDef installed fill:#34ebba; C(v2.0.3) o--o |>= 2.0.4 < 3.0.0| D(v3.0.1) -{{}} | An upgrade path to skip a range of operator bundle versions using the [CSV][csv-definition] `olm.skipRange` annotation. (E.g `annotations.olm.skipRange: '>= 2.0.4 < 3.0.0'`) More info: [here][upgrade-path-skiprange]. | +{{}} | An upgrade path to skip a range of operator bundle versions using the [`olm.channel`][olm-channel] `skipRange` field. More info: [here][upgrade-path-skiprange]. | | {{}} flowchart TB classDef head fill:#ffbfcf; @@ -235,7 +235,7 @@ flowchart TB E(v0.0.2 \n fa:fa-tag label=value) {{}} | -[csv-definition]:/docs/concepts/crds/clusterserviceversion/ +[olm-channel]:/docs/reference/file-based-catalogs/#olmchannel [upgrade-path-replaces]:/docs/concepts/olm-architecture/operator-catalog/creating-an-update-graph/#replaces [upgrade-path-skips]:/docs/concepts/olm-architecture/operator-catalog/creating-an-update-graph/#skips [upgrade-path-skiprange]:/docs/concepts/olm-architecture/operator-catalog/creating-an-update-graph/#skiprange \ No newline at end of file