Propose convention for storing operator bundle #27
Conversation
openshift-ci-robot
added
the
size/L
|
On first pass this looks pretty reasonable to me. I do notice that while OCI containers are referenced the majority of the examples rely on the In terms of the format version in use, how would one figure that out? Would there be a specific label associated with the formats version that someone would use |
|
/assign @jwforres |
| * The value `manifests+metadata` implies that this bundle contains both operator metadata and manifets. | ||
| * The label `operators.coreos.com.bundle.mediatype` reflects the media type of the operator bundle. It could be helm charts, plain kubernetes manifests etc. | ||
|
|
||
| The labels will also be put inside a YAML file, as shown below. |
There was a problem hiding this comment.
which is authoritative if they do not match?
There was a problem hiding this comment.
Ideally, we would like to have these annotations defined in one place. In case of a mismatch, the annotations.yaml file is authoritative because on-cluster operator-registry ( component of olm ) that relies on these annotations has access to the yaml file only.
The potential use case for the LABELS is - an external off-cluster tool can inspect the image to check the type of bundle image without downloading the content.
I will add this info to the proposal.
There was a problem hiding this comment.
@tkashem I think we should add the reasoning for why OLM doesn't just use the labels itself. As I understand it, there are certain rules around how OpenShift components interact with container images, and that precludes inspecting images directly -- does that sound correct?
olm/operator-bundle.md
Outdated
| ADD test/annotations.yaml /metadata/annotations.yaml | ||
| ``` | ||
|
|
||
| Below is the directory layout of the operator bundle insdie the image. |
There was a problem hiding this comment.
will be fixed in the next push.
| ```bash | ||
| $ tree | ||
| / | ||
| ├── manifests |
There was a problem hiding this comment.
Can you describe what you would do with this? It doesn't look like a straight apply of the directory content. It is simply free-form?
There was a problem hiding this comment.
In this example, I have used the format used by operator-registry - https://github.com/operator-framework/operator-registry#manifest-format. This is the format olm uses to package operator manifests today. The directory content in this case is not directly kubectl applyable. While the operator is being installed, olm manipulates the content before applying it to the cluster.
I think, we can also package content in the operator bundle image that are straight apply.
There was a problem hiding this comment.
In this example, I have used the format used by operator-registry - https://github.com/operator-framework/operator-registry#manifest-format. This is the format olm uses to package operator manifests today. The directory content in this case is not directly
kubectl applyable. While the operator is being installed,olmmanipulates the content before applying it to the cluster.I think, we can also package content in the operator bundle image that are straight
apply.
please include a link for reference.
There was a problem hiding this comment.
The directory content in this case is not directly kubectl applyable
The content should be kubectl applyable, right?
There was a problem hiding this comment.
It would be great if it supported arbitrary resources. A challenge SRE has faced developing operators for managing OSD is how to get all required resources installed. We have a limited set of resources we can include in a grpc bundle today. Resources off the top of my head that we can't bundle but need to deploy w/ the operator: Service, ServiceMonitor, PrometheusRule
olm/operator-bundle.md
Outdated
| operators.coreos.com.bundle.mediatype: bundle | ||
| ``` | ||
|
|
||
| The package manifest YAML file that defines a set of `channel`(s) will not be included in the bundle. Operator registry will provide machineries to define `package` and `channel` that can be used while index is being built. |
There was a problem hiding this comment.
can you link out to the channel and package you're describing?
There was a problem hiding this comment.
This also doesn't explain what the index is - or link to something that explains the index concept. I'd be fine linking to another enhancement proposal PR for now if that is where we describe the index.
There was a problem hiding this comment.
can you link out to the channel and package you're describing?
- https://github.com/operator-framework/operator-lifecycle-manager/blob/master/doc/design/how-to-update-operators.md
- https://github.com/operator-framework/operator-lifecycle-manager#discovery-catalogs-and-automated-upgrades
- https://github.com/operator-framework/operator-lifecycle-manager/blob/master/doc/design/architecture.md#catalog-registry-design
I will add the links to the proposal also.
This also doesn't explain what the index is - or link to something that explains the index concept. I'd be fine linking to another enhancement proposal PR for now if that is where we describe the index.
Enhancement proposal that covers index - #24.
We still have some open questions whether channel and package definitions should be included inside the operator bundle image.
olm/operator-bundle.md
Outdated
|
|
||
| ##### Build, Push, Pull Operator Bundle Image | ||
| We use the following labels to annotate the operator bundle image. | ||
| * The label `operators.coreos.com.bundle.resources` represents the bundle type: |
There was a problem hiding this comment.
Is this a good opportunity for us to switch to an operator framework based API domain, operators.operatorframework.io or something. Ideas?
There was a problem hiding this comment.
operators.operatorframework.io sounds good to me. I am happy to update the proposal with this new domain.
At some point, we need to plan to version our APIs to be consistent across?
olm/operator-bundle.md
Outdated
| operators.coreos.com.bundle.mediatype: bundle | ||
| ``` | ||
|
|
||
| The package manifest YAML file that defines a set of `channel`(s) will not be included in the bundle. Operator registry will provide machineries to define `package` and `channel` that can be used while index is being built. |
There was a problem hiding this comment.
This also doesn't explain what the index is - or link to something that explains the index concept. I'd be fine linking to another enhancement proposal PR for now if that is where we describe the index.
| ```bash | ||
| cat <<EOF | kubectl apply -f - | ||
| apiVersion: operators.coreos.com/v2alpha1 | ||
| kind: Operator |
There was a problem hiding this comment.
What component would watch for this Operator resource and translate this into a running operator.
There was a problem hiding this comment.
This example shows how a user can run the operator using olm. This is being covered here - #28.
There was a problem hiding this comment.
@tkashem I think it would also be useful to write out manual application by pulling the bundle image locally, unpacking it, and applying to the cluster.
olm/operator-bundle.md
Outdated
| $ cd test | ||
|
|
||
| # the following command generates the necessary scaffolding. | ||
| $ operator-framework bundle init --type=operator-registry --bundle-folder=0.1.0 |
There was a problem hiding this comment.
Do we have existing values for this --type flag today? If so what are they? I'm not convinced that operator-registry is the right thing to put here, but I'm missing some context to say what might be better.
There was a problem hiding this comment.
https://github.com/operator-framework/operator-registry#manifest-format - operator registry bundle format is what olm uses today to pack/unpack operators. We wanted to take an incremental approach:
- Quick win: package operator-registry into the bundle image, this gets us off of
app-registrywith relatively less efforts and sooner. - Long term: We can focus our efforts on a different format where operator metadata and manifests are separately defined and stored.
csvis decomposed into plain kube manifests. We can also focus on adding support for other packaging format like helm.
The values for --type can be - operator-registry, plain ( straight apply kube manifets ), helm.
Good point, we can use |
olm/operator-bundle.md
Outdated
| $ cd test | ||
|
|
||
| # the following command generates the necessary scaffolding. | ||
| $ operator-framework bundle init --type=operator-registry --bundle-folder=0.1.0 |
There was a problem hiding this comment.
If we merge this proposal are we committing to the operator-framework cli name?
There was a problem hiding this comment.
No, it's just an example. :) It's an open question whether we have a new cli, or use operator-sdk or operator-registry.
tkashem
force-pushed
the
operator-bundle
branch
5 times, most recently
from
2c0c623
to
dec2726
Compare
olm/operator-bundle.md
Outdated
| We use the following labels to annotate the operator bundle image. | ||
| * The label `operators.operatorframework.io.bundle.resources` represents the bundle type: | ||
| * The value `manifests` implies that this bundle contains operator manifests only. | ||
| * The value `metadada` implies that this bundle has operator metadata only. |
There was a problem hiding this comment.
| * The value `metadada` implies that this bundle has operator metadata only. | |
| * The value `metadata` implies that this bundle has operator metadata only. |
olm/operator-bundle.md
Outdated
| * The label `operators.operatorframework.io.bundle.resources` represents the bundle type: | ||
| * The value `manifests` implies that this bundle contains operator manifests only. | ||
| * The value `metadada` implies that this bundle has operator metadata only. | ||
| * The value `manifests+metadata` implies that this bundle contains both operator metadata and manifets. |
There was a problem hiding this comment.
| * The value `manifests+metadata` implies that this bundle contains both operator metadata and manifets. | |
| * The value `manifests+metadata` implies that this bundle contains both operator metadata and manifests. |
olm/operator-bundle.md
Outdated
| By standardizing on a container format for this data, we get many other features for free, such as: identity, distribution, replication, deduplication, signing, and ingress. | ||
|
|
||
| ### Goals | ||
| * Define a convention for storing operator manifetss and metadata with container image. |
There was a problem hiding this comment.
| * Define a convention for storing operator manifetss and metadata with container image. | |
| * Define a convention for storing operator manifests and metadata with container image. |
olm/operator-bundle.md
Outdated
| * The `/manifests` folder is expected to contain resources that can be applied to the cluster using standard tooling like `kubectl`. | ||
| * The `/metadata` folder is expected to contain resources that are not directly appliable. It can be used to store supporting metadata associated with the operator. | ||
| * The image is not runnable, it is built from `scratch`. | ||
| * The first image layer is expected to contain all of `/manifests` |
There was a problem hiding this comment.
What's the reason for this requirement?
There was a problem hiding this comment.
I removed it, so unpacking the bundle content will now involve going through each image layer and extracting it.
| $ cd test | ||
|
|
||
| # the following command generates the necessary scaffolding. | ||
| $ operator-framework bundle init --type=registry --bundle-folder=0.1.0 |
There was a problem hiding this comment.
Why not add this to operator-sdk?
There was a problem hiding this comment.
It will be - we're building the libraries and tooling so that operator-sdk can just wrap them. I believe this is just a strawman, but we'll likely have an entry point outside of sdk as well.
olm/operator-bundle.md
Outdated
| Once the scaffolding is generated the user can do a `docker build` to create an operator bundle image. | ||
|
|
||
| #### Validate an Operator Bundle | ||
| As an operator author I want to validate an operator bundle image so that I can ensure it runs as expected on a cluster. |
There was a problem hiding this comment.
As stated above, these images do not and cannot run, so consider rewording this. You could just leave it at "As an operator author I want to validate an operator bundle image."
There was a problem hiding this comment.
it's a typo, it should be I can ensure my operator runs as expected on a cluster.
olm/operator-bundle.md
Outdated
| ``` | ||
|
|
||
| The validate command will do the following: | ||
| * Make sure the image `label` and `annotations.yaml` are appropriately configured. If there is any mismatch, the tool shoud generate apropriate error message. |
| ├── 0.1.0 | ||
| │ ├── testbackup.crd.yaml | ||
| │ ├── testcluster.crd.yaml | ||
| │ ├── testoperator.v0.1.0.clusterserviceversion.yaml |
There was a problem hiding this comment.
Do CSVs still include the "placeholder" namespace? This seems to assume that's not a problem in this example.
There was a problem hiding this comment.
We have a separate effort to enable "directly applying CSVs" that this section assumes is already in place.
But I think the example probably should include a namespace flag?
olm/operator-bundle.md
Outdated
| * The label `operators.operatorframework.io.bundle.resources` represents the bundle type: | ||
| * The value `manifests` implies that this bundle contains operator manifests only. | ||
| * The value `metadada` implies that this bundle has operator metadata only. | ||
| * The value `manifests+metadata` implies that this bundle contains both operator metadata and manifets. |
There was a problem hiding this comment.
What's the use case for having a bundle with only one or the other? It seems like not a lot of data; why not "bundle" it together all the time and not worry about this difference?
There was a problem hiding this comment.
The spec keeps room for separation, on the other hand our implementation which is already in progress "bundles" all of manifests/metadata together.
This enhancement proposes standards and conventions for storing operator manifests and metadata as conatiner images.
tkashem
force-pushed
the
operator-bundle
branch
from
dec2726
to
909628c
Compare
| * The label `operators.operatorframework.io.bundle.resources` represents the bundle type: | ||
| * The value `manifests` implies that this bundle contains operator manifests only. | ||
| * The value `metadata` implies that this bundle has operator metadata only. | ||
| * The value `manifests+metadata` implies that this bundle contains both operator metadata and manifests. |
There was a problem hiding this comment.
If there's not a direct need for this now, I suggest leaving it out of the initial implementation. I'm sure you will version the format you're creating, and if this turns out to be a valuable distinction, you can add it in a later version.
Otherwise, this adds a fair amount of complexity that may not be worthwhile.
There was a problem hiding this comment.
I think this falls under the inspect portion of the Build, Push, Pull Operator Bundle story. It's necessary if a user wants to identify a bundle image before pulling its layers.
There was a problem hiding this comment.
I agree with Michael, this is not something that we need to be focusing on.
|
|
||
| We would like to be able to talk about a set of metadata and manifests, outside the context of a cluster, as representing a particular application or service (in this case, an operator). | ||
|
|
||
| By standardizing on a container format for this data, we get many other features for free, such as: identity, distribution, replication, deduplication, signing, and ingress. |
There was a problem hiding this comment.
... on a container format ...
Would "on an OCI-compliant format" be more clear?
| By standardizing on a container format for this data, we get many other features for free, such as: identity, distribution, replication, deduplication, signing, and ingress. | ||
|
|
||
| ### Goals | ||
| * Define a convention for storing operator manifests and metadata with container image. |
There was a problem hiding this comment.
| * Define a convention for storing operator manifests and metadata with container image. | |
| * Define a convention for storing operator manifests and metadata with a container image. |
| * Build on-cluster tooling to interact with bundles | ||
|
|
||
| ## Proposal | ||
| We delineate the operator metadata from the operator manifests. The operator manifests refers to a set of kubernetes manifest(s) the defines the deployment and RBAC model of the operator. The operator metadata on the other hand are, but not limited to: |
There was a problem hiding this comment.
| We delineate the operator metadata from the operator manifests. The operator manifests refers to a set of kubernetes manifest(s) the defines the deployment and RBAC model of the operator. The operator metadata on the other hand are, but not limited to: | |
| We delineate operator metadata from operator manifests. We define _operator manifests_ as a set of kubernetes manifests that compose an operator's deployment (`Deployment`, RBAC, etc), and _operator metadata_ as including: |
|
|
||
| *Constraints*: | ||
| * An operator bundle (including both manifests and metadata) should be identifiable using a single versioned identifier. | ||
| * For an operator The metadata can be downloaded independently of the manifest. |
There was a problem hiding this comment.
| * For an operator The metadata can be downloaded independently of the manifest. | |
| * An operator's metadata can be downloaded independently of its manifests. |
| * The label `operators.operatorframework.io.bundle.resources` represents the bundle type: | ||
| * The value `manifests` implies that this bundle contains operator manifests only. | ||
| * The value `metadata` implies that this bundle has operator metadata only. | ||
| * The value `manifests+metadata` implies that this bundle contains both operator metadata and manifests. |
There was a problem hiding this comment.
I think this falls under the inspect portion of the Build, Push, Pull Operator Bundle story. It's necessary if a user wants to identify a bundle image before pulling its layers.
| * The value `manifests` implies that this bundle contains operator manifests only. | ||
| * The value `metadata` implies that this bundle has operator metadata only. | ||
| * The value `manifests+metadata` implies that this bundle contains both operator metadata and manifests. | ||
| * The label `operators.operatorframework.io.bundle.mediatype` reflects the media type or format of the operator bundle. It could be helm charts, plain kubernetes manifests etc. |
There was a problem hiding this comment.
Are we sure that manifests and metadata will always be of the same mediatype?
| * `registry`: Format used by [Operator Registry](https://github.com/operator-framework/operator-registry#manifest-format) to package an operator. | ||
| * `helm`: Can be used to package a helm chart inside an operator bundle. | ||
| * `plain`: Can be used to package plain k8s manifests inside an operator bundle. |
There was a problem hiding this comment.
Can we call out kustomize too?
| $ cd test | ||
|
|
||
| # the following command generates the necessary scaffolding. | ||
| $ operator-framework bundle init --type=registry --bundle-folder=0.1.0 |
There was a problem hiding this comment.
What registry version would --type=registry use, the latest the tool is privy to? Should we always require the proposed version suffix (+<v>)?
| * Verify that the format of the bundle is valid. If the bundle is of `registry` format, we should verify that the bundle conforms to operator-registry standards. | ||
|
|
||
| #### Run the Operator from the Bundle Image | ||
| As an operator author I want to run my operator directly from the bundle image. Once an operator is packaged into a bundle image, we want to give the author ability to run it using `olm` directly from the bundle image. |
There was a problem hiding this comment.
| As an operator author I want to run my operator directly from the bundle image. Once an operator is packaged into a bundle image, we want to give the author ability to run it using `olm` directly from the bundle image. | |
| As an operator author I want to install my operator directly using only its bundle image. |
| ```bash | ||
| cat <<EOF | kubectl apply -f - | ||
| apiVersion: operators.coreos.com/v2alpha1 | ||
| kind: Operator |
|
Did you consider putting the metadata into a LABEL on the image? That would simplify retrieval of metadata from a bundle. That approach was used successfully with APBs. |
|
|
||
| `Dockerfile` for operator bundle | ||
| ``` | ||
| FROM scratch |
There was a problem hiding this comment.
Have we considered that this is included in the operator image in a well-defined place?
Seems to me the only issue would be the SHA that is used in the deployment manifest but could OLM/Index Image replace that?
There was a problem hiding this comment.
As discussed off-line - this is a feature we could build as part of the index in the future, but it would make it harder to audit images that are running on-cluster. We should be able to layer it on to the current proposal easily, but we need to weigh building that vs. the UX of understanding what's actually running on your cluster.
|
@mhrivnak Labels would be ideal, but we need the metadata on-cluster, and we won't have access to the actual (docker v2-2) manifest on-cluster. There's nothing stopping us from replicating the metadata into labels, much like we're doing for the media types here, as need arises. |
| * The potential use case for the `LABELS` is - an external off-cluster tool can inspect the image to check the type of a given bundle image without downloading the content. | ||
|
|
||
| ###### Format | ||
| We can use the following values for `mediatype`: |
There was a problem hiding this comment.
Does this indicate that we will have to write the ability to deploy with these types? Can we say that these are potential types and not implement them until later? I think that we just focus on registry for now.
|
/approve The remaining issues will be addressed in a followup PR. |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: ecordell, shawn-hurley, tkashem The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci-robot
added
the
approved
This enhancement proposes standards and conventions for storing
operator manifests and metadata as conatiner images.