Add documentation about CSI #8783
Conversation
openshift-ci-robot
added
do-not-merge/work-in-progress
|
@jsafrane Sure, I can review the content. For the image, please see these guidelines: https://mojo.redhat.com/docs/DOC-1025233. You can send a request to our CCS designers to create the image. |
|
@tkatarki, this is CSI docs for OpenShift. Work in progress. |
ghost
self-assigned this
|
|
||
| [NOTE] | ||
| ==== | ||
| CSI volumes is an alpha feature and may change in a future release of {product-title}. See xref:local-volume-alpha-feature-status[Feature Status(Local Volume)] section for details on known issues and workarounds. |
There was a problem hiding this comment.
s/for details on known.../for information about known...
|
|
||
| [NOTE] | ||
| ==== | ||
| {product-title} does not ship any CSI drivers. It is recommended to use CSI drivers provided by link:https://kubernetes-csi.github.io/docs/Drivers.html[community or storage vendors]. |
There was a problem hiding this comment.
Should this be "does not ship with any CSI drivers"? Are the drivers installed or configured?
s/use CSI drivers/use the CSI drivers
| [[install-config-persistent-storage-csi-architecture]] | ||
| == Architecture | ||
|
|
||
| CSI drivers are typically shipped as container images. These containers are typically not aware of {product-title} where they run. In order to use CSI-compatible storage backed in {product-title}, cluster administrator must deploy several components that sever as a bridge between {product-title} and CSI storage driver. |
There was a problem hiding this comment.
s/In order to use/To use
s/cluster administrator/the cluster administrator
s/sever/serve
s/CSI storage driver/the storage driver
|
|
||
| CSI drivers are typically shipped as container images. These containers are typically not aware of {product-title} where they run. In order to use CSI-compatible storage backed in {product-title}, cluster administrator must deploy several components that sever as a bridge between {product-title} and CSI storage driver. | ||
|
|
||
| Following picture provides high-level overview about these components running in pods in {product-title} cluster. |
There was a problem hiding this comment.
s/Following picture/The following diagram
|
|
||
| image::csi.png["Architecture of CSI components"] | ||
|
|
||
| It is possible to run multiple CSI drivers for different storage backends. Each driver needs its own external attacher StatefulSet, external provisioner Deployment and DaemonSet with the driver and CSI registrar. |
There was a problem hiding this comment.
s/external provisioner Deployment/external provisioner deployment
|
|
||
| It is possible to run multiple CSI drivers for different storage backends. Each driver needs its own external attacher StatefulSet, external provisioner Deployment and DaemonSet with the driver and CSI registrar. | ||
|
|
||
| === External CSI attacher StatefulSet |
There was a problem hiding this comment.
s/External CSI attacher StatefulSet/External CSI Attacher StatefulSet
| * External CSI attacher container itself that translates `attach` and `detach` calls from {product-title} to respective `ControllerPublish` and `ControllerUnpublish` calls to CSI driver. | ||
| * CSI driver container. | ||
|
|
||
| CSI attacher container talks to CSI driver container using Unix Domain Sockets, ensuring that no CSI communication leaves the pod. CSI driver is not accessible from outside of the pod at all. |
There was a problem hiding this comment.
s/CSI attacher/The CSI attacher
s/talks to CSI/talks to the CSI
s/CSI driver/The CSI driver
s/pod at all/pod
|
|
||
| CSI attacher container talks to CSI driver container using Unix Domain Sockets, ensuring that no CSI communication leaves the pod. CSI driver is not accessible from outside of the pod at all. | ||
|
|
||
| StatefulSet is used to ensure that exactly one instance of the pod runs at any given time. External attacher pod does not scale in {product-title} {product-version}. |
There was a problem hiding this comment.
s/that exactly/that only
s/External attacher/The external attacher
|
|
||
| [NOTE] | ||
| ==== | ||
| `attach` and `detach` operations typically require CSI driver to use credentials to the storage backend. It is recommended to run external attacher pod runs xref:../../install_config/install/advanced_install.adoc#configuring-dedicated-infrastructure-nodes[infrastructure nodes] so the credentials can never leak to user processes even in case of catastrophic security breach on a node. |
There was a problem hiding this comment.
s/require CSI driver/require the CSI driver
The second sentence is not clear. Should it be "It is recommended to run the external attacher pod infrastructure nodes so that the credentials never leak to the user processes, even in the event of a catastrophic security breach on a node."
|
|
||
| [NOTE] | ||
| ==== | ||
| External attacher must run also for CSI drivers that don't support 3rd party attach/detach operations. External attacher will not issue any `ControllerPublish` or `ControllerUnpublish` to CSI driver, however it still must run to implement necessary {product-title} attachment API. |
There was a problem hiding this comment.
s/External attacher/The external attacher
s/must run also/must also run
s/don't/do not
s/3rd party/third-party
s/ControllerUnpublish/ControllerUnpublish operations
s/,however/,however,
s/still must run/must still run
s/implement necessary/to implement the necessary
| External attacher must run also for CSI drivers that don't support 3rd party attach/detach operations. External attacher will not issue any `ControllerPublish` or `ControllerUnpublish` to CSI driver, however it still must run to implement necessary {product-title} attachment API. | ||
| ==== | ||
|
|
||
| === External CSI provisioner deployment |
There was a problem hiding this comment.
s/External CSI provisioner deployment/External CSI Provisioner Deployment
|
|
||
| === External CSI provisioner deployment | ||
|
|
||
| External CSI provisioner is a Deployment that deploys single or several pods with two containers: |
| * External CSI provisioner container itself that translates `provision` and `delete` calls from {product-title} to respective `CreateVolume` and `DeleteVolume` calls to CSI driver. | ||
| * CSI driver container. | ||
|
|
||
| CSI provisioner container talks to CSI driver container using Unix Domain Sockets, ensuring that no CSI communication leaves the pod. CSI driver is not accessible from outside of the pod at all. |
There was a problem hiding this comment.
s/CSI provisioner/The CSI provisioner
s/talks to CSI/talks to the CSI
s/CSI driver/The CSI driver
s/pod at all/pod
|
|
||
| CSI provisioner container talks to CSI driver container using Unix Domain Sockets, ensuring that no CSI communication leaves the pod. CSI driver is not accessible from outside of the pod at all. | ||
|
|
||
| Deployment can be used to scale up/down number of provisioners. |
There was a problem hiding this comment.
"Deployment can be used to scale the number of provisioners up or down."
|
|
||
| [NOTE] | ||
| ==== | ||
| `provision` and `delete` operations typically require CSI driver to use credentials to the storage backend. It is recommended to run external provisioner pods on xref:../../install_config/install/advanced_install.adoc#configuring-dedicated-infrastructure-nodes[infrastructure nodes] so the credentials can never leak to user processes even in case of catastrophic security breach on a node. |
There was a problem hiding this comment.
s/require CSI driver/require the CSI driver
The second sentence is not clear. Should it be "It is recommended to run the external attacher pod infrastructure nodes so that the credentials never leak to the user processes, even in the event of a catastrophic security breach on a node."
| `provision` and `delete` operations typically require CSI driver to use credentials to the storage backend. It is recommended to run external provisioner pods on xref:../../install_config/install/advanced_install.adoc#configuring-dedicated-infrastructure-nodes[infrastructure nodes] so the credentials can never leak to user processes even in case of catastrophic security breach on a node. | ||
| ==== | ||
|
|
||
| === CSI driver DaemonSet |
There was a problem hiding this comment.
s/CSI driver DaemonSet/CSI Driver DaemonSet
|
|
||
| === CSI driver DaemonSet | ||
|
|
||
| Finally, CSI driver DaemonSet runs a pod on every node that allows {product-title} to mount storage provided by CSI driver to the node and use it in user workloads (Pods) as persistent volumes. The pod with CSI driver contains these containers: |
There was a problem hiding this comment.
s/CSI driver/the CSI driver
s/Pods/pods
s/persistent volumes/persistent volumes (PVs)
s/The pod with CSI driver/The pod with the CSI driver installed
s/contains these containers:/contains the following containers:
|
|
||
| Finally, CSI driver DaemonSet runs a pod on every node that allows {product-title} to mount storage provided by CSI driver to the node and use it in user workloads (Pods) as persistent volumes. The pod with CSI driver contains these containers: | ||
|
|
||
| * CSI driver registrar, that registers the CSI driver into `openshift-node` service running on the node. `openshift-node` process running on the node then directly talks to CSI driver using Unix Domain Socket available on the node. |
There was a problem hiding this comment.
s/registrar, that registers/registrar, which registers
s/into openshift-node/into the openshift-node
s/openshift-node/The openshift-node
s/CSI driver/the CSI driver
s/using Unix/using the Unix
| Finally, CSI driver DaemonSet runs a pod on every node that allows {product-title} to mount storage provided by CSI driver to the node and use it in user workloads (Pods) as persistent volumes. The pod with CSI driver contains these containers: | ||
|
|
||
| * CSI driver registrar, that registers the CSI driver into `openshift-node` service running on the node. `openshift-node` process running on the node then directly talks to CSI driver using Unix Domain Socket available on the node. | ||
| * CSI driver itself. |
| * CSI driver registrar, that registers the CSI driver into `openshift-node` service running on the node. `openshift-node` process running on the node then directly talks to CSI driver using Unix Domain Socket available on the node. | ||
| * CSI driver itself. | ||
|
|
||
| CSI driver deployed on the node should have as least credentials to the storage backend as possible. {product-title} will use just Node plugin set of CSI calls such as `NodePublish`/`NodeUnpublish` and `NodeStage`/`NodeUnstage` (if implemented). |
There was a problem hiding this comment.
The first sentence is not clear. Should it be "The CSI driver deployed on the node should have as few credentials to the storage backend as possible."
s/will use just Node/will only use the node plugin set
|
|
||
| CSI driver deployed on the node should have as least credentials to the storage backend as possible. {product-title} will use just Node plugin set of CSI calls such as `NodePublish`/`NodeUnpublish` and `NodeStage`/`NodeUnstage` (if implemented). | ||
|
|
||
| == Example deployment |
|
|
||
| == Example deployment | ||
|
|
||
| Since {product-title} does not ship any CSI driver, this example shows how to deploy community driver for OpenStack Cinder in {product-title}. |
There was a problem hiding this comment.
s/does not ship any/does not ship with any CSI driver installed
s/shows how to deploy community driver/shows how to deploy a community driver
|
|
||
| TODO: example yaml file that uses OSE images + shell commands how to install it. | ||
|
|
||
| == Dynamic provisioning |
There was a problem hiding this comment.
s/Dynamic provisioning/Dynamic Provisioning
|
|
||
| == Dynamic provisioning | ||
|
|
||
| Dynamic provisioning of persistent storage depends on capabilities of CSI driver and underlying storage backend. Provider of a CSI driver should document how to create StorageClass in {product-title} and what parameters are available for configuration. |
There was a problem hiding this comment.
s/depends on capabilities/of CSI driver/depends on the capabilities of the CSI driver
s/Provider of a CSI driver/The provider of the CSI driver
|
|
||
| Dynamic provisioning of persistent storage depends on capabilities of CSI driver and underlying storage backend. Provider of a CSI driver should document how to create StorageClass in {product-title} and what parameters are available for configuration. | ||
|
|
||
| In case of our OpenShift Cinder example, we can deploy this storage class to enable dynamic provisioning: |
|
I added |
| [NOTE] | ||
| ==== | ||
| CSI volumes is a beta feature and may change in a future release of {product-title}. Due to its beta status, it does not support SELinux yet. | ||
| ==== |
There was a problem hiding this comment.
s/not support SELinux yet./not support SELinux at this time.
There was a problem hiding this comment.
I removed the whole sentence, we do support SELinux now.
|
|
||
| [NOTE] | ||
| ==== | ||
| {product-title} {product-version} supports version 0.2.0 of link:https://github.com/container-storage-interface/spec[CSI specification]. |
| == Architecture | ||
|
|
||
| CSI drivers are typically shipped as container images. These containers are not aware of {product-title} where they run. To use CSI-compatible storage backed in {product-title}, the cluster administrator must deploy several components that serve as a bridge between {product-title} and the storage driver. | ||
|
|
|
|
||
| CSI drivers are typically shipped as container images. These containers are not aware of {product-title} where they run. To use CSI-compatible storage backed in {product-title}, the cluster administrator must deploy several components that serve as a bridge between {product-title} and the storage driver. | ||
|
|
||
| The following diagram provides high-level overview about these components running in pods in {product-title} cluster. |
There was a problem hiding this comment.
s/provides/provides a
s/about these/about the
s/{product-title}/the {product-title}
| image::OpenShift_CSI_Arch_470752_0518.png["Architecture of CSI components"] | ||
|
|
||
| It is possible to run multiple CSI drivers for different storage backends. Each driver needs its own external controllers deployment, and DaemonSet with the driver and CSI registrar. | ||
|
|
There was a problem hiding this comment.
Suggest changing the second sentence to "Each driver needs its own external controllers' deployment and DaemonSet with the driver and CSI registrar."
|
|
||
| === External CSI Controllers | ||
|
|
||
| External CSI Controllers is a deployment that deploys one or several pods with two containers: |
|
|
||
| * External CSI attacher container that translates `attach` and `detach` calls from {product-title} to respective `ControllerPublish` and `ControllerUnpublish` calls to CSI driver. | ||
| * External CSI provisioner container that translates `provision` and `delete` calls from {product-title} to respective `CreateVolume` and `DeleteVolume` calls to CSI driver. | ||
| * CSI driver container. |
There was a problem hiding this comment.
Remove the period (.) from each item in the list above. These are not complete sentences, so punctuation can be omitted.
|
|
||
| [NOTE] | ||
| ==== | ||
| `attach`, `detach`, `provision` and `delete` operations typically require the CSI driver to use credentials to the storage backend. Run the CSI controller pods on xref:../../install_config/install/advanced_install.adoc#configuring-dedicated-infrastructure-nodes[infrastructure nodes] so the credentials never leak to user processes, even in the event of a catastrophic security breach on a compute node. |
| [NOTE] | ||
| ==== | ||
| The external attacher must also run for CSI drivers that do not support third-party attach/detach operations. The external attacher will not issue any `ControllerPublish` or `ControllerUnpublish` operations to CSI driver. However, it still must run to implement the necessary {product-title} attachment API. | ||
| ==== |
There was a problem hiding this comment.
s/operations to CSI driver/operations to the CSI driver
|
|
||
| === CSI Driver DaemonSet | ||
|
|
||
| Finally, the CSI driver DaemonSet runs a pod on every node that allows {product-title} to mount storage provided by CSI driver to the node and use it in user workloads (pods) as persistent volumes (PVs). The pod with the CSI driver installed contains the following containers: |
| scc "privileged" added to: ["system:serviceaccount:csi:cinder-csi"] | ||
| ---- | ||
|
|
||
| . Apply this YAML file to create the Deployment with the external CSI attacher and provisioner, and DaemonSet with the CSI driver. |
| hostPath: | ||
| path: /dev | ||
| ---- | ||
| <1> Replace with `cloud.conf` for your OpenStack as described in xref:../../install_config/configuring_openstack.adoc#configuring-openstack-variables[OpenStack configuration]. The Secret can be generated e.g. by `oc create secret generic cloudconfig --from-file cloud.conf --dry-run -o yaml`. |
There was a problem hiding this comment.
s/OpenStack as described in/OpenStack deployment, as described in
Suggest changing the second sentence to "For example, the Secret can be generated using the oc create secret generic cloudconfig --from-file cloud.conf --dry-run -o yaml.
| == Dynamic Provisioning | ||
|
|
||
| Dynamic provisioning of persistent storage depends on the capabilities of the CSI driver and underlying storage backend. The provider of the CSI driver should document how to create StorageClass in {product-title} and what parameters are available for configuration. | ||
|
|
There was a problem hiding this comment.
s/create StorageClass in/a StorageClass in
s/and what parameters are/and the parameters available
| Dynamic provisioning of persistent storage depends on the capabilities of the CSI driver and underlying storage backend. The provider of the CSI driver should document how to create StorageClass in {product-title} and what parameters are available for configuration. | ||
|
|
||
| As seen in the OpenStack Cinder example, you can deploy this StorageClass to enable dynamic provisioning. The following example creates a new default storage class that ensures that all PVCs that do not require any special storage class are provisioned by the installed CSI driver: | ||
|
|
| [[install-config-persistent-storage-csi-usage]] | ||
| == Usage | ||
|
|
||
| Once the CSI driver is deployed and the StorageClass for dynamic provisioning is created, {product-title} can use CSI without any other configuration. The following example installs default MySQL template without any modification of the template: |
There was a problem hiding this comment.
Suggest changing to "Once the CSI driver is deployed and the StorageClass for dynamic provisioning is created, {product-title} is ready to use CSI."
s/installs default/installs a default
s/without any modification of the template:/without any changes to the template:
|
@jsafrane I think the length of the YAML file is OK, but I did suggest additional edits. Thanks. |
|
I fixed all your comments and pushed them in a separate commit for easier review. Let me know when it's ready to be squashed and merged, thanks. |
|
@jsafrane Thanks, you can squash the extra commit. |
|
Suqashed |
ghost
merged commit
|
/cherrypick enterprise-3.10 |
|
@tmorriso-rh: new pull request created: #9669 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
This is the first draft of CSI docs, I'll finish it later with yaml files and so on.
@tmorriso-rh, PTAL. I know the image is too big with too small text and very rough. Do we have any designers or artists that can make it beautiful? I am engineer, not Michelangelo nor Shakespeare :-)
@humblec, can I use you as a guinea pig for proof reading? Is the Architecure understandable and complete?