Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JBOD support documentation #1280

Merged
merged 9 commits into from Feb 19, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
21 changes: 9 additions & 12 deletions documentation/book/assembly-storage.adoc
Expand Up @@ -10,33 +10,30 @@
[id='assembly-storage-{context}']
= Kafka and Zookeeper storage

Kafka brokers and Zookeeper are stateful applications.
They need to store data on disks.
{ProductName} allows you to configure the type of storage, which they want to use for Kafka and Zookeeper.
Storage configuration is mandatory and has to be specified in every `Kafka` resource.
As stateful applications, Kafka and Zookeeper need to store data on disk. {ProductName} supports three different types of storage for this data: ephemeral, persistent, and JBOD (Just a Bunch of Disks) storage.

Storage can be configured using the `storage` property in following resources:
NOTE: JBOD storage is only supported for Kafka, not for Zookeeper.

When configuring a `Kafka` resource, you can specify the type of storage used by the Kafka broker and its corresponding Zookeeper node. You configure the storage type using the `storage` property in the following resources:

* `Kafka.spec.kafka`
* `Kafka.spec.zookeeper`

{ProductName} supports two types of storage:

* Ephemeral
* Persistent
The storage type is configured in the `type` field.

The type of storage is specified in the `type` field.

IMPORTANT: Once the Kafka cluster is deployed, the storage cannot be changed.
WARNING: The storage type cannot be changed after a Kafka cluster is deployed.

include::ref-storage-ephemeral.adoc[leveloffset=+1]

include::ref-storage-persistent.adoc[leveloffset=+1]

include::ref-storage-jbod.adoc[leveloffset=+1]

.Additional resources

* For more information about ephemeral storage, see xref:type-EphemeralStorage-reference[ephemeral storage schema reference].
* For more information about persistent storage, see xref:type-PersistentClaimStorage-reference[persistent storage schema reference].
* For more information about JBOD storage, see xref:type-JbodStorage-reference[JBOD schema reference].
* For more information about the schema for `Kafka`, see xref:type-Kafka-reference[`Kafka` schema reference].

:context: {parent-context}
Expand Up @@ -38,4 +38,5 @@ The following resources will created by the Cluster Operator in the {ProductPlat
`_cluster-name_-clients-ca-cert`:: Secret with the Clients CA public key. This key can be used to verify the identity of the Kafka brokers.
`_cluster-name_-cluster-operator-certs`:: Secret with Cluster operators keys for communication with Kafka and Zookeeper.
`data-_cluster-name_-kafka-_idx_`:: Persistent Volume Claim for the volume used for storing data for the Kafka broker pod `_idx_`. This resource will be created only if persistent storage is selected for provisioning persistent volumes to store data.
`data-_id_-_cluster-name_-kafka-_idx_`:: Persistent Volume Claim for the volume `_id_` used for storing data for the Kafka broker pod `_idx_`. This resource is only created if persistent storage is selected for JBOD volumes when provisioning persistent volumes to store data.
`data-_cluster-name_-zookeeper-_idx_`:: Persistent Volume Claim for the volume used for storing data for the Zookeeper node pod `_idx_`. This resource will be created only if persistent storage is selected for provisioning persistent volumes to store data.
43 changes: 43 additions & 0 deletions documentation/book/ref-storage-jbod.adoc
@@ -0,0 +1,43 @@
// Module included in the following assemblies:
//
// assembly-storage.adoc

[id='ref-jbod-storage-{context}']
= JBOD storage overview

You can configure {ProductName} to use JBOD, a data storage configuration in which multiple disks are combined to create a single logical disk, or volume. JBOD is one approach to providing increased data storage for Kafka brokers. It can also improve performance.

A JBOD configuration is described by one or more volumes, each of which can be either xref:ref-ephemeral-storage-{context}[ephemeral] or xref:ref-persistent-storage-{context}[persistent]. The rules and constraints for JBOD volume declarations are the same as those for ephemeral and persistent storage. For example, you cannot change the size of a persistent storage volume after it has been provisioned.

== JBOD configuration

To use JBOD with {ProductName}, the storage `type` must be set to `jbod`. The `volumes` property allows you to describe the disks that make up your JBOD storage array or configuration. The following fragment shows an example JBOD configuration:

[source,yaml]
----
# ...
storage:
type: jbod
volumes:
- id: 0
type: persistent-claim
size: 100Gi
deleteClaim: false
- id: 1
type: persistent-claim
size: 100Gi
deleteClaim: false
# ...
----

The ids cannot be changed once the JBOD volumes are created.

NOTE: Adding and removing volumes from a JBOD configuration is not currently supported.

== JBOD and Persistent Volume Claims

When persistent storage is used to declare JBOD volumes, the naming scheme of the resulting Persistent Volume Claims is as follows:

`data-_id_-_cluster-name_-kafka-_idx_`::

Where `_id_` is the ID of the volume used for storing data for Kafka broker pod `_idx_`.
4 changes: 4 additions & 0 deletions documentation/book/ref-storage-persistent.adoc
Expand Up @@ -12,6 +12,10 @@ The data types which can be used with persistent volume claims include many type
To use persistent storage, the `type` has to be set to `persistent-claim`.
Persistent storage supports additional configuration options:

`id` (optional)::
Storage identification number. This option is mandatory for storage volumes defined in a JBOD storage declaration.
Default is `0`.

`size` (required)::
Defines the size of the persistent volume claim, for example, "1000Gi".

Expand Down