From 2855ba4fd9632bd6d5928f9593e626b9f88ff1b4 Mon Sep 17 00:00:00 2001 From: Nate Thornton Date: Tue, 13 Dec 2022 13:34:00 -0600 Subject: [PATCH 1/4] First iteration of High Availability Cluster user guide Signed-off-by: Nate Thornton --- docs/guides/ha-cluster/readme.md | 62 ++++++++++++++++++++++++++++++++ docs/guides/index.md | 1 + mkdocs.yml | 2 ++ 3 files changed, 65 insertions(+) create mode 100644 docs/guides/ha-cluster/readme.md diff --git a/docs/guides/ha-cluster/readme.md b/docs/guides/ha-cluster/readme.md new file mode 100644 index 00000000..bae7fd94 --- /dev/null +++ b/docs/guides/ha-cluster/readme.md @@ -0,0 +1,62 @@ +--- +authors: Nate Thornton +categories: setup +--- + +# High Availability Cluster + +Rabbit software supports provisioning of Red Hat GFS2 (Global File System 2) storage. Per Red Hat, +> GFS2 allows multiple nodes to share storage at a block level as if the storage were connected to locally to each cluster node. GFS2 cluster file system requires a cluster infrastructure + +Therefore, in order to use of the GFS2, the Rabbit and its associated compute nodes must form a high-availability cluster. + +## Cluster Setup + +Red Hat provides instructions for [creating a high availability cluster with Pacemaker](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_high_availability_clusters/assembly_creating-high-availability-cluster-configuring-and-managing-high-availability-clusters#doc-wrapper), including instructions for [installing cluster software](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_high_availability_clusters/assembly_creating-high-availability-cluster-configuring-and-managing-high-availability-clusters#proc_installing-cluster-software-creating-high-availability-cluster) and +[creating a high availability cluster](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_high_availability_clusters/assembly_creating-high-availability-cluster-configuring-and-managing-high-availability-clusters) + + +## Fencing Agents + +Fencing is the process of restricting and releasing access to resources that a failed cluster node may have access to. Since a failed node may be unresponsive, an external device must exist that can restrict access to shared resources of that node, or to issue a hard reboot of the node. More information can be found form Red Hat: [1.2.1 Fencing](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_high_availability_clusters/assembly_overview-of-high-availability-configuring-and-managing-high-availability-clusters#fencing) + +HPE hardware implements software known as the Hardware System Supervisor (HSS), which itself conforms to the SNIA Redfish/Swordfish standard. This provides the means to manage hardware outside the host OS. + +### Compute Fencing + +!!! warning + Usage of the Redfish fencing agent is not yet verified + +The [Redfish fencing agent](https://github.com/ClusterLabs/fence-agents/tree/main/agents/redfish) from [ClusterLabs](https://github.com/ClusterLabs/fence-agents) should be used for Compute nodes in the cluster. Configure the agent with the following parameters. + +| Argument | Definition | +| -------- | ---------- | +| `--ip=[ADDRESS]` | The IP address or hostname of the compute node's HSS node controller | +|`--systems-uri=[URI]` | The URI of the Systems object. Must be `/redfish/v1/Systems/Node0` | +|`--ssl-insecure` | Instructs the use of an insecure SSL exchange | + + +### Rabbit Fencing + +!!! info + Rabbit fencing agent is in active development; the description below is subject to change. + +Since the Rabbit node is connected to 16 compute blades, careful coordination around fencing of a Rabbit node is required to minimize the impact of the outage. When a Rabbit node is fenced, the corresponding Kubernetes Storage resource (`storages.dws.cray.hpe.com`) is updated with a status of 'Fenced'. The workload manager must observe this change and handle the movement of resources off the Rabbit node and clear the 'Fenced' status before forcibly rebooting the node. + +Configure the Rabbit agent with the follow parameters + +| Argument | Definition | +| -------- | ---------- | +| `--kubernetes-service-host=[ADDRESS]` | The IP address of the kubeapi server | +| `--kubernetes-service-port=[PORT]` | The listening port of the kube api server | +| `--service-token-file=[PATH]` | The location of the service token file. The file must be present on all nodes within the cluster | +| `--service-cert-file=[PATH]` | The location of the service certificate file. The file must be present on all nodes within the cluster | +| `--nnf-node-name=[RABBIT-NODE-NAME]` | Name of the rabbit node | + +### Dummy Fencing + +The [dummy fencing agent](https://github.com/ClusterLabs/fence-agents/tree/main/agents/dummy) from ClusterLabs can be used for nodes in the cluster for an early access development system. + +## Configuring a GFS2 file system in a cluster + +Follow steps 1-8 of the procedure from Red Hat: [Configuring a GFS2 file system in a cluster](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_high_availability_clusters/assembly_configuring-gfs2-in-a-cluster-configuring-and-managing-high-availability-clusters#doc-wrapper) diff --git a/docs/guides/index.md b/docs/guides/index.md index fe222cdb..c07f1fc4 100644 --- a/docs/guides/index.md +++ b/docs/guides/index.md @@ -5,6 +5,7 @@ * [Initial Setup](initial-setup/readme.md) * [Compute Daemons](compute-daemons/readme.md) * [Firmware Upgrade](firmware-upgrade/readme.md) +* [High Availability Cluster](ha-cluster/readme.md) ## Provisioning diff --git a/mkdocs.yml b/mkdocs.yml index 4c57147b..eb755f78 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -11,6 +11,7 @@ nav: - 'Initial Setup': 'guides/initial-setup/readme.md' - 'Compute Daemons': 'guides/compute-daemons/readme.md' - 'Firmware Upgrade': 'guides/firmware-upgrade/readme.md' + - 'High Availability Cluster': 'guides/ha-cluster/readme.md' - 'Storage Profiles': 'guides/storage-profiles/readme.md' - 'RFCs': - rfcs/index.md @@ -27,6 +28,7 @@ plugins: markdown_extensions: - pymdownx.highlight: anchor_linenums: true + - admonition - pymdownx.details - pymdownx.inlinehilite - pymdownx.snippets From 71a826088a234071b5a2e0bdb4b4c0e13b50064c Mon Sep 17 00:00:00 2001 From: Nate Thornton Date: Tue, 13 Dec 2022 18:15:12 -0600 Subject: [PATCH 2/4] fix typos Signed-off-by: Nate Thornton --- docs/guides/ha-cluster/readme.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/guides/ha-cluster/readme.md b/docs/guides/ha-cluster/readme.md index bae7fd94..af39379d 100644 --- a/docs/guides/ha-cluster/readme.md +++ b/docs/guides/ha-cluster/readme.md @@ -6,9 +6,9 @@ categories: setup # High Availability Cluster Rabbit software supports provisioning of Red Hat GFS2 (Global File System 2) storage. Per Red Hat, -> GFS2 allows multiple nodes to share storage at a block level as if the storage were connected to locally to each cluster node. GFS2 cluster file system requires a cluster infrastructure +> GFS2 allows multiple nodes to share storage at a block level as if the storage were connected locally to each cluster node. GFS2 cluster file system requires a cluster infrastructure -Therefore, in order to use of the GFS2, the Rabbit and its associated compute nodes must form a high-availability cluster. +Therefore, in order to use GFS2, the Rabbit and its associated compute nodes must form a high availability cluster. ## Cluster Setup From d4bb211cf4663e6d830727781b67995154cb0592 Mon Sep 17 00:00:00 2001 From: Nate Thornton Date: Wed, 14 Dec 2022 11:46:57 -0600 Subject: [PATCH 3/4] blakes changes Co-authored-by: Blake Devcich <89158881+bdevcich-hpe@users.noreply.github.com> --- docs/guides/ha-cluster/readme.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/guides/ha-cluster/readme.md b/docs/guides/ha-cluster/readme.md index af39379d..dff69c86 100644 --- a/docs/guides/ha-cluster/readme.md +++ b/docs/guides/ha-cluster/readme.md @@ -5,20 +5,20 @@ categories: setup # High Availability Cluster -Rabbit software supports provisioning of Red Hat GFS2 (Global File System 2) storage. Per Red Hat, -> GFS2 allows multiple nodes to share storage at a block level as if the storage were connected locally to each cluster node. GFS2 cluster file system requires a cluster infrastructure +Rabbit software supports provisioning of Red Hat GFS2 (Global File System 2) storage. Per Red Hat: +> GFS2 allows multiple nodes to share storage at a block level as if the storage were connected locally to each cluster node. GFS2 cluster file system requires a cluster infrastructure. Therefore, in order to use GFS2, the Rabbit and its associated compute nodes must form a high availability cluster. ## Cluster Setup Red Hat provides instructions for [creating a high availability cluster with Pacemaker](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_high_availability_clusters/assembly_creating-high-availability-cluster-configuring-and-managing-high-availability-clusters#doc-wrapper), including instructions for [installing cluster software](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_high_availability_clusters/assembly_creating-high-availability-cluster-configuring-and-managing-high-availability-clusters#proc_installing-cluster-software-creating-high-availability-cluster) and -[creating a high availability cluster](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_high_availability_clusters/assembly_creating-high-availability-cluster-configuring-and-managing-high-availability-clusters) +[creating a high availability cluster](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_high_availability_clusters/assembly_creating-high-availability-cluster-configuring-and-managing-high-availability-clusters). ## Fencing Agents -Fencing is the process of restricting and releasing access to resources that a failed cluster node may have access to. Since a failed node may be unresponsive, an external device must exist that can restrict access to shared resources of that node, or to issue a hard reboot of the node. More information can be found form Red Hat: [1.2.1 Fencing](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_high_availability_clusters/assembly_overview-of-high-availability-configuring-and-managing-high-availability-clusters#fencing) +Fencing is the process of restricting and releasing access to resources that a failed cluster node may have access to. Since a failed node may be unresponsive, an external device must exist that can restrict access to shared resources of that node, or to issue a hard reboot of the node. More information can be found form Red Hat: [1.2.1 Fencing](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_high_availability_clusters/assembly_overview-of-high-availability-configuring-and-managing-high-availability-clusters#fencing). HPE hardware implements software known as the Hardware System Supervisor (HSS), which itself conforms to the SNIA Redfish/Swordfish standard. This provides the means to manage hardware outside the host OS. @@ -27,7 +27,7 @@ HPE hardware implements software known as the Hardware System Supervisor (HSS), !!! warning Usage of the Redfish fencing agent is not yet verified -The [Redfish fencing agent](https://github.com/ClusterLabs/fence-agents/tree/main/agents/redfish) from [ClusterLabs](https://github.com/ClusterLabs/fence-agents) should be used for Compute nodes in the cluster. Configure the agent with the following parameters. +The [Redfish fencing agent](https://github.com/ClusterLabs/fence-agents/tree/main/agents/redfish) from [ClusterLabs](https://github.com/ClusterLabs/fence-agents) should be used for Compute nodes in the cluster. Configure the agent with the following parameters: | Argument | Definition | | -------- | ---------- | @@ -43,7 +43,7 @@ The [Redfish fencing agent](https://github.com/ClusterLabs/fence-agents/tree/mai Since the Rabbit node is connected to 16 compute blades, careful coordination around fencing of a Rabbit node is required to minimize the impact of the outage. When a Rabbit node is fenced, the corresponding Kubernetes Storage resource (`storages.dws.cray.hpe.com`) is updated with a status of 'Fenced'. The workload manager must observe this change and handle the movement of resources off the Rabbit node and clear the 'Fenced' status before forcibly rebooting the node. -Configure the Rabbit agent with the follow parameters +Configure the Rabbit agent with the following parameters: | Argument | Definition | | -------- | ---------- | @@ -59,4 +59,4 @@ The [dummy fencing agent](https://github.com/ClusterLabs/fence-agents/tree/main/ ## Configuring a GFS2 file system in a cluster -Follow steps 1-8 of the procedure from Red Hat: [Configuring a GFS2 file system in a cluster](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_high_availability_clusters/assembly_configuring-gfs2-in-a-cluster-configuring-and-managing-high-availability-clusters#doc-wrapper) +Follow steps 1-8 of the procedure from Red Hat: [Configuring a GFS2 file system in a cluster](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_high_availability_clusters/assembly_configuring-gfs2-in-a-cluster-configuring-and-managing-high-availability-clusters#doc-wrapper). From 91021257524c98557f2e893c94ce80c2d1b25460 Mon Sep 17 00:00:00 2001 From: Nate Thornton Date: Wed, 14 Dec 2022 11:47:36 -0600 Subject: [PATCH 4/4] fix kubeapi typo Signed-off-by: Nate Thornton --- docs/guides/ha-cluster/readme.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/ha-cluster/readme.md b/docs/guides/ha-cluster/readme.md index dff69c86..46044712 100644 --- a/docs/guides/ha-cluster/readme.md +++ b/docs/guides/ha-cluster/readme.md @@ -48,7 +48,7 @@ Configure the Rabbit agent with the following parameters: | Argument | Definition | | -------- | ---------- | | `--kubernetes-service-host=[ADDRESS]` | The IP address of the kubeapi server | -| `--kubernetes-service-port=[PORT]` | The listening port of the kube api server | +| `--kubernetes-service-port=[PORT]` | The listening port of the kubeapi server | | `--service-token-file=[PATH]` | The location of the service token file. The file must be present on all nodes within the cluster | | `--service-cert-file=[PATH]` | The location of the service certificate file. The file must be present on all nodes within the cluster | | `--nnf-node-name=[RABBIT-NODE-NAME]` | Name of the rabbit node |