From f5bf6435987cef673608c70fe0c0accd3029d09f Mon Sep 17 00:00:00 2001 From: Antoine Le Morvan Date: Fri, 4 Feb 2022 17:49:00 +0100 Subject: [PATCH 1/3] writing glusterfs doc --- docs/guides/file_sharing/glusterfs.md | 324 ++++++++++++++++++++++++++ 1 file changed, 324 insertions(+) create mode 100644 docs/guides/file_sharing/glusterfs.md diff --git a/docs/guides/file_sharing/glusterfs.md b/docs/guides/file_sharing/glusterfs.md new file mode 100644 index 0000000000..568d0b6b1c --- /dev/null +++ b/docs/guides/file_sharing/glusterfs.md @@ -0,0 +1,324 @@ +# High availability cluster with GlusterFS + +## Prerequisites + +* Proficiency with a command-line editor (we are using _vi_ in this example) +* A heavy comfort level with issuing commands from the command-line, viewing logs, and other general systems administrator duties +* All commands are run as the root user or sudo + +## Introduction + +GlusterFS is a distributed file system. + +It allows to store a large amount of data distributed on clusters of servers with a very high availability. + +It is composed of a server part to be installed on all the nodes of the server clusters. + +Clients can access the data via the glusterfs client or the mount command. + +Cluster can operate 2 modes: + + * A replicated mode: each node of the cluster has all the data. + * A distributed mode: no data redundancy. If a storage fails, the data on the failed node is lost. + +Both modes can be used together to provide both a replicated and distributed file system as long as you have the right number of servers. + +Data are store inside bricks. + +> A Brick is the basic unit of storage in GlusterFS, represented by an export directory on a server in the trusted storage pool. + +## Test platform + +Our fictitious platform is composed of 2 servers and a client, all Rocky Linux servers. + +* First node: node1.cluster.local - 192.168.1.10 +* Second node: node2.cluster.local - 192.168.1.11 +* Client1: client1.clients.local - 192.168.1.12 + +!!! Note + + Make sure you have the necessary bandwidth between the servers of the cluster. + +Each server in the cluster has a second disk for data storage. + +## Preparation of the disks + +We will create a new LVM logical volum that will be mount on `/data/glusterfs/vol0` on both cluster's servers. + +``` +$ sudo pvcreate /dev/sdb +$ sudo vgcreate vg_data /dev/sdb +$ sudo lvcreate -l 100%FREE -n lv_data vg_data +$ sudo mkfs.xfs /dev/vg_data/lv_data +$ sudo mkdir -p /data/glusterfs/volume1 +``` + +!!! Note + + If LVM is not available on your servers, just install it with the following command: + + ``` + $ sudo dnf install lvm2 + ``` + +We can add now that logical volume to the `/etc/fstab` file: + +``` +/dev/mapper/vg_data-lv_data /data/glusterfs/volume1 xfs defaults 1 2 +``` + +And mount it: + +``` +$ sudo mount -a +``` + +As the data is stored in a sub-volume called brick, we can create a directory in this new data space dedicated to it: + +``` +$ sudo mkdir /data/glusterfs/volume1/brick0 +``` + +## Installation + +At the time of writing this documentation, the original CentOS Storage SIG repository is no longer available and the RockyLinux repository is not yet available. + +However, we will use (for the time being) the archived version. + +First of all, it is necessary to add the dedicated repository to gluster (in version 9) on both servers: + +``` +sudo dnf install centos-release-gluster9 +``` + +!!! Note + + Later, when it is ready on the Rocky side, we can change the name of this package. + +As the repo list and url is not available anymore, let's change the content of the `/etc/yum.repos.d/CentOS-Gluster-9.repo`: + +``` +[centos-gluster9] +name=CentOS-$releasever - Gluster 9 +#mirrorlist=http://mirrorlist.centos.org?arch=$basearch&release=$releasever&repo=storage-gluster-9 +baseurl=https://dl.rockylinux.org/vault/centos/8.5.2111/storage/x86_64/gluster-9/ +gpgcheck=1 +enabled=1 +gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Storage +``` + +We are now ready to install the glusterfs server: + +``` +$ sudo dnf install glusterfs glusterfs-libs glusterfs-server +``` + +## Firewall rules + +A few rules are necessary for the service to work: + +``` +$ sudo firewall-cmd --zone=public --add-port=24007-24008/tcp --permanent +$ sudo firewall-cmd --zone=public --add-port=49152/tcp --permanent +$ sudo firewall-cmd --reload +``` + +## Name resolution + +You can let DNS handle the name resolution of the servers in your cluster, or you can choose to relieve the servers of this task by inserting records for each of them in your `/etc/hosts` files. This will also keep things running even in the event of a DNS failure. + +``` +192.168.10.10 node1.cluster.local +192.168.10.11 node2.cluster.local +``` + +## Starting the service + + +Without further delay start the service: + +``` +$ sudo systemctl enable glusterfsd.service +$ sudo systemctl start glusterfsd.service +``` + +We are ready to join the 2 nodes to the same pool. + +This command is to be performed only once on a single node (here on node1): + +``` +sudo gluster peer probe node2.cluster.local +peer probe: success +``` + +Verify: + +``` +node1 $ sudo gluster peer status +Number of Peers: 1 + +Hostname: node2.cluster.local +Uuid: c4ff108d-0682-43b2-bc0c-311a0417fae2 +State: Peer in Cluster (Connected) +Other names: +192.168.10.11 + +``` + +``` +node2 $ sudo gluster peer status +Number of Peers: 1 + +Hostname: node1.cluster.local +Uuid: 6375e3c2-4f25-42de-bbb6-ab6a859bf55f +State: Peer in Cluster (Connected) +Other names: +192.168.10.10 +``` + +We can now create a volume with 2 replica: + +``` +$ sudo gluster volume create volume1 replica 2 node1.cluster.local:/data/glusterfs/volume1/brick0/ node2.cluster.local:/data/glusterfs/volume1/brick0/ +Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to avoid this. See: http://docs.gluster.org/en/latest/Administrator%20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/. +Do you still want to continue? + (y/n) y +volume create: volume1: success: please start the volume to access data +``` + +!!! Note + + As the return command says, a 2-node cluster is not the best idea in the world against split brain. But this will suffice for the purposes of our test platform. + +We can now start the volume to access data: + +``` +$ sudo gluster volume start volume1 + +volume start: volume1: success +``` + +Check the volume state: + +``` +$ sudo gluster volume status +Status of volume: volume1 +Gluster process TCP Port RDMA Port Online Pid +------------------------------------------------------------------------------ +Brick node1.cluster.local:/data/glusterfs/v +olume1/brick0 49152 0 Y 1210 +Brick node2.cluster.local:/data/glusterfs/v +olume1/brick0 49152 0 Y 1135 +Self-heal Daemon on localhost N/A N/A Y 1227 +Self-heal Daemon on node2.cluster.local N/A N/A Y 1152 + +Task Status of Volume volume1 +------------------------------------------------------------------------------ +There are no active volume tasks +``` + +``` +$ sudo gluster volume info + +Volume Name: volume1 +Type: Replicate +Volume ID: f51ca783-e815-4474-b256-3444af2c40c4 +Status: Started +Snapshot Count: 0 +Number of Bricks: 1 x 2 = 2 +Transport-type: tcp +Bricks: +Brick1: node1.cluster.local:/data/glusterfs/volume1/brick0 +Brick2: node2.cluster.local:/data/glusterfs/volume1/brick0 +Options Reconfigured: +cluster.granular-entry-heal: on +storage.fips-mode-rchecksum: on +transport.address-family: inet +nfs.disable: on +performance.client-io-threads: off +``` + +The status must be Started. + +We can already restrict access on the volume a little bit: + +``` +$ sudo gluster volume set volume1 auth.allow 192.168.10.* +``` + +It's as simple as that + +## Clients access + +There are several ways to access our data from a client. + +The preferred method: + +``` +$ sudo dnf install glusterfs-client +$ sudo mkdir /data +$ sudo mount.glusterfs node1.cluster.local:/volume1 /data +``` + +There are no additional repositories to configure. The client is already present in the base repos. + +Create a file and check that it is present on all the nodes of the cluster: + +On client: + +``` +sudo touch /data/test +``` + +On both servers: + +``` +$ ll /data/glusterfs/volume1/brick0/ +total 0 +-rw-r--r--. 2 root root 0 Feb 3 19:21 test +``` + +Sound good! But what happens if the node 1 falls? It is the one that was specified when mounting the remote access. + +Let's stop the node one: + +``` +$ sudo shutdown -h now +``` + +Check status on node2: + +``` +$ sudo gluster peer status +Number of Peers: 1 + +Hostname: node1.cluster.local +Uuid: 6375e3c2-4f25-42de-bbb6-ab6a859bf55f +State: Peer in Cluster (Disconnected) +Other names: +192.168.10.10 +[antoine@node2 ~]$ sudo gluster volume status +Status of volume: volume1 +Gluster process TCP Port RDMA Port Online Pid +------------------------------------------------------------------------------ +Brick node2.cluster.local:/data/glusterfs/v +olume1/brick0 49152 0 Y 1135 +Self-heal Daemon on localhost N/A N/A Y 1152 + +Task Status of Volume volume1 +------------------------------------------------------------------------------ +There are no active volume tasks +``` + +The node1 is away. + +And on client: + +``` +$ ll /data/test +-rw-r--r--. 1 root root 0 Feb 4 16:41 /data/test +``` + +File is already there. + +Upon connection, the glusterfs client receives a list of nodes it can address, which explains the transparent switchover we just witnessed. From 5b22a214d6c3bc2fcef1e502f4ca1288650442ce Mon Sep 17 00:00:00 2001 From: Antoine Le Morvan Date: Fri, 11 Feb 2022 12:39:26 +0100 Subject: [PATCH 2/3] use services if possible in firewalld-cmd --- docs/guides/file_sharing/glusterfs.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/guides/file_sharing/glusterfs.md b/docs/guides/file_sharing/glusterfs.md index 568d0b6b1c..a6754c69d4 100644 --- a/docs/guides/file_sharing/glusterfs.md +++ b/docs/guides/file_sharing/glusterfs.md @@ -117,6 +117,13 @@ $ sudo dnf install glusterfs glusterfs-libs glusterfs-server A few rules are necessary for the service to work: +``` +$ sudo firewall-cmd --zone=public --add-service=glusterfs --permanent +$ sudo firewall-cmd --reload +``` + +or: + ``` $ sudo firewall-cmd --zone=public --add-port=24007-24008/tcp --permanent $ sudo firewall-cmd --zone=public --add-port=49152/tcp --permanent From 3e2a3cfbca2f186145d47dc1b7dba11d087d69d2 Mon Sep 17 00:00:00 2001 From: Steven Spencer Date: Fri, 11 Feb 2022 10:07:28 -0600 Subject: [PATCH 3/3] Editing pass - GlusterFS document * read for spelling, word usage, readability * check punctuation, etc. * change some wording to make the document more clear * added meta information to shorten the title for the side menu * added a conclusion section --- docs/guides/file_sharing/glusterfs.md | 40 +++++++++++++++++---------- 1 file changed, 25 insertions(+), 15 deletions(-) diff --git a/docs/guides/file_sharing/glusterfs.md b/docs/guides/file_sharing/glusterfs.md index a6754c69d4..d510f2dfd0 100644 --- a/docs/guides/file_sharing/glusterfs.md +++ b/docs/guides/file_sharing/glusterfs.md @@ -1,3 +1,10 @@ +--- +title: Clustering-GlusterFS +author: Antoine Le Morvan +contributors: Steven Spencer +update: 11-Feb-2022 +--- + # High availability cluster with GlusterFS ## Prerequisites @@ -10,26 +17,26 @@ GlusterFS is a distributed file system. -It allows to store a large amount of data distributed on clusters of servers with a very high availability. +It allows for storage of large amount of data distributed across clusters of servers with a very high availability. It is composed of a server part to be installed on all the nodes of the server clusters. -Clients can access the data via the glusterfs client or the mount command. +Clients can access the data via the `glusterfs` client or the `mount` command. -Cluster can operate 2 modes: +GlusterFS can operate in two modes: - * A replicated mode: each node of the cluster has all the data. - * A distributed mode: no data redundancy. If a storage fails, the data on the failed node is lost. + * replicated mode: each node of the cluster has all the data. + * distributed mode: no data redundancy. If a storage fails, the data on the failed node is lost. Both modes can be used together to provide both a replicated and distributed file system as long as you have the right number of servers. -Data are store inside bricks. +Data is stored inside bricks. > A Brick is the basic unit of storage in GlusterFS, represented by an export directory on a server in the trusted storage pool. ## Test platform -Our fictitious platform is composed of 2 servers and a client, all Rocky Linux servers. +Our fictitious platform is composed of two servers and a client, all Rocky Linux servers. * First node: node1.cluster.local - 192.168.1.10 * Second node: node2.cluster.local - 192.168.1.11 @@ -43,7 +50,7 @@ Each server in the cluster has a second disk for data storage. ## Preparation of the disks -We will create a new LVM logical volum that will be mount on `/data/glusterfs/vol0` on both cluster's servers. +We will create a new LVM logical volume that will be mounted on `/data/glusterfs/vol0` on both of the cluster's servers: ``` $ sudo pvcreate /dev/sdb @@ -61,7 +68,7 @@ $ sudo mkdir -p /data/glusterfs/volume1 $ sudo dnf install lvm2 ``` -We can add now that logical volume to the `/etc/fstab` file: +We can now add that logical volume to the `/etc/fstab` file: ``` /dev/mapper/vg_data-lv_data /data/glusterfs/volume1 xfs defaults 1 2 @@ -141,15 +148,14 @@ You can let DNS handle the name resolution of the servers in your cluster, or yo ## Starting the service - -Without further delay start the service: +Without further delay, let's start the service: ``` $ sudo systemctl enable glusterfsd.service $ sudo systemctl start glusterfsd.service ``` -We are ready to join the 2 nodes to the same pool. +We are ready to join the two nodes to the same pool. This command is to be performed only once on a single node (here on node1): @@ -183,7 +189,7 @@ Other names: 192.168.10.10 ``` -We can now create a volume with 2 replica: +We can now create a volume with 2 replicas: ``` $ sudo gluster volume create volume1 replica 2 node1.cluster.local:/data/glusterfs/volume1/brick0/ node2.cluster.local:/data/glusterfs/volume1/brick0/ @@ -245,7 +251,7 @@ nfs.disable: on performance.client-io-threads: off ``` -The status must be Started. +The status must be "Started". We can already restrict access on the volume a little bit: @@ -285,7 +291,7 @@ total 0 -rw-r--r--. 2 root root 0 Feb 3 19:21 test ``` -Sound good! But what happens if the node 1 falls? It is the one that was specified when mounting the remote access. +Sound good! But what happens if the node 1 fails? It is the one that was specified when mounting the remote access. Let's stop the node one: @@ -329,3 +335,7 @@ $ ll /data/test File is already there. Upon connection, the glusterfs client receives a list of nodes it can address, which explains the transparent switchover we just witnessed. + +## Conclusions + +While there are no current repositories, using the archived repositories that CentOS had for GlusterFS will still work. As outlined, GlusterFS is pretty easy to install and maintain. Using the command line tools is a pretty straight forward process. GlusterFS will help with creating and maintaining high-availability clusters for data storage and redundancy. You can find more information on GlusterFS and tool usage from the [official documentation pages.](https://docs.gluster.org/en/latest/)