Rfc0003 fencing agent for gfs2

docs/rfcs/0003/readme.md

@@ -0,0 +1,39 @@
+---
+authors: Bill Johnson <billj@hpe.com>
+state: discussion
+discussion: 
+---
+GFS2 Cluster Fencing Agents
+==================================
+The GFS2 file systems requires a cluster in order to avoid split-brained behavior when 2 or more nodes are accessing the file system.  In order to achieve this, NNF makes use of a Pacemaker Cluster in combination with CoroSync for inter-node communication.  This cluster requires fencing agents to be defined for each node in the cluster.  These agents are responsible for "fencing" off a node that has been "voted off of the island".
+
+In the Near Node Flash (Rabbit) System, there are 2 types of nodes in a PCS Cluster:
+* 1 Rabbit Node
+* Upto 16 Compute Nodes
+Note: In a given chassis it is expected that the Rabbit and all 16 compute nodes are part of a cluster.  There may be exceptions for UAN/FEN nodes in a given chassis, they may be exempted from the cluster.
+
+There are a number of different approaches to fencing misbehaving nodes:
+* Power the node off - This is the typical approach to fencing
+* Do nothing and report the node
+* Mark the node in some way as unhealthy
+
+After a bit of discussion with our customer it has been decided that we will mark a misbehaving node as degraded within Kubernetes.  We have engineered the Storage object Spec section to expose a region where nodes may be marked as degraded by a fencing agent.  The basic flow is as follows:
+
+1. The cluster achieves a quorum that a specific node is to be fenced.
+2. The node assigned to fence the offending node calls the fencing agent running on that node, passing it the name of the node to be fenced.
+3. The fencing agent queries the Storage object via the Kubernetes API.
+4. The fencing agent marks the node as Degraded in the Spec and updates the Storage object via the Kubernetes API.
+
+It is the responsibility of the WLM, Flux in this case, to watch for these changes and to take the appropriate action.  In the case of Rabbit, this may mean removing the Rabbit from the schedule and allowing any remaining jobs to complete/drain.  Ultimately it may require an administrator to analyze the degraded Rabbit for any malfunctions before it can be placed back into service.
+
+Sample Agents
+==================================
+Contained within this RFC are 2 sample fencing agents.  Both agents are written in python3 and require the fencing.py python module in order to function.
+
+## fence_hpe_hss_ssh
+
+**fence_hpe_hss_ssh** is a fencing agent that connects to the HSS via SSH and can power a node on or off using the redfish tool present on the managing HSS.  This fencing agent is basically functional and will install and run in a Pacemaker cluster.
+
+## fence_hpe_nnf
+
+**fence_hpe_nnf** is a fencing agent that is intended to mark an NNF node as Degraded in the Storage CR, **it will not power down a node that has been selected for fencing**.  At the time of this writing, this fencing agent only has the structure framed in and is awaiting the specific kubernetes objects and logic in order to become functional.

docs/rfcs/0003/src/fence_hpe_hss_ssh/Readme.md

@@ -0,0 +1,58 @@
+# fence_hpe_hss_ssh.py
+
+## Overview
+fence_hpe_hss_ssh.py is a fencing agent intended to run in a PaceMaker cluster environment.  This agent uses SSH to interact with the HSS redfish CLI to power a given node on or off.  The HSS redfish CLI offers the following usage:
+
+    redfish node status
+    redfish node [0-3] [on|off|forceoff]
+
+It makes use of the fencing agent python library created by ClusterLabs to abstract much of the boilerplate so that only the actual fencing logic appears in the agent code.  The ClusterLabs github repository is located here:
+
+https://github.com/ClusterLabs/fence-agents
+
+The fencing library tends to live in /usr/share/fence, however installation locations may vary:
+
+    # ls -l /usr/share/fence/fenc*
+    -rwxr-xr-x. 1 root root 52344 Feb 11  2022 /usr/share/fence/fencing.py
+    -rwxr-xr-x. 1 root root  3753 Feb 11  2022 /usr/share/fence/fencing_snmp.py
+
+This agent works with Pacemaker 2.1 on TOSS 4.x
+    # stonith_admin --version
+    Pacemaker 2.1.0-8.el8
+    # uname -a
+    Linux rabbit-compute-3 4.18.0-348.20.1.2toss.t4.x86_64 #1 SMP Wed Apr 6 17:46:46 PDT 2022 x86_64 x86_64 x86_64 GNU/Linux    
+
+## Installation    
+
+It is recommended that you install the fencing agent package, which will include the appropriate python modules.
+
+    dnf install fence-agents-common
+
+To list recognized stonith agents:
+
+    # pcs stonith list
+    fence_ssh - Basic fencing agent that uses SSH
+    fence_watchdog - Dummy watchdog fence agent
+
+
+For a TOSS system, copy the fencing agent into /usr/sbin and make it executable.
+
+    #
+    # cp fence_hpe_hss_ssh.py /usr/sbin/fence_hpe_hss_ssh
+    # chmod 755 /usr/sbin/fence_hpe_hss_ssh
+    #
+    # pcs stonith list
+    fence_hpe_hss_ssh - Fence agent for HP HSS over SSH
+    fence_ssh - Basic fencing agent that uses SSH
+    fence_watchdog - Dummy watchdog fence agent
+
+
+Typically the fencing agent is selected and setup using the PaceMaker CLI tool in the following fashion:
+
+    # pcs stonith create gfs2_rabbit_hss fence_hpe_hss_ssh ip="10.1.1.6" pcmk_host_list="rabbit-node-1" ssh=1 username="<username>" password="<password>" op monitor interval=30s
+    # pcs stonith status
+      * rabbit-node-1       (stonith:fence_ssh):     Started rabbit-compute-2
+      * rabbit-compute-2    (stonith:fence_ssh):     Started rabbit-compute-3
+      * rabbit-compute-3    (stonith:fence_ssh):     Started rabbit-node-1
+      * gfs2_rabbit_hss     (stonith:fence_hpe_hss_ssh):     Started rabbit-compute-3 (Monitoring)
+

docs/rfcs/0003/src/fence_hpe_hss_ssh/fence_hpe_hss_ssh.py

@@ -0,0 +1,138 @@
+#!/usr/bin/python3
+# -*- coding: utf-8 -*-
+#
+# Copyright 2021, 2022 Hewlett Packard Enterprise Development LP
+# Other additional copyright holders may be indicated within.
+#
+# The entirety of this work is licensed under the Apache License,
+# Version 2.0 (the "License"); you may not use this file except
+# in compliance with the License.
+#
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys, re, time
+import atexit
+from os.path import exists
+
+sys.path.append("@FENCEAGENTSLIBDIR@")
+
+# In some cases, we do not wish to allow power off
+allow_powerOff = False
+
+try:
+	from fencing import *
+	from fencing import fail, EC_STATUS
+except:
+	if exists("/usr/share/fence/fencing.py"):
+		sys.path.append("/usr/share/fence/")
+		try:
+			from fencing import *
+			from fencing import fail, EC_STATUS
+		except:
+			fence_import_error()
+	else:
+		fence_import_error()
+
+
+def fence_import_error():
+	print("Error importing fencing libs, may need to update python path")
+	print("e.g. export PYTHONPATH=/usr/share/fence/")
+	exit(1)
+
+def get_power_status(conn, options):
+	"""
+	Checks the power state of node 0
+
+	Returns:
+	"on" - Node 0 is powered on
+	"off" - Any power state other than ON
+	"""
+
+	# TODO: Perhaps drive this off of pmck_host_map
+	conn.send_eol("redfish node status | grep 'node 0'")
+
+	re_state = re.compile('\[(.*)', re.IGNORECASE)
+	conn.log_expect(re_state, int(options["--shell-timeout"]))
+
+	status = conn.match.group(1).lower()
+
+	if status.startswith("on"):
+		return "on"
+	else:
+		return "off"
+
+def set_power_status(conn, options):
+	"""
+	Sets the power state of Node 0 to 'on' or 'off'
+	"""
+
+	if options["--action"] == "on":
+		conn.send_eol("redfish node 0 on")
+	else:
+		if allow_powerOff:
+			conn.send_eol("redfish node 0 forceoff")
+
+	conn.log_expect(options["--command-prompt"], int(options["--power-timeout"]))
+
+	return
+
+def reboot_cycle(conn, options):
+	"""
+	Powers off the node (it does not actually power cycle)
+	"""
+
+	conn.send_eol("redfish node 0 forceoff")
+	conn.log_expect(options["--command-prompt"], int(options["--power-timeout"]))
+
+	if get_power_status(conn, options) == "off":
+		logging.error("Timed out waiting to power ON\n")
+
+	return True
+
+# From the HSS redfish utility
+# nC usage:
+# redfish node status
+# redfish node [0-3] [on|off|forceoff]
+
+def main():
+	"""
+	Agent entry point
+	"""
+
+	device_opt = ["ipaddr", "login", "passwd", "cmd_prompt", "secure"]
+
+	atexit.register(atexit_handler)
+
+	all_opt["cmd_prompt"]["default"] = ["nC-RBTP:>"]
+	all_opt["power_wait"]["default"] = 5
+
+	options = check_input(device_opt, process_input(device_opt))
+
+	docs = {}
+	docs["shortdesc"] = "Fence agent for HPE HSS over SSH"
+	docs["longdesc"] = "fence_hpe_hss_ssh is a fence agent that connects to the HSS device. It logs into \
+device via ssh and interacts with a specific node.\n"
+	docs["vendorurl"] = "http://www.hpe.com"
+	docs["symlink"] = [("fence_hpe_hss_ssh", "Fence agent for HPE HSS over SSH")]
+	show_docs(options, docs)
+
+	##
+	## Operate the fencing device
+	####
+	conn = fence_login(options)
+
+	result = fence_action(conn, options, set_power_status, get_power_status, get_power_status)
+
+	fence_logout(conn, "exit")
+	sys.exit(result)
+
+if __name__ == "__main__":
+	main()

docs/rfcs/0003/src/fence_hpe_nnf/Readme.md

@@ -0,0 +1,44 @@
+# fence_hpe_hss_ssh.py
+
+## This agent is a work in progress and is not currently operational, it requires the following to be complete:
+1. Logic to identify the Rabbit node based on the information passed in the pmck_host_list
+2. A Kubernetes...
+   1. Role setup to read/write Storage CRDs
+   2. Service Account for this agent
+   3. RoleBinding to marry 1 & 2 above
+   4. The SA cert & token to enable access to the Kubernetes API and Storage CR
+3. Properly formed k8s request to pull the Storage Spec
+4. Logic to update the Storage Spec
+5. Properly formed k8s request to update the Storage Spec
+
+## Overview
+fence_hpe_nnf.py is a fencing agent intended to run in a PaceMaker cluster environment.  This agent uses the Kubernetes API to mark a specific NNF node as Degrated.
+
+It makes use of the fencing agent python library created by ClusterLabs to abstract much of the boilerplate so that only the actual fencing logic appears in the agent code.  The ClusterLabs github repository is located here:
+
+https://github.com/ClusterLabs/fence-agents
+
+The fencing library tends to live in /usr/share/fence, however installation locations may vary:
+
+    # ls -l /usr/share/fence/fenc*
+    -rwxr-xr-x. 1 root root 52344 Feb 11  2022 /usr/share/fence/fencing.py
+    -rwxr-xr-x. 1 root root  3753 Feb 11  2022 /usr/share/fence/fencing_snmp.py
+
+This agent works with Pacemaker 2.1 on TOSS 4.x
+
+    # stonith_admin --version
+    Pacemaker 2.1.0-8.el8
+    # uname -a
+    Linux rabbit-compute-3 4.18.0-348.20.1.2toss.t4.x86_64 #1 SMP Wed Apr 6 17:46:46 PDT 2022 x86_64 x86_64 x86_64 GNU/Linux    
+
+## Installation    
+
+*See fence_hpe_hss for more information on installation*
+
+Typically the fencing agent is selected and setup using the PaceMaker CLI tool in the following fashion:
+
+    # pcs stonith create gfs2_rabbit_nnf fence_hpe_nnf ip="10.1.1.6" pcmk_host_list="rabbit-node-1" ssh=1 username="<username>" password="<username>" op monitor interval=30s
+
+    Note:
+    * They don't make it easy to pass custom arguments in this framework, so recommend creating a config file under /etc for any additional info such as certs and tokens.
+