FenceAgentAPI.md

This page describes how to implement a fencing agent, and how agents are called from the cluster software.

Fencing, generally, is a way to prevent an ill-behaved cluster member from accessing shared data in a way which would cause data or file system corruption. The canonical case where fencing is required is something like this: Node1 live-hangs with a lock on a GFS file system. Node2 thinks node1 is dead, takes a lock, and begins accessing the same data. Node1 wakes up and continues what it was doing. Because we can not predict when node1 would wake up or prevent it from issuing I/Os immediately after waking up, we need a way to prevent its I/Os from completing even if it does wake up.

Definitions

• I/O Fencing is, in short, the act of preventing a node from issuing I/Os (usually to shared storage). It is also known as STOMITH or STONITH.
• Fencing devices are hardware components used to prevent a node from issuing I/Os.
• Fencing agents are software components used to communicate with fencing devices in order to perform I/O Fencing. In the context of this project, they are standalone applications which are spawned by the cluster software (compared to, for example, Linux-HA which uses dynamically-loaded modules).
• Power fencing is when a node is power-cycled, reset, or turned off to prevent it from issuing I/Os
• Fabric fencing is when a node's access to the shared data is cut off at the device-level. Disabling a port on a fibre channel switch (zoning), revoking a SCSI3 group reservation, or disabling access at the SAN itself from a given initiator's GUID are all examples of fabric fencing.
• Manual fencing or meatware is when an administrator must manually power-cycle a machine (or unplug its storage cables) and follow up with the cluster, notifying the cluster that the machine has been fenced. This is never recommended.

What is given to fencing agents by fenced & stonithd

When the cluster decides a node must be fenced, a node is chosen to perform the task. The fence daemon ("fenced") is responsible for performing the request, and it spawns agents listed for a given node as noted in cluster.conf. When called by other pieces of software (fenced, fence_node), fencing agents take arguments from standard input (as opposed to the command line) in the following format:

argument=value
#this line is ignored
argument2=value2

• Lines beginning with a # should be ignored
• One line per argument
• Argument and value are separated by an equals sign
• If argument is specified several times only last value is used

The following things are not allowed in this input:

• Spaces in the argument (or anything else which isn't allowed by XML standards for an attribute name)
• Spaces between the argument and the equals sign (=)
• Newlines in the value (or anything else which isn't allowed by XML standards for an attribute value)

The following are guidelines for parsing the arguments:

• Quotation marks around the value is not necessary, and are not provided by the fencing system. Simply parse from the equals sign to the newline.
• Equals signs are allowed in the argument, but it is not recommended.
• Arguments which are not recognized should be ignored by the fencing agent.

Example cluster.conf

<clusternodes>
    <clusternode name="red.lab.boston.redhat.com" nodeid="1" votes="1">
        <fence>
            <method name="1">
                <device name="ips-rack9" port="1" action="reboot"/>
            </method>
        </fence>
    </clusternode>
    ...
</clusternodes>
<fencedevices>
    <fencedevice agent="fence_wti" name="ips-rack9" passwd="wti" ipaddr="ips-rack9"/>
</fencedevices>

Example input given to a fence_agent

Given the previous cluster.conf example, the following is sent to the agent named fence_wti when the cluster decides to fence red.lab.boston.redhat.com:

agent=fence_wti
name=ips-rack9
passwd=wti
ipaddr=ips-rack9
port=1
action=reboot
nodename=red.lab.boston.redhat.com

Note that fenced always passes the name of the node to be fenced via the nodename argument.

Agent Operations and Return Values

• off - fence / turn off / etc. This operation is required. Return values:
  • 0 if the operation was successful, or
  • 1 if not successful or verification could not be performed. This includes inability to contact the fence device at any point.
• on - un-fence / turn on / etc. Return values:
  • 0 if the operation was successful, or
  • 1 if not successful or verification could not be performed. This includes inability to contact the fence device at any point.
• reboot - this is normally an off followed by an on, but not always. It is important to note that in some cases, this operation is not verifiable. For example, if you use an integrated power management feature like iLO or IPMI to reset the node, there is no point at which the node has lost power. Return values:
  • 0 if the off portion was successful (the on portion failing is a don't-care case for the cluster, as it can recover safely)
  • 1 if the off portion was unsuccessful (see above for reasons).
• status - this is used by pacemaker to verify that the agent is working. It is not required by 'fenced'. Use is encouraged. Return values:
  • 0 if the fence device is reachable and the port is in the on state
  • 1 if the fence device could not be contacted
  • 2 if the fence device is reachable but is in the off state
• monitor - Attempt to contact the fencing device. Typically, 'status' for one-port hardware, and 'list' for multi-port hardware. Return values:
  • 0 if the fence device is reachable and working properly
  • 1 if the fence device could not be contacted
  • 2 if the fence device is reachable but is in the off state (single-port hardware only)
• list - Multi-port fencing devices only. Prints a list of port names and assignments Return values:
  • 0 if the fence device is reachable and working properly
  • 1 if the fence device could not be contacted

Attribute Specifications

These attributes should be used when implementing new agents, but this is not an exhaustive list. Some agents may have other arguments which are not covered here.

• action - the operation (noted previously) to perform. This is one of the following (case insensitive): on, off, reboot, monitor, list, or status
• option - (DEPRECATED; use action) - same as action
• ipaddr - for a hostname or IP address
• login - for a username or login name
• passwd - for a password
• passwd_script - if your agent supports storing passwords outside of cluster.conf, this is a script used to retrieve your password (details on how this works will be added later). Generally, this script simply echoes the password to standard output (and is read in by the agent at run-time).
• port - if you have to specify a plug or port (for example, on a network-enabled PDU with 8 ports)
• nodename - if the agent fences by node name, this is the parameter to use (e.g. instead of port). In the event that both nodename and port are specified, the preference is given to port.

Implementation Best-Practices

Currently, this project has a number of requirements which should be common to all agents (even if not currently):

• Verifiable operation - When a node has been fenced by a particular device, the agent should query the device to ensure the new state has taken effect. For example, if you turn the a power plug "off", the agent should, after doing this, query the device and verify that the plug is in the "off" state. This is not necessary for the "on" case or when you "un-fence" a node.
• Timeout is a failure, not success - A timeout is an assumption, and we want a guarantee, which is why we generally verify operations from within the agents after performing them.
• Fabric fencing must never have a reboot operation - Don't waste time implementing one.
• There should be a command line mode, too - for debugging, your agent should be able to operate using arguments passed in via the command line as well. How the mapping is done between the command line arguments and stdin arguments is implementation dependent, i.e., it's up to you ;)
• Whitespace in stdin - existing agents tend to strip leading whitespace (before the argument) when processing arguments from standard input, but this is not a requirement. Whether whitespace is stripped between the equals sign and newline in the value is implementation dependent.
• Output of fence_agent -o metadata should validate against RelaxNG schema available at lib/metadata.rng