Hare User Guide

Welcome

Hare is an experiment in Social Architecture, disguised as a software project.

What Hare does?

1. Configures Motr object store.
2. Starts/stops Motr services.
3. Notifies Motr of service and device faults.

Hare implementation uses Consul key-value store and health-checking mechanisms.

Installation

Building from source

• Download Hare.

  git clone https://github.com/Seagate/cortx-hare.git hare
  cd hare

• Install Python (≥ 3.6), libraries and header files needed to compile Python extensions.

  sudo yum -y install python3 python3-devel

• Install Consul.

  sudo yum -y install yum-utils
  sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
  sudo yum -y install consul-1.7.8

• Install Motr.

  • .. from RPMs

    sudo yum -y install cortx-motr cortx-motr-devel

  • .. or from sources

    git clone --recursive https://github.com/Seagate/cortx-motr.git motr
    cd motr
    
    scripts/m0 make
    sudo scripts/install-motr-service --link
    
    export M0_SRC_DIR=$PWD
    cd -

• Build and install Hare.

  make
  sudo make devinstall

• Add current user to hare group.

  sudo usermod --append --groups hare $USER

  Log out and log back in.

Quick start

☑️ Checklist

Before starting the cluster as <user> at <origin> machine, ensure that

#	Check	Where
1	passwordless sudo works for <user>	all machines
2	<user> can ssh from <origin> to other machines	<origin>
3	cortx-hare and cortx-s3server RPMs are installed	all machines
4	/opt/seagate/cortx/hare/bin is in <user>'s PATH	all machines
5	<user> is a member of hare group	all machines
6	CDF exists and corresponds to the actual cluster configuration	<origin>

Prepare the CDF

If you are starting the cluster for the first time, you will need a cluster description file (CDF).

See cfgen --help-schema for the description of CDF format.

You can make a copy of /opt/seagate/cortx/hare/share/cfgen/examples/singlenode.yaml (single-node setup) or /opt/seagate/cortx/hare/share/cfgen/examples/ldr1-cluster.yaml (dual-node setup) and edit it as necessary.

cp /opt/seagate/cortx/hare/share/cfgen/examples/singlenode.yaml ~/CDF.yaml
vi ~/CDF.yaml

You will probably need to modify host, data_iface, and io_disks values.

data_iface

• Make sure that data_iface value refers to existing network interface (it should be present in the output of ip a command).

• This network interface must be configured for LNet. If you can see its IP address in the output of sudo lctl list_nids command, you are all set. Otherwise, configure LNet by executing this code snippet on each node:

  IFACE=eth1  # XXX `data_iface` value from the CDF
  sudo tee /etc/modprobe.d/lnet.conf <<< \
      "options lnet networks=tcp($IFACE) config_on_load=1"

io_disks

• Devices specified in io_disks section must exist.

• Sometimes it is convenient to use loop devices instead of actual disks:

  sudo mkdir -p /var/motr
  for i in {0..9}; do
      sudo dd if=/dev/zero of=/var/motr/disk$i.img bs=1M seek=9999 count=1
      sudo losetup /dev/loop$i /var/motr/disk$i.img
  done

Hare we go

• Start the cluster.

  hctl bootstrap --mkfs ~/CDF.yaml

• Run I/O test.

  /opt/seagate/cortx/hare/libexec/m0crate-io-conf >/tmp/m0crate-io.yaml
  dd if=/dev/urandom of=/tmp/128M bs=1M count=128
  sudo m0crate -S /tmp/m0crate-io.yaml

• Stop the cluster.

  hctl shutdown

Reporting problems

To request changes or report a bug, please log an issue and describe the problem you are facing.

When reporting a bug, consider running

hctl reportbug

to collect forensic data. Run this command on every node of the cluster and attach generated files to the GitHub issue.

Troubleshooting

LNet is not configured

• To check, run

  sudo lctl list_nids

  This command should show network identifiers.

• If it doesn't, try to start LNet manually:

  sudo modprobe lnet
  sudo lctl network up

  Run sudo lctl list_nids again.

• Still no luck? Perhaps /etc/modprobe.d/lnet.conf file is missing or corrupted. Create it with these commands:

  IFACE=eth1  # XXX `data_iface` value from the CDF
  sudo tee /etc/modprobe.d/lnet.conf <<< \
      "options lnet networks=tcp($IFACE) config_on_load=1"

  Try to start LNet one more time.

RC Leader cannot be elected

If hctl bootstrap cannot complete and keeps printing dots..........

2020-01-14 10:57:25: Generating cluster configuration... Ok.
2020-01-14 10:57:26: Starting Consul server agent on this node.......... Ok.
2020-01-14 10:57:34: Importing configuration into the KV Store... Ok.
2020-01-14 10:57:35: Starting Consul agents on remaining cluster nodes... Ok.
2020-01-14 10:57:35: Update Consul agents configs from the KV Store... Ok.
2020-01-14 10:57:36: Install Motr configuration files... Ok.
2020-01-14 10:57:36: Waiting for the RC Leader to get elected..................[goes on forever]

try these commands

hctl shutdown
sudo systemctl reset-failed hare-hax

and bootstrap again.

See also

• Hare RFCs
• Hare wiki