Cluster Management Backends

Sheepdog uses a cluster management backend to manage memebership and broadcast messages to the cluster nodes.

For now, sheepdog can use corosync (the default), zookeeper and Accord.

Corosync

Corosync runs on every node as a dademon.

Corosync can only works reliably with less than 15 nodes, due to its implementation and design goal for small sized cluster.

Zookeeper

Zookeeper runs as a standalone cluster. This means you need to set up zookeeper cluster first, then pass the IP:PORT list to sheep start-up option.

If you want to add more nodes into sheepdog cluster, you should run sheepdog against zookeeper. Some users (eg. at Taobao.com) have been working with the scalability of the sheepdog and currently running with around 1000 nodes in their test environment. For several month expediencies with zookeeper, they have found that it works well for node number below 1000, with object cache enabled.

To enable zookeeper support (Suppose we have a zookeeper cluster with 3 nodes):

Install zookeeper (by source tarball)

Fetch the newest tarball(>= 3.3.4)

$ wget http://mirror.bjtu.edu.cn/apache/zookeeper/zookeeper-3.3.4/zookeeper-3.3.4.tar.gz

Install zookeeper C client library(used by sheep)

$ tar -zxvf zookeeper-3.3.4.tar.gz
$ cd zookeeper-3.3.4/src/c
$ ./configure
$ make
$ make install

Set configuration

$ cd zookeeper-3.3.4/conf
$ mv zoo_sample.cfg zoo.cfg
  --You can learn more options from here: http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_configuration
$ mkdir -p /tmp/zookeeper

Start zookeeper

$ sudo ./bin/zkServer.sh start 

Start sheepdog

$ sudo sheep -d /store/29 -z 29 -p 7029 -c zookeeper:127.0.0.1:2181

Install zookeeper (Debian-based distribution)

$ sudo apt-get install zookeeper

Start the zookeeper

$ /usr/share/zookeeper/bin/zkServer.sh start # The default port is 2181

To compile sheep, firstly install zookeeper files (Debian-based distribution)

$ sudo apt-get install libzookeeper-dev

Then configure, make the sheep source

$ ./configure --enable-zookeeper
$ make
$ sudo make install

Enable zookeeper driver in sheep

$ git clone git://github.com/collie/sheepdog.git
$ cd sheepdog/
$ ./configure --enable-debug --enable-zookeeper
$ make
$ sudo make install

Start the sheep

$ sheep -c zookeeper:IP1:PORT1,IP2:PORT2,IP3:PORT3 ...other...option...

Accord

For a cluster more than 1000 nodes, I think Accord would come up to our rescue, but it is currently in a unstable development state. When the sheepdog scales up to 1000 nodes reliably, we might go to look at Accord and refine it to be a working state.