Getting Started with BFT SMaRt

snakejerusalem edited this page Apr 4, 2016 · 29 revisions

BFT-SMaRt provides a middleware to replicate request from multiple clients to multiple servers. This page has the instructions to download and install BFT-SMaRt.

Download

First, download the latest stable version of BFT-SMaRt from our site or from the repository.

Installation

BFT-SMaRt's code must be installed in each replica and client that will use it. To do so, first, extract the downloaded archive. After that, copy the following files and folders to each of the replicas and clients:

  • bin/BFT-SMaRt.jar
  • config/
  • lib/

After copying the files to the replicas, the first step is to get the IP addresses from each replica and define a port for each one to receive the messages from other replicas. After that, edit the file config/hosts.config in each replica to set the IP address and port for each one. The information must be the same across all replicas. Let's use as example this configuration:

0 127.0.0.1 10001
1 127.0.0.2 10001
2 127.0.0.3 10001
3 127.0.0.4 10001

For each line, the first parameter is the replica ID. The second parameter is the IP address and the third is the port. This information must be the same across all replicas.

If you wish to install all BFT-SMaRt replicas in the same host, do not forget to assign different ports to each replica, like this:

0 127.0.0.1 10000
1 127.0.0.1 11000
2 127.0.0.1 12000
3 127.0.0.1 13000

By now you should be able to start replicas and run demo examples included in BFT-SMaRt binary. Instructions to run these demos can be found at our Demos page.

Below, we describe how to use BFT-SMaRt to build your own application. We will take as an example a implementation of the java.util.Map interface. In this example, multiple clients will be able to perform operations in a TreeMap, whose data will be replicated among several replicas.

Client code

Lets start by creating the client implementation of the Map interface:

package foo.gettingstarted.client;

// This is the class which sends requests to replicas
import bftsmart.tom.ServiceProxy;

// Classes that need to be declared to implement this
// replicated Map
import foo.gettingstarted.RequestType;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.IOException;
import java.util.Collection;
import java.util.Map;
import java.util.Set;

public class MapClient implements Map<String, String> {
...

The interface java.util.Map defines several methods to be implemented. The objective of this example is to show how to extend BFT-SMaRt to do your own application, so, we will only implement the methods put(String, String), get(Object), remove(Object) and size().

The TreeMap we are creating will be implemented as a BFT-SMARt client that interact with replicas through the class bftsmart.tom.ServiceProxy, which can interact with the replicas in a transparent way. We will also create a constructor to pass the client ID as a parameter, so the requests performed by this client will be uniquely identified.

ServiceProxy clientProxy = null;

public MapClient(int clientId) {
	clientProxy = new ServiceProxy(clientId);
}

To make the code clearer we will create a class to indicate the request type.

 package foo.gettingstarted;
 
 public class RequestType {
     public static final int PUT = 1;
     public static final int GET = 2;
     public static final int REMOVE = 3;
     public static final int SIZE = 4;
}

We will now implement the put method. We use ByteArrayOutputStream and DataOutputStream to encode client's key/value pairs into a byte array, since the ServiceProxy.invokeOrdered method takes a byte array as parameter. We are using the invokeOrdered method because put is a write operation, hence it is necessary to guarantee that all replicas see the same sequence of operations.

    @Override
public String put(String key, String value) {
	ByteArrayOutputStream out = new ByteArrayOutputStream();
	DataOutputStream dos = new DataOutputStream(out);
	try {
		dos.writeInt(RequestType.PUT);
		dos.writeUTF(key);
		dos.writeUTF(value);
		byte[] reply = clientProxy.invokeOrdered(out.toByteArray());
		if(reply != null) {
			String previousValue = new String(reply);
			return previousValue;
		}
		return null;
	} catch(IOException ioe) {
		System.out.println("Exception putting value into hashmap: " + ioe.getMessage());
		return null;
	}
}

The get method is similar to the put method. The difference is that get invokes theServiceProxy.invokeUnordered method to read values. The invokeUnordered method will not trigger the BFT-SMaRt's protocol to order the requests since it is a read-only operation, which does not change the state of the application.

@Override
public String get(Object key) {
	try {
		ByteArrayOutputStream out = new ByteArrayOutputStream();
		DataOutputStream dos = new DataOutputStream(out);
		dos.writeInt(RequestType.GET);
		dos.writeUTF(String.valueOf(key));
		byte[] reply = clientProxy.invokeUnordered(out.toByteArray());
		String value = new String(reply);
		return value;
	} catch(IOException ioe) {
		System.out.println("Exception getting value from the hashmap: " + ioe.getMessage());
		return null;
	}
}

The remove method will look similar to the put method, since removing a key is also a write operation.

@Override
public String remove(Object key) {
	ByteArrayOutputStream out = new ByteArrayOutputStream();
	DataOutputStream dos = new DataOutputStream(out);
	try {
		dos.writeInt(RequestType.REMOVE);
		dos.writeUTF(String.valueOf(key));
		byte[] reply = clientProxy.invokeOrdered(out.toByteArray());
		if(reply != null) {
			String removedValue = new String(reply);
			return removedValue;
		}
		return null;
	} catch(IOException ioe) {
		System.out.println("Exception removing value from the hashmap: " + ioe.getMessage());
		return null;
	}
}

The size method is almost the same as get, with the difference that it does not pass any parameter.

@Override
public int size() {
	try {
		ByteArrayOutputStream out = new ByteArrayOutputStream();
		DataOutputStream dos = new DataOutputStream(out);
		dos.writeInt(RequestType.SIZE);
		byte[] reply = clientProxy.invokeUnordered(out.toByteArray());
		ByteArrayInputStream in = new ByteArrayInputStream(reply);
		DataInputStream dis = new DataInputStream(in);
		int size = dis.readInt();
		return size;
	} catch(IOException ioe) {
		System.out.println("Exception getting the size the hashmap: " + ioe.getMessage());
		return -1;
	}
}

Given that in this example we do not aim to support all methods provided by the Map interface, we need to implement the remaining methods using the following code:

throw new UnsupportedOperationException("Not supported yet.");

Replica code

To implement the application on replicas, an object must be instantiated to deliver client requests to that application.

BFT-SMaRt requires the replicas to implement the Executable and Recoverable interfaces. The Executable interface defines methods to process ordered and unordered requests. The application must implement it's business logic within these methods.

The Recoverable interface defines methods to manage the application's state. It is used to store the state during system execution and fetch it back if a replica fails or is added into the system on-the-fly.

In our example, we extend the class DefaultRecoverable, which provides a basic state management strategy. It extends the BatchExecutable interface, that receives entire batches of requests from BFT-SMaRt.

Request processing

package foo.gettingstarted.server;

// These are the classes which receive requests from clients
import bftsmart.tom.MessageContext;
import bftsmart.tom.ServiceReplica;
import bftsmart.tom.server.defaultservices.DefaultRecoverable;
import foo.gettingstarted.RequestType;

// Classes that need to be declared to implement this
// replicated Map
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.IOException;
import java.io.ObjectInput;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.util.TreeMap;
import java.util.Map;

public class TreeMapServer extends DefaultRecoverable {

    Map<String, String> table;

    public TreeMapServer(int id) {
        table = new TreeMap<>();
        new ServiceReplica(id, this, this);
    }

    public static void main(String[] args) {
        if (args.length < 1) {
            System.out.println("Usage: HashMapServer <server id>");
            System.exit(0);
        }

        new TreeMapServer(Integer.parseInt(args[0]));
    }

The Map<String, String> table is the actual object in which the data will be stored in the server. The ServiceReplica object is responsible for delivering requests and return replies. We also need to implement the main method, since TreeMapServer is also responsible for launching the processes associated with the application and the BFT-SMaRt replication protocol.

Because DefaultRecoverable delivers a batch of commands, we need to implement the abstract method appExecuteBatch, that delivers an array of commands that need to be execute in a deterministic order and also produce deterministic replies. We can enforce a deterministic order of processing as follows:

    @Override
    public byte[][] appExecuteBatch(byte[][] command, MessageContext[] mcs) {

        byte[][] replies = new byte[command.length][];
        for (int i = 0; i < command.length; i++) {
            replies[i] = executeSingle(command[i], mcs[i]);
        }

        return replies;
    }

All requests delivered within a batch are ordered request. Unordered requests are never included in batches.

Lets start by implementing the executeSingle method used in the above snippet, which processes those ordered requests. As we explained earlier, these requests represent either put or remove operation. Therefore, we implement half of the application's business logic in this method.

The implementation of executeSingle parses the request to identify it's operation type (put or remove). After that, it reads the respective arguments, applies the operations into the TreeMap and returns the result that will be delivered to the client that issued the request.

    private byte[] executeSingle(byte[] command, MessageContext msgCtx) {
        ByteArrayInputStream in = new ByteArrayInputStream(command);
        DataInputStream dis = new DataInputStream(in);
        int reqType;
        try {
            reqType = dis.readInt();
            if (reqType == RequestType.PUT) {
                String key = dis.readUTF();
                String value = dis.readUTF();
                String oldValue = table.put(key, value);
                byte[] resultBytes = null;
                if (oldValue != null) {
                    resultBytes = oldValue.getBytes();
                }
                return resultBytes;
            } else if (reqType == RequestType.REMOVE) {
                String key = dis.readUTF();
                String removedValue = table.remove(key);
                byte[] resultBytes = null;
                if (removedValue != null) {
                    resultBytes = removedValue.getBytes();
                }
                return resultBytes;
            } else {
                System.out.println("Unknown request type: " + reqType);
                return null;
            }
        } catch (IOException e) {
            System.out.println("Exception reading data in the replica: " + e.getMessage());
            e.printStackTrace();
            return null;
        }
    }

To process unordered requests, it is necessary to implement the abstract method appExecuteUnordered. As explained earlier, these unordered operations are get and size.

    @Override
    public byte[] appExecuteUnordered(byte[] command, MessageContext msgCtx) {
        ByteArrayInputStream in = new ByteArrayInputStream(command);
        DataInputStream dis = new DataInputStream(in);
        int reqType;
        try {
            reqType = dis.readInt();
            if (reqType == RequestType.GET) {
                String key = dis.readUTF();
                String readValue = table.get(key);
                byte[] resultBytes = null;
                if (readValue != null) {
                    resultBytes = readValue.getBytes();
                }
                return resultBytes;
            } else if (reqType == RequestType.SIZE) {
                int size = table.size();

                ByteArrayOutputStream out = new ByteArrayOutputStream();
                DataOutputStream dos = new DataOutputStream(out);
                dos.writeInt(size);
                byte[] sizeInBytes = out.toByteArray();

                return sizeInBytes;
            } else {
                System.out.println("Unknown request type: " + reqType);
                return null;
            }
        } catch (IOException e) {
            System.out.println("Exception reading data in the replica: " + e.getMessage());
            e.printStackTrace();
            return null;
        }
    }

State management

State management is needed to support recovery of replicas. DefaultRecoverable provides developers with a basic state management strategy, by periodically performing a checkpoint. This checkpoint is done by taking a snapshot of the application state and storing the following requests into a log. Furthermore, DefaultRecoverable also sends the snapshot and correspondent log to replicas that request the state.

It is the onus of the developer to define how the state is encoded (when taking a periodic snapshot) and decoded (when installing a snapshot on the recovered replica). This must be implemented, respectively, in the abstract methods getSnapshot and installSnapshot. Whereas getSnapshot is invoked periodically, installSnapshot might be invoked when (1) replicas that are to late to process requests jump to a more up-to-date state, or (2) replicas that were once failed/crashed re-start their execution from scratch.

Below there are simple implementations of these methods, using a methodology similar to the one used to send and receive client requests.

@Override
public void installSnapshot(byte[] state) {
	ByteArrayInputStream bis = new ByteArrayInputStream(state);
	try {
		ObjectInput in = new ObjectInputStream(bis);
		table = (Map<String, String>)in.readObject();
		in.close();
		bis.close();
	} catch (ClassNotFoundException e) {
		System.out.print("Coudn't find Map: " + e.getMessage());
		e.printStackTrace();
	} catch (IOException e) {
		System.out.print("Exception installing the application state: " + e.getMessage());
		e.printStackTrace();
	}
}

@Override
public byte[] getSnapshot() {
	try {
		ByteArrayOutputStream bos = new ByteArrayOutputStream();
		ObjectOutputStream out = new ObjectOutputStream(bos);
		out.writeObject(table);
		out.flush();
		out.close();
		bos.close();
		return bos.toByteArray();
	} catch (IOException e) {
		System.out.println("Exception when trying to take a + " +
				"snapshot of the application state" + e.getMessage());
		e.printStackTrace();
		return new byte[0];
	}
}

Test

To test this replicated Map implementation, you can use a small console application to perform operations in the classes we described. This example is similar to the demo found in bftsmart.demo.bftmap.BFTMapInteractiveClient.

package foo.gettingstarted.client;

import java.io.Console;
import java.util.Scanner;

public class ConsoleClient {

	public static void main(String[] args) {
		if(args.length < 1) {
			System.out.println("Usage: ConsoleClient <client id>");
		}

		HashMapClient client = new HashMapClient(Integer.parseInt(args[0]));
		Console console = System.console();

		Scanner sc = new Scanner(System.in);

		while(true) {
			System.out.println("Select an option:");
			System.out.println("1. ADD A KEY AND VALUE TO THE MAP");
			System.out.println("2. READ A VALUE FROM THE MAP");
			System.out.println("3. REMOVE AND ENTRY FROM THE MAP");
			System.out.println("4. GET THE SIZE OF THE MAP");

			int cmd = sc.nextInt();

			switch(cmd) {
			case 1:
				System.out.println("Putting value in the map");
				String key = console.readLine("Enter the key:");
				String value = console.readLine("Enter the value:");
				String result =  client.put(key, value);
				System.out.println("Previous value: " + result);
				break;
			case 2:
				System.out.println("Reading value from the map");
				key = console.readLine("Enter the key:");
				result =  client.get(key);
				System.out.println("Value read: " + result);
				break;
			case 3:
				System.out.println("Removing value in the map");
				key = console.readLine("Enter the key:");
				result =  client.remove(key);
				System.out.println("Value removed: " + result);
				break;
			case 4:
				System.out.println("Getting the map size");
				int size = client.size();
				System.out.println("Map size: " + size);
				break;
			}
		}
	}
}

Running the test

To execute this test it is necessary to reference all the libraries and binaries to the java classpath.

Lets assume ./foo to be the directory were the test is executed. It is necessary to have in this directory the BFT-SMaRt binary (which we assume it is placed in ./foo/bin/BFT-SMaRt.jar), configuration files (./foo/config), the libraries (./foo/lib) and the custom application code we developed (which we assume it is already compiled into a jar file and placed in ./foo/dist/foo.jar).

The ./foo/lib/ directory must contain all libraries from BFT-SMaRt' zip file. The same must be ensured for ./foo/config/, that must contain the configuration files included in the zip file, as well as the modifications to hosts.config that we described in the beginning of this tutorial.

If you are running the test in multiple hosts, please remember to do this process in all replicas and also to set hosts.config up to reflect the correct IPs for the replicas.

Assuming that the current directory is ./foo/, the command to start the replicas is:

$ java -cp bin/BFT-SMaRt.jar:lib/*:dist/foo.jar foo.gettingstarted.server.TreeMapServer

It is necessary to instantiate 3f+1 or more replicas to perform this test. The parameter 'f' represents the number of faults we wish to tolerate. TAssuming that the protocol will tolerate one faulty replica, it is necessary to launch four replicas, so you need to repeat the command above with replicas ID from 0 to 3. Replicas must be launched in that order.

To run the console client, the command line is:

$ java -cp bin/BFT-SMaRt.jar:lib/*:dist/foo.jar foo.gettingstarted.client.ConsoleClient 1001

After that the text interface will be displayed and it will be possible to test the implemented TreeMap operations.

Code

The complete source code for this example can be found here, and 'foo.jar' can be downloaded here.

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.