D2 Quickstart Tutorial

In this tutorial, we will explain the basic concepts of D2 using a simple client server project. Apache Zookeeper (http://zookeeper.apache.org/) is required for doing this tutorial. The completed code for this tutorial is available in rest.li’s examples/d2-quickstart.

What is D2 in a nutshell?

Imagine we have a Service Oriented Architecture. Let’s say we have hundreds of servers. Each server can host different set of services. Some of those services maybe partitioned so a server may belong to some specific partitions for those services. We use D2 to store information about which server can serve what service. So this means with D2, a client requesting a particular service doesn’t need to know where the physical servers are. The client can ask D2 to route a request to the right server. So D2 is similar to DNS in some ways. At the core, what D2 does is nothing but a layer of indirection between a client and a server. But D2 supports many other goodies like client side load balancing, partitioning and multi-data-center routing.

Our smallest unit of indirection is called a service. A service can be a URL endpoint, a restli resource, it can be anything as long as the name of the service unique. A collection of services is called a cluster. A service that belongs to one cluster cannot belong to a different cluster. A cluster has one-to-many relationship to a service. All this information about clusters and services is stored in zookeeper.

A server joins a cluster by creating an ephemeral node in zookeeper. When a server dies, zookeeper will notice because the heart beat message is not refreshed, then the ephemeral node is automatically removed.

A client attempting to send request to a service first consults zookeeper to find out which cluster owns the service. Then the client queries zookeeper for all the ephemeral nodes (servers) for that cluster. Given a list of ephemeral node, the client will deliberately choose a server to send the request to.

That is all you need to know about D2 in a nutshell.

The tutorial

We will create a basic client server application in Java. We use gradle for our build process. The top level structure of our project will have 3 subdirectories:

/server
/client
/config

You also need a settings.gradle and build.gradle file in the root directory.
- For settings.gradle -

 
include 'server'
include 'client'
include 'config'

This will tell gradle that gradle should search for ‘server’, ‘client’, ‘config’ directories and mark them as part of the project.
- For build.gradle -

 
allprojects {
    apply plugin: 'idea'
    apply plugin: 'eclipse'
}
final pegasusVersion = ‘1.15.9’

ext.spec = [

    ‘product’ : [

        ‘pegasus’ : [

                ‘r2’ : ‘com.linkedin.pegasus:r2:’ + pegasusVersion,

                ‘d2’ : ‘com.linkedin.pegasus:d2:’ + pegasusVersion

        ]

    ]

]
subprojects {

    repositories {

        mavenLocal()

        mavenCentral()

    }

}

This tells gradle that it should use pegasus artifact version 1.15.9 from maven central repository. This also tells gradle we have dependency to r2 and d2 libraries.

Step 1. Create a server

Create the following project structure for ‘server’ subdirectory. We need EchoServer.java and ExampleD2Server.java.

• d2-quickstart/
  • client/
  • config/
  • server/
    • src/
      • main/
        • java/
          • com/
            • example/
              • d2/
                • server/
                  • EchoServer.java
                  • ExampleD2Server.java
        • config/
          • server.json

For this example we are creating an echo server to represent a real production server. The echo server always returns 200 success and prints out to stdout when a request comes in. Here is the implementation of the echo server

public class EchoServer
{
  private final int        _port;
  private final HttpServer _server;

  public EchoServer (int port, final String name, List<String> contextPaths)
      throws IOException
  {
    _port = port;
    _server = HttpServer.create(new InetSocketAddress(_port), 0);
    for (String contextPath : contextPaths)
    {
      _server.createContext(contextPath, new MyHandler(contextPath, name));
    }
    _server.setExecutor(null);
  }

  static class MyHandler implements HttpHandler
  {
    private final String _name;
    private final String _serverName;

    private MyHandler(String name, String serverName)
    {
      _name = name;
      _serverName = serverName;
    }

    public void handle(HttpExchange t) throws IOException
    {
      System.out.println(new Date().toString() + ": " + _serverName
                             + " received a request for the context handler = " + _name );
      String response = "Successfully contacted server " + _serverName;
      t.sendResponseHeaders(200, response.length());
      OutputStream os = t.getResponseBody();
      os.write(response.getBytes());
      os.close();
    }
  }

  public void start()
      throws IOException
  {
    _server.start();
  }

  public void stop()
      throws IOException
  {
    _server.stop(0);
  }

}

Step 2. Create a config runner

• d2-quickstart/
  • client/
  • config/
    • src/
      • main/
        • java/
          • com/
            • example/
              • d2/
                • config/
                  • ConfigRunner.java
        • d2Config/
          • d2Config.json
  • server/

Step 3. Create a client

• d2-quickstart/
  • client/
    • src/
      • main/
        • java/
          • com/
            • example/
              • d2/
                • client/
                  • ExampleD2Client.java
        • config/
          • client.json
  • config/
  • server/