-
Notifications
You must be signed in to change notification settings - Fork 0
D2 Quickstart Tutorial
In this tutorial, we will explain the basic concepts of D2 using a simple client server project. Apache Zookeeper (http://zookeeper.apache.org/) is required for doing this tutorial. The completed code for this tutorial is available in rest.li’s examples/d2-quickstart.
Imagine we have a Service Oriented Architecture. Let’s say we have hundreds of servers. Each server can host different set of services. Some of those services maybe partitioned so a server may belong to some specific partitions for those services. We use D2 to store information about which server can serve what service. So this means with D2, a client requesting a particular service doesn’t need to know where the physical servers are. The client can ask D2 to route a request to the right server. So D2 is similar to DNS in some ways. At the core, what D2 does is nothing but a layer of indirection between a client and a server. But D2 supports many other goodies like client side load balancing, partitioning and multi-data-center routing.
Our smallest unit of indirection is called a service. A service can be a URL endpoint, a restli resource, it can be anything as long as the name of the service unique. A collection of services is called a cluster. A service that belongs to one cluster cannot belong to a different cluster. A cluster has one-to-many relationship to a service. All this information about clusters and services is stored in zookeeper.
A server joins a cluster by creating an ephemeral node in zookeeper. When a server dies, zookeeper will notice because the heart beat message is not refreshed, then the ephemeral node is automatically removed.
A client attempting to send request to a service first consults zookeeper to find out which cluster owns the service. Then the client queries zookeeper for all the ephemeral nodes (servers) for that cluster. Given a list of ephemeral node, the client will deliberately choose a server to send the request to.
That is all you need to know about D2 in a nutshell.
We will create a basic client server application in Java. We use gradle for our build process. The top level structure of our project will have 3 subdirectories:
/server
/client
/config
You also need a settings.gradle and build.gradle file in the root directory.
- For settings.gradle -
include 'server'
include 'client'
include 'config'
This will tell gradle that gradle should search for ‘server’, ‘client’, ‘config’ directories and mark them as part of the project.
- For build.gradle -
allprojects {
apply plugin: 'idea'
apply plugin: 'eclipse'
}
final pegasusVersion = ‘1.15.9’
ext.spec = [
‘product’ : [
‘pegasus’ : [
‘r2’ : ‘com.linkedin.pegasus:r2:’ + pegasusVersion,
‘d2’ : ‘com.linkedin.pegasus:d2:’ + pegasusVersion
]
]
]
subprojects {
repositories {
mavenLocal()
mavenCentral()
}
}
This tells gradle that it should use pegasus artifact version 1.15.9 from maven central repository. This also tells gradle we have dependency to r2 and d2 libraries.
Create the following project structure for ‘server’ subdirectory. We need EchoServer.java and ExampleD2Server.java.
-
d2-quickstart/
- client/
- config/
-
server/
-
src/
-
main/
-
java/
-
com/
-
example/
-
d2/
-
server/
- EchoServer.java
- ExampleD2Server.java
-
server/
-
d2/
-
example/
-
com/
-
config/
- server.json
-
java/
-
main/
-
src/
In this example we are creating an echo server to illustrate a real production server. The echo server always returns http status code 200 and prints out to stdout when a request comes in.
First we create build.gradle to declare java library dependencies
apply plugin: 'java'
dependencies {
compile 'com.googlecode.json-simple:json-simple:1.1.1'
compile spec.product.pegasus.r2
compile spec.product.pegasus.d2
}
Here is the implementation of the echo server
package com.example.d2.server;
import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpHandler;
import com.sun.net.httpserver.HttpServer;
import java.io.IOException;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.util.Date;
import java.util.List;
public class EchoServer
{
private final int _port;
private final HttpServer _server;
public EchoServer (int port, final String name, List<String> contextPaths)
throws IOException
{
_port = port;
_server = HttpServer.create(new InetSocketAddress(_port), 0);
for (String contextPath : contextPaths)
{
_server.createContext(contextPath, new MyHandler(contextPath, name));
}
_server.setExecutor(null);
}
static class MyHandler implements HttpHandler
{
private final String _name;
private final String _serverName;
private MyHandler(String name, String serverName)
{
_name = name;
_serverName = serverName;
}
public void handle(HttpExchange t) throws IOException
{
System.out.println(new Date().toString() + ": " + _serverName
+ " received a request for the context handler = " + _name );
String response = "Successfully contacted server " + _serverName;
t.sendResponseHeaders(200, response.length());
OutputStream os = t.getResponseBody();
os.write(response.getBytes());
os.close();
}
}
public void start()
throws IOException
{
_server.start();
}
public void stop()
throws IOException
{
_server.stop(0);
}
}
We store the configuration for the servers in server.json. Here is the content of server.json:
{
"echoServers" :
[
{
"name" : "RecommendationService-1",
"port" : 39901,
"threadPoolSize" : 1,
"contextPaths" : [
"/articleRecommendation",
"/jobRecommendation"
]
},
{
"name" : "RecommendationService-2",
"port" : 39902,
"threadPoolSize" : 1,
"contextPaths" : [
"/articleRecommendation",
"/jobRecommendation"
]
},
{
"name" : "RecommendationService-3",
"port" : 39903,
"threadPoolSize" : 1,
"contextPaths" : [
"/articleRecommendation",
"/jobRecommendation"
]
},
{
"name" : "NewsService-1",
"port" : 39904,
"threadPoolSize" : 1,
"contextPaths" : [
"/newsArticle"
]
},
{
"name" : "NewsService-2",
"port" : 39905,
"threadPoolSize" : 1,
"contextPaths" : [
"/newsArticle"
]
},
{
"name" : "NewsService-3",
"port" : 39906,
"threadPoolSize" : 1,
"contextPaths" : [
"/newsArticle"
]
}
],
"d2Servers" :
[
{
"serverUri" : "http://localhost:39901",
"d2Cluster" : "RecommendationService",
"partitionData" : {
"0" : {
"weight" : "1.0"
}
}
},
{
"serverUri" : "http://localhost:39902",
"d2Cluster" : "RecommendationService",
"partitionData" : {
"0" : {
"weight" : "1.0"
}
}
},
{
"serverUri" : "http://localhost:39903",
"d2Cluster" : "RecommendationService",
"partitionData" : {
"0" : {
"weight" : "1.0"
}
}
},
{
"serverUri" : "http://localhost:39904",
"d2Cluster" : "NewsService",
"partitionData" : {
"0" : {
"weight" : "1.0"
}
}
},
{
"serverUri" : "http://localhost:39905",
"d2Cluster" : "NewsService",
"partitionData" : {
"0" : {
"weight" : "1.0"
}
}
},
{
"serverUri" : "http://localhost:39906",
"d2Cluster" : "NewsService",
"partitionData" : {
"0" : {
"weight" : "1.0"
}
}
}
],
"zkConnectString" : "localhost:2181",
"zkSessionTimeout" : 5000,
"zkBasePath" : "/d2",
"zkRetryLimit" : 10,
"announcerStartTimeout" : 5000,
"announcerShutdownTimeout" : 5000
}
In the configuration above we have 6 echo servers and 6 d2 announcers. The first 3 echo servers belong to RecommendationService and the remaining echo servers belong to NewsService.
Finally, we add the task of running this server to build.gradle.
task runServer(type: JavaExec) {
main = 'com.example.d2.server.ExampleD2Server'
classpath = sourceSets.main.runtimeClasspath
standardInput = System.in
}
In order to run the server you run this command
../../gradlew runServer
-
d2-quickstart/
- client/
-
config/
-
src/
-
main/
-
java/
-
com/
-
example/
-
d2/
-
config/
- ConfigRunner.java
-
config/
-
d2/
-
example/
-
com/
-
d2Config/
- d2Config.json
-
java/
-
main/
-
src/
- server/
-
d2-quickstart/
-
client/
-
src/
-
main/
-
java/
-
com/
-
example/
-
d2/
-
client/
- ExampleD2Client.java
-
client/
-
d2/
-
example/
-
com/
-
config/
- client.json
-
java/
-
main/
-
src/
- config/
- server/
-
client/