Addressed comments from Siva.

linkedin · Apr 29, 2016 · 916e5a2 · 916e5a2
2 parents d6e384e + 615b5f8
commit 916e5a2
Show file tree

Hide file tree

Showing 488 changed files with 2,108 additions and 1,739 deletions.
diff --git a/HEADER b/HEADER
@@ -1,5 +1,5 @@
 /**
- * Copyright 2015 LinkedIn Corp. All rights reserved.
+ * Copyright 2016 LinkedIn Corp. All rights reserved.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.

diff --git a/LICENSE b/LICENSE
@@ -186,7 +186,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright 2015 LinkedIn Corp. All rights reserved.
+   Copyright 2016 LinkedIn Corp. All rights reserved.
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

diff --git a/NOTICE b/NOTICE
@@ -0,0 +1,40 @@
+(c) 2016 LinkedIn Corp. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use
+this file except in compliance with the License. You may obtain a copy of the
+License at  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed
+under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
+CONDITIONS OF ANY KIND, either express or implied.
+
+This product includes software developed by The Apache Software
+Foundation (http://www.apache.org/).
+
+This product includes/uses Codehale Metrics (https://github.com/dropwizard/metrics)
+Copyright (C) 2010 Code Hale, Yammer.com
+License: Apache 2.0
+
+This product includes/uses SLF4J (http://slf4j.org)
+Copyright (c) 2004 QOS.ch
+License: MIT
+
+This product includes/uses JOpt Simple (http://jopt-simple.sourceforge.net)
+Copyright (C) 2016 JOpt Simple 
+License: MIT
+
+This product includes/uses Bouncy Castle (http://www.bouncycastle.org/)
+Copyright (C) 2015 The Legion of the Bouncy Castle Inc. (http://www.bouncycastle.org)
+License: MIT
+
+This product includes/uses Netty (http://netty.io/)
+Copyright (C) 2016 The Netty project
+License: Apache 2.0
+
+This product includes/uses Log4j2 (http://logging.apache.org/log4j/2.x/)
+Copyright (C) 2015 Apache Log4j
+License: Apache 2.0
+
+This product includes/uses Apache commons codec (https://commons.apache.org/proper/commons-codec/)
+Copyright (C) 2014 Apache Commons
+License: Apache 2.0
diff --git a/README.md b/README.md
@@ -1,4 +1,90 @@
-ambry
-========
+# Ambry
+Ambry is a distributed object store that supports storage of trillion of small immutable objects (50K -100K) as well as billions of large objects. It was specifically designed to store and serve media objects in web companies. However, it can be used as a general purpose storage system to store DB backups, search indexes or business reports. The system has the following characterisitics: 
+
+1. Highly available and horizontally scalable
+2. Low latency and high throughput
+3. Optimized for both small and large objects
+4. Cost effective
+5. Easy to use
+
+## Documentation
+Detailed documentation is available at https://github.com/linkedin/ambry/wiki
+
+## Getting Started
+##### Step 1: Download the code, build it and prepare for deployment.
+To get the latest code and build it, do
+
+    $ git clone git@github.com:linkedin/ambry.git
+    $ cd ambry
+    $ ./gradlew allJar
+    $ cd target
+    $ mkdir logs
+Ambry uses files that provide information about the cluster to route requests from the frontend to servers and for replication between servers. We will use a simple clustermap that contains a single server with one partition. The partition will use `/tmp` as the mount point.
+##### Step 2: Deploy a server.
+    $ nohup java -Dlog4j.configuration=file:../config/log4j.properties -jar ambry.jar --serverPropsFilePath ../config/server.properties --hardwareLayoutFilePath ../config/HardwareLayout.json --partitionLayoutFilePath ../config/PartitionLayout.json > logs/server.log &
+
+Through this command, we configure the log4j properties, provide the server with configuration options and cluster definitions and redirect output to a log. Note down the process ID returned (`serverProcessID`) because it will be needed for shutdown.  
+The log will be available at `logs/server.log`. Alternately, you can change the log4j properties to write the log messages to a file instead of standard output.
+##### Step 3: Deploy a frontend.
+    $ nohup java -Dlog4j.configuration=file:../config/log4j.properties -cp "*" com.github.ambry.frontend.AmbryFrontendMain --serverPropsFilePath ../config/frontend.properties --hardwareLayoutFilePath ../config/HardwareLayout.json --partitionLayoutFilePath ../config/PartitionLayout.json > logs/frontend.log &
+
+Note down the process ID returned (`frontendProcessID`) because it will be needed for shutdown. Make sure that the frontend is ready to receive requests.
+
+    $ curl http://localhost:1174/healthCheck
+    GOOD
+The log will be available at `logs/frontend.log`. Alternately, you can change the log4j properties to write the log messages to a file instead of standard output.
+##### Step 4: Interact with Ambry !
+We are now ready to store and retrieve data from Ambry. Let us start by storing a simple image. For demonstration purposes, we will use an image `demo.gif` that has been copied into the `target` folder.
+###### POST
+    $ curl -i -H "x-ambry-blob-size : `wc -c demo.gif | xargs | cut -d" " -f1`" -H "x-ambry-service-id : CUrlUpload"  -H "x-ambry-owner-id : `whoami`" -H "x-ambry-content-type : image/gif" -H "x-ambry-um-description : Demonstration Image" http://localhost:1174/ --data-binary @demo.gif
+    HTTP/1.1 201 Created
+    Location: AmbryID
+    Content-Length: 0
+The CUrl command creates a `POST` request that contains the binary data in demo.gif. Along with the file data, we provide headers that act as blob properties. These include the size of the blob, the service ID, the owner ID and the content type.  
+In addition to these properties, Ambry also has a provision for arbitrary user defined metadata. We provide `x-ambry-um-description` as user metadata. Ambry does not interpret this data and it is purely for user annotation.
+The `Location` header in the response is the blob ID of the blob we just uploaded.
+###### GET - Blob Info
+Now that we stored a blob, let us verify some properties of the blob we uploaded.
+
+    $ curl -i http://localhost:1174/AmbryID/BlobInfo
+    HTTP/1.1 200 OK
+    x-ambry-blob-size: {Blob size}
+    x-ambry-service-id: CUrlUpload
+    x-ambry-creation-time: {Creation time}
+    x-ambry-private: false
+    x-ambry-content-type: image/gif
+    x-ambry-owner-id: {username}
+    x-ambry-um-desc: Demonstration Image
+    Content-Length: 0
+###### GET - Blob
+Now that we have verified that Ambry returns properties correctly, let us obtain the actual blob.
+
+    $ curl http://localhost:1174/AmbryID > demo-downloaded.gif
+    $ diff demo.gif demo-downloaded.gif 
+    $
+This confirms that the data that was sent in the `POST` request matches what we received in the `GET`. If you would like to see the image, simply point your browser to `http://localhost:1174/AmbryID` and you should see the image that was uploaded !
+###### DELETE
+Ambry is an immutable store and blobs cannot be updated but they can be deleted in order to make them irretrievable. Let us go ahead and delete the blob we just created.
+
+    $ curl -i -X DELETE http://localhost:1174/AmbryID
+    HTTP/1.1 202 Accepted
+    Content-Length: 0
+You will no longer be able to retrieve the blob properties or data.
+
+    $ curl -i http://localhost:1174/AmbryID/BlobInfo
+    HTTP/1.1 410 Gone
+    Content-Type: text/plain; charset=UTF-8
+    Content-Length: 17
+    Connection: close
+
+    Failure: 410 Gone
+##### Step 5: Stop the frontend and server.
+    $ kill -15 frontendProcessID
+    $ kill -15 serverProcessID
+You can confirm that the services have been shut down by looking at the logs.
+##### Additional information:
+In addition to the simple APIs demonstrated above, Ambry provides support for `GET` of only user metadata and `HEAD`. In addition to the `POST` of binary data that was demonstrated, Ambry also supports `POST` of `multipart/form-data` via CUrl or web forms.
+Other features of interest include:
+* **Time To Live (TTL)**: During `POST`, a TTL in seconds can be provided through the addition of a header named `x-ambry-ttl`. This means that Ambry will stop serving the blob after the TTL has expired. On `GET`, expired blobs behave the same way as deleted blobs.
+* **Private**: During `POST`, providing a header named `x-ambry-private` with the value `true` will mark the blob as private. API behavior can be configured based on whether a blob is public or private.
 
-An immutable object store
diff --git a/ambry-admin/src/main/java/com.github.ambry.admin/AdminBlobStorageService.java b/ambry-admin/src/main/java/com.github.ambry.admin/AdminBlobStorageService.java
@@ -1,5 +1,5 @@
 /**
- * Copyright 2015 LinkedIn Corp. All rights reserved.
+ * Copyright 2016 LinkedIn Corp. All rights reserved.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -34,7 +34,6 @@
 import java.util.Date;
 import java.util.GregorianCalendar;
 import java.util.HashSet;
-import java.util.Map;
 import java.util.Set;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -640,7 +639,7 @@ public void onCompletion(BlobInfo result, Exception exception) {
       restResponseChannel.setHeader(RestUtils.Headers.DATE, new GregorianCalendar().getTime());
       if (exception == null && result != null) {
         logger.trace("Successful HEAD of {}", blobId);
-        setResponseHeaders(result);
+        setBlobPropertiesResponseHeaders(result);
       } else if (exception == null) {
         exception = new IllegalStateException("Both response and exception are null for HeadCallback");
       }
@@ -663,11 +662,11 @@ public void onCompletion(BlobInfo result, Exception exception) {
   }
 
   /**
-   * Sets the required headers in the response.
+   * Sets the required blob properties headers in the response.
    * @param blobInfo the {@link BlobInfo} to refer to while setting headers.
    * @throws RestServiceException if there was any problem setting the headers.
    */
-  private void setResponseHeaders(BlobInfo blobInfo)
+  private void setBlobPropertiesResponseHeaders(BlobInfo blobInfo)
       throws RestServiceException {
     BlobProperties blobProperties = blobInfo.getBlobProperties();
     restResponseChannel.setStatus(ResponseStatus.Ok);
@@ -689,10 +688,5 @@ private void setResponseHeaders(BlobInfo blobInfo)
     if (blobProperties.getOwnerId() != null) {
       restResponseChannel.setHeader(RestUtils.Headers.OWNER_ID, blobProperties.getOwnerId());
     }
-    byte[] userMetadataArray = blobInfo.getUserMetadata();
-    Map<String, String> userMetadata = RestUtils.buildUserMetadata(userMetadataArray);
-    for (Map.Entry<String, String> entry : userMetadata.entrySet()) {
-      restResponseChannel.setHeader(entry.getKey(), entry.getValue());
-    }
   }
 }
diff --git a/ambry-admin/src/main/java/com.github.ambry.admin/AdminBlobStorageServiceFactory.java b/ambry-admin/src/main/java/com.github.ambry.admin/AdminBlobStorageServiceFactory.java
@@ -1,5 +1,5 @@
 /**
- * Copyright 2015 LinkedIn Corp. All rights reserved.
+ * Copyright 2016 LinkedIn Corp. All rights reserved.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.

diff --git a/ambry-admin/src/main/java/com.github.ambry.admin/AdminConfig.java b/ambry-admin/src/main/java/com.github.ambry.admin/AdminConfig.java
@@ -1,5 +1,5 @@
 /**
- * Copyright 2015 LinkedIn Corp. All rights reserved.
+ * Copyright 2016 LinkedIn Corp. All rights reserved.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.

diff --git a/ambry-admin/src/main/java/com.github.ambry.admin/AdminMetrics.java b/ambry-admin/src/main/java/com.github.ambry.admin/AdminMetrics.java
@@ -1,5 +1,5 @@
 /**
- * Copyright 2015 LinkedIn Corp. All rights reserved.
+ * Copyright 2016 LinkedIn Corp. All rights reserved.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.

diff --git a/ambry-admin/src/main/java/com.github.ambry.admin/AdminRestServerMain.java b/ambry-admin/src/main/java/com.github.ambry.admin/AdminRestServerMain.java
@@ -1,5 +1,5 @@
 /**
- * Copyright 2015 LinkedIn Corp. All rights reserved.
+ * Copyright 2016 LinkedIn Corp. All rights reserved.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.

diff --git a/ambry-admin/src/main/java/com.github.ambry.admin/EchoHandler.java b/ambry-admin/src/main/java/com.github.ambry.admin/EchoHandler.java
@@ -1,5 +1,5 @@
 /**
- * Copyright 2015 LinkedIn Corp. All rights reserved.
+ * Copyright 2016 LinkedIn Corp. All rights reserved.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.

diff --git a/ambry-admin/src/main/java/com.github.ambry.admin/GetReplicasForBlobIdHandler.java b/ambry-admin/src/main/java/com.github.ambry.admin/GetReplicasForBlobIdHandler.java
@@ -1,5 +1,5 @@
 /**
- * Copyright 2015 LinkedIn Corp. All rights reserved.
+ * Copyright 2016 LinkedIn Corp. All rights reserved.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.

diff --git a/ambry-admin/src/test/java/com.github.ambry.admin/AdminBlobStorageServiceFactoryTest.java b/ambry-admin/src/test/java/com.github.ambry.admin/AdminBlobStorageServiceFactoryTest.java
@@ -1,5 +1,5 @@
 /**
- * Copyright 2015 LinkedIn Corp. All rights reserved.
+ * Copyright 2016 LinkedIn Corp. All rights reserved.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.

diff --git a/ambry-admin/src/test/java/com.github.ambry.admin/AdminBlobStorageServiceTest.java b/ambry-admin/src/test/java/com.github.ambry.admin/AdminBlobStorageServiceTest.java
@@ -1,5 +1,5 @@
 /**
- * Copyright 2015 LinkedIn Corp. All rights reserved.
+ * Copyright 2016 LinkedIn Corp. All rights reserved.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -52,7 +52,6 @@
 import java.net.URISyntaxException;
 import java.nio.ByteBuffer;
 import java.util.HashMap;
-import java.util.Iterator;
 import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
@@ -1102,6 +1101,8 @@ private void getHeadAndVerify(String blobId, JSONObject expectedHeaders)
     checkCommonGetHeadHeaders(restResponseChannel, expectedHeaders);
     assertEquals("Content-Length does not match blob size", expectedHeaders.getString(RestUtils.Headers.BLOB_SIZE),
         restResponseChannel.getHeader(RestUtils.Headers.CONTENT_LENGTH));
+    assertEquals("Blob size does not match ", expectedHeaders.getString(RestUtils.Headers.BLOB_SIZE),
+        restResponseChannel.getHeader(RestUtils.Headers.BLOB_SIZE));
     assertEquals(RestUtils.Headers.SERVICE_ID + " does not match",
         expectedHeaders.getString(RestUtils.Headers.SERVICE_ID),
         restResponseChannel.getHeader(RestUtils.Headers.SERVICE_ID));
@@ -1121,25 +1122,6 @@ private void getHeadAndVerify(String blobId, JSONObject expectedHeaders)
           expectedHeaders.getString(RestUtils.Headers.OWNER_ID),
           restResponseChannel.getHeader(RestUtils.Headers.OWNER_ID));
     }
-    verifyUserMetadataHeaders(expectedHeaders, restResponseChannel);
-  }
-
-  /**
-   * Verifies User metadata headers from output, to that sent in during input
-   * @param expectedHeaders the expected headers in the response.
-   * @param restResponseChannel the {@link RestResponseChannel} which contains the response.
-   * @throws JSONException
-   */
-  private void verifyUserMetadataHeaders(JSONObject expectedHeaders, MockRestResponseChannel restResponseChannel)
-      throws JSONException {
-    Iterator itr = expectedHeaders.keys();
-    while (itr.hasNext()) {
-      String key = (String) itr.next();
-      if (key.startsWith(RestUtils.Headers.USER_META_DATA_HEADER_PREFIX)) {
-        String outValue = restResponseChannel.getHeader(key);
-        assertEquals("Value for " + key + "does not match in user metadata", expectedHeaders.getString(key), outValue);
-      }
-    }
   }
 
   /**

diff --git a/ambry-admin/src/test/java/com.github.ambry.admin/AdminIntegrationTest.java b/ambry-admin/src/test/java/com.github.ambry.admin/AdminIntegrationTest.java
@@ -1,5 +1,5 @@
 /**
- * Copyright 2015 LinkedIn Corp. All rights reserved.
+ * Copyright 2016 LinkedIn Corp. All rights reserved.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -42,7 +42,6 @@
 import java.io.IOException;
 import java.nio.ByteBuffer;
 import java.util.List;
-import java.util.Map;
 import java.util.Properties;
 import java.util.Queue;
 import java.util.Random;
@@ -352,6 +351,8 @@ private void getHeadAndVerify(String blobId, HttpHeaders expectedHeaders)
     checkCommonGetHeadHeaders(response.headers(), expectedHeaders);
     assertEquals("Content-Length does not match blob size",
         Long.parseLong(expectedHeaders.get(RestUtils.Headers.BLOB_SIZE)), HttpHeaders.getContentLength(response));
+    assertEquals("Blob size does not match", expectedHeaders.get(RestUtils.Headers.BLOB_SIZE),
+        HttpHeaders.getHeader(response, RestUtils.Headers.BLOB_SIZE));
     assertEquals(RestUtils.Headers.SERVICE_ID + " does not match", expectedHeaders.get(RestUtils.Headers.SERVICE_ID),
         HttpHeaders.getHeader(response, RestUtils.Headers.SERVICE_ID));
     assertEquals(RestUtils.Headers.PRIVATE + " does not match", expectedHeaders.get(RestUtils.Headers.PRIVATE),
@@ -369,22 +370,6 @@ private void getHeadAndVerify(String blobId, HttpHeaders expectedHeaders)
       assertEquals(RestUtils.Headers.OWNER_ID + " does not match", expectedHeaders.get(RestUtils.Headers.OWNER_ID),
           HttpHeaders.getHeader(response, RestUtils.Headers.OWNER_ID));
     }
-    verifyUserMetadataHeaders(expectedHeaders, response);
-  }
-
-  /**
-   * Verifies User metadata headers from output, to that sent in during input
-   * @param expectedHeaders the expected headers in the response.
-   * @param response the {@link HttpResponse} which contains the headers of the response.
-   */
-  private void verifyUserMetadataHeaders(HttpHeaders expectedHeaders, HttpResponse response) {
-    for (Map.Entry<String, String> header : expectedHeaders) {
-      String key = header.getKey();
-      if (key.startsWith(RestUtils.Headers.USER_META_DATA_HEADER_PREFIX)) {
-        assertEquals("Value for " + key + "does not match in user metadata", header.getValue(),
-            HttpHeaders.getHeader(response, key));
-      }
-    }
   }
 
   /**

diff --git a/ambry-admin/src/test/java/com.github.ambry.admin/TestAdminRestServerMain.java b/ambry-admin/src/test/java/com.github.ambry.admin/TestAdminRestServerMain.java
@@ -1,5 +1,5 @@
 /**
- * Copyright 2015 LinkedIn Corp. All rights reserved.
+ * Copyright 2016 LinkedIn Corp. All rights reserved.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.

diff --git a/ambry-api/src/main/java/com.github.ambry/clustermap/ClusterMap.java b/ambry-api/src/main/java/com.github.ambry/clustermap/ClusterMap.java
@@ -1,5 +1,5 @@
 /**
- * Copyright 2015 LinkedIn Corp. All rights reserved.
+ * Copyright 2016 LinkedIn Corp. All rights reserved.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.

diff --git a/ambry-api/src/main/java/com.github.ambry/clustermap/DataNodeId.java b/ambry-api/src/main/java/com.github.ambry/clustermap/DataNodeId.java
@@ -1,5 +1,5 @@
 /**
- * Copyright 2015 LinkedIn Corp. All rights reserved.
+ * Copyright 2016 LinkedIn Corp. All rights reserved.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -72,4 +72,12 @@ public abstract class DataNodeId implements Resource, Comparable<DataNodeId> {
    * @return name of the Datacenter
    */
   public abstract String getDatacenterName();
+
+  /**
+   * Get the DataNodeId's server rack ID.  If there is no rack ID for this node,
+   * -1 will be returned, so the caller must check that the returned value is non-negative.
+   *
+   * @return a valid rack ID, or a negative number if no rack ID is assigned
+   */
+  public abstract long getRackId();
 }
diff --git a/ambry-api/src/main/java/com.github.ambry/clustermap/DiskId.java b/ambry-api/src/main/java/com.github.ambry/clustermap/DiskId.java
@@ -1,5 +1,5 @@
 /**
- * Copyright 2015 LinkedIn Corp. All rights reserved.
+ * Copyright 2016 LinkedIn Corp. All rights reserved.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.

diff --git a/ambry-api/src/main/java/com.github.ambry/clustermap/HardwareState.java b/ambry-api/src/main/java/com.github.ambry/clustermap/HardwareState.java
@@ -1,5 +1,5 @@
 /**
- * Copyright 2015 LinkedIn Corp. All rights reserved.
+ * Copyright 2016 LinkedIn Corp. All rights reserved.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.