From 1358f95deb52a9d96f69fcf71eaf2ba0684cc5f3 Mon Sep 17 00:00:00 2001
From: Sean Gallagher <sean.gallagher@elasticsearch.com>
Date: Tue, 1 Apr 2014 12:28:08 -0400
Subject: [PATCH 01/54] Author: Sean Gallagher Date: Tue Apr 1 12:28:00 2014

Added upgrade.asciidoc and links to it from setup.asciidoc
---
 README.textile                | 4 ++++
 docs/reference/setup.asciidoc | 2 ++
 2 files changed, 6 insertions(+)

diff --git a/README.textile b/README.textile
index 9d8fba6b45d8b..86c4651a3255c 100644
--- a/README.textile
+++ b/README.textile
@@ -206,6 +206,10 @@ The distribution will be created under @target/releases@.
 See the "TESTING":TESTING.asciidoc file for more information about
 running the Elasticsearch test suite.
 
+h3. Upgrading to Elasticsearch 1.0?
+
+In order to ensure a smooth upgrade process from earlier versions of Elasticsearch (< 0.90.x), it is recommended to perform a full cluster restart. Please see the "Upgrading" section of the "setup reference":http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html.
+
 h1. License
 
 <pre>
diff --git a/docs/reference/setup.asciidoc b/docs/reference/setup.asciidoc
index cf413cd918a67..12324483f89c2 100644
--- a/docs/reference/setup.asciidoc
+++ b/docs/reference/setup.asciidoc
@@ -61,3 +61,5 @@ include::setup/as-a-service-win.asciidoc[]
 include::setup/dir-layout.asciidoc[]
 
 include::setup/repositories.asciidoc[]
+
+include::setup/upgrade.asciidoc[]

From e13d759ccf23ff453b5c0c072b36fb00f7366494 Mon Sep 17 00:00:00 2001
From: Sean Gallagher <sean.gallagher@elasticsearch.com>
Date: Tue, 1 Apr 2014 12:30:15 -0400
Subject: [PATCH 02/54] Author: Sean Gallagher Date: Apr 1 2014

Added upgrade.asciidoc
---
 docs/reference/setup/upgrade.asciidoc | 72 +++++++++++++++++++++++++++
 1 file changed, 72 insertions(+)
 create mode 100644 docs/reference/setup/upgrade.asciidoc

diff --git a/docs/reference/setup/upgrade.asciidoc b/docs/reference/setup/upgrade.asciidoc
new file mode 100644
index 0000000000000..0d1300ffe6bd1
--- /dev/null
+++ b/docs/reference/setup/upgrade.asciidoc
@@ -0,0 +1,72 @@
+[[setup-upgrade]]
+== Upgrading Elasticsearch
+
+[float]
+=== Rolling Upgrades
+Within major releases (0.90.x to 0.90.x, 1.0 to all later releases), rolling upgrades are supported.  To perform a rolling upgrade:
+
+* Shut down a single node within the cluster.
+
+--------------------------------------------
+curl -XPOST 'http://localhost:9200/_cluster/nodes/_local/_shutdown'
+--------------------------------------------
+
+* Confirm that all shards are correctly reallocated to the remaining running nodes.
+
+* Back up the data directory on the stopped node (optional).  This step is performed in order to allow for a smoother rollback in the event of problems during the upgrade.
+
+* Upgrade the stopped node.  To upgrade using a zip or compressed tarball from elasticsearch.org:
+** Extract the zip or tarball to a new directory, usually in the same volume as the current Elasticsearch installation.  Do not overwrite the existing installation, as the downloaded archive will contain a default elasticsearch.yml file and will overwrite your existing configuration.
+** Copy the configuration files from the old Elasticsearch installation's config directory to the new Elasticsearch installation's config directory.  Copy data files if necessary.
+** The simplest solution for moving from one version to another is to have a symbolic link for 'elasticsearch' that points to the currently running version.  This link can be easily updated and will provide a stable access point to the most recent version.  Update this symbolic link if it is being used.
+
+To upgrade using a .deb or .rpm package:
+
+* Use rpm or deb to install the new package.  All files should be placed in their proper locations, and config files should not be overwritten.
+
+* Start the now upgraded node.  Observe that all shards are properly allocated back to the node and that it joins the cluster smoothly.
+
+Repeat this process for all remaining nodes.
+
+It may be possible to perform the upgrade by installing the new software while the service is running.  This would reduce downtime by ensuring the service was ready to run on the new version as soon as it is stopped on the node being upgraded.  This can be done by installing the new version in its own directory and using the symbolic link method outlined above.  It is important to test this procedure first to be sure that site-specific configuration data and production indices will not be overwritten during the upgrade process.
+
+[float]
+=== Upgrading from 0.90.x or earlier to 1.0 or later
+Elasticsearch releases prior to 1.0 and releases after 1.0 are not compatible with each other, so a rolling upgrade is not possible.  In order to upgrade a pre-1.0 system to 1.0 or later, a full cluster stop and start is required.  In order to perform this upgrade:
+
+* Disable shard reallocation (optional).  This is done to allow for a faster startup after cluster shutdown.  If this step is not performed, the nodes will immediately start trying to replicate shards to each other on startup and will spend a lot of time on wasted I/O.  With shard reallocation disabled, the nodes will join the cluster with their indices intact, without attempting to rebalance.  After startup is complete, reallocation will be turned back on.
+
+This syntax is from versions prior to 1.0:
+------------------------------------------------------------
+	curl -XPUT localhost:9200/_cluster/settings -d '{
+		"persistent" : {
+		"cluster.routing.allocation.disable_allocation" : true
+		}
+	}'
+------------------------------------------------------------
+
+* Stop all Elasticsearch services on all nodes in the cluster.
+------------------------------------------------------
+	curl -XPOST 'http://localhost:9200/_shutdown'
+------------------------------------------------------
+
+* Backup all data directories on all nodes (optional).  This step is performed to allow for a smoother rollback in the event of problems during the upgrade.
+
+* On the first node to be upgraded, extract the archive or install the new package as described above in the Rolling Upgrades section.  Repeat for all nodes.
+
+* After upgrading Elasticsearch on all nodes is complete, the cluster can be started by starting each node individually.
+** Start master-eligible nodes first, one at a time.  Verify that a quorum has been reached and a master has been elected before proceeding.
+** Start data and client nodes one at a time, verifying that they successfully join the cluster.
+
+* When the cluster is up and running, shard reallocation can be enabled.
+
+This synax is from release 1.0 and later:
+------------------------------------------------------
+	curl -XPUT localhost:9200/_cluster/settings -d '{
+     		"persistent" : {
+         	"cluster.routing.allocation.enable" : "all"
+     		}
+ 	}'
+------------------------------------------------------
+
+The cluster upgrade can be streamlined by installing the software before stopping cluster services.  If this is done, testing must be performed to ensure that no production data or configuration files are overwritten prior to restart.

From 31f7696e653ba9ba6593843787c60144e3f316a1 Mon Sep 17 00:00:00 2001
From: Sean Gallagher <sean.gallagher@elasticsearch.com>
Date: Fri, 4 Apr 2014 16:57:51 -0400
Subject: [PATCH 03/54] Add upgrade instructions Author: Sean Gallagher Date:
 4/4/14 Closes issue #5651

---
 README.textile                        |  2 +-
 docs/reference/setup/upgrade.asciidoc | 84 +++++++++++++++++++++++----
 2 files changed, 73 insertions(+), 13 deletions(-)

diff --git a/README.textile b/README.textile
index 86c4651a3255c..fcf53bdba8c4d 100644
--- a/README.textile
+++ b/README.textile
@@ -206,7 +206,7 @@ The distribution will be created under @target/releases@.
 See the "TESTING":TESTING.asciidoc file for more information about
 running the Elasticsearch test suite.
 
-h3. Upgrading to Elasticsearch 1.0?
+h3. Upgrading to Elasticsearch 1.x?
 
 In order to ensure a smooth upgrade process from earlier versions of Elasticsearch (< 0.90.x), it is recommended to perform a full cluster restart. Please see the "Upgrading" section of the "setup reference":http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html.
 
diff --git a/docs/reference/setup/upgrade.asciidoc b/docs/reference/setup/upgrade.asciidoc
index 0d1300ffe6bd1..b52d8eeab215d 100644
--- a/docs/reference/setup/upgrade.asciidoc
+++ b/docs/reference/setup/upgrade.asciidoc
@@ -1,57 +1,116 @@
 [[setup-upgrade]]
-== Upgrading Elasticsearch
+== Upgrading
+
+Elasticsearch can usually be upgraded using a rolling upgrade process, resulting in no interruption of service.  This section details how to perform both rolling and restart upgrades.
+
+[float]
+[[backup]]
+=== Back Up Your Data!
+
+Before performing an upgrade, it's a good idea to back up the data on your system.  This will allow you to roll back in the event of a problem with the upgrade.  The upgrades sometimes include upgrades to the Lucene libraries used by Elasticsearch to access the index files, and after an index file has been updated to work with a new version of Lucene, it may not be accessible to the versions of Lucene present in earlier Elasticsearch releases.
+
+[float]
+==== 0.90.x and earlier
+
+To back up a running 0.90.x system, first disable index flushing.  This will prevent indices from being flushed to disk while the backup is in process:
+
+[source,sh]
+-----------------------------------
+$ curl -XPUT 'http://localhost:9200/_all/_settings' -d '{
+    "index": {
+        "translog.disable_flush": "true"
+    }
+}'
+-----------------------------------
+
+Then disable reallocation.  This will prevent the cluster from moving data files from one node to another while the backup is in process:
+
+[source,sh]
+-----------------------------------
+$ curl -XPUT 'http://localhost:9200/_cluster/settings' -d '{
+    "transient" : {
+        "cluster.routing.allocation.disable_allocation": "true"
+    }
+}'
+-----------------------------------
+
+After reallocation and index flushing are disabled, initiate a backup of Elasticsearch's data path using your favorite backup method (tar, storage array snapshots, backup software).  When the backup is complete and data no longer needs to be read from the Elasticsearch data path, reallocation and index flushing must be re-enabled:
+
+[source,sh]
+-----------------------------------
+$ curl -XPUT 'http://localhost:9200/_all/_settings' -d '{
+    "index": {
+        "translog.disable_flush": "false"
+    }
+}'
+
+$ curl -XPUT 'http://localhost:9200/_cluster/settings' -d '{
+    "transient" : {
+        "cluster.routing.allocation.disable_allocation": "false"
+    }
+}'
+-----------------------------------
+
+[float]
+==== 1.0 and later
+
+To back up a running 1.0 or later system, it is simplest to use the snapshot feature.  Complete instructions for backup and restore with snapshots are available http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html[here].
 
 [float]
+[[rolling-upgrades]]
 === Rolling Upgrades
-Within major releases (0.90.x to 0.90.x, 1.0 to all later releases), rolling upgrades are supported.  To perform a rolling upgrade:
+
+Within major releases after release 1.0, rolling upgrades are supported.  To perform a rolling upgrade:
 
 * Shut down a single node within the cluster.
 
+[source,sh]
 --------------------------------------------
 curl -XPOST 'http://localhost:9200/_cluster/nodes/_local/_shutdown'
 --------------------------------------------
 
 * Confirm that all shards are correctly reallocated to the remaining running nodes.
 
-* Back up the data directory on the stopped node (optional).  This step is performed in order to allow for a smoother rollback in the event of problems during the upgrade.
-
 * Upgrade the stopped node.  To upgrade using a zip or compressed tarball from elasticsearch.org:
 ** Extract the zip or tarball to a new directory, usually in the same volume as the current Elasticsearch installation.  Do not overwrite the existing installation, as the downloaded archive will contain a default elasticsearch.yml file and will overwrite your existing configuration.
-** Copy the configuration files from the old Elasticsearch installation's config directory to the new Elasticsearch installation's config directory.  Copy data files if necessary.
+** Copy the configuration files from the old Elasticsearch installation's config directory to the new Elasticsearch installation's config directory.  Copy data files if necessary.  If data files are not located within the tarball's extraction directory, they will not have to be moved.
 ** The simplest solution for moving from one version to another is to have a symbolic link for 'elasticsearch' that points to the currently running version.  This link can be easily updated and will provide a stable access point to the most recent version.  Update this symbolic link if it is being used.
 
-To upgrade using a .deb or .rpm package:
+*To upgrade using a .deb or .rpm package:
 
-* Use rpm or deb to install the new package.  All files should be placed in their proper locations, and config files should not be overwritten.
+** Use rpm or deb to install the new package.  All files should be placed in their proper locations, and config files should not be overwritten.
 
-* Start the now upgraded node.  Observe that all shards are properly allocated back to the node and that it joins the cluster smoothly.
+** Start the now upgraded node.  Observe that all shards are properly allocated back to the node and that it joins the cluster smoothly.
 
 Repeat this process for all remaining nodes.
 
 It may be possible to perform the upgrade by installing the new software while the service is running.  This would reduce downtime by ensuring the service was ready to run on the new version as soon as it is stopped on the node being upgraded.  This can be done by installing the new version in its own directory and using the symbolic link method outlined above.  It is important to test this procedure first to be sure that site-specific configuration data and production indices will not be overwritten during the upgrade process.
 
 [float]
+[[restart-upgrade]]
 === Upgrading from 0.90.x or earlier to 1.0 or later
+
 Elasticsearch releases prior to 1.0 and releases after 1.0 are not compatible with each other, so a rolling upgrade is not possible.  In order to upgrade a pre-1.0 system to 1.0 or later, a full cluster stop and start is required.  In order to perform this upgrade:
 
 * Disable shard reallocation (optional).  This is done to allow for a faster startup after cluster shutdown.  If this step is not performed, the nodes will immediately start trying to replicate shards to each other on startup and will spend a lot of time on wasted I/O.  With shard reallocation disabled, the nodes will join the cluster with their indices intact, without attempting to rebalance.  After startup is complete, reallocation will be turned back on.
 
 This syntax is from versions prior to 1.0:
-------------------------------------------------------------
+
+[source,sh]
+--------------------------------------------------
 	curl -XPUT localhost:9200/_cluster/settings -d '{
 		"persistent" : {
 		"cluster.routing.allocation.disable_allocation" : true
 		}
 	}'
-------------------------------------------------------------
+--------------------------------------------------
 
 * Stop all Elasticsearch services on all nodes in the cluster.
+[source,sh]
 ------------------------------------------------------
 	curl -XPOST 'http://localhost:9200/_shutdown'
 ------------------------------------------------------
 
-* Backup all data directories on all nodes (optional).  This step is performed to allow for a smoother rollback in the event of problems during the upgrade.
-
 * On the first node to be upgraded, extract the archive or install the new package as described above in the Rolling Upgrades section.  Repeat for all nodes.
 
 * After upgrading Elasticsearch on all nodes is complete, the cluster can be started by starting each node individually.
@@ -61,6 +120,7 @@ This syntax is from versions prior to 1.0:
 * When the cluster is up and running, shard reallocation can be enabled.
 
 This synax is from release 1.0 and later:
+[source,sh]
 ------------------------------------------------------
 	curl -XPUT localhost:9200/_cluster/settings -d '{
      		"persistent" : {

From 124d370b5fc8dde000344f272e91f4f7b6d437b1 Mon Sep 17 00:00:00 2001
From: Simon Willnauer <simonw@apache.org>
Date: Sat, 5 Apr 2014 22:18:38 +0200
Subject: [PATCH 04/54] [TEST] cleanup secondary cluster properly in Tribe
 tests.

---
 .../java/org/elasticsearch/tribe/TribeTests.java  | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/src/test/java/org/elasticsearch/tribe/TribeTests.java b/src/test/java/org/elasticsearch/tribe/TribeTests.java
index 8ea5453ce3e40..dbc0e5de2ec46 100644
--- a/src/test/java/org/elasticsearch/tribe/TribeTests.java
+++ b/src/test/java/org/elasticsearch/tribe/TribeTests.java
@@ -66,19 +66,26 @@ public static void setupSecondCluster() throws Exception {
     @AfterClass
     public static void tearDownSecondCluster() {
         if (cluster2 != null) {
-            cluster2.afterTest();
-            cluster2.close();
-            cluster2 = null;
+            try {
+                cluster2.close();
+            } finally {
+                cluster2 = null;
+            }
         }
     }
 
     @After
     public void tearDownTribeNode() {
         if (cluster2 != null) {
-            cluster2.client().admin().indices().prepareDelete("_all").execute().actionGet();
+            try {
+                cluster2.wipe();
+            } finally {
+                cluster2.afterTest();
+            }
         }
         if (tribeNode != null) {
             tribeNode.close();
+            tribeNode = null;
         }
     }
 

From a5aafbb04c06d691be3e70ac53b7dfc24f84455e Mon Sep 17 00:00:00 2001
From: Simon Willnauer <simonw@apache.org>
Date: Sat, 5 Apr 2014 22:35:55 +0200
Subject: [PATCH 05/54] [TEST] Prevent RelocationTests from going crazy when
 relocations take time

---
 .../java/org/elasticsearch/recovery/RelocationTests.java  | 5 +++--
 .../java/org/elasticsearch/test/BackgroundIndexer.java    | 8 ++++----
 2 files changed, 7 insertions(+), 6 deletions(-)

diff --git a/src/test/java/org/elasticsearch/recovery/RelocationTests.java b/src/test/java/org/elasticsearch/recovery/RelocationTests.java
index 314a097e708d6..dc3fa203488d1 100644
--- a/src/test/java/org/elasticsearch/recovery/RelocationTests.java
+++ b/src/test/java/org/elasticsearch/recovery/RelocationTests.java
@@ -21,6 +21,7 @@
 
 import com.carrotsearch.hppc.IntOpenHashSet;
 import com.carrotsearch.hppc.procedures.IntProcedure;
+import com.carrotsearch.randomizedtesting.RandomizedTest;
 import org.apache.lucene.util.LuceneTestCase.Slow;
 import org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse;
 import org.elasticsearch.action.search.SearchPhaseExecutionException;
@@ -128,8 +129,8 @@ public void testRelocationWhileIndexingRandom() throws Exception {
             }
         }
 
-        try (BackgroundIndexer indexer = new BackgroundIndexer("test", "type1", client())) {
-            final int numDocs = scaledRandomIntBetween(200, 2500);
+        final int numDocs = scaledRandomIntBetween(200, 2500);
+        try (BackgroundIndexer indexer = new BackgroundIndexer("test", "type1", client(), scaledRandomIntBetween(2, 5), true, numDocs * 2)) {
             logger.info("--> waiting for {} docs to be indexed ...", numDocs);
             waitForDocs(numDocs, indexer);
             logger.info("--> {} docs indexed", numDocs);
diff --git a/src/test/java/org/elasticsearch/test/BackgroundIndexer.java b/src/test/java/org/elasticsearch/test/BackgroundIndexer.java
index a906789758bd1..fa432ce7042f5 100644
--- a/src/test/java/org/elasticsearch/test/BackgroundIndexer.java
+++ b/src/test/java/org/elasticsearch/test/BackgroundIndexer.java
@@ -54,10 +54,10 @@ public BackgroundIndexer(String index, String type, Client client) {
     }
 
     public BackgroundIndexer(String index, String type, Client client, int writerCount) {
-        this(index, type, client, writerCount, true);
+        this(index, type, client, writerCount, true, Integer.MAX_VALUE);
     }
 
-    public BackgroundIndexer(final String index, final String type, final Client client, final int writerCount, boolean autoStart) {
+    public BackgroundIndexer(final String index, final String type, final Client client, final int writerCount, boolean autoStart, final int maxNumDocs) {
 
         failures = new CopyOnWriteArrayList<>();
         writers = new Thread[writerCount];
@@ -73,7 +73,7 @@ public void run() {
                     try {
                         startLatch.await();
                         logger.info("**** starting indexing thread {}", indexerId);
-                        while (!stop.get()) {
+                        while (!stop.get() && indexCounter.get() < maxNumDocs) {  // step out once we reach the hard limit
                             if (batch) {
                                 int batchSize = RandomizedTest.getRandom().nextInt(20) + 1;
                                 BulkRequestBuilder bulkRequest = client.prepareBulk();
@@ -97,7 +97,7 @@ public void run() {
                                 indexCounter.incrementAndGet();
                             }
                         }
-                        logger.info("**** done indexing thread {}", indexerId);
+                        logger.info("**** done indexing thread {}  stop: {} numDocsIndexed: {} maxNumDocs: {}", indexerId, stop.get(), indexCounter.get(), maxNumDocs);
                     } catch (Throwable e) {
                         failures.add(e);
                         logger.warn("**** failed indexing thread {} on doc id {}", e, indexerId, id);

From d26a9562312c97b27f25997b1cc683b77c6338f9 Mon Sep 17 00:00:00 2001
From: Shay Banon <kimchy@gmail.com>
Date: Fri, 4 Apr 2014 20:44:12 +0200
Subject: [PATCH 06/54] releasable bytes output + use in transport / translog
 create a new releasable bytes output, that can be recycled, and use it in
 netty and the translog, 2 areas where the recycling will help nicely. Note,
 opted for statically typed enforced releasble bytes output, to make sure
 people take the extra care to control when the bytes reference are released. 
 Also, the mock page/array classes were fixed to not take into account
 potential recycling going during teardown, for example, on a shared cluster
 ping requests still happen, so recycling happen actively during teardown.
 closes #5691

---
 .../common/bytes/PagedBytesReference.java     |   8 +-
 .../bytes/ReleasableBytesReference.java       |  28 +++++
 .../bytes/ReleasablePagedBytesReference.java  |  46 +++++++
 .../common/io/ReleasableBytesStream.java      |  30 +++++
 .../common/io/stream/BytesStreamOutput.java   |  30 ++---
 .../stream/ReleasableBytesStreamOutput.java   |  48 ++++++++
 .../index/translog/fs/FsTranslog.java         |  25 ++--
 .../transport/netty/NettyTransport.java       | 114 ++++++++++--------
 .../netty/NettyTransportChannel.java          |  96 +++++++++------
 .../BenchmarkNettyLargeMessages.java          |   5 +-
 .../transport/TransportBenchmark.java         |   3 +-
 .../common/util/BigArraysTests.java           |   2 +-
 .../zen/ping/unicast/UnicastZenPingTests.java |   5 +-
 .../test/ElasticsearchTestCase.java           |   6 +-
 .../test/cache/recycler/MockBigArrays.java    |  32 ++++-
 .../cache/recycler/MockPageCacheRecycler.java |  32 ++++-
 .../recycler/MockPageCacheRecyclerModule.java |   1 -
 .../netty/SimpleNettyTransportTests.java      |   3 +-
 18 files changed, 379 insertions(+), 135 deletions(-)
 create mode 100644 src/main/java/org/elasticsearch/common/bytes/ReleasableBytesReference.java
 create mode 100644 src/main/java/org/elasticsearch/common/bytes/ReleasablePagedBytesReference.java
 create mode 100644 src/main/java/org/elasticsearch/common/io/ReleasableBytesStream.java
 create mode 100644 src/main/java/org/elasticsearch/common/io/stream/ReleasableBytesStreamOutput.java
 rename src/test/java/org/elasticsearch/{ => test}/cache/recycler/MockPageCacheRecycler.java (75%)

diff --git a/src/main/java/org/elasticsearch/common/bytes/PagedBytesReference.java b/src/main/java/org/elasticsearch/common/bytes/PagedBytesReference.java
index 8ea6f8fc8ef51..664f041f61b1a 100644
--- a/src/main/java/org/elasticsearch/common/bytes/PagedBytesReference.java
+++ b/src/main/java/org/elasticsearch/common/bytes/PagedBytesReference.java
@@ -36,13 +36,17 @@
 import java.nio.channels.GatheringByteChannel;
 import java.util.Arrays;
 
-public final class PagedBytesReference implements BytesReference {
+/**
+ * A page based bytes reference, internally holding the bytes in a paged
+ * data structure.
+ */
+public class PagedBytesReference implements BytesReference {
 
     private static final int PAGE_SIZE = BigArrays.BYTE_PAGE_SIZE;
     private static final int NIO_GATHERING_LIMIT = 524288;
 
     private final BigArrays bigarrays;
-    private final ByteArray bytearray;
+    protected final ByteArray bytearray;
     private final int offset;
     private final int length;
     private int hash = 0;
diff --git a/src/main/java/org/elasticsearch/common/bytes/ReleasableBytesReference.java b/src/main/java/org/elasticsearch/common/bytes/ReleasableBytesReference.java
new file mode 100644
index 0000000000000..749a83a67f42c
--- /dev/null
+++ b/src/main/java/org/elasticsearch/common/bytes/ReleasableBytesReference.java
@@ -0,0 +1,28 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.common.bytes;
+
+import org.elasticsearch.common.lease.Releasable;
+
+/**
+ * A bytes reference that needs to be released once its usage is done.
+ */
+public interface ReleasableBytesReference extends BytesReference, Releasable {
+}
diff --git a/src/main/java/org/elasticsearch/common/bytes/ReleasablePagedBytesReference.java b/src/main/java/org/elasticsearch/common/bytes/ReleasablePagedBytesReference.java
new file mode 100644
index 0000000000000..f660cbd022d6b
--- /dev/null
+++ b/src/main/java/org/elasticsearch/common/bytes/ReleasablePagedBytesReference.java
@@ -0,0 +1,46 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.common.bytes;
+
+import org.elasticsearch.ElasticsearchException;
+import org.elasticsearch.common.lease.Releasables;
+import org.elasticsearch.common.util.BigArrays;
+import org.elasticsearch.common.util.ByteArray;
+
+/**
+ * An extension to {@link PagedBytesReference} that requires releasing its content. This
+ * class exists to make it explicit when a bytes reference needs to be released, and when not.
+ */
+public class ReleasablePagedBytesReference extends PagedBytesReference implements ReleasableBytesReference {
+
+    public ReleasablePagedBytesReference(BigArrays bigarrays, ByteArray bytearray, int length) {
+        super(bigarrays, bytearray, length);
+    }
+
+    public ReleasablePagedBytesReference(BigArrays bigarrays, ByteArray bytearray, int from, int length) {
+        super(bigarrays, bytearray, from, length);
+    }
+
+    @Override
+    public boolean release() throws ElasticsearchException {
+        Releasables.release(bytearray);
+        return true;
+    }
+}
diff --git a/src/main/java/org/elasticsearch/common/io/ReleasableBytesStream.java b/src/main/java/org/elasticsearch/common/io/ReleasableBytesStream.java
new file mode 100644
index 0000000000000..d6971b52d0c05
--- /dev/null
+++ b/src/main/java/org/elasticsearch/common/io/ReleasableBytesStream.java
@@ -0,0 +1,30 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.common.io;
+
+import org.elasticsearch.common.bytes.ReleasableBytesReference;
+
+/**
+ * A bytes stream that requires its bytes to be released once no longer used.
+ */
+public interface ReleasableBytesStream extends BytesStream {
+
+    ReleasableBytesReference bytes();
+}
diff --git a/src/main/java/org/elasticsearch/common/io/stream/BytesStreamOutput.java b/src/main/java/org/elasticsearch/common/io/stream/BytesStreamOutput.java
index 899ecf2d87315..6f51f07a0d498 100644
--- a/src/main/java/org/elasticsearch/common/io/stream/BytesStreamOutput.java
+++ b/src/main/java/org/elasticsearch/common/io/stream/BytesStreamOutput.java
@@ -33,37 +33,31 @@
  */
 public class BytesStreamOutput extends StreamOutput implements BytesStream {
 
-    /**
-     * Factory/manager for our ByteArray
-     */
-    private final BigArrays bigarrays;
+    protected final BigArrays bigarrays;
 
-    /**
-     * The internal list of pages.
-     */
-    private ByteArray bytes;
+    protected ByteArray bytes;
+    protected int count;
 
     /**
-     * The number of valid bytes in the buffer.
-     */
-    private int count;
-
-    /**
-     * Create a nonrecycling {@link BytesStreamOutput} with 1 initial page acquired.
+     * Create a non recycling {@link BytesStreamOutput} with 1 initial page acquired.
      */
     public BytesStreamOutput() {
         this(BigArrays.PAGE_SIZE_IN_BYTES);
     }
 
     /**
-     * Create a nonrecycling {@link BytesStreamOutput} with enough initial pages acquired
-     * to satisfy the capacity given by {@link expectedSize}.
+     * Create a non recycling {@link BytesStreamOutput} with enough initial pages acquired
+     * to satisfy the capacity given by expected size.
      * 
      * @param expectedSize the expected maximum size of the stream in bytes.
      */
     public BytesStreamOutput(int expectedSize) {
-        bigarrays = BigArrays.NON_RECYCLING_INSTANCE;
-        bytes = bigarrays.newByteArray(expectedSize);
+        this(expectedSize, BigArrays.NON_RECYCLING_INSTANCE);
+    }
+
+    protected BytesStreamOutput(int expectedSize, BigArrays bigarrays) {
+        this.bigarrays = bigarrays;
+        this.bytes = bigarrays.newByteArray(expectedSize);
     }
 
     @Override
diff --git a/src/main/java/org/elasticsearch/common/io/stream/ReleasableBytesStreamOutput.java b/src/main/java/org/elasticsearch/common/io/stream/ReleasableBytesStreamOutput.java
new file mode 100644
index 0000000000000..0ead43059f043
--- /dev/null
+++ b/src/main/java/org/elasticsearch/common/io/stream/ReleasableBytesStreamOutput.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.common.io.stream;
+
+import org.elasticsearch.common.bytes.ReleasableBytesReference;
+import org.elasticsearch.common.bytes.ReleasablePagedBytesReference;
+import org.elasticsearch.common.io.ReleasableBytesStream;
+import org.elasticsearch.common.util.BigArrays;
+
+/**
+ * An bytes stream output that allows providing a {@link BigArrays} instance
+ * expecting it to require releasing its content ({@link #bytes()}) once done.
+ * <p/>
+ * Please note, its is the responsibility of the caller to make sure the bytes
+ * reference do not "escape" and are released only once.
+ */
+public class ReleasableBytesStreamOutput extends BytesStreamOutput implements ReleasableBytesStream {
+
+    public ReleasableBytesStreamOutput(BigArrays bigarrays) {
+        super(BigArrays.PAGE_SIZE_IN_BYTES, bigarrays);
+    }
+
+    public ReleasableBytesStreamOutput(int expectedSize, BigArrays bigarrays) {
+        super(expectedSize, bigarrays);
+    }
+
+    @Override
+    public ReleasableBytesReference bytes() {
+        return new ReleasablePagedBytesReference(bigarrays, bytes, count);
+    }
+}
diff --git a/src/main/java/org/elasticsearch/index/translog/fs/FsTranslog.java b/src/main/java/org/elasticsearch/index/translog/fs/FsTranslog.java
index d485e695905c9..528c5d66d5cc8 100644
--- a/src/main/java/org/elasticsearch/index/translog/fs/FsTranslog.java
+++ b/src/main/java/org/elasticsearch/index/translog/fs/FsTranslog.java
@@ -21,12 +21,14 @@
 
 import jsr166y.ThreadLocalRandom;
 import org.elasticsearch.ElasticsearchException;
-import org.elasticsearch.common.bytes.BytesReference;
+import org.elasticsearch.common.bytes.ReleasableBytesReference;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.io.FileSystemUtils;
-import org.elasticsearch.common.io.stream.BytesStreamOutput;
+import org.elasticsearch.common.io.stream.ReleasableBytesStreamOutput;
+import org.elasticsearch.common.lease.Releasables;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.unit.ByteSizeValue;
+import org.elasticsearch.common.util.BigArrays;
 import org.elasticsearch.env.NodeEnvironment;
 import org.elasticsearch.index.settings.IndexSettings;
 import org.elasticsearch.index.settings.IndexSettingsService;
@@ -62,6 +64,7 @@ public void onRefreshSettings(Settings settings) {
     }
 
     private final IndexSettingsService indexSettingsService;
+    private final BigArrays bigArrays;
 
     private final ReadWriteLock rwl = new ReentrantReadWriteLock();
     private final File[] locations;
@@ -79,9 +82,10 @@ public void onRefreshSettings(Settings settings) {
     private final ApplySettings applySettings = new ApplySettings();
 
     @Inject
-    public FsTranslog(ShardId shardId, @IndexSettings Settings indexSettings, IndexSettingsService indexSettingsService, NodeEnvironment nodeEnv) {
+    public FsTranslog(ShardId shardId, @IndexSettings Settings indexSettings, IndexSettingsService indexSettingsService, NodeEnvironment nodeEnv, BigArrays bigArrays) {
         super(shardId, indexSettings);
         this.indexSettingsService = indexSettingsService;
+        this.bigArrays = bigArrays;
         File[] shardLocations = nodeEnv.shardLocations(shardId);
         this.locations = new File[shardLocations.length];
         for (int i = 0; i < shardLocations.length; i++) {
@@ -101,6 +105,7 @@ public FsTranslog(ShardId shardId, @IndexSettings Settings indexSettings, File l
         this.indexSettingsService = null;
         this.locations = new File[]{location};
         FileSystemUtils.mkdirs(location);
+        this.bigArrays = BigArrays.NON_RECYCLING_INSTANCE;
 
         this.type = FsTranslogFile.Type.fromString(componentSettings.get("type", FsTranslogFile.Type.BUFFERED.name()));
     }
@@ -335,8 +340,9 @@ public byte[] read(Location location) {
     @Override
     public Location add(Operation operation) throws TranslogException {
         rwl.readLock().lock();
+        ReleasableBytesStreamOutput out = new ReleasableBytesStreamOutput(bigArrays);
+        boolean released = false;
         try {
-            BytesStreamOutput out = new BytesStreamOutput();
             out.writeInt(0); // marker for the size...
             TranslogStreams.writeTranslogOperation(out, operation);
             out.flush();
@@ -345,11 +351,11 @@ public Location add(Operation operation) throws TranslogException {
             int size = out.size();
             out.seek(0);
             out.writeInt(size - 4);
-            
+
             // seek back to end
             out.seek(size);
 
-            BytesReference bytes = out.bytes();
+            ReleasableBytesReference bytes = out.bytes();
             Location location = current.add(bytes);
             if (syncOnEachOperation) {
                 current.sync();
@@ -362,11 +368,16 @@ public Location add(Operation operation) throws TranslogException {
                     // ignore
                 }
             }
+            Releasables.release(bytes);
+            released = true;
             return location;
-        } catch (Exception e) {
+        } catch (Throwable e) {
             throw new TranslogException(shardId, "Failed to write operation [" + operation + "]", e);
         } finally {
             rwl.readLock().unlock();
+            if (!released) {
+                Releasables.release(out.bytes());
+            }
         }
     }
 
diff --git a/src/main/java/org/elasticsearch/transport/netty/NettyTransport.java b/src/main/java/org/elasticsearch/transport/netty/NettyTransport.java
index eeaee80354888..deaf4dd098b04 100644
--- a/src/main/java/org/elasticsearch/transport/netty/NettyTransport.java
+++ b/src/main/java/org/elasticsearch/transport/netty/NettyTransport.java
@@ -27,15 +27,19 @@
 import org.elasticsearch.Version;
 import org.elasticsearch.cluster.node.DiscoveryNode;
 import org.elasticsearch.common.Strings;
+import org.elasticsearch.common.bytes.ReleasableBytesReference;
 import org.elasticsearch.common.component.AbstractLifecycleComponent;
 import org.elasticsearch.common.compress.CompressorFactory;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.io.stream.BytesStreamOutput;
 import org.elasticsearch.common.io.stream.HandlesStreamOutput;
+import org.elasticsearch.common.io.stream.ReleasableBytesStreamOutput;
 import org.elasticsearch.common.io.stream.StreamOutput;
+import org.elasticsearch.common.lease.Releasables;
 import org.elasticsearch.common.math.MathUtils;
 import org.elasticsearch.common.netty.NettyStaticSetup;
 import org.elasticsearch.common.netty.OpenChannelsHandler;
+import org.elasticsearch.common.netty.ReleaseChannelFutureListener;
 import org.elasticsearch.common.network.NetworkService;
 import org.elasticsearch.common.network.NetworkUtils;
 import org.elasticsearch.common.settings.Settings;
@@ -45,6 +49,7 @@
 import org.elasticsearch.common.transport.TransportAddress;
 import org.elasticsearch.common.unit.ByteSizeValue;
 import org.elasticsearch.common.unit.TimeValue;
+import org.elasticsearch.common.util.BigArrays;
 import org.elasticsearch.common.util.concurrent.EsExecutors;
 import org.elasticsearch.common.util.concurrent.KeyedLock;
 import org.elasticsearch.monitor.jvm.JvmInfo;
@@ -140,6 +145,8 @@ public class NettyTransport extends AbstractLifecycleComponent<Transport> implem
     final ByteSizeValue maxCumulationBufferCapacity;
     final int maxCompositeBufferComponents;
 
+    final BigArrays bigArrays;
+
     private final ThreadPool threadPool;
 
     private volatile OpenChannelsHandler serverOpenChannels;
@@ -165,10 +172,11 @@ public class NettyTransport extends AbstractLifecycleComponent<Transport> implem
     private final ReadWriteLock globalLock = new ReentrantReadWriteLock();
 
     @Inject
-    public NettyTransport(Settings settings, ThreadPool threadPool, NetworkService networkService, Version version) {
+    public NettyTransport(Settings settings, ThreadPool threadPool, NetworkService networkService, BigArrays bigArrays, Version version) {
         super(settings);
         this.threadPool = threadPool;
         this.networkService = networkService;
+        this.bigArrays = bigArrays;
         this.version = version;
 
         if (settings.getAsBoolean("netty.epollBugWorkaround", false)) {
@@ -547,58 +555,58 @@ public void sendRequest(final DiscoveryNode node, final long requestId, final St
         byte status = 0;
         status = TransportStatus.setRequest(status);
 
-        BytesStreamOutput bStream = new BytesStreamOutput();
-        bStream.skip(NettyHeader.HEADER_SIZE);
-        StreamOutput stream = bStream;
-        // only compress if asked, and, the request is not bytes, since then only
-        // the header part is compressed, and the "body" can't be extracted as compressed
-        if (options.compress() && (!(request instanceof BytesTransportRequest))) {
-            status = TransportStatus.setCompress(status);
-            stream = CompressorFactory.defaultCompressor().streamOutput(stream);
-        }
-        stream = new HandlesStreamOutput(stream);
-
-        // we pick the smallest of the 2, to support both backward and forward compatibility
-        // note, this is the only place we need to do this, since from here on, we use the serialized version
-        // as the version to use also when the node receiving this request will send the response with
-        Version version = Version.smallest(this.version, node.version());
-
-        stream.setVersion(version);
-        stream.writeString(action);
-
-        ChannelBuffer buffer;
-        // it might be nice to somehow generalize this optimization, maybe a smart "paged" bytes output
-        // that create paged channel buffers, but its tricky to know when to do it (where this option is
-        // more explicit).
-        if (request instanceof BytesTransportRequest) {
-            BytesTransportRequest bRequest = (BytesTransportRequest) request;
-            assert node.version().equals(bRequest.version());
-            bRequest.writeThin(stream);
-            stream.close();
-            ChannelBuffer headerBuffer = bStream.bytes().toChannelBuffer();
-            ChannelBuffer contentBuffer = bRequest.bytes().toChannelBuffer();
-            // false on gathering, cause gathering causes the NIO layer to combine the buffers into a single direct buffer....
-            buffer = new CompositeChannelBuffer(headerBuffer.order(), ImmutableList.<ChannelBuffer>of(headerBuffer, contentBuffer), false);
-        } else {
-            request.writeTo(stream);
-            stream.close();
-            buffer = bStream.bytes().toChannelBuffer();
-        }
-        NettyHeader.writeHeader(buffer, requestId, status, version);
-        targetChannel.write(buffer);
-
-        // We handle close connection exception in the #exceptionCaught method, which is the main reason we want to add this future
-//        channelFuture.addListener(new ChannelFutureListener() {
-//            @Override public void operationComplete(ChannelFuture future) throws Exception {
-//                if (!future.isSuccess()) {
-//                    // maybe add back the retry?
-//                    TransportResponseHandler handler = transportServiceAdapter.remove(requestId);
-//                    if (handler != null) {
-//                        handler.handleException(new RemoteTransportException("Failed write request", new SendRequestTransportException(node, action, future.getCause())));
-//                    }
-//                }
-//            }
-//        });
+        ReleasableBytesStreamOutput bStream = new ReleasableBytesStreamOutput(bigArrays);
+        boolean addedReleaseListener = false;
+        try {
+            bStream.skip(NettyHeader.HEADER_SIZE);
+            StreamOutput stream = bStream;
+            // only compress if asked, and, the request is not bytes, since then only
+            // the header part is compressed, and the "body" can't be extracted as compressed
+            if (options.compress() && (!(request instanceof BytesTransportRequest))) {
+                status = TransportStatus.setCompress(status);
+                stream = CompressorFactory.defaultCompressor().streamOutput(stream);
+            }
+            stream = new HandlesStreamOutput(stream);
+
+            // we pick the smallest of the 2, to support both backward and forward compatibility
+            // note, this is the only place we need to do this, since from here on, we use the serialized version
+            // as the version to use also when the node receiving this request will send the response with
+            Version version = Version.smallest(this.version, node.version());
+
+            stream.setVersion(version);
+            stream.writeString(action);
+
+            ReleasableBytesReference bytes;
+            ChannelBuffer buffer;
+            // it might be nice to somehow generalize this optimization, maybe a smart "paged" bytes output
+            // that create paged channel buffers, but its tricky to know when to do it (where this option is
+            // more explicit).
+            if (request instanceof BytesTransportRequest) {
+                BytesTransportRequest bRequest = (BytesTransportRequest) request;
+                assert node.version().equals(bRequest.version());
+                bRequest.writeThin(stream);
+                stream.close();
+                bytes = bStream.bytes();
+                ChannelBuffer headerBuffer = bytes.toChannelBuffer();
+                ChannelBuffer contentBuffer = bRequest.bytes().toChannelBuffer();
+                // false on gathering, cause gathering causes the NIO layer to combine the buffers into a single direct buffer....
+                buffer = new CompositeChannelBuffer(headerBuffer.order(), ImmutableList.<ChannelBuffer>of(headerBuffer, contentBuffer), false);
+            } else {
+                request.writeTo(stream);
+                stream.close();
+                bytes = bStream.bytes();
+                buffer = bytes.toChannelBuffer();
+            }
+            NettyHeader.writeHeader(buffer, requestId, status, version);
+            ChannelFuture future = targetChannel.write(buffer);
+            ReleaseChannelFutureListener listener = new ReleaseChannelFutureListener(bytes);
+            future.addListener(listener);
+            addedReleaseListener = true;
+        } finally {
+            if (!addedReleaseListener) {
+                Releasables.release(bStream.bytes());
+            }
+        }
     }
 
     @Override
diff --git a/src/main/java/org/elasticsearch/transport/netty/NettyTransportChannel.java b/src/main/java/org/elasticsearch/transport/netty/NettyTransportChannel.java
index 5d50d4e3158bf..c915ccea7c63c 100644
--- a/src/main/java/org/elasticsearch/transport/netty/NettyTransportChannel.java
+++ b/src/main/java/org/elasticsearch/transport/netty/NettyTransportChannel.java
@@ -20,15 +20,19 @@
 package org.elasticsearch.transport.netty;
 
 import org.elasticsearch.Version;
+import org.elasticsearch.common.bytes.ReleasableBytesReference;
 import org.elasticsearch.common.compress.CompressorFactory;
 import org.elasticsearch.common.io.ThrowableObjectOutputStream;
-import org.elasticsearch.common.io.stream.BytesStreamOutput;
 import org.elasticsearch.common.io.stream.HandlesStreamOutput;
+import org.elasticsearch.common.io.stream.ReleasableBytesStreamOutput;
 import org.elasticsearch.common.io.stream.StreamOutput;
+import org.elasticsearch.common.lease.Releasables;
+import org.elasticsearch.common.netty.ReleaseChannelFutureListener;
 import org.elasticsearch.transport.*;
 import org.elasticsearch.transport.support.TransportStatus;
 import org.jboss.netty.buffer.ChannelBuffer;
 import org.jboss.netty.channel.Channel;
+import org.jboss.netty.channel.ChannelFuture;
 
 import java.io.IOException;
 import java.io.NotSerializableException;
@@ -71,47 +75,69 @@ public void sendResponse(TransportResponse response, TransportResponseOptions op
         byte status = 0;
         status = TransportStatus.setResponse(status);
 
-        BytesStreamOutput bStream = new BytesStreamOutput();
-        bStream.skip(NettyHeader.HEADER_SIZE);
-        StreamOutput stream = bStream;
-        if (options.compress()) {
-            status = TransportStatus.setCompress(status);
-            stream = CompressorFactory.defaultCompressor().streamOutput(stream);
-        }
-        stream = new HandlesStreamOutput(stream);
-        stream.setVersion(version);
-        response.writeTo(stream);
-        stream.close();
+        ReleasableBytesStreamOutput bStream = new ReleasableBytesStreamOutput(transport.bigArrays);
+        boolean addedReleaseListener = false;
+        try {
+            bStream.skip(NettyHeader.HEADER_SIZE);
+            StreamOutput stream = bStream;
+            if (options.compress()) {
+                status = TransportStatus.setCompress(status);
+                stream = CompressorFactory.defaultCompressor().streamOutput(stream);
+            }
+            stream = new HandlesStreamOutput(stream);
+            stream.setVersion(version);
+            response.writeTo(stream);
+            stream.close();
 
-        ChannelBuffer buffer = bStream.bytes().toChannelBuffer();
-        NettyHeader.writeHeader(buffer, requestId, status, version);
-        channel.write(buffer);
+            ReleasableBytesReference bytes = bStream.bytes();
+            ChannelBuffer buffer = bytes.toChannelBuffer();
+            NettyHeader.writeHeader(buffer, requestId, status, version);
+            ChannelFuture future = channel.write(buffer);
+            ReleaseChannelFutureListener listener = new ReleaseChannelFutureListener(bytes);
+            future.addListener(listener);
+            addedReleaseListener = true;
+        } finally {
+            if (!addedReleaseListener) {
+                Releasables.release(bStream.bytes());
+            }
+        }
     }
 
     @Override
     public void sendResponse(Throwable error) throws IOException {
-        BytesStreamOutput stream = new BytesStreamOutput();
+        ReleasableBytesStreamOutput stream = new ReleasableBytesStreamOutput(transport.bigArrays);
+        boolean addedReleaseListener = false;
         try {
-            stream.skip(NettyHeader.HEADER_SIZE);
-            RemoteTransportException tx = new RemoteTransportException(transport.nodeName(), transport.wrapAddress(channel.getLocalAddress()), action, error);
-            ThrowableObjectOutputStream too = new ThrowableObjectOutputStream(stream);
-            too.writeObject(tx);
-            too.close();
-        } catch (NotSerializableException e) {
-            stream.reset();
-            stream.skip(NettyHeader.HEADER_SIZE);
-            RemoteTransportException tx = new RemoteTransportException(transport.nodeName(), transport.wrapAddress(channel.getLocalAddress()), action, new NotSerializableTransportException(error));
-            ThrowableObjectOutputStream too = new ThrowableObjectOutputStream(stream);
-            too.writeObject(tx);
-            too.close();
-        }
+            try {
+                stream.skip(NettyHeader.HEADER_SIZE);
+                RemoteTransportException tx = new RemoteTransportException(transport.nodeName(), transport.wrapAddress(channel.getLocalAddress()), action, error);
+                ThrowableObjectOutputStream too = new ThrowableObjectOutputStream(stream);
+                too.writeObject(tx);
+                too.close();
+            } catch (NotSerializableException e) {
+                stream.reset();
+                stream.skip(NettyHeader.HEADER_SIZE);
+                RemoteTransportException tx = new RemoteTransportException(transport.nodeName(), transport.wrapAddress(channel.getLocalAddress()), action, new NotSerializableTransportException(error));
+                ThrowableObjectOutputStream too = new ThrowableObjectOutputStream(stream);
+                too.writeObject(tx);
+                too.close();
+            }
 
-        byte status = 0;
-        status = TransportStatus.setResponse(status);
-        status = TransportStatus.setError(status);
+            byte status = 0;
+            status = TransportStatus.setResponse(status);
+            status = TransportStatus.setError(status);
 
-        ChannelBuffer buffer = stream.bytes().toChannelBuffer();
-        NettyHeader.writeHeader(buffer, requestId, status, version);
-        channel.write(buffer);
+            ReleasableBytesReference bytes = stream.bytes();
+            ChannelBuffer buffer = bytes.toChannelBuffer();
+            NettyHeader.writeHeader(buffer, requestId, status, version);
+            ChannelFuture future = channel.write(buffer);
+            ReleaseChannelFutureListener listener = new ReleaseChannelFutureListener(bytes);
+            future.addListener(listener);
+            addedReleaseListener = true;
+        } finally {
+            if (!addedReleaseListener) {
+                Releasables.release(stream.bytes());
+            }
+        }
     }
 }
diff --git a/src/test/java/org/elasticsearch/benchmark/transport/BenchmarkNettyLargeMessages.java b/src/test/java/org/elasticsearch/benchmark/transport/BenchmarkNettyLargeMessages.java
index 84bf8d9cc8def..2b355676c3da5 100644
--- a/src/test/java/org/elasticsearch/benchmark/transport/BenchmarkNettyLargeMessages.java
+++ b/src/test/java/org/elasticsearch/benchmark/transport/BenchmarkNettyLargeMessages.java
@@ -28,6 +28,7 @@
 import org.elasticsearch.common.transport.InetSocketTransportAddress;
 import org.elasticsearch.common.unit.ByteSizeUnit;
 import org.elasticsearch.common.unit.ByteSizeValue;
+import org.elasticsearch.common.util.BigArrays;
 import org.elasticsearch.threadpool.ThreadPool;
 import org.elasticsearch.transport.*;
 import org.elasticsearch.transport.netty.NettyTransport;
@@ -53,8 +54,8 @@ public static void main(String[] args) throws InterruptedException {
         NetworkService networkService = new NetworkService(settings);
 
         final ThreadPool threadPool = new ThreadPool();
-        final TransportService transportServiceServer = new TransportService(new NettyTransport(settings, threadPool, networkService, Version.CURRENT), threadPool).start();
-        final TransportService transportServiceClient = new TransportService(new NettyTransport(settings, threadPool, networkService, Version.CURRENT), threadPool).start();
+        final TransportService transportServiceServer = new TransportService(new NettyTransport(settings, threadPool, networkService, BigArrays.NON_RECYCLING_INSTANCE, Version.CURRENT), threadPool).start();
+        final TransportService transportServiceClient = new TransportService(new NettyTransport(settings, threadPool, networkService, BigArrays.NON_RECYCLING_INSTANCE, Version.CURRENT), threadPool).start();
 
         final DiscoveryNode bigNode = new DiscoveryNode("big", new InetSocketTransportAddress("localhost", 9300), Version.CURRENT);
 //        final DiscoveryNode smallNode = new DiscoveryNode("small", new InetSocketTransportAddress("localhost", 9300));
diff --git a/src/test/java/org/elasticsearch/benchmark/transport/TransportBenchmark.java b/src/test/java/org/elasticsearch/benchmark/transport/TransportBenchmark.java
index 3c206ef61cc70..39a44e9a9552e 100644
--- a/src/test/java/org/elasticsearch/benchmark/transport/TransportBenchmark.java
+++ b/src/test/java/org/elasticsearch/benchmark/transport/TransportBenchmark.java
@@ -27,6 +27,7 @@
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.unit.ByteSizeUnit;
 import org.elasticsearch.common.unit.ByteSizeValue;
+import org.elasticsearch.common.util.BigArrays;
 import org.elasticsearch.threadpool.ThreadPool;
 import org.elasticsearch.transport.*;
 import org.elasticsearch.transport.local.LocalTransport;
@@ -50,7 +51,7 @@ public Transport newTransport(Settings settings, ThreadPool threadPool) {
         NETTY {
             @Override
             public Transport newTransport(Settings settings, ThreadPool threadPool) {
-                return new NettyTransport(settings, threadPool, new NetworkService(ImmutableSettings.EMPTY), Version.CURRENT);
+                return new NettyTransport(settings, threadPool, new NetworkService(ImmutableSettings.EMPTY), BigArrays.NON_RECYCLING_INSTANCE, Version.CURRENT);
             }
         };
 
diff --git a/src/test/java/org/elasticsearch/common/util/BigArraysTests.java b/src/test/java/org/elasticsearch/common/util/BigArraysTests.java
index 1b82f010d6703..781ab143d9dcc 100644
--- a/src/test/java/org/elasticsearch/common/util/BigArraysTests.java
+++ b/src/test/java/org/elasticsearch/common/util/BigArraysTests.java
@@ -20,7 +20,7 @@
 package org.elasticsearch.common.util;
 
 import org.apache.lucene.util.BytesRef;
-import org.elasticsearch.cache.recycler.MockPageCacheRecycler;
+import org.elasticsearch.test.cache.recycler.MockPageCacheRecycler;
 import org.elasticsearch.cache.recycler.PageCacheRecycler;
 import org.elasticsearch.common.settings.ImmutableSettings;
 import org.elasticsearch.test.ElasticsearchTestCase;
diff --git a/src/test/java/org/elasticsearch/discovery/zen/ping/unicast/UnicastZenPingTests.java b/src/test/java/org/elasticsearch/discovery/zen/ping/unicast/UnicastZenPingTests.java
index 7a553e2b3d8f5..b5b0b4c0b4649 100644
--- a/src/test/java/org/elasticsearch/discovery/zen/ping/unicast/UnicastZenPingTests.java
+++ b/src/test/java/org/elasticsearch/discovery/zen/ping/unicast/UnicastZenPingTests.java
@@ -28,6 +28,7 @@
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.transport.InetSocketTransportAddress;
 import org.elasticsearch.common.unit.TimeValue;
+import org.elasticsearch.common.util.BigArrays;
 import org.elasticsearch.discovery.zen.DiscoveryNodesProvider;
 import org.elasticsearch.discovery.zen.ping.ZenPing;
 import org.elasticsearch.node.service.NodeService;
@@ -55,13 +56,13 @@ public void testSimplePings() {
         ClusterName clusterName = new ClusterName("test");
         NetworkService networkService = new NetworkService(settings);
 
-        NettyTransport transportA = new NettyTransport(settings, threadPool, networkService, Version.CURRENT);
+        NettyTransport transportA = new NettyTransport(settings, threadPool, networkService, BigArrays.NON_RECYCLING_INSTANCE, Version.CURRENT);
         final TransportService transportServiceA = new TransportService(transportA, threadPool).start();
         final DiscoveryNode nodeA = new DiscoveryNode("UZP_A", transportServiceA.boundAddress().publishAddress(), Version.CURRENT);
 
         InetSocketTransportAddress addressA = (InetSocketTransportAddress) transportA.boundAddress().publishAddress();
 
-        NettyTransport transportB = new NettyTransport(settings, threadPool, networkService, Version.CURRENT);
+        NettyTransport transportB = new NettyTransport(settings, threadPool, networkService, BigArrays.NON_RECYCLING_INSTANCE, Version.CURRENT);
         final TransportService transportServiceB = new TransportService(transportB, threadPool).start();
         final DiscoveryNode nodeB = new DiscoveryNode("UZP_B", transportServiceA.boundAddress().publishAddress(), Version.CURRENT);
 
diff --git a/src/test/java/org/elasticsearch/test/ElasticsearchTestCase.java b/src/test/java/org/elasticsearch/test/ElasticsearchTestCase.java
index 7402676deb04e..1d13ebc4b5737 100644
--- a/src/test/java/org/elasticsearch/test/ElasticsearchTestCase.java
+++ b/src/test/java/org/elasticsearch/test/ElasticsearchTestCase.java
@@ -30,7 +30,7 @@
 import org.apache.lucene.util.LuceneTestCase;
 import org.apache.lucene.util.TimeUnits;
 import org.elasticsearch.Version;
-import org.elasticsearch.cache.recycler.MockPageCacheRecycler;
+import org.elasticsearch.test.cache.recycler.MockPageCacheRecycler;
 import org.elasticsearch.client.Requests;
 import org.elasticsearch.common.Strings;
 import org.elasticsearch.common.logging.ESLogger;
@@ -134,7 +134,7 @@ public void resetPageTracking() {
     }
 
     @After
-    public void ensureAllPagesReleased() {
+    public void ensureAllPagesReleased() throws Exception {
         MockPageCacheRecycler.ensureAllPagesAreReleased();
     }
 
@@ -145,7 +145,7 @@ public void resetArrayTracking() {
     }
 
     @After
-    public void ensureAllArraysReleased() {
+    public void ensureAllArraysReleased() throws Exception {
         MockBigArrays.ensureAllArraysAreReleased();
     }
 
diff --git a/src/test/java/org/elasticsearch/test/cache/recycler/MockBigArrays.java b/src/test/java/org/elasticsearch/test/cache/recycler/MockBigArrays.java
index af7c364725ed7..5d6b4b5b626e0 100644
--- a/src/test/java/org/elasticsearch/test/cache/recycler/MockBigArrays.java
+++ b/src/test/java/org/elasticsearch/test/cache/recycler/MockBigArrays.java
@@ -21,12 +21,17 @@
 
 import com.carrotsearch.randomizedtesting.RandomizedContext;
 import com.carrotsearch.randomizedtesting.SeedUtils;
+import com.google.common.base.Predicate;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
 import org.apache.lucene.util.BytesRef;
 import org.elasticsearch.cache.recycler.PageCacheRecycler;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.util.*;
+import org.elasticsearch.test.ElasticsearchTestCase;
 
+import java.util.Map;
 import java.util.Random;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.ConcurrentMap;
@@ -56,12 +61,31 @@ public static void reset() {
         ACQUIRED_ARRAYS.clear();
     }
 
-    public static void ensureAllArraysAreReleased() {
+    public static void ensureAllArraysAreReleased() throws Exception {
         if (DISCARD) {
             DISCARD = false;
-        } else if (ACQUIRED_ARRAYS.size() > 0) {
-            final Object cause = ACQUIRED_ARRAYS.entrySet().iterator().next().getValue();
-            throw new RuntimeException(ACQUIRED_ARRAYS.size() + " arrays have not been released", cause instanceof Throwable ? (Throwable) cause : null);
+        } else {
+            final Map<Object, Object> masterCopy = Maps.newHashMap(ACQUIRED_ARRAYS);
+            if (masterCopy.isEmpty()) {
+                return;
+            }
+            // not empty, we might be executing on a shared cluster that keeps on obtaining
+            // and releasing arrays, lets make sure that after a reasonable timeout, all master
+            // copy (snapshot) have been released
+            boolean success = ElasticsearchTestCase.awaitBusy(new Predicate<Object>() {
+                @Override
+                public boolean apply(Object input) {
+                    return Sets.intersection(masterCopy.keySet(), ACQUIRED_ARRAYS.keySet()).isEmpty();
+                }
+            });
+            if (success) {
+                return;
+            }
+            masterCopy.keySet().retainAll(ACQUIRED_ARRAYS.keySet());
+            if (!masterCopy.isEmpty()) {
+                final Object cause = masterCopy.entrySet().iterator().next().getValue();
+                throw new RuntimeException(masterCopy.size() + " arrays have not been released", cause instanceof Throwable ? (Throwable) cause : null);
+            }
         }
     }
 
diff --git a/src/test/java/org/elasticsearch/cache/recycler/MockPageCacheRecycler.java b/src/test/java/org/elasticsearch/test/cache/recycler/MockPageCacheRecycler.java
similarity index 75%
rename from src/test/java/org/elasticsearch/cache/recycler/MockPageCacheRecycler.java
rename to src/test/java/org/elasticsearch/test/cache/recycler/MockPageCacheRecycler.java
index f3904800c836e..2866c71e1bdfe 100644
--- a/src/test/java/org/elasticsearch/cache/recycler/MockPageCacheRecycler.java
+++ b/src/test/java/org/elasticsearch/test/cache/recycler/MockPageCacheRecycler.java
@@ -17,17 +17,22 @@
  * under the License.
  */
 
-package org.elasticsearch.cache.recycler;
+package org.elasticsearch.test.cache.recycler;
 
+import com.google.common.base.Predicate;
 import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
 import org.elasticsearch.ElasticsearchException;
+import org.elasticsearch.cache.recycler.PageCacheRecycler;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.recycler.Recycler.V;
 import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.test.ElasticsearchTestCase;
 import org.elasticsearch.test.TestCluster;
 import org.elasticsearch.threadpool.ThreadPool;
 
 import java.lang.reflect.Array;
+import java.util.Map;
 import java.util.Random;
 import java.util.concurrent.ConcurrentMap;
 
@@ -39,10 +44,27 @@ public static void reset() {
         ACQUIRED_PAGES.clear();
     }
 
-    public static void ensureAllPagesAreReleased() {
-        if (ACQUIRED_PAGES.size() > 0) {
-            final Throwable t = ACQUIRED_PAGES.entrySet().iterator().next().getValue();
-            throw new RuntimeException(ACQUIRED_PAGES.size() + " pages have not been released", t);
+    public static void ensureAllPagesAreReleased() throws Exception {
+        final Map<Object, Throwable> masterCopy = Maps.newHashMap(ACQUIRED_PAGES);
+        if (masterCopy.isEmpty()) {
+            return;
+        }
+        // not empty, we might be executing on a shared cluster that keeps on obtaining
+        // and releasing pages, lets make sure that after a reasonable timeout, all master
+        // copy (snapshot) have been released
+        boolean success = ElasticsearchTestCase.awaitBusy(new Predicate<Object>() {
+            @Override
+            public boolean apply(Object input) {
+                return Sets.intersection(masterCopy.keySet(), ACQUIRED_PAGES.keySet()).isEmpty();
+            }
+        });
+        if (success) {
+            return;
+        }
+        masterCopy.keySet().retainAll(ACQUIRED_PAGES.keySet());
+        if (!masterCopy.isEmpty()) {
+            final Throwable t = masterCopy.entrySet().iterator().next().getValue();
+            throw new RuntimeException(masterCopy.size() + " pages have not been released", t);
         }
     }
 
diff --git a/src/test/java/org/elasticsearch/test/cache/recycler/MockPageCacheRecyclerModule.java b/src/test/java/org/elasticsearch/test/cache/recycler/MockPageCacheRecyclerModule.java
index 3f5b2a4429778..339fb949699fb 100644
--- a/src/test/java/org/elasticsearch/test/cache/recycler/MockPageCacheRecyclerModule.java
+++ b/src/test/java/org/elasticsearch/test/cache/recycler/MockPageCacheRecyclerModule.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.test.cache.recycler;
 
-import org.elasticsearch.cache.recycler.MockPageCacheRecycler;
 import org.elasticsearch.cache.recycler.PageCacheRecycler;
 import org.elasticsearch.common.inject.AbstractModule;
 
diff --git a/src/test/java/org/elasticsearch/transport/netty/SimpleNettyTransportTests.java b/src/test/java/org/elasticsearch/transport/netty/SimpleNettyTransportTests.java
index 17db10094780b..6215ffcae384c 100644
--- a/src/test/java/org/elasticsearch/transport/netty/SimpleNettyTransportTests.java
+++ b/src/test/java/org/elasticsearch/transport/netty/SimpleNettyTransportTests.java
@@ -25,6 +25,7 @@
 import org.elasticsearch.common.settings.ImmutableSettings;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.transport.InetSocketTransportAddress;
+import org.elasticsearch.common.util.BigArrays;
 import org.elasticsearch.test.transport.MockTransportService;
 import org.elasticsearch.transport.AbstractSimpleTransportTests;
 import org.elasticsearch.transport.ConnectTransportException;
@@ -37,7 +38,7 @@ protected MockTransportService build(Settings settings, Version version) {
         int startPort = 11000 + randomIntBetween(0, 255);
         int endPort = startPort + 10;
         settings = ImmutableSettings.builder().put(settings).put("transport.tcp.port", startPort + "-" + endPort).build();
-        MockTransportService transportService = new MockTransportService(settings, new NettyTransport(settings, threadPool, new NetworkService(settings), version), threadPool);
+        MockTransportService transportService = new MockTransportService(settings, new NettyTransport(settings, threadPool, new NetworkService(settings), BigArrays.NON_RECYCLING_INSTANCE, version), threadPool);
         transportService.start();
         return transportService;
     }

From 7b9df39800ff290daa339946e8d0a15d3591233b Mon Sep 17 00:00:00 2001
From: Boaz Leskes <b.leskes@gmail.com>
Date: Sun, 6 Apr 2014 00:41:24 +0200
Subject: [PATCH 07/54] [Test] Added better control over the number of
 documents indexed by BackgroundIndexer

Used the new controls to reduce indexing activity in RelocationTests and RecoveryWhileUnderLoadTests

Closes #5696
---
 .../recovery/RecoveryWhileUnderLoadTests.java |  37 ++++--
 .../recovery/RelocationTests.java             |   9 +-
 .../elasticsearch/test/BackgroundIndexer.java | 120 ++++++++++++++++--
 .../test/ElasticsearchIntegrationTest.java    |  13 +-
 4 files changed, 150 insertions(+), 29 deletions(-)

diff --git a/src/test/java/org/elasticsearch/recovery/RecoveryWhileUnderLoadTests.java b/src/test/java/org/elasticsearch/recovery/RecoveryWhileUnderLoadTests.java
index 1980cfa7c6462..537c4ef6d82fa 100644
--- a/src/test/java/org/elasticsearch/recovery/RecoveryWhileUnderLoadTests.java
+++ b/src/test/java/org/elasticsearch/recovery/RecoveryWhileUnderLoadTests.java
@@ -58,23 +58,29 @@ public void recoverWhileUnderLoadAllocateBackupsTest() throws Exception {
         assertAcked(prepareCreate("test", 1, settingsBuilder().put(SETTING_NUMBER_OF_SHARDS, numberOfShards).put(SETTING_NUMBER_OF_REPLICAS, 1)));
 
         final int totalNumDocs = scaledRandomIntBetween(200, 20000);
-        try (BackgroundIndexer indexer = new BackgroundIndexer("test", "type", client())) {
-            int waitFor = totalNumDocs / 10;
+        int waitFor = totalNumDocs / 10;
+        int extraDocs = waitFor;
+        try (BackgroundIndexer indexer = new BackgroundIndexer("test", "type", client(), extraDocs)) {
             logger.info("--> waiting for {} docs to be indexed ...", waitFor);
             waitForDocs(waitFor, indexer);
             indexer.assertNoFailures();
             logger.info("--> {} docs indexed", waitFor);
 
+            extraDocs = totalNumDocs / 10;
+            waitFor += extraDocs;
+            indexer.continueIndexing(extraDocs);
             logger.info("--> flushing the index ....");
             // now flush, just to make sure we have some data in the index, not just translog
             client().admin().indices().prepareFlush().execute().actionGet();
 
-            waitFor += totalNumDocs / 10;
             logger.info("--> waiting for {} docs to be indexed ...", waitFor);
             waitForDocs(waitFor, indexer);
             indexer.assertNoFailures();
             logger.info("--> {} docs indexed", waitFor);
 
+            extraDocs = totalNumDocs - waitFor;
+            indexer.continueIndexing(extraDocs);
+
             logger.info("--> allow 2 nodes for index [test] ...");
             // now start another node, while we index
             allowNodes("test", 2);
@@ -108,22 +114,28 @@ public void recoverWhileUnderLoadAllocateBackupsRelocatePrimariesTest() throws E
         assertAcked(prepareCreate("test", 1, settingsBuilder().put(SETTING_NUMBER_OF_SHARDS, numberOfShards).put(SETTING_NUMBER_OF_REPLICAS, 1)));
 
         final int totalNumDocs = scaledRandomIntBetween(200, 20000);
-        try (BackgroundIndexer indexer = new BackgroundIndexer("test", "type", client())) {
-            int waitFor = totalNumDocs / 10;
+        int waitFor = totalNumDocs / 10;
+        int extraDocs = waitFor;
+        try (BackgroundIndexer indexer = new BackgroundIndexer("test", "type", client(), extraDocs)) {
             logger.info("--> waiting for {} docs to be indexed ...", waitFor);
             waitForDocs(waitFor, indexer);
             indexer.assertNoFailures();
             logger.info("--> {} docs indexed", waitFor);
 
+            extraDocs = totalNumDocs / 10;
+            waitFor += extraDocs;
+            indexer.continueIndexing(extraDocs);
             logger.info("--> flushing the index ....");
             // now flush, just to make sure we have some data in the index, not just translog
             client().admin().indices().prepareFlush().execute().actionGet();
 
-            waitFor += totalNumDocs / 10;
             logger.info("--> waiting for {} docs to be indexed ...", waitFor);
             waitForDocs(waitFor, indexer);
             indexer.assertNoFailures();
             logger.info("--> {} docs indexed", waitFor);
+
+            extraDocs = totalNumDocs - waitFor;
+            indexer.continueIndexing(extraDocs);
             logger.info("--> allow 4 nodes for index [test] ...");
             allowNodes("test", 4);
 
@@ -156,24 +168,29 @@ public void recoverWhileUnderLoadWithNodeShutdown() throws Exception {
         assertAcked(prepareCreate("test", 2, settingsBuilder().put(SETTING_NUMBER_OF_SHARDS, numberOfShards).put(SETTING_NUMBER_OF_REPLICAS, 1)));
 
         final int totalNumDocs = scaledRandomIntBetween(200, 20000);
-        try (BackgroundIndexer indexer = new BackgroundIndexer("test", "type", client())) {
-            int waitFor = totalNumDocs / 10;
+        int waitFor = totalNumDocs / 10;
+        int extraDocs = waitFor;
+        try (BackgroundIndexer indexer = new BackgroundIndexer("test", "type", client(), extraDocs)) {
             logger.info("--> waiting for {} docs to be indexed ...", waitFor);
             waitForDocs(waitFor, indexer);
             indexer.assertNoFailures();
             logger.info("--> {} docs indexed", waitFor);
 
+            extraDocs = totalNumDocs / 10;
+            waitFor += extraDocs;
+            indexer.continueIndexing(extraDocs);
             logger.info("--> flushing the index ....");
             // now flush, just to make sure we have some data in the index, not just translog
             client().admin().indices().prepareFlush().execute().actionGet();
 
-            waitFor += totalNumDocs / 10;
             logger.info("--> waiting for {} docs to be indexed ...", waitFor);
             waitForDocs(waitFor, indexer);
             indexer.assertNoFailures();
             logger.info("--> {} docs indexed", waitFor);
 
             // now start more nodes, while we index
+            extraDocs = totalNumDocs - waitFor;
+            indexer.continueIndexing(extraDocs);
             logger.info("--> allow 4 nodes for index [test] ...");
             allowNodes("test", 4);
 
@@ -227,7 +244,7 @@ public void recoverWhileRelocating() throws Exception {
 
         final int numDocs = scaledRandomIntBetween(200, 50000);
 
-        try (BackgroundIndexer indexer = new BackgroundIndexer("test", "type", client())) {
+        try (BackgroundIndexer indexer = new BackgroundIndexer("test", "type", client(), numDocs)) {
 
             for (int i = 0; i < numDocs; i += scaledRandomIntBetween(100, Math.min(1000, numDocs))) {
                 indexer.assertNoFailures();
diff --git a/src/test/java/org/elasticsearch/recovery/RelocationTests.java b/src/test/java/org/elasticsearch/recovery/RelocationTests.java
index dc3fa203488d1..3f739353be125 100644
--- a/src/test/java/org/elasticsearch/recovery/RelocationTests.java
+++ b/src/test/java/org/elasticsearch/recovery/RelocationTests.java
@@ -21,7 +21,6 @@
 
 import com.carrotsearch.hppc.IntOpenHashSet;
 import com.carrotsearch.hppc.procedures.IntProcedure;
-import com.carrotsearch.randomizedtesting.RandomizedTest;
 import org.apache.lucene.util.LuceneTestCase.Slow;
 import org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse;
 import org.elasticsearch.action.search.SearchPhaseExecutionException;
@@ -129,8 +128,8 @@ public void testRelocationWhileIndexingRandom() throws Exception {
             }
         }
 
-        final int numDocs = scaledRandomIntBetween(200, 2500);
-        try (BackgroundIndexer indexer = new BackgroundIndexer("test", "type1", client(), scaledRandomIntBetween(2, 5), true, numDocs * 2)) {
+        int numDocs = scaledRandomIntBetween(200, 2500);
+        try (BackgroundIndexer indexer = new BackgroundIndexer("test", "type1", client(), numDocs)) {
             logger.info("--> waiting for {} docs to be indexed ...", numDocs);
             waitForDocs(numDocs, indexer);
             logger.info("--> {} docs indexed", numDocs);
@@ -142,6 +141,9 @@ public void testRelocationWhileIndexingRandom() throws Exception {
                 int toNode = fromNode == 0 ? 1 : 0;
                 fromNode += nodeShiftBased;
                 toNode += nodeShiftBased;
+                numDocs = scaledRandomIntBetween(200, 1000);
+                logger.debug("--> Allow indexer to index [{}] documents", numDocs);
+                indexer.continueIndexing(numDocs);
                 logger.info("--> START relocate the shard from {} to {}", nodes[fromNode], nodes[toNode]);
                 client().admin().cluster().prepareReroute()
                         .add(new MoveAllocationCommand(new ShardId("test", 0), nodes[fromNode], nodes[toNode]))
@@ -154,6 +156,7 @@ public void testRelocationWhileIndexingRandom() throws Exception {
                 assertThat(clusterHealthResponse.isTimedOut(), equalTo(false));
                 clusterHealthResponse = client().admin().cluster().prepareHealth().setWaitForEvents(Priority.LANGUID).setWaitForRelocatingShards(0).setTimeout(ACCEPTABLE_RELOCATION_TIME).execute().actionGet();
                 assertThat(clusterHealthResponse.isTimedOut(), equalTo(false));
+                indexer.pauseIndexing();
                 logger.info("--> DONE relocate the shard from {} to {}", fromNode, toNode);
             }
             logger.info("--> done relocations");
diff --git a/src/test/java/org/elasticsearch/test/BackgroundIndexer.java b/src/test/java/org/elasticsearch/test/BackgroundIndexer.java
index fa432ce7042f5..3326cb8d167dc 100644
--- a/src/test/java/org/elasticsearch/test/BackgroundIndexer.java
+++ b/src/test/java/org/elasticsearch/test/BackgroundIndexer.java
@@ -30,6 +30,7 @@
 
 import java.util.concurrent.CopyOnWriteArrayList;
 import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.Semaphore;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.concurrent.atomic.AtomicLong;
@@ -48,21 +49,64 @@ public class BackgroundIndexer implements AutoCloseable {
     final AtomicLong idGenerator = new AtomicLong();
     final AtomicLong indexCounter = new AtomicLong();
     final CountDownLatch startLatch = new CountDownLatch(1);
-
+    final AtomicBoolean hasBudget = new AtomicBoolean(false); // when set to true, writers will acquire writes from a semaphore
+    final Semaphore availableBudget = new Semaphore(0);
+
+    /**
+     * Start indexing in the background using a random number of threads.
+     *
+     * @param index  index name to index into
+     * @param type   document type
+     * @param client client to use
+     */
     public BackgroundIndexer(String index, String type, Client client) {
-        this(index, type, client, RandomizedTest.scaledRandomIntBetween(2, 5));
+        this(index, type, client, -1);
+    }
+
+    /**
+     * Start indexing in the background using a random number of threads. Indexing will be paused after numOfDocs docs has
+     * been indexed.
+     *
+     * @param index     index name to index into
+     * @param type      document type
+     * @param client    client to use
+     * @param numOfDocs number of document to index before pausing. Set to -1 to have no limit.
+     */
+    public BackgroundIndexer(String index, String type, Client client, int numOfDocs) {
+        this(index, type, client, numOfDocs, RandomizedTest.scaledRandomIntBetween(2, 5));
     }
 
-    public BackgroundIndexer(String index, String type, Client client, int writerCount) {
-        this(index, type, client, writerCount, true, Integer.MAX_VALUE);
+    /**
+     * Start indexing in the background using a given number of threads. Indexing will be paused after numOfDocs docs has
+     * been indexed.
+     *
+     * @param index       index name to index into
+     * @param type        document type
+     * @param client      client to use
+     * @param numOfDocs   number of document to index before pausing. Set to -1 to have no limit.
+     * @param writerCount number of indexing threads to use
+     */
+    public BackgroundIndexer(String index, String type, Client client, int numOfDocs, final int writerCount) {
+        this(index, type, client, numOfDocs, writerCount, true);
     }
 
-    public BackgroundIndexer(final String index, final String type, final Client client, final int writerCount, boolean autoStart, final int maxNumDocs) {
+    /**
+     * Start indexing in the background using a given number of threads. Indexing will be paused after numOfDocs docs has
+     * been indexed.
+     *
+     * @param index       index name to index into
+     * @param type        document type
+     * @param client      client to use
+     * @param numOfDocs   number of document to index before pausing. Set to -1 to have no limit.
+     * @param writerCount number of indexing threads to use
+     * @param autoStart   set to true to start indexing as soon as all threads have been created.
+     */
+    public BackgroundIndexer(final String index, final String type, final Client client, final int numOfDocs, final int writerCount, boolean autoStart) {
 
         failures = new CopyOnWriteArrayList<>();
         writers = new Thread[writerCount];
         stopLatch = new CountDownLatch(writers.length);
-        logger.info("--> starting {} indexing threads", writerCount);
+        logger.info("--> creating {} indexing threads (auto start: [{}], numOfDocs: [{}])", writerCount, autoStart, numOfDocs);
         for (int i = 0; i < writers.length; i++) {
             final int indexerId = i;
             final boolean batch = RandomizedTest.getRandom().nextBoolean();
@@ -73,9 +117,17 @@ public void run() {
                     try {
                         startLatch.await();
                         logger.info("**** starting indexing thread {}", indexerId);
-                        while (!stop.get() && indexCounter.get() < maxNumDocs) {  // step out once we reach the hard limit
+                        while (!stop.get()) {
                             if (batch) {
                                 int batchSize = RandomizedTest.getRandom().nextInt(20) + 1;
+                                if (hasBudget.get()) {
+                                    batchSize = Math.max(Math.min(batchSize, availableBudget.availablePermits()), 1);// always try to get at least one
+                                    if (!availableBudget.tryAcquire(batchSize, 250, TimeUnit.MILLISECONDS)) {
+                                        // time out -> check if we have to stop.
+                                        continue;
+                                    }
+
+                                }
                                 BulkRequestBuilder bulkRequest = client.prepareBulk();
                                 for (int i = 0; i < batchSize; i++) {
                                     id = idGenerator.incrementAndGet();
@@ -92,12 +144,17 @@ public void run() {
                                 }
 
                             } else {
+
+                                if (hasBudget.get() && !availableBudget.tryAcquire(250, TimeUnit.MILLISECONDS)) {
+                                    // time out -> check if we have to stop.
+                                    continue;
+                                }
                                 id = idGenerator.incrementAndGet();
                                 client.prepareIndex(index, type, Long.toString(id) + "-" + indexerId).setSource("test", "value" + id).get();
                                 indexCounter.incrementAndGet();
                             }
                         }
-                        logger.info("**** done indexing thread {}  stop: {} numDocsIndexed: {} maxNumDocs: {}", indexerId, stop.get(), indexCounter.get(), maxNumDocs);
+                        logger.info("**** done indexing thread {}  stop: {} numDocsIndexed: {}", indexerId, stop.get(), indexCounter.get());
                     } catch (Throwable e) {
                         failures.add(e);
                         logger.warn("**** failed indexing thread {} on doc id {}", e, indexerId, id);
@@ -110,20 +167,63 @@ public void run() {
         }
 
         if (autoStart) {
-            startLatch.countDown();
+            start(numOfDocs);
         }
     }
 
+    private void setBudget(int numOfDocs) {
+        logger.debug("updating budget to [{}]", numOfDocs);
+        if (numOfDocs >= 0) {
+            hasBudget.set(true);
+            availableBudget.release(numOfDocs);
+        } else {
+            hasBudget.set(false);
+        }
+
+    }
+
+    /** Start indexing with no limit to the number of documents */
     public void start() {
+        start(-1);
+    }
+
+    /**
+     * Start indexing
+     *
+     * @param numOfDocs number of document to index before pausing. Set to -1 to have no limit.
+     */
+    public void start(int numOfDocs) {
+        assert !stop.get() : "background indexer can not be started after it has stopped";
+        setBudget(numOfDocs);
         startLatch.countDown();
     }
 
+    /** Pausing indexing by setting current document limit to 0 */
+    public void pauseIndexing() {
+        availableBudget.drainPermits();
+        setBudget(0);
+    }
+
+    /** Continue indexing after it has paused. No new document limit will be set */
+    public void continueIndexing() {
+        continueIndexing(-1);
+    }
+
+    /**
+     * Continue indexing after it has paused.
+     *
+     * @param numOfDocs number of document to index before pausing. Set to -1 to have no limit.
+     */
+    public void continueIndexing(int numOfDocs) {
+        setBudget(numOfDocs);
+    }
+
+    /** Stop all background threads **/
     public void stop() throws InterruptedException {
         if (stop.get()) {
             return;
         }
         stop.set(true);
-
         Assert.assertThat("timeout while waiting for indexing threads to stop", stopLatch.await(6, TimeUnit.MINUTES), equalTo(true));
         assertNoFailures();
     }
diff --git a/src/test/java/org/elasticsearch/test/ElasticsearchIntegrationTest.java b/src/test/java/org/elasticsearch/test/ElasticsearchIntegrationTest.java
index 80eddf841a313..e3acdaf6b0bf1 100644
--- a/src/test/java/org/elasticsearch/test/ElasticsearchIntegrationTest.java
+++ b/src/test/java/org/elasticsearch/test/ElasticsearchIntegrationTest.java
@@ -207,7 +207,7 @@ private static void initializeGlobalCluster() {
                     }
                     try {
                         transportAddresses[i++] = new InetSocketTransportAddress(split[0], Integer.valueOf(split[1]));
-                    } catch(NumberFormatException e) {
+                    } catch (NumberFormatException e) {
                         throw new IllegalArgumentException("port is not valid, expected number but was [" + split[1] + "]");
                     }
                 }
@@ -319,7 +319,7 @@ public static ImmutableTestCluster immutableCluster() {
     }
 
     public static TestCluster cluster() {
-        if (!(currentCluster instanceof  TestCluster)) {
+        if (!(currentCluster instanceof TestCluster)) {
             throw new UnsupportedOperationException("current test cluster is immutable");
         }
         return (TestCluster) currentCluster;
@@ -558,7 +558,7 @@ public long waitForDocs(final long numDocs, int maxWaitTime, TimeUnit maxWaitTim
         Predicate<Object> testDocs = new Predicate<Object>() {
             public boolean apply(Object o) {
                 lastKnownCount[0] = indexer.totalIndexedDocs();
-                if (lastKnownCount[0] > numDocs) {
+                if (lastKnownCount[0] >= numDocs) {
                     long count = client().prepareCount().setQuery(matchAllQuery()).execute().actionGet().getCount();
                     if (count == lastKnownCount[0]) {
                         // no progress - try to refresh for the next time
@@ -569,7 +569,7 @@ public boolean apply(Object o) {
                 } else {
                     logger.debug("[{}] docs indexed. waiting for [{}]", lastKnownCount[0], numDocs);
                 }
-                return lastKnownCount[0] > numDocs;
+                return lastKnownCount[0] >= numDocs;
             }
         };
 
@@ -1088,7 +1088,7 @@ private static boolean runTestScopeLifecycle() {
     @Before
     public final void before() throws IOException {
         if (runTestScopeLifecycle()) {
-          beforeInternal();
+            beforeInternal();
         }
     }
 
@@ -1139,7 +1139,8 @@ private final static void initializeSuiteScope() throws Exception {
      *
      * @see SuiteScopeTest
      */
-    protected void setupSuiteScopeCluster() throws Exception {}
+    protected void setupSuiteScopeCluster() throws Exception {
+    }
 
     private static boolean isSuiteScope(Class<?> clazz) {
         if (clazz == Object.class || clazz == ElasticsearchIntegrationTest.class) {

From ade1d0ef571e8eb49ca9e75d16c712e92a3c79c2 Mon Sep 17 00:00:00 2001
From: Martijn van Groningen <martijn.v.groningen@gmail.com>
Date: Mon, 17 Feb 2014 02:23:38 +0100
Subject: [PATCH 08/54] Added global ordinals (unique incremental numbering for
 terms) to fielddata. Added a terms aggregation implementations that work on
 global ordinals, which is also the default.

Closes #5672
---
 .../index-modules/fielddata.asciidoc          |  52 ++
 .../bucket/terms-aggregation.asciidoc         |  17 +-
 .../common/lucene/TopReaderContextAware.java  |  29 ++
 .../index/engine/internal/InternalEngine.java |   1 +
 .../fielddata/AbstractIndexFieldData.java     |   4 +
 .../index/fielddata/AtomicFieldData.java      |  15 +-
 .../index/fielddata/FieldDataType.java        |   2 -
 .../index/fielddata/IndexFieldData.java       |  13 +-
 .../index/fielddata/IndexFieldDataCache.java  |  50 +-
 .../fielddata/IndexFieldDataService.java      |  15 +-
 .../index/fielddata/RamUsage.java             |  30 ++
 .../index/fielddata/ShardFieldData.java       |  16 +-
 .../ordinals/GlobalOrdinalsBuilder.java       |  35 ++
 .../GlobalOrdinalsIndexFieldData.java         | 201 ++++++++
 .../InternalGlobalOrdinalsBuilder.java        | 466 ++++++++++++++++++
 .../plain/AbstractBytesIndexFieldData.java    |  39 +-
 .../AtomicFieldDataWithOrdinalsTermsEnum.java | 149 ++++++
 .../plain/BinaryDVIndexFieldData.java         |   5 +-
 .../plain/BinaryDVNumericIndexFieldData.java  |   5 +-
 .../plain/DisabledIndexFieldData.java         |   3 +-
 .../plain/DocValuesIndexFieldData.java        |  21 +-
 .../plain/DoubleArrayIndexFieldData.java      |   3 +-
 .../plain/FSTBytesAtomicFieldData.java        |   6 +-
 .../plain/FSTBytesIndexFieldData.java         |  10 +-
 .../plain/FloatArrayIndexFieldData.java       |   3 +-
 .../plain/GeoPointBinaryDVIndexFieldData.java |   9 +-
 .../GeoPointCompressedIndexFieldData.java     |   3 +-
 .../GeoPointDoubleArrayIndexFieldData.java    |   3 +-
 .../plain/NumericDVIndexFieldData.java        |   5 +-
 .../plain/PackedArrayIndexFieldData.java      |   3 +-
 .../plain/PagedBytesAtomicFieldData.java      |   6 +
 .../plain/PagedBytesIndexFieldData.java       |  13 +-
 .../plain/ParentChildIndexFieldData.java      |   3 +-
 .../plain/SortedSetDVAtomicFieldData.java     |  19 +
 .../SortedSetDVBytesAtomicFieldData.java      |   1 +
 .../plain/SortedSetDVBytesIndexFieldData.java |  42 +-
 .../index/mapper/FieldMapper.java             |   9 +
 .../elasticsearch/index/shard/ShardUtils.java |   8 +
 .../cache/IndicesFieldDataCache.java          |  63 ++-
 .../cache/IndicesFieldDataCacheListener.java  |   7 +-
 .../indices/warmer/IndicesWarmer.java         |  18 +
 .../indices/warmer/InternalIndicesWarmer.java |  33 +-
 .../percolator/QueryCollector.java            |   1 +
 .../elasticsearch/search/SearchService.java   |  52 +-
 .../search/aggregations/AggregationPhase.java |   1 +
 .../bucket/BucketsAggregator.java             |  18 +
 .../terms/AbstractStringTermsAggregator.java  |  57 +++
 .../GlobalOrdinalsStringTermsAggregator.java  | 168 +++++++
 .../bucket/terms/StringTermsAggregator.java   |  13 +-
 .../bucket/terms/TermsAggregatorFactory.java  |  89 ++--
 .../support/AggregationContext.java           |  12 +
 .../aggregations/support/ValuesSource.java    |  41 +-
 .../aggregations/GlobalOrdinalsBenchmark.java | 277 +++++++++++
 ...AggregationSearchAndIndexingBenchmark.java | 354 +++++++++++++
 .../TermsAggregationSearchBenchmark.java      |  91 +++-
 .../fielddata/AbstractFieldDataTests.java     |  14 +-
 .../AbstractStringFieldDataTests.java         | 156 ++++++
 .../index/fielddata/DuelFieldDataTests.java   |  52 ++
 .../NoOrdinalsStringFieldDataTests.java       |  11 +
 .../search/aggregations/RandomTests.java      |  53 +-
 .../aggregations/bucket/StringTermsTests.java |  32 +-
 .../org/elasticsearch/test/TestCluster.java   |   6 +-
 62 files changed, 2738 insertions(+), 195 deletions(-)
 create mode 100644 src/main/java/org/elasticsearch/common/lucene/TopReaderContextAware.java
 create mode 100644 src/main/java/org/elasticsearch/index/fielddata/RamUsage.java
 create mode 100644 src/main/java/org/elasticsearch/index/fielddata/ordinals/GlobalOrdinalsBuilder.java
 create mode 100644 src/main/java/org/elasticsearch/index/fielddata/ordinals/GlobalOrdinalsIndexFieldData.java
 create mode 100644 src/main/java/org/elasticsearch/index/fielddata/ordinals/InternalGlobalOrdinalsBuilder.java
 create mode 100644 src/main/java/org/elasticsearch/index/fielddata/plain/AtomicFieldDataWithOrdinalsTermsEnum.java
 create mode 100644 src/main/java/org/elasticsearch/search/aggregations/bucket/terms/AbstractStringTermsAggregator.java
 create mode 100644 src/main/java/org/elasticsearch/search/aggregations/bucket/terms/GlobalOrdinalsStringTermsAggregator.java
 create mode 100644 src/test/java/org/elasticsearch/benchmark/search/aggregations/GlobalOrdinalsBenchmark.java
 create mode 100644 src/test/java/org/elasticsearch/benchmark/search/aggregations/TermsAggregationSearchAndIndexingBenchmark.java

diff --git a/docs/reference/index-modules/fielddata.asciidoc b/docs/reference/index-modules/fielddata.asciidoc
index c958dcbc037e2..57afa043fa558 100644
--- a/docs/reference/index-modules/fielddata.asciidoc
+++ b/docs/reference/index-modules/fielddata.asciidoc
@@ -124,6 +124,41 @@ field data format.
 `doc_values`::
     Computes and stores field data data-structures on disk at indexing time.
 
+[float]
+==== Global ordinals
+
+coming[1.2.0]
+
+Global ordinals is a data-structure on top of field data, that maintains an
+incremental numbering for all the terms in field data in a lexicographic order.
+Each term has a unique number and the number of term 'A' is lower than the number
+of term 'B'. Global ordinals are only supported on string fields.
+
+Field data on string also has ordinals, which is a unique numbering for all terms
+in a particular segment and field. Global ordinals just build on top of this,
+by providing a mapping between the segment ordinals and the global ordinals.
+The latter being unique across the entire shard.
+
+Global ordinals can be beneficial in search features that use segment ordinals already
+such as the terms aggregator to improve the execution time. Often these search features
+need to merge the segment ordinal results to a cross segment terms result. With
+global ordinals this mapping happens during field data load time instead of during each
+query execution. With global ordinals search features only need to resolve the actual
+term when building the (shard) response, but during the execution there is no need
+at all to use the actual terms and the unique numbering global ordinals provided is
+sufficient and improves the execution time.
+
+Global ordinals for a specified field are tied to all the segments of a shard (Lucene index),
+which is different than for field data for a specific field which is tied to a single segment.
+For this reason global ordinals need to be rebuilt in its entirety once new segments
+become visible. This one time cost would happen anyway without global ordinals, but
+then it would happen for each search execution instead!
+
+The loading time of global ordinals depends on the number of terms in a field, but in general
+it is low, since it source field data has already been loaded. The memory overhead of global
+ordinals is a small because it is very efficiently compressed. Eager loading of global ordinals
+can move the loading time from the first search request, to the refresh itself.
+
 [float]
 === Fielddata loading
 
@@ -147,6 +182,23 @@ It is possible to force field data to be loaded and cached eagerly through the
 }
 --------------------------------------------------
 
+Global ordinals can also be eagerly loaded:
+
+[source,js]
+--------------------------------------------------
+{
+    category: {
+        type:      "string",
+        fielddata: {
+            loading: "eager_global_ordinals"
+        }
+    }
+}
+--------------------------------------------------
+
+With the above setting both field data and global ordinals for a specific field
+are eagerly loaded.
+
 [float]
 ==== Disabling field data loading
 
diff --git a/docs/reference/search/aggregations/bucket/terms-aggregation.asciidoc b/docs/reference/search/aggregations/bucket/terms-aggregation.asciidoc
index 09583e290c180..cea5409d8dde9 100644
--- a/docs/reference/search/aggregations/bucket/terms-aggregation.asciidoc
+++ b/docs/reference/search/aggregations/bucket/terms-aggregation.asciidoc
@@ -310,12 +310,15 @@ http://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html#UNIX_LINES
 
 ==== Execution hint
 
-There are two mechanisms by which terms aggregations can be executed: either by using field values directly in order to aggregate
-data per-bucket (`map`), or by using ordinals of the field values instead of the values themselves (`ordinals`). Although the
-latter execution mode can be expected to be slightly faster, it is only available for use when the underlying data source exposes
-those terms ordinals. Moreover, it may actually be slower if most field values are unique. Elasticsearch tries to have sensible
-defaults when it comes to the execution mode that should be used, but in case you know that one execution mode may perform better
-than the other one, you have the ability to "hint" it to Elasticsearch:
+coming[1.2.0] The `global_ordinals` execution mode
+
+There are three mechanisms by which terms aggregations can be executed: either by using field values directly in order to aggregate
+data per-bucket (`map`), by using ordinals of the field values instead of the values themselves (`ordinals`) or by using global
+ordinals of the field (`global_ordinals`). The latter is faster, especially for fields with many unique
+values. However it can be slower if only a few documents match, when for example a terms aggregator is nested in another
+aggregator, this applies for both `ordinals` and `global_ordinals` execution modes. Elasticsearch tries to have sensible
+defaults when it comes to the execution mode that should be used, but  in case you know that one execution mode may
+perform better than the other one, you have the ability to "hint" it to Elasticsearch:
 
 [source,js]
 --------------------------------------------------
@@ -331,6 +334,6 @@ than the other one, you have the ability to "hint" it to Elasticsearch:
 }
 --------------------------------------------------
 
-<1> the possible values are `map` and `ordinals`
+<1> the possible values are `map`, `ordinals` and `global_ordinals`
 
 Please note that Elasticsearch will ignore this execution hint if it is not applicable.
diff --git a/src/main/java/org/elasticsearch/common/lucene/TopReaderContextAware.java b/src/main/java/org/elasticsearch/common/lucene/TopReaderContextAware.java
new file mode 100644
index 0000000000000..ea11025319932
--- /dev/null
+++ b/src/main/java/org/elasticsearch/common/lucene/TopReaderContextAware.java
@@ -0,0 +1,29 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.elasticsearch.common.lucene;
+
+import org.apache.lucene.index.IndexReaderContext;
+
+/**
+ *
+ */
+public interface TopReaderContextAware {
+
+    public void setNextReader(IndexReaderContext reader);
+}
diff --git a/src/main/java/org/elasticsearch/index/engine/internal/InternalEngine.java b/src/main/java/org/elasticsearch/index/engine/internal/InternalEngine.java
index d069fd5525def..7fe8849ce27d4 100644
--- a/src/main/java/org/elasticsearch/index/engine/internal/InternalEngine.java
+++ b/src/main/java/org/elasticsearch/index/engine/internal/InternalEngine.java
@@ -1560,6 +1560,7 @@ public IndexSearcher newSearcher(IndexReader reader) throws IOException {
                                 new SimpleSearcher("warmer", newSearcher));
                         warmer.warm(context);
                     }
+                    warmer.warmTop(new IndicesWarmer.WarmerContext(shardId, searcher.getIndexReader()));
                 } catch (Throwable e) {
                     if (!closed) {
                         logger.warn("failed to prepare/warm", e);
diff --git a/src/main/java/org/elasticsearch/index/fielddata/AbstractIndexFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/AbstractIndexFieldData.java
index 34e1ef80a29ce..c370b9cc8a5c3 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/AbstractIndexFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/AbstractIndexFieldData.java
@@ -53,6 +53,10 @@ public FieldMapper.Names getFieldNames() {
         return this.fieldNames;
     }
 
+    public FieldDataType getFieldDataType() {
+        return fieldDataType;
+    }
+
     @Override
     public void clear() {
         cache.clear(fieldNames.indexName());
diff --git a/src/main/java/org/elasticsearch/index/fielddata/AtomicFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/AtomicFieldData.java
index 9845608d664cb..cc3975ebdfccc 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/AtomicFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/AtomicFieldData.java
@@ -19,10 +19,12 @@
 
 package org.elasticsearch.index.fielddata;
 
+import org.apache.lucene.index.TermsEnum;
+
 /**
  * The thread safe {@link org.apache.lucene.index.AtomicReader} level cache of the data.
  */
-public interface AtomicFieldData<Script extends ScriptDocValues> {
+public interface AtomicFieldData<Script extends ScriptDocValues> extends RamUsage {
 
     /**
      * If this method returns false, this means that no document has multiple values. However this method may return true even if all
@@ -46,11 +48,6 @@ public interface AtomicFieldData<Script extends ScriptDocValues> {
      */
     long getNumberUniqueValues();
 
-    /**
-     * Size (in bytes) of memory used by this field data.
-     */
-    long getMemorySizeInBytes();
-
     /**
      * Use a non thread safe (lightweight) view of the values as bytes.
      *
@@ -78,6 +75,12 @@ interface WithOrdinals<Script extends ScriptDocValues> extends AtomicFieldData<S
          * @param needsHashes
          */
         BytesValues.WithOrdinals getBytesValues(boolean needsHashes);
+
+        /**
+         * Returns a terms enum to iterate over all the underlying values.
+         */
+        TermsEnum getTermsEnum();
+
     }
 
     /**
diff --git a/src/main/java/org/elasticsearch/index/fielddata/FieldDataType.java b/src/main/java/org/elasticsearch/index/fielddata/FieldDataType.java
index 559dde870b3ea..98cd00ac79bf5 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/FieldDataType.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/FieldDataType.java
@@ -27,8 +27,6 @@
  */
 public class FieldDataType {
 
-
-
     public static final String FORMAT_KEY = "format";
     public static final String DOC_VALUES_FORMAT_VALUE = "doc_values";
 
diff --git a/src/main/java/org/elasticsearch/index/fielddata/IndexFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/IndexFieldData.java
index c0f71334b1879..f7ff4dd142888 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/IndexFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/IndexFieldData.java
@@ -30,6 +30,7 @@
 import org.elasticsearch.index.Index;
 import org.elasticsearch.index.IndexComponent;
 import org.elasticsearch.index.fielddata.fieldcomparator.SortMode;
+import org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsBuilder;
 import org.elasticsearch.index.mapper.FieldMapper;
 import org.elasticsearch.index.mapper.MapperService;
 import org.elasticsearch.index.settings.IndexSettings;
@@ -56,6 +57,11 @@ public static boolean removeOrdsOnSingleValue(FieldDataType fieldDataType) {
      */
     FieldMapper.Names getFieldNames();
 
+    /**
+     * The field data type.
+     */
+    FieldDataType getFieldDataType();
+
     /**
      * Are the values ordered? (in ascending manner).
      */
@@ -173,7 +179,7 @@ protected final Object missingObject(Object missingValue, boolean reversed) {
     interface Builder {
 
         IndexFieldData build(Index index, @IndexSettings Settings indexSettings, FieldMapper<?> mapper, IndexFieldDataCache cache,
-                             CircuitBreakerService breakerService, MapperService mapperService);
+                             CircuitBreakerService breakerService, MapperService mapperService, GlobalOrdinalsBuilder globalOrdinalBuilder);
     }
 
     public interface WithOrdinals<FD extends AtomicFieldData.WithOrdinals> extends IndexFieldData<FD> {
@@ -187,6 +193,11 @@ public interface WithOrdinals<FD extends AtomicFieldData.WithOrdinals> extends I
          * Loads directly the atomic field data for the reader, ignoring any caching involved.
          */
         FD loadDirect(AtomicReaderContext context) throws Exception;
+
+        WithOrdinals loadGlobal(IndexReader indexReader);
+
+        WithOrdinals localGlobalDirect(IndexReader indexReader) throws Exception;
+
     }
 
 }
diff --git a/src/main/java/org/elasticsearch/index/fielddata/IndexFieldDataCache.java b/src/main/java/org/elasticsearch/index/fielddata/IndexFieldDataCache.java
index 6d36c302f9153..5553cecb8bee8 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/IndexFieldDataCache.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/IndexFieldDataCache.java
@@ -24,9 +24,10 @@
 import com.google.common.cache.RemovalListener;
 import com.google.common.cache.RemovalNotification;
 import org.apache.lucene.index.AtomicReaderContext;
+import org.apache.lucene.index.IndexReader;
 import org.apache.lucene.index.SegmentReader;
-import org.elasticsearch.common.Nullable;
 import org.elasticsearch.common.lucene.SegmentReaderUtils;
+import org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsIndexFieldData;
 import org.elasticsearch.index.mapper.FieldMapper;
 import org.elasticsearch.index.service.IndexService;
 import org.elasticsearch.index.shard.ShardId;
@@ -45,6 +46,8 @@ public interface IndexFieldDataCache {
 
     <FD extends AtomicFieldData, IFD extends IndexFieldData<FD>> FD load(AtomicReaderContext context, IFD indexFieldData) throws Exception;
 
+    <IFD extends IndexFieldData.WithOrdinals<?>> IFD load(final IndexReader indexReader, final IFD indexFieldData) throws Exception;
+
     /**
      * Clears all the field data stored cached in on this index.
      */
@@ -59,19 +62,19 @@ public interface IndexFieldDataCache {
 
     interface Listener {
 
-        void onLoad(FieldMapper.Names fieldNames, FieldDataType fieldDataType, AtomicFieldData fieldData);
+        void onLoad(FieldMapper.Names fieldNames, FieldDataType fieldDataType, RamUsage ramUsage);
 
-        void onUnload(FieldMapper.Names fieldNames, FieldDataType fieldDataType, boolean wasEvicted, long sizeInBytes, @Nullable AtomicFieldData fieldData);
+        void onUnload(FieldMapper.Names fieldNames, FieldDataType fieldDataType, boolean wasEvicted, long sizeInBytes);
     }
 
     /**
      * The resident field data cache is a *per field* cache that keeps all the values in memory.
      */
-    static abstract class FieldBased implements IndexFieldDataCache, SegmentReader.CoreClosedListener, RemovalListener<FieldBased.Key, AtomicFieldData> {
+    static abstract class FieldBased implements IndexFieldDataCache, SegmentReader.CoreClosedListener, RemovalListener<FieldBased.Key, RamUsage>, IndexReader.ReaderClosedListener {
         private final IndexService indexService;
         private final FieldMapper.Names fieldNames;
         private final FieldDataType fieldDataType;
-        private final Cache<Key, AtomicFieldData> cache;
+        private final Cache<Key, RamUsage> cache;
         private final IndicesFieldDataCacheListener indicesFieldDataCacheListener;
 
         protected FieldBased(IndexService indexService, FieldMapper.Names fieldNames, FieldDataType fieldDataType, CacheBuilder cache, IndicesFieldDataCacheListener indicesFieldDataCacheListener) {
@@ -86,18 +89,18 @@ protected FieldBased(IndexService indexService, FieldMapper.Names fieldNames, Fi
         }
 
         @Override
-        public void onRemoval(RemovalNotification<Key, AtomicFieldData> notification) {
+        public void onRemoval(RemovalNotification<Key, RamUsage> notification) {
             final Key key = notification.getKey();
             assert key != null && key.listeners != null;
 
-            final AtomicFieldData value = notification.getValue();
+            final RamUsage value = notification.getValue();
             long sizeInBytes = key.sizeInBytes;
             assert sizeInBytes >= 0 || value != null : "Expected size [" + sizeInBytes + "] to be positive or value [" + value + "] to be non-null";
             if (sizeInBytes == -1 && value != null) {
                 sizeInBytes = value.getMemorySizeInBytes();
             }
             for (Listener listener : key.listeners) {
-                listener.onUnload(fieldNames, fieldDataType, notification.wasEvicted(), sizeInBytes, value);
+                listener.onUnload(fieldNames, fieldDataType, notification.wasEvicted(), sizeInBytes);
             }
         }
 
@@ -127,6 +130,32 @@ public AtomicFieldData call() throws Exception {
             });
         }
 
+        public <IFD extends IndexFieldData.WithOrdinals<?>> IFD load(final IndexReader indexReader, final IFD indexFieldData) throws Exception {
+            final Key key = new Key(indexReader.getCoreCacheKey());
+            //noinspection unchecked
+            return (IFD) cache.get(key, new Callable<RamUsage>() {
+                @Override
+                public GlobalOrdinalsIndexFieldData call() throws Exception {
+                    indexReader.addReaderClosedListener(FieldBased.this);
+                    GlobalOrdinalsIndexFieldData ifd = (GlobalOrdinalsIndexFieldData) indexFieldData.localGlobalDirect(indexReader);
+                    key.sizeInBytes = ifd.getMemorySizeInBytes();
+                    key.listeners.add(indicesFieldDataCacheListener);
+                    final ShardId shardId = ShardUtils.extractShardId(indexReader);
+                    if (shardId != null) {
+                        IndexShard shard = indexService.shard(shardId.id());
+                        if (shard != null) {
+                            key.listeners.add(shard.fieldData());
+                        }
+                    }
+                    for (Listener listener : key.listeners) {
+                        listener.onLoad(fieldNames, fieldDataType, ifd);
+                    }
+
+                    return ifd;
+                }
+            });
+        }
+
         @Override
         public void clear() {
             cache.invalidateAll();
@@ -147,6 +176,11 @@ public void onClose(Object coreCacheKey) {
             cache.invalidate(new Key(coreCacheKey));
         }
 
+        @Override
+        public void onClose(IndexReader reader) {
+            cache.invalidate(reader.getCoreCacheKey());
+        }
+
         static class Key {
             final Object readerKey;
             final List<Listener> listeners = new ArrayList<>();
diff --git a/src/main/java/org/elasticsearch/index/fielddata/IndexFieldDataService.java b/src/main/java/org/elasticsearch/index/fielddata/IndexFieldDataService.java
index 5eb6eaa23ab67..735ff5322d695 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/IndexFieldDataService.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/IndexFieldDataService.java
@@ -31,15 +31,16 @@
 import org.elasticsearch.common.util.concurrent.ConcurrentCollections;
 import org.elasticsearch.index.AbstractIndexComponent;
 import org.elasticsearch.index.Index;
-import org.elasticsearch.index.fielddata.plain.ParentChildIndexFieldData;
+import org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsBuilder;
+import org.elasticsearch.index.fielddata.ordinals.InternalGlobalOrdinalsBuilder;
 import org.elasticsearch.index.fielddata.plain.*;
 import org.elasticsearch.index.mapper.FieldMapper;
 import org.elasticsearch.index.mapper.internal.ParentFieldMapper;
 import org.elasticsearch.index.service.IndexService;
 import org.elasticsearch.index.settings.IndexSettings;
-import org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCacheListener;
-import org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache;
 import org.elasticsearch.indices.fielddata.breaker.CircuitBreakerService;
+import org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache;
+import org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCacheListener;
 
 import java.util.Map;
 import java.util.concurrent.ConcurrentMap;
@@ -134,6 +135,11 @@ public IndexFieldDataService(Index index, CircuitBreakerService circuitBreakerSe
         this(index, ImmutableSettings.Builder.EMPTY_SETTINGS, new IndicesFieldDataCache(ImmutableSettings.Builder.EMPTY_SETTINGS, new IndicesFieldDataCacheListener(circuitBreakerService)), circuitBreakerService, new IndicesFieldDataCacheListener(circuitBreakerService));
     }
 
+    // public for testing
+    public IndexFieldDataService(Index index, CircuitBreakerService circuitBreakerService, IndicesFieldDataCache indicesFieldDataCache) {
+        this(index, ImmutableSettings.Builder.EMPTY_SETTINGS, indicesFieldDataCache, circuitBreakerService, new IndicesFieldDataCacheListener(circuitBreakerService));
+    }
+
     @Inject
     public IndexFieldDataService(Index index, @IndexSettings Settings indexSettings, IndicesFieldDataCache indicesFieldDataCache,
                                  CircuitBreakerService circuitBreakerService, IndicesFieldDataCacheListener indicesFieldDataCacheListener) {
@@ -241,7 +247,8 @@ public <IFD extends IndexFieldData<?>> IFD getForField(FieldMapper<?> mapper) {
                         fieldDataCaches.put(fieldNames.indexName(), cache);
                     }
 
-                    fieldData = builder.build(index, indexSettings, mapper, cache, circuitBreakerService, indexService.mapperService());
+                    GlobalOrdinalsBuilder globalOrdinalBuilder = new InternalGlobalOrdinalsBuilder(index(), indexSettings);
+                    fieldData = builder.build(index, indexSettings, mapper, cache, circuitBreakerService, indexService.mapperService(), globalOrdinalBuilder);
                     loadedFieldData.put(fieldNames.indexName(), fieldData);
                 }
             }
diff --git a/src/main/java/org/elasticsearch/index/fielddata/RamUsage.java b/src/main/java/org/elasticsearch/index/fielddata/RamUsage.java
new file mode 100644
index 0000000000000..035041a17a72b
--- /dev/null
+++ b/src/main/java/org/elasticsearch/index/fielddata/RamUsage.java
@@ -0,0 +1,30 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.elasticsearch.index.fielddata;
+
+/**
+ */
+public interface RamUsage {
+
+    /**
+     * Size (in bytes) of memory used by this particular instance.
+     */
+    long getMemorySizeInBytes();
+
+}
diff --git a/src/main/java/org/elasticsearch/index/fielddata/ShardFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/ShardFieldData.java
index 198d8762d84a7..35cbf32f4d31b 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/ShardFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/ShardFieldData.java
@@ -20,7 +20,6 @@
 package org.elasticsearch.index.fielddata;
 
 import com.carrotsearch.hppc.ObjectLongOpenHashMap;
-import org.elasticsearch.common.Nullable;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.metrics.CounterMetric;
 import org.elasticsearch.common.regex.Regex;
@@ -72,27 +71,24 @@ public FieldDataStats stats(String... fields) {
     }
 
     @Override
-    public void onLoad(FieldMapper.Names fieldNames, FieldDataType fieldDataType, AtomicFieldData fieldData) {
-        long sizeInBytes = fieldData.getMemorySizeInBytes();
-
-        totalMetric.inc(sizeInBytes);
-
+    public void onLoad(FieldMapper.Names fieldNames, FieldDataType fieldDataType, RamUsage ramUsage) {
+        totalMetric.inc(ramUsage.getMemorySizeInBytes());
         String keyFieldName = fieldNames.indexName();
         CounterMetric total = perFieldTotals.get(keyFieldName);
         if (total != null) {
-            total.inc(sizeInBytes);
+            total.inc(ramUsage.getMemorySizeInBytes());
         } else {
             total = new CounterMetric();
-            total.inc(sizeInBytes);
+            total.inc(ramUsage.getMemorySizeInBytes());
             CounterMetric prev = perFieldTotals.putIfAbsent(keyFieldName, total);
             if (prev != null) {
-                prev.inc(sizeInBytes);
+                prev.inc(ramUsage.getMemorySizeInBytes());
             }
         }
     }
 
     @Override
-    public void onUnload(FieldMapper.Names fieldNames, FieldDataType fieldDataType, boolean wasEvicted, long sizeInBytes, @Nullable AtomicFieldData fieldData) {
+    public void onUnload(FieldMapper.Names fieldNames, FieldDataType fieldDataType, boolean wasEvicted, long sizeInBytes) {
         if (wasEvicted) {
             evictionsMetric.inc();
         }
diff --git a/src/main/java/org/elasticsearch/index/fielddata/ordinals/GlobalOrdinalsBuilder.java b/src/main/java/org/elasticsearch/index/fielddata/ordinals/GlobalOrdinalsBuilder.java
new file mode 100644
index 0000000000000..9af55b16c2baf
--- /dev/null
+++ b/src/main/java/org/elasticsearch/index/fielddata/ordinals/GlobalOrdinalsBuilder.java
@@ -0,0 +1,35 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.index.fielddata.ordinals;
+
+import org.apache.lucene.index.IndexReader;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.index.fielddata.IndexFieldData;
+import org.elasticsearch.indices.fielddata.breaker.CircuitBreakerService;
+
+import java.io.IOException;
+
+/**
+*/
+public interface GlobalOrdinalsBuilder {
+
+    IndexFieldData.WithOrdinals build(IndexReader indexReader, IndexFieldData.WithOrdinals indexFieldData, Settings settings, CircuitBreakerService breakerService) throws IOException;
+
+}
diff --git a/src/main/java/org/elasticsearch/index/fielddata/ordinals/GlobalOrdinalsIndexFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/ordinals/GlobalOrdinalsIndexFieldData.java
new file mode 100644
index 0000000000000..bcd670f4dac5b
--- /dev/null
+++ b/src/main/java/org/elasticsearch/index/fielddata/ordinals/GlobalOrdinalsIndexFieldData.java
@@ -0,0 +1,201 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.elasticsearch.index.fielddata.ordinals;
+
+import org.apache.lucene.index.AtomicReaderContext;
+import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.TermsEnum;
+import org.apache.lucene.util.BytesRef;
+import org.apache.lucene.util.LongValues;
+import org.elasticsearch.common.Nullable;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.index.AbstractIndexComponent;
+import org.elasticsearch.index.Index;
+import org.elasticsearch.index.fielddata.*;
+import org.elasticsearch.index.fielddata.fieldcomparator.SortMode;
+import org.elasticsearch.index.fielddata.plain.AtomicFieldDataWithOrdinalsTermsEnum;
+import org.elasticsearch.index.mapper.FieldMapper;
+
+import static org.elasticsearch.index.fielddata.ordinals.InternalGlobalOrdinalsBuilder.OrdinalMappingSource;
+
+/**
+ * {@link IndexFieldData} impl based on global ordinals.
+ */
+public final class GlobalOrdinalsIndexFieldData extends AbstractIndexComponent implements IndexFieldData.WithOrdinals, RamUsage {
+
+    private final FieldMapper.Names fieldNames;
+    private final FieldDataType fieldDataType;
+    private final Atomic[] atomicReaders;
+    private final long memorySizeInBytes;
+
+    public GlobalOrdinalsIndexFieldData(Index index, Settings settings, FieldMapper.Names fieldNames, FieldDataType fieldDataType, AtomicFieldData.WithOrdinals[] segmentAfd, LongValues globalOrdToFirstSegment, LongValues globalOrdToFirstSegmentDelta, OrdinalMappingSource[] segmentOrdToGlobalOrds, long memorySizeInBytes) {
+        super(index, settings);
+        this.fieldNames = fieldNames;
+        this.fieldDataType = fieldDataType;
+        this.atomicReaders = new Atomic[segmentAfd.length];
+        for (int i = 0; i < segmentAfd.length; i++) {
+            atomicReaders[i] = new Atomic(segmentAfd[i], globalOrdToFirstSegment, globalOrdToFirstSegmentDelta, segmentOrdToGlobalOrds[i]);
+        }
+        this.memorySizeInBytes = memorySizeInBytes;
+    }
+
+    @Override
+    public AtomicFieldData.WithOrdinals load(AtomicReaderContext context) {
+        return atomicReaders[context.ord];
+    }
+
+    @Override
+    public AtomicFieldData.WithOrdinals loadDirect(AtomicReaderContext context) throws Exception {
+        return load(context);
+    }
+
+    @Override
+    public WithOrdinals loadGlobal(IndexReader indexReader) {
+        return this;
+    }
+
+    @Override
+    public WithOrdinals localGlobalDirect(IndexReader indexReader) throws Exception {
+        return this;
+    }
+
+    @Override
+    public FieldMapper.Names getFieldNames() {
+        return fieldNames;
+    }
+
+    @Override
+    public FieldDataType getFieldDataType() {
+        return fieldDataType;
+    }
+
+    @Override
+    public boolean valuesOrdered() {
+        return false;
+    }
+
+    @Override
+    public XFieldComparatorSource comparatorSource(@Nullable Object missingValue, SortMode sortMode) {
+        throw new UnsupportedOperationException("no global ordinals sorting yet");
+    }
+
+    @Override
+    public void clear() {
+        // no need to clear, because this is cached and cleared in AbstractBytesIndexFieldData
+    }
+
+    @Override
+    public void clear(IndexReader reader) {
+        // no need to clear, because this is cached and cleared in AbstractBytesIndexFieldData
+    }
+
+    @Override
+    public long getMemorySizeInBytes() {
+        return memorySizeInBytes;
+    }
+
+    private final class Atomic implements AtomicFieldData.WithOrdinals {
+
+        private final AtomicFieldData.WithOrdinals afd;
+        private final OrdinalMappingSource segmentOrdToGlobalOrdLookup;
+        private final LongValues globalOrdToFirstSegment;
+        private final LongValues globalOrdToFirstSegmentDelta;
+
+        private Atomic(WithOrdinals afd, LongValues globalOrdToFirstSegment, LongValues globalOrdToFirstSegmentDelta, OrdinalMappingSource segmentOrdToGlobalOrdLookup) {
+            this.afd = afd;
+            this.segmentOrdToGlobalOrdLookup = segmentOrdToGlobalOrdLookup;
+            this.globalOrdToFirstSegment = globalOrdToFirstSegment;
+            this.globalOrdToFirstSegmentDelta = globalOrdToFirstSegmentDelta;
+        }
+
+        @Override
+        public BytesValues.WithOrdinals getBytesValues(boolean needsHashes) {
+            BytesValues.WithOrdinals values = afd.getBytesValues(false);
+            Ordinals.Docs segmentOrdinals = values.ordinals();
+            Ordinals.Docs globalOrdinals = segmentOrdToGlobalOrdLookup.globalOrdinals(segmentOrdinals);
+
+            final BytesValues.WithOrdinals[] bytesValues = new BytesValues.WithOrdinals[atomicReaders.length];
+            for (int i = 0; i < bytesValues.length; i++) {
+                bytesValues[i] = atomicReaders[i].afd.getBytesValues(false);
+            }
+            return new BytesValues.WithOrdinals(globalOrdinals) {
+
+                int readerIndex;
+
+                @Override
+                public BytesRef getValueByOrd(long globalOrd) {
+                    final long segmentOrd = globalOrd - globalOrdToFirstSegmentDelta.get(globalOrd);
+                    readerIndex = (int) globalOrdToFirstSegment.get(globalOrd);
+                    return bytesValues[readerIndex].getValueByOrd(segmentOrd);
+                }
+
+                @Override
+                public BytesRef copyShared() {
+                    return bytesValues[readerIndex].copyShared();
+                }
+
+                @Override
+                public int currentValueHash() {
+                    return bytesValues[readerIndex].currentValueHash();
+                }
+            };
+        }
+
+        @Override
+        public boolean isMultiValued() {
+            return afd.isMultiValued();
+        }
+
+        @Override
+        public boolean isValuesOrdered() {
+            return afd.isValuesOrdered();
+        }
+
+        @Override
+        public int getNumDocs() {
+            return afd.getNumDocs();
+        }
+
+        @Override
+        public long getNumberUniqueValues() {
+            return afd.getNumberUniqueValues();
+        }
+
+        @Override
+        public long getMemorySizeInBytes() {
+            return afd.getMemorySizeInBytes();
+        }
+
+        @Override
+        public ScriptDocValues getScriptValues() {
+            throw new UnsupportedOperationException("Script values not supported on global ordinals");
+        }
+
+        @Override
+        public TermsEnum getTermsEnum() {
+            return new AtomicFieldDataWithOrdinalsTermsEnum(this);
+        }
+
+        @Override
+        public void close() {
+        }
+
+    }
+
+}
diff --git a/src/main/java/org/elasticsearch/index/fielddata/ordinals/InternalGlobalOrdinalsBuilder.java b/src/main/java/org/elasticsearch/index/fielddata/ordinals/InternalGlobalOrdinalsBuilder.java
new file mode 100644
index 0000000000000..acbc3120e021d
--- /dev/null
+++ b/src/main/java/org/elasticsearch/index/fielddata/ordinals/InternalGlobalOrdinalsBuilder.java
@@ -0,0 +1,466 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.index.fielddata.ordinals;
+
+import org.apache.lucene.index.AtomicReaderContext;
+import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.TermsEnum;
+import org.apache.lucene.util.BytesRef;
+import org.apache.lucene.util.BytesRefIterator;
+import org.apache.lucene.util.LongsRef;
+import org.apache.lucene.util.PriorityQueue;
+import org.apache.lucene.util.packed.AppendingPackedLongBuffer;
+import org.apache.lucene.util.packed.MonotonicAppendingLongBuffer;
+import org.apache.lucene.util.packed.PackedInts;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.index.AbstractIndexComponent;
+import org.elasticsearch.index.Index;
+import org.elasticsearch.index.fielddata.AtomicFieldData;
+import org.elasticsearch.index.fielddata.FieldDataType;
+import org.elasticsearch.index.fielddata.IndexFieldData;
+import org.elasticsearch.index.settings.IndexSettings;
+import org.elasticsearch.indices.fielddata.breaker.CircuitBreakerService;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+
+/**
+ */
+public class InternalGlobalOrdinalsBuilder extends AbstractIndexComponent implements GlobalOrdinalsBuilder {
+
+    public final static int ORDINAL_MAPPING_THRESHOLD_DEFAULT = 2048;
+    public final static String ORDINAL_MAPPING_THRESHOLD_KEY = "global_ordinals_compress_threshold";
+    public final static String ORDINAL_MAPPING_THRESHOLD_INDEX_SETTING_KEY = "index." + ORDINAL_MAPPING_THRESHOLD_KEY;
+
+    public InternalGlobalOrdinalsBuilder(Index index, @IndexSettings Settings indexSettings) {
+        super(index, indexSettings);
+    }
+
+    @Override
+    public IndexFieldData.WithOrdinals build(final IndexReader indexReader, IndexFieldData.WithOrdinals indexFieldData, Settings settings, CircuitBreakerService breakerService) throws IOException {
+        assert indexReader.leaves().size() > 1;
+        long startTime = System.currentTimeMillis();
+
+        // It makes sense to make the overhead ratio configurable for the mapping from segment ords to global ords
+        // However, other mappings are never the bottleneck and only used to get the original value from an ord, so
+        // it makes sense to force COMPACT for them
+        final float acceptableOverheadRatio = settings.getAsFloat("acceptable_overhead_ratio", PackedInts.FAST);
+        final AppendingPackedLongBuffer globalOrdToFirstSegment = new AppendingPackedLongBuffer(PackedInts.COMPACT);
+        globalOrdToFirstSegment.add(0);
+        final MonotonicAppendingLongBuffer globalOrdToFirstSegmentDelta = new MonotonicAppendingLongBuffer(PackedInts.COMPACT);
+        globalOrdToFirstSegmentDelta.add(0);
+
+        FieldDataType fieldDataType = indexFieldData.getFieldDataType();
+        int defaultThreshold = settings.getAsInt(ORDINAL_MAPPING_THRESHOLD_INDEX_SETTING_KEY, ORDINAL_MAPPING_THRESHOLD_DEFAULT);
+        int threshold = fieldDataType.getSettings().getAsInt(ORDINAL_MAPPING_THRESHOLD_KEY, defaultThreshold);
+        OrdinalMappingSourceBuilder ordinalMappingBuilder = new OrdinalMappingSourceBuilder(
+                indexReader.leaves().size(), acceptableOverheadRatio, threshold
+        );
+
+        long currentGlobalOrdinal = 0;
+        final AtomicFieldData.WithOrdinals[] withOrdinals = new AtomicFieldData.WithOrdinals[indexReader.leaves().size()];
+        TermIterator termIterator = new TermIterator(indexFieldData, indexReader.leaves(), withOrdinals);
+        for (BytesRef term = termIterator.next(); term != null; term = termIterator.next()) {
+            currentGlobalOrdinal++;
+            globalOrdToFirstSegment.add(termIterator.firstReaderIndex());
+            long globalOrdinalDelta = currentGlobalOrdinal - termIterator.firstLocalOrdinal();
+            globalOrdToFirstSegmentDelta.add(globalOrdinalDelta);
+            for (TermIterator.LeafSource leafSource : termIterator.competitiveLeafs()) {
+                ordinalMappingBuilder.onOrdinal(leafSource.context.ord, leafSource.tenum.ord(), currentGlobalOrdinal);
+            }
+        }
+
+        // ram used for the globalOrd to segmentOrd and segmentOrd to firstReaderIndex lookups
+        long memorySizeInBytesCounter = 0;
+        globalOrdToFirstSegment.freeze();
+        memorySizeInBytesCounter += globalOrdToFirstSegment.ramBytesUsed();
+        globalOrdToFirstSegmentDelta.freeze();
+        memorySizeInBytesCounter += globalOrdToFirstSegmentDelta.ramBytesUsed();
+
+        final long maxOrd = currentGlobalOrdinal + 1;
+        OrdinalMappingSource[] segmentOrdToGlobalOrdLookups = ordinalMappingBuilder.build(maxOrd);
+        // add ram used for the main segmentOrd to globalOrd lookups
+        memorySizeInBytesCounter += ordinalMappingBuilder.getMemorySizeInBytes();
+
+        final long memorySizeInBytes = memorySizeInBytesCounter;
+        breakerService.getBreaker().addWithoutBreaking(memorySizeInBytes);
+
+        if (logger.isDebugEnabled()) {
+            String implName = segmentOrdToGlobalOrdLookups[0].getClass().getSimpleName();
+            logger.debug(
+                    "Global-ordinals[{}][{}][{}] took {} ms",
+                    implName,
+                    indexFieldData.getFieldNames().fullName(),
+                    maxOrd,
+                    (System.currentTimeMillis() - startTime)
+            );
+        }
+        return new GlobalOrdinalsIndexFieldData(indexFieldData.index(), settings, indexFieldData.getFieldNames(),
+                fieldDataType, withOrdinals, globalOrdToFirstSegment, globalOrdToFirstSegmentDelta,
+                segmentOrdToGlobalOrdLookups, memorySizeInBytes
+        );
+    }
+
+    public interface OrdinalMappingSource {
+
+        Ordinals.Docs globalOrdinals(Ordinals.Docs segmentOrdinals);
+
+    }
+
+    private static abstract class GlobalOrdinalMapping implements Ordinals.Docs {
+
+        protected final Ordinals.Docs segmentOrdinals;
+        private final long memorySizeInBytes;
+        protected final long maxOrd;
+
+        protected long currentGlobalOrd;
+
+        private GlobalOrdinalMapping(Ordinals.Docs segmentOrdinals, long memorySizeInBytes, long maxOrd) {
+            this.segmentOrdinals = segmentOrdinals;
+            this.memorySizeInBytes = memorySizeInBytes;
+            this.maxOrd = maxOrd;
+        }
+
+        @Override
+        public final Ordinals ordinals() {
+            return new Ordinals() {
+                @Override
+                public long getMemorySizeInBytes() {
+                    return memorySizeInBytes;
+                }
+
+                @Override
+                public boolean isMultiValued() {
+                    return GlobalOrdinalMapping.this.isMultiValued();
+                }
+
+                @Override
+                public int getNumDocs() {
+                    return GlobalOrdinalMapping.this.getNumDocs();
+                }
+
+                @Override
+                public long getNumOrds() {
+                    return GlobalOrdinalMapping.this.getNumOrds();
+                }
+
+                @Override
+                public long getMaxOrd() {
+                    return GlobalOrdinalMapping.this.getMaxOrd();
+                }
+
+                @Override
+                public Docs ordinals() {
+                    return GlobalOrdinalMapping.this;
+                }
+            };
+        }
+
+        @Override
+        public final int getNumDocs() {
+            return segmentOrdinals.getNumDocs();
+        }
+
+        @Override
+        public final long getNumOrds() {
+            return maxOrd - Ordinals.MIN_ORDINAL;
+        }
+
+        @Override
+        public final long getMaxOrd() {
+            return maxOrd;
+        }
+
+        @Override
+        public final boolean isMultiValued() {
+            return segmentOrdinals.isMultiValued();
+        }
+
+        @Override
+        public final int setDocument(int docId) {
+            return segmentOrdinals.setDocument(docId);
+        }
+
+        @Override
+        public final long currentOrd() {
+            return currentGlobalOrd;
+        }
+
+        @Override
+        public final long getOrd(int docId) {
+            long segmentOrd = segmentOrdinals.getOrd(docId);
+            return currentGlobalOrd = getGlobalOrd(segmentOrd);
+        }
+
+        @Override
+        public final LongsRef getOrds(int docId) {
+            LongsRef refs = segmentOrdinals.getOrds(docId);
+            for (int i = refs.offset; i < refs.length; i++) {
+                refs.longs[i] = getGlobalOrd(refs.longs[i]);
+            }
+            return refs;
+        }
+
+        @Override
+        public final long nextOrd() {
+            long segmentOrd = segmentOrdinals.nextOrd();
+            return currentGlobalOrd = getGlobalOrd(segmentOrd);
+        }
+
+        protected abstract long getGlobalOrd(long segmentOrd);
+
+    }
+
+    private final static class OrdinalMappingSourceBuilder {
+
+        final MonotonicAppendingLongBuffer[] segmentOrdToGlobalOrdDeltas;
+        final float acceptableOverheadRatio;
+        final int numSegments;
+        final int threshold;
+
+        long memorySizeInBytesCounter;
+
+        private OrdinalMappingSourceBuilder(int numSegments, float acceptableOverheadRatio, int threshold) {
+            segmentOrdToGlobalOrdDeltas = new MonotonicAppendingLongBuffer[numSegments];
+            for (int i = 0; i < segmentOrdToGlobalOrdDeltas.length; i++) {
+                segmentOrdToGlobalOrdDeltas[i] = new MonotonicAppendingLongBuffer(acceptableOverheadRatio);
+                segmentOrdToGlobalOrdDeltas[i].add(0);
+            }
+            this.numSegments = numSegments;
+            this.acceptableOverheadRatio = acceptableOverheadRatio;
+            this.threshold = threshold;
+        }
+
+        public void onOrdinal(int readerIndex, long segmentOrdinal, long globalOrdinal) {
+            long delta = globalOrdinal - segmentOrdinal;
+            segmentOrdToGlobalOrdDeltas[readerIndex].add(delta);
+        }
+
+        public OrdinalMappingSource[] build(long maxOrd) {
+            // If we find out that there are less then predefined number of ordinals, it is better to put the the
+            // segment ordinal to global ordinal mapping in a packed ints, since the amount values are small and
+            // will most likely fit in the CPU caches and MonotonicAppendingLongBuffer's compression will just be
+            // unnecessary.
+
+            if (maxOrd <= threshold) {
+                // Rebuilding from MonotonicAppendingLongBuffer to PackedInts.Mutable is fast
+                PackedInts.Mutable[] newSegmentOrdToGlobalOrdDeltas = new PackedInts.Mutable[numSegments];
+                for (int i = 0; i < segmentOrdToGlobalOrdDeltas.length; i++) {
+                    newSegmentOrdToGlobalOrdDeltas[i] = PackedInts.getMutable((int) segmentOrdToGlobalOrdDeltas[i].size(), PackedInts.bitsRequired(maxOrd), acceptableOverheadRatio);
+                }
+
+                for (int readerIndex = 0; readerIndex < segmentOrdToGlobalOrdDeltas.length; readerIndex++) {
+                    MonotonicAppendingLongBuffer segmentOrdToGlobalOrdDelta = segmentOrdToGlobalOrdDeltas[readerIndex];
+
+                    for (long ordIndex = 0; ordIndex < segmentOrdToGlobalOrdDelta.size(); ordIndex++) {
+                        long ordDelta = segmentOrdToGlobalOrdDelta.get(ordIndex);
+                        newSegmentOrdToGlobalOrdDeltas[readerIndex].set((int) ordIndex, ordDelta);
+                    }
+                }
+
+                PackedIntOrdinalMappingSource[] sources = new PackedIntOrdinalMappingSource[numSegments];
+                for (int i = 0; i < newSegmentOrdToGlobalOrdDeltas.length; i++) {
+                    PackedInts.Reader segmentOrdToGlobalOrdDelta = newSegmentOrdToGlobalOrdDeltas[i];
+                    long ramUsed = segmentOrdToGlobalOrdDelta.ramBytesUsed();
+                    sources[i] = new PackedIntOrdinalMappingSource(segmentOrdToGlobalOrdDelta, ramUsed, maxOrd);
+                    memorySizeInBytesCounter += ramUsed;
+                }
+                return sources;
+            } else {
+                OrdinalMappingSource[] sources = new OrdinalMappingSource[segmentOrdToGlobalOrdDeltas.length];
+                for (int i = 0; i < segmentOrdToGlobalOrdDeltas.length; i++) {
+                    MonotonicAppendingLongBuffer segmentOrdToGlobalOrdLookup = segmentOrdToGlobalOrdDeltas[i];
+                    segmentOrdToGlobalOrdLookup.freeze();
+                    long ramUsed = segmentOrdToGlobalOrdLookup.ramBytesUsed();
+                    sources[i] = new CompressedOrdinalMappingSource(segmentOrdToGlobalOrdLookup, ramUsed, maxOrd);
+                    memorySizeInBytesCounter += ramUsed;
+                }
+                return sources;
+            }
+        }
+
+        public long getMemorySizeInBytes() {
+            return memorySizeInBytesCounter;
+        }
+    }
+
+    private final static class CompressedOrdinalMappingSource implements OrdinalMappingSource {
+
+        private final MonotonicAppendingLongBuffer globalOrdinalMapping;
+        private final long memorySizeInBytes;
+        private final long maxOrd;
+
+        private CompressedOrdinalMappingSource(MonotonicAppendingLongBuffer globalOrdinalMapping, long memorySizeInBytes, long maxOrd) {
+            this.globalOrdinalMapping = globalOrdinalMapping;
+            this.memorySizeInBytes = memorySizeInBytes;
+            this.maxOrd = maxOrd;
+        }
+
+        @Override
+        public Ordinals.Docs globalOrdinals(Ordinals.Docs segmentOrdinals) {
+            return new GlobalOrdinalsDocs(segmentOrdinals, globalOrdinalMapping, memorySizeInBytes, maxOrd);
+        }
+
+        private final static class GlobalOrdinalsDocs extends GlobalOrdinalMapping {
+
+            private final MonotonicAppendingLongBuffer segmentOrdToGlobalOrdLookup;
+
+            private GlobalOrdinalsDocs(Ordinals.Docs segmentOrdinals, MonotonicAppendingLongBuffer segmentOrdToGlobalOrdLookup, long memorySizeInBytes, long maxOrd) {
+                super(segmentOrdinals, memorySizeInBytes, maxOrd);
+                this.segmentOrdToGlobalOrdLookup = segmentOrdToGlobalOrdLookup;
+            }
+
+            @Override
+            protected long getGlobalOrd(long segmentOrd) {
+                return segmentOrd + segmentOrdToGlobalOrdLookup.get(segmentOrd);
+            }
+        }
+
+    }
+
+    private static final class PackedIntOrdinalMappingSource implements OrdinalMappingSource {
+
+        private final PackedInts.Reader segmentOrdToGlobalOrdLookup;
+        private final long memorySizeInBytes;
+        private final long maxOrd;
+
+        private PackedIntOrdinalMappingSource(PackedInts.Reader segmentOrdToGlobalOrdLookup, long memorySizeInBytes, long maxOrd) {
+            this.segmentOrdToGlobalOrdLookup = segmentOrdToGlobalOrdLookup;
+            this.memorySizeInBytes = memorySizeInBytes;
+            this.maxOrd = maxOrd;
+        }
+
+        @Override
+        public Ordinals.Docs globalOrdinals(Ordinals.Docs segmentOrdinals) {
+            return new GlobalOrdinalsDocs(segmentOrdinals, memorySizeInBytes, maxOrd, segmentOrdToGlobalOrdLookup);
+        }
+
+        private final static class GlobalOrdinalsDocs extends GlobalOrdinalMapping {
+
+            private final PackedInts.Reader segmentOrdToGlobalOrdLookup;
+
+            private GlobalOrdinalsDocs(Ordinals.Docs segmentOrdinals, long memorySizeInBytes, long maxOrd, PackedInts.Reader segmentOrdToGlobalOrdLookup) {
+                super(segmentOrdinals, memorySizeInBytes, maxOrd);
+                this.segmentOrdToGlobalOrdLookup = segmentOrdToGlobalOrdLookup;
+            }
+
+            @Override
+            protected long getGlobalOrd(long segmentOrd) {
+                return segmentOrd + segmentOrdToGlobalOrdLookup.get((int) segmentOrd);
+            }
+        }
+
+    }
+
+    private final static class TermIterator implements BytesRefIterator {
+
+        private final LeafSourceQueue sources;
+        private final List<LeafSource> competitiveLeafs = new ArrayList<>();
+
+        private TermIterator(IndexFieldData.WithOrdinals indexFieldData, List<AtomicReaderContext> leaves, AtomicFieldData.WithOrdinals[] withOrdinals) throws IOException {
+            this.sources = new LeafSourceQueue(leaves.size());
+            for (int i = 0; i < leaves.size(); i++) {
+                AtomicReaderContext atomicReaderContext = leaves.get(i);
+                AtomicFieldData.WithOrdinals afd = indexFieldData.load(atomicReaderContext);
+                withOrdinals[i] = afd;
+                LeafSource leafSource = new LeafSource(afd, atomicReaderContext);
+                if (leafSource.current != null) {
+                    sources.add(leafSource);
+                }
+            }
+        }
+
+        public BytesRef next() throws IOException {
+            for (LeafSource top : competitiveLeafs) {
+                if (top.next() != null) {
+                    sources.add(top);
+                }
+            }
+            competitiveLeafs.clear();
+            if (sources.size() == 0) {
+                return null;
+            }
+
+            do {
+                LeafSource competitiveLeaf = sources.pop();
+                competitiveLeafs.add(competitiveLeaf);
+            } while (sources.size() > 0 && competitiveLeafs.get(0).current.equals(sources.top().current));
+            return competitiveLeafs.get(0).current;
+        }
+
+        @Override
+        public Comparator<BytesRef> getComparator() {
+            return BytesRef.getUTF8SortedAsUnicodeComparator();
+        }
+
+        List<LeafSource> competitiveLeafs() throws IOException {
+            return competitiveLeafs;
+        }
+
+        int firstReaderIndex() {
+            return competitiveLeafs.get(0).context.ord;
+        }
+
+        long firstLocalOrdinal() throws IOException {
+            return competitiveLeafs.get(0).tenum.ord();
+        }
+
+        private static class LeafSource {
+
+            final TermsEnum tenum;
+            final AtomicReaderContext context;
+
+            BytesRef current;
+
+            private LeafSource(AtomicFieldData.WithOrdinals afd, AtomicReaderContext context) throws IOException {
+                this.tenum = afd.getTermsEnum();
+                this.context = context;
+                this.current = tenum.next();
+            }
+
+            BytesRef next() throws IOException {
+                return current = tenum.next();
+            }
+
+        }
+
+        private final static class LeafSourceQueue extends PriorityQueue<LeafSource> {
+
+            private final Comparator<BytesRef> termComp = BytesRef.getUTF8SortedAsUnicodeComparator();
+
+            LeafSourceQueue(int size) {
+                super(size);
+            }
+
+            @Override
+            protected boolean lessThan(LeafSource termsA, LeafSource termsB) {
+                final int cmp = termComp.compare(termsA.current, termsB.current);
+                if (cmp != 0) {
+                    return cmp < 0;
+                } else {
+                    return termsA.context.ord < termsB.context.ord;
+                }
+            }
+        }
+
+    }
+}
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/AbstractBytesIndexFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/AbstractBytesIndexFieldData.java
index 136986174bf4b..a2b5587d201da 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/AbstractBytesIndexFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/AbstractBytesIndexFieldData.java
@@ -18,20 +18,20 @@
  */
 package org.elasticsearch.index.fielddata.plain;
 
-import org.apache.lucene.index.AtomicReader;
-import org.apache.lucene.index.FilteredTermsEnum;
-import org.apache.lucene.index.Terms;
-import org.apache.lucene.index.TermsEnum;
+import org.apache.lucene.index.*;
 import org.apache.lucene.util.BytesRef;
 import org.apache.lucene.util.CharsRef;
 import org.apache.lucene.util.UnicodeUtil;
+import org.elasticsearch.ElasticsearchException;
 import org.elasticsearch.common.Nullable;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.index.Index;
 import org.elasticsearch.index.fielddata.*;
 import org.elasticsearch.index.fielddata.fieldcomparator.BytesRefFieldComparatorSource;
 import org.elasticsearch.index.fielddata.fieldcomparator.SortMode;
+import org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsBuilder;
 import org.elasticsearch.index.mapper.FieldMapper.Names;
+import org.elasticsearch.indices.fielddata.breaker.CircuitBreakerService;
 
 import java.io.IOException;
 import java.util.Map;
@@ -42,14 +42,17 @@ public abstract class AbstractBytesIndexFieldData<FD extends AtomicFieldData.Wit
 
     protected Settings frequency;
     protected Settings regex;
+    protected final GlobalOrdinalsBuilder globalOrdinalsBuilder;
+    protected final CircuitBreakerService breakerService;
 
     protected AbstractBytesIndexFieldData(Index index, Settings indexSettings, Names fieldNames, FieldDataType fieldDataType,
-            IndexFieldDataCache cache) {
+                                          IndexFieldDataCache cache, GlobalOrdinalsBuilder globalOrdinalsBuilder, CircuitBreakerService breakerService) {
         super(index, indexSettings, fieldNames, fieldDataType, cache);
         final Map<String, Settings> groups = fieldDataType.getSettings().getGroups("filter");
         frequency = groups.get("frequency");
         regex = groups.get("regex");
-       
+        this.globalOrdinalsBuilder = globalOrdinalsBuilder;
+        this.breakerService = breakerService;
     }
     
     @Override
@@ -61,7 +64,29 @@ public final boolean valuesOrdered() {
     public XFieldComparatorSource comparatorSource(@Nullable Object missingValue, SortMode sortMode) {
         return new BytesRefFieldComparatorSource(this, missingValue, sortMode);
     }
-    
+
+    @Override
+    public WithOrdinals loadGlobal(IndexReader indexReader) {
+        if (indexReader.leaves().size() <= 1) {
+            // ordinals are already global
+            return this;
+        }
+        try {
+            return cache.load(indexReader, this);
+        } catch (Throwable e) {
+            if (e instanceof ElasticsearchException) {
+                throw (ElasticsearchException) e;
+            } else {
+                throw new ElasticsearchException(e.getMessage(), e);
+            }
+        }
+    }
+
+    @Override
+    public WithOrdinals localGlobalDirect(IndexReader indexReader) throws Exception {
+        return globalOrdinalsBuilder.build(indexReader, this, indexSettings, breakerService);
+    }
+
     protected TermsEnum filter(Terms terms, AtomicReader reader) throws IOException {
         TermsEnum iterator = terms.iterator(null);
         if (iterator == null) {
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/AtomicFieldDataWithOrdinalsTermsEnum.java b/src/main/java/org/elasticsearch/index/fielddata/plain/AtomicFieldDataWithOrdinalsTermsEnum.java
new file mode 100644
index 0000000000000..2d60bbf091e1a
--- /dev/null
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/AtomicFieldDataWithOrdinalsTermsEnum.java
@@ -0,0 +1,149 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.index.fielddata.plain;
+
+import org.apache.lucene.index.DocsAndPositionsEnum;
+import org.apache.lucene.index.DocsEnum;
+import org.apache.lucene.index.TermsEnum;
+import org.apache.lucene.util.Bits;
+import org.apache.lucene.util.BytesRef;
+import org.elasticsearch.index.fielddata.AtomicFieldData;
+import org.elasticsearch.index.fielddata.BytesValues;
+import org.elasticsearch.index.fielddata.ordinals.Ordinals;
+
+import java.io.IOException;
+import java.util.Comparator;
+
+/**
+ * A general {@link org.apache.lucene.index.TermsEnum} to iterate over terms from a {@link AtomicFieldData.WithOrdinals}
+ * instance.
+ */
+public class AtomicFieldDataWithOrdinalsTermsEnum extends TermsEnum {
+
+    private final BytesValues.WithOrdinals bytesValues;
+    private final Ordinals.Docs ordinals;
+    private final long maxOrd;
+
+    private long currentOrd = Ordinals.MISSING_ORDINAL;
+    private BytesRef currentTerm;
+
+    public AtomicFieldDataWithOrdinalsTermsEnum(AtomicFieldData.WithOrdinals afd) {
+        this.bytesValues = afd.getBytesValues(false);
+        this.ordinals = bytesValues.ordinals();
+        this.maxOrd = ordinals.getMaxOrd();
+    }
+
+    @Override
+    public SeekStatus seekCeil(BytesRef text) throws IOException {
+        long ord = binarySearch(bytesValues, text);
+        if (ord >= 0) {
+            currentOrd = ord;
+            currentTerm = bytesValues.getValueByOrd(currentOrd);
+            return SeekStatus.FOUND;
+        } else {
+            currentOrd = -ord - 1;
+            if (ord >= maxOrd) {
+                return SeekStatus.END;
+            } else {
+                currentTerm = bytesValues.getValueByOrd(currentOrd);
+                return SeekStatus.NOT_FOUND;
+            }
+        }
+    }
+
+    @Override
+    public void seekExact(long ord) throws IOException {
+        assert ord >= 0 && ord < ordinals.getMaxOrd();
+        currentOrd = ord;
+        if (currentOrd == Ordinals.MISSING_ORDINAL) {
+            currentTerm = null;
+        } else {
+            currentTerm = bytesValues.getValueByOrd(currentOrd);
+        }
+    }
+
+    @Override
+    public BytesRef term() throws IOException {
+        return currentTerm;
+    }
+
+    @Override
+    public long ord() throws IOException {
+        return currentOrd;
+    }
+
+    @Override
+    public int docFreq() throws IOException {
+        throw new UnsupportedOperationException("docFreq not supported");
+    }
+
+    @Override
+    public long totalTermFreq() throws IOException {
+        return -1;
+    }
+
+    @Override
+    public DocsEnum docs(Bits liveDocs, DocsEnum reuse, int flags) throws IOException {
+        throw new UnsupportedOperationException("docs not supported");
+    }
+
+    @Override
+    public DocsAndPositionsEnum docsAndPositions(Bits liveDocs, DocsAndPositionsEnum reuse, int flags) throws IOException {
+        throw new UnsupportedOperationException("docsAndPositions not supported");
+    }
+
+    @Override
+    public BytesRef next() throws IOException {
+        if (++currentOrd < maxOrd) {
+            return currentTerm = bytesValues.getValueByOrd(currentOrd);
+        } else {
+            return null;
+        }
+    }
+
+    @Override
+    public Comparator<BytesRef> getComparator() {
+        return BytesRef.getUTF8SortedAsUnicodeComparator();
+    }
+
+    final private static long binarySearch(BytesValues.WithOrdinals a, BytesRef key) {
+        long low = 1;
+        long high = a.ordinals().getMaxOrd();
+        while (low <= high) {
+            long mid = (low + high) >>> 1;
+            BytesRef midVal = a.getValueByOrd(mid);
+            int cmp;
+            if (midVal != null) {
+                cmp = midVal.compareTo(key);
+            } else {
+                cmp = -1;
+            }
+
+            if (cmp < 0)
+                low = mid + 1;
+            else if (cmp > 0)
+                high = mid - 1;
+            else
+                return mid;
+        }
+        return -(low + 1);
+    }
+
+}
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/BinaryDVIndexFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/BinaryDVIndexFieldData.java
index 4029b56bd28be..63f93b8292afd 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/BinaryDVIndexFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/BinaryDVIndexFieldData.java
@@ -21,6 +21,7 @@
 
 import org.apache.lucene.index.AtomicReaderContext;
 import org.elasticsearch.index.Index;
+import org.elasticsearch.index.fielddata.FieldDataType;
 import org.elasticsearch.index.fielddata.IndexFieldData;
 import org.elasticsearch.index.fielddata.fieldcomparator.BytesRefFieldComparatorSource;
 import org.elasticsearch.index.fielddata.fieldcomparator.SortMode;
@@ -28,8 +29,8 @@
 
 public class BinaryDVIndexFieldData extends DocValuesIndexFieldData implements IndexFieldData<BinaryDVAtomicFieldData> {
 
-    public BinaryDVIndexFieldData(Index index, Names fieldNames) {
-        super(index, fieldNames);
+    public BinaryDVIndexFieldData(Index index, Names fieldNames, FieldDataType fieldDataType) {
+        super(index, fieldNames, fieldDataType);
     }
 
     @Override
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/BinaryDVNumericIndexFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/BinaryDVNumericIndexFieldData.java
index ee495a8d44c2f..ff89ccbfa411c 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/BinaryDVNumericIndexFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/BinaryDVNumericIndexFieldData.java
@@ -23,6 +23,7 @@
 import org.apache.lucene.index.AtomicReaderContext;
 import org.elasticsearch.ElasticsearchIllegalStateException;
 import org.elasticsearch.index.Index;
+import org.elasticsearch.index.fielddata.FieldDataType;
 import org.elasticsearch.index.fielddata.IndexNumericFieldData;
 import org.elasticsearch.index.fielddata.fieldcomparator.DoubleValuesComparatorSource;
 import org.elasticsearch.index.fielddata.fieldcomparator.FloatValuesComparatorSource;
@@ -36,8 +37,8 @@ public class BinaryDVNumericIndexFieldData extends DocValuesIndexFieldData imple
 
     private final NumericType numericType;
 
-    public BinaryDVNumericIndexFieldData(Index index, Names fieldNames, NumericType numericType) {
-        super(index, fieldNames);
+    public BinaryDVNumericIndexFieldData(Index index, Names fieldNames, NumericType numericType, FieldDataType fieldDataType) {
+        super(index, fieldNames, fieldDataType);
         Preconditions.checkArgument(numericType != null, "numericType must be non-null");
         this.numericType = numericType;
     }
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/DisabledIndexFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/DisabledIndexFieldData.java
index 687f775f0da12..1b5f5c454aa3f 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/DisabledIndexFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/DisabledIndexFieldData.java
@@ -25,6 +25,7 @@
 import org.elasticsearch.index.Index;
 import org.elasticsearch.index.fielddata.*;
 import org.elasticsearch.index.fielddata.fieldcomparator.SortMode;
+import org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsBuilder;
 import org.elasticsearch.index.mapper.FieldMapper;
 import org.elasticsearch.index.mapper.FieldMapper.Names;
 import org.elasticsearch.index.mapper.MapperService;
@@ -40,7 +41,7 @@ public final class DisabledIndexFieldData extends AbstractIndexFieldData<AtomicF
     public static class Builder implements IndexFieldData.Builder {
         @Override
         public IndexFieldData<AtomicFieldData<?>> build(Index index, @IndexSettings Settings indexSettings, FieldMapper<?> mapper,
-                                                        IndexFieldDataCache cache, CircuitBreakerService breakerService, MapperService mapperService) {
+                                                        IndexFieldDataCache cache, CircuitBreakerService breakerService, MapperService mapperService, GlobalOrdinalsBuilder globalOrdinalBuilder) {
             // Ignore Circuit Breaker
             return new DisabledIndexFieldData(index, indexSettings, mapper.names(), mapper.fieldDataType(), cache);
         }
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/DocValuesIndexFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/DocValuesIndexFieldData.java
index 4b5a464253539..cf0988211e253 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/DocValuesIndexFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/DocValuesIndexFieldData.java
@@ -24,9 +24,11 @@
 import org.elasticsearch.ElasticsearchIllegalArgumentException;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.index.Index;
+import org.elasticsearch.index.fielddata.FieldDataType;
 import org.elasticsearch.index.fielddata.IndexFieldData;
 import org.elasticsearch.index.fielddata.IndexFieldDataCache;
 import org.elasticsearch.index.fielddata.IndexNumericFieldData.NumericType;
+import org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsBuilder;
 import org.elasticsearch.index.mapper.FieldMapper;
 import org.elasticsearch.index.mapper.FieldMapper.Names;
 import org.elasticsearch.index.mapper.MapperService;
@@ -43,17 +45,23 @@ public abstract class DocValuesIndexFieldData {
 
     protected final Index index;
     protected final Names fieldNames;
+    protected final FieldDataType fieldDataType;
 
-    public DocValuesIndexFieldData(Index index, Names fieldNames) {
+    public DocValuesIndexFieldData(Index index, Names fieldNames, FieldDataType fieldDataType) {
         super();
         this.index = index;
         this.fieldNames = fieldNames;
+        this.fieldDataType = fieldDataType;
     }
 
     public final Names getFieldNames() {
         return fieldNames;
     }
 
+    public final FieldDataType getFieldDataType() {
+        return fieldDataType;
+    }
+
     public final void clear() {
         // can't do
     }
@@ -80,7 +88,8 @@ public Builder numericType(NumericType type) {
 
         @Override
         public IndexFieldData<?> build(Index index, Settings indexSettings, FieldMapper<?> mapper, IndexFieldDataCache cache,
-                                       CircuitBreakerService breakerService, MapperService mapperService) {
+                                       CircuitBreakerService breakerService, MapperService mapperService,
+                                       GlobalOrdinalsBuilder globalOrdinalBuilder) {
             // Ignore Circuit Breaker
             final FieldMapper.Names fieldNames = mapper.names();
             final Settings fdSettings = mapper.fieldDataType().getSettings();
@@ -91,14 +100,14 @@ public IndexFieldData<?> build(Index index, Settings indexSettings, FieldMapper<
 
             if (BINARY_INDEX_FIELD_NAMES.contains(fieldNames.indexName())) {
                 assert numericType == null;
-                return new BinaryDVIndexFieldData(index, fieldNames);
+                return new BinaryDVIndexFieldData(index, fieldNames, mapper.fieldDataType());
             } else if (NUMERIC_INDEX_FIELD_NAMES.contains(fieldNames.indexName())) {
                 assert !numericType.isFloatingPoint();
-                return new NumericDVIndexFieldData(index, fieldNames);
+                return new NumericDVIndexFieldData(index, fieldNames, mapper.fieldDataType());
             } else if (numericType != null) {
-                return new BinaryDVNumericIndexFieldData(index, fieldNames, numericType);
+                return new BinaryDVNumericIndexFieldData(index, fieldNames, numericType, mapper.fieldDataType());
             } else {
-                return new SortedSetDVBytesIndexFieldData(index, fieldNames);
+                return new SortedSetDVBytesIndexFieldData(index, cache, indexSettings, fieldNames, globalOrdinalBuilder,breakerService, mapper.fieldDataType());
             }
         }
 
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/DoubleArrayIndexFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/DoubleArrayIndexFieldData.java
index f68faa4ab7faa..bb708a49f969c 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/DoubleArrayIndexFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/DoubleArrayIndexFieldData.java
@@ -30,6 +30,7 @@
 import org.elasticsearch.index.fielddata.*;
 import org.elasticsearch.index.fielddata.fieldcomparator.DoubleValuesComparatorSource;
 import org.elasticsearch.index.fielddata.fieldcomparator.SortMode;
+import org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsBuilder;
 import org.elasticsearch.index.fielddata.ordinals.Ordinals;
 import org.elasticsearch.index.fielddata.ordinals.Ordinals.Docs;
 import org.elasticsearch.index.fielddata.ordinals.OrdinalsBuilder;
@@ -48,7 +49,7 @@ public static class Builder implements IndexFieldData.Builder {
 
         @Override
         public IndexFieldData<?> build(Index index, @IndexSettings Settings indexSettings, FieldMapper<?> mapper, IndexFieldDataCache cache,
-                                       CircuitBreakerService breakerService, MapperService mapperService) {
+                                       CircuitBreakerService breakerService, MapperService mapperService, GlobalOrdinalsBuilder globalOrdinalBuilder) {
             return new DoubleArrayIndexFieldData(index, indexSettings, mapper.names(), mapper.fieldDataType(), cache, breakerService);
         }
     }
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/FSTBytesAtomicFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/FSTBytesAtomicFieldData.java
index d728aa029db16..981068adaf2e2 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/FSTBytesAtomicFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/FSTBytesAtomicFieldData.java
@@ -18,6 +18,7 @@
  */
 package org.elasticsearch.index.fielddata.plain;
 
+import org.apache.lucene.index.TermsEnum;
 import org.apache.lucene.util.BytesRef;
 import org.apache.lucene.util.IntsRef;
 import org.apache.lucene.util.fst.*;
@@ -125,7 +126,10 @@ public ScriptDocValues.Strings getScriptValues() {
         return new ScriptDocValues.Strings(getBytesValues(false));
     }
 
-
+    @Override
+    public TermsEnum getTermsEnum() {
+        return new AtomicFieldDataWithOrdinalsTermsEnum(this);
+    }
 
     static class BytesValues extends org.elasticsearch.index.fielddata.BytesValues.WithOrdinals {
 
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/FSTBytesIndexFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/FSTBytesIndexFieldData.java
index d15fad5726d43..f97bdc4d18563 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/FSTBytesIndexFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/FSTBytesIndexFieldData.java
@@ -30,6 +30,7 @@
 import org.elasticsearch.index.fielddata.FieldDataType;
 import org.elasticsearch.index.fielddata.IndexFieldData;
 import org.elasticsearch.index.fielddata.IndexFieldDataCache;
+import org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsBuilder;
 import org.elasticsearch.index.fielddata.ordinals.Ordinals;
 import org.elasticsearch.index.fielddata.ordinals.OrdinalsBuilder;
 import org.elasticsearch.index.mapper.FieldMapper;
@@ -47,14 +48,15 @@ public static class Builder implements IndexFieldData.Builder {
 
         @Override
         public IndexFieldData<FSTBytesAtomicFieldData> build(Index index, @IndexSettings Settings indexSettings, FieldMapper<?> mapper,
-                                                             IndexFieldDataCache cache, CircuitBreakerService breakerService, MapperService mapperService) {
-            return new FSTBytesIndexFieldData(index, indexSettings, mapper.names(), mapper.fieldDataType(), cache, breakerService);
+                                                             IndexFieldDataCache cache, CircuitBreakerService breakerService, MapperService mapperService,
+                                                             GlobalOrdinalsBuilder globalOrdinalBuilder) {
+            return new FSTBytesIndexFieldData(index, indexSettings, mapper.names(), mapper.fieldDataType(), cache, breakerService, globalOrdinalBuilder);
         }
     }
 
     FSTBytesIndexFieldData(Index index, @IndexSettings Settings indexSettings, FieldMapper.Names fieldNames, FieldDataType fieldDataType,
-                           IndexFieldDataCache cache, CircuitBreakerService breakerService) {
-        super(index, indexSettings, fieldNames, fieldDataType, cache);
+                           IndexFieldDataCache cache, CircuitBreakerService breakerService, GlobalOrdinalsBuilder globalOrdinalsBuilder) {
+        super(index, indexSettings, fieldNames, fieldDataType, cache, globalOrdinalsBuilder, breakerService);
         this.breakerService = breakerService;
     }
 
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/FloatArrayIndexFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/FloatArrayIndexFieldData.java
index 36f1cbabf6c97..202fa5d1650df 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/FloatArrayIndexFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/FloatArrayIndexFieldData.java
@@ -29,6 +29,7 @@
 import org.elasticsearch.index.fielddata.*;
 import org.elasticsearch.index.fielddata.fieldcomparator.FloatValuesComparatorSource;
 import org.elasticsearch.index.fielddata.fieldcomparator.SortMode;
+import org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsBuilder;
 import org.elasticsearch.index.fielddata.ordinals.Ordinals;
 import org.elasticsearch.index.fielddata.ordinals.Ordinals.Docs;
 import org.elasticsearch.index.fielddata.ordinals.OrdinalsBuilder;
@@ -47,7 +48,7 @@ public static class Builder implements IndexFieldData.Builder {
 
         @Override
         public IndexFieldData<?> build(Index index, @IndexSettings Settings indexSettings, FieldMapper<?> mapper, IndexFieldDataCache cache,
-                                       CircuitBreakerService breakerService, MapperService mapperService) {
+                                       CircuitBreakerService breakerService, MapperService mapperService, GlobalOrdinalsBuilder globalOrdinalBuilder) {
             return new FloatArrayIndexFieldData(index, indexSettings, mapper.names(), mapper.fieldDataType(), cache, breakerService);
         }
     }
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/GeoPointBinaryDVIndexFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/GeoPointBinaryDVIndexFieldData.java
index eedb7bedc5fbe..5e87872b78d1f 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/GeoPointBinaryDVIndexFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/GeoPointBinaryDVIndexFieldData.java
@@ -27,6 +27,7 @@
 import org.elasticsearch.index.Index;
 import org.elasticsearch.index.fielddata.*;
 import org.elasticsearch.index.fielddata.fieldcomparator.SortMode;
+import org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsBuilder;
 import org.elasticsearch.index.mapper.FieldMapper;
 import org.elasticsearch.index.mapper.FieldMapper.Names;
 import org.elasticsearch.index.mapper.MapperService;
@@ -36,8 +37,8 @@
 
 public class GeoPointBinaryDVIndexFieldData extends DocValuesIndexFieldData implements IndexGeoPointFieldData<AtomicGeoPointFieldData<ScriptDocValues>> {
 
-    public GeoPointBinaryDVIndexFieldData(Index index, Names fieldNames) {
-        super(index, fieldNames);
+    public GeoPointBinaryDVIndexFieldData(Index index, Names fieldNames, FieldDataType fieldDataType) {
+        super(index, fieldNames, fieldDataType);
     }
 
     @Override
@@ -68,10 +69,10 @@ public static class Builder implements IndexFieldData.Builder {
 
         @Override
         public IndexFieldData<?> build(Index index, Settings indexSettings, FieldMapper<?> mapper, IndexFieldDataCache cache,
-                                       CircuitBreakerService breakerService, MapperService mapperService) {
+                                       CircuitBreakerService breakerService, MapperService mapperService, GlobalOrdinalsBuilder globalOrdinalBuilder) {
             // Ignore breaker
             final FieldMapper.Names fieldNames = mapper.names();
-            return new GeoPointBinaryDVIndexFieldData(index, fieldNames);
+            return new GeoPointBinaryDVIndexFieldData(index, fieldNames, mapper.fieldDataType());
         }
 
     }
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/GeoPointCompressedIndexFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/GeoPointCompressedIndexFieldData.java
index 5b3a80a55b0e6..16f755f732e3f 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/GeoPointCompressedIndexFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/GeoPointCompressedIndexFieldData.java
@@ -31,6 +31,7 @@
 import org.elasticsearch.common.util.BigArrays;
 import org.elasticsearch.index.Index;
 import org.elasticsearch.index.fielddata.*;
+import org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsBuilder;
 import org.elasticsearch.index.fielddata.ordinals.Ordinals;
 import org.elasticsearch.index.fielddata.ordinals.Ordinals.Docs;
 import org.elasticsearch.index.fielddata.ordinals.OrdinalsBuilder;
@@ -52,7 +53,7 @@ public static class Builder implements IndexFieldData.Builder {
 
         @Override
         public IndexFieldData<?> build(Index index, @IndexSettings Settings indexSettings, FieldMapper<?> mapper, IndexFieldDataCache cache,
-                                       CircuitBreakerService breakerService, MapperService mapperService) {
+                                       CircuitBreakerService breakerService, MapperService mapperService, GlobalOrdinalsBuilder globalOrdinalBuilder) {
             FieldDataType type = mapper.fieldDataType();
             final String precisionAsString = type.getSettings().get(PRECISION_KEY);
             final Distance precision;
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/GeoPointDoubleArrayIndexFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/GeoPointDoubleArrayIndexFieldData.java
index ebc583bfaae1e..cc0d14cdb5973 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/GeoPointDoubleArrayIndexFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/GeoPointDoubleArrayIndexFieldData.java
@@ -27,6 +27,7 @@
 import org.elasticsearch.common.util.BigDoubleArrayList;
 import org.elasticsearch.index.Index;
 import org.elasticsearch.index.fielddata.*;
+import org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsBuilder;
 import org.elasticsearch.index.fielddata.ordinals.Ordinals;
 import org.elasticsearch.index.fielddata.ordinals.Ordinals.Docs;
 import org.elasticsearch.index.fielddata.ordinals.OrdinalsBuilder;
@@ -45,7 +46,7 @@ public static class Builder implements IndexFieldData.Builder {
 
         @Override
         public IndexFieldData<?> build(Index index, @IndexSettings Settings indexSettings, FieldMapper<?> mapper, IndexFieldDataCache cache,
-                                       CircuitBreakerService breakerService, MapperService mapperService) {
+                                       CircuitBreakerService breakerService, MapperService mapperService, GlobalOrdinalsBuilder globalOrdinalBuilder) {
             return new GeoPointDoubleArrayIndexFieldData(index, indexSettings, mapper.names(), mapper.fieldDataType(), cache, breakerService);
         }
     }
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/NumericDVIndexFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/NumericDVIndexFieldData.java
index 20043376e4ce5..442f9e50b6eba 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/NumericDVIndexFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/NumericDVIndexFieldData.java
@@ -21,6 +21,7 @@
 
 import org.apache.lucene.index.AtomicReaderContext;
 import org.elasticsearch.index.Index;
+import org.elasticsearch.index.fielddata.FieldDataType;
 import org.elasticsearch.index.fielddata.IndexNumericFieldData;
 import org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource;
 import org.elasticsearch.index.fielddata.fieldcomparator.SortMode;
@@ -28,8 +29,8 @@
 
 public class NumericDVIndexFieldData extends DocValuesIndexFieldData implements IndexNumericFieldData<NumericDVAtomicFieldData> {
 
-    public NumericDVIndexFieldData(Index index, Names fieldNames) {
-        super(index, fieldNames);
+    public NumericDVIndexFieldData(Index index, Names fieldNames, FieldDataType fieldDataType) {
+        super(index, fieldNames, fieldDataType);
     }
 
     @Override
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/PackedArrayIndexFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/PackedArrayIndexFieldData.java
index 0c895f49accaa..76846f39c3ddd 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/PackedArrayIndexFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/PackedArrayIndexFieldData.java
@@ -38,6 +38,7 @@
 import org.elasticsearch.index.fielddata.*;
 import org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource;
 import org.elasticsearch.index.fielddata.fieldcomparator.SortMode;
+import org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsBuilder;
 import org.elasticsearch.index.fielddata.ordinals.Ordinals;
 import org.elasticsearch.index.fielddata.ordinals.Ordinals.Docs;
 import org.elasticsearch.index.fielddata.ordinals.OrdinalsBuilder;
@@ -65,7 +66,7 @@ public Builder setNumericType(NumericType numericType) {
 
         @Override
         public IndexFieldData<AtomicNumericFieldData> build(Index index, @IndexSettings Settings indexSettings, FieldMapper<?> mapper,
-                                                            IndexFieldDataCache cache, CircuitBreakerService breakerService, MapperService mapperService) {
+                                                            IndexFieldDataCache cache, CircuitBreakerService breakerService, MapperService mapperService, GlobalOrdinalsBuilder globalOrdinalBuilder) {
             return new PackedArrayIndexFieldData(index, indexSettings, mapper.names(), mapper.fieldDataType(), cache, numericType, breakerService);
         }
     }
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/PagedBytesAtomicFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/PagedBytesAtomicFieldData.java
index 3664f95087160..a7e3549522220 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/PagedBytesAtomicFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/PagedBytesAtomicFieldData.java
@@ -18,6 +18,7 @@
  */
 package org.elasticsearch.index.fielddata.plain;
 
+import org.apache.lucene.index.TermsEnum;
 import org.apache.lucene.util.BytesRef;
 import org.apache.lucene.util.PagedBytes;
 import org.apache.lucene.util.PagedBytes.Reader;
@@ -120,6 +121,11 @@ public ScriptDocValues.Strings getScriptValues() {
         return new ScriptDocValues.Strings(getBytesValues(false));
     }
 
+    @Override
+    public TermsEnum getTermsEnum() {
+        return new AtomicFieldDataWithOrdinalsTermsEnum(this);
+    }
+
     static class BytesValues extends org.elasticsearch.index.fielddata.BytesValues.WithOrdinals {
 
         protected final PagedBytes.Reader bytes;
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/PagedBytesIndexFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/PagedBytesIndexFieldData.java
index d06d5dfa9566a..0f6ea5d6fec45 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/PagedBytesIndexFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/PagedBytesIndexFieldData.java
@@ -28,6 +28,7 @@
 import org.elasticsearch.index.Index;
 import org.elasticsearch.index.fielddata.RamAccountingTermsEnum;
 import org.elasticsearch.index.fielddata.*;
+import org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsBuilder;
 import org.elasticsearch.index.fielddata.ordinals.Ordinals;
 import org.elasticsearch.index.fielddata.ordinals.OrdinalsBuilder;
 import org.elasticsearch.index.mapper.FieldMapper;
@@ -41,21 +42,21 @@
  */
 public class PagedBytesIndexFieldData extends AbstractBytesIndexFieldData<PagedBytesAtomicFieldData> {
 
-    private final CircuitBreakerService breakerService;
 
     public static class Builder implements IndexFieldData.Builder {
 
         @Override
         public IndexFieldData<PagedBytesAtomicFieldData> build(Index index, @IndexSettings Settings indexSettings, FieldMapper<?> mapper,
-                                                               IndexFieldDataCache cache, CircuitBreakerService breakerService, MapperService mapperService) {
-            return new PagedBytesIndexFieldData(index, indexSettings, mapper.names(), mapper.fieldDataType(), cache, breakerService);
+                                                               IndexFieldDataCache cache, CircuitBreakerService breakerService, MapperService mapperService,
+                                                               GlobalOrdinalsBuilder globalOrdinalBuilder) {
+            return new PagedBytesIndexFieldData(index, indexSettings, mapper.names(), mapper.fieldDataType(), cache, breakerService, globalOrdinalBuilder);
         }
     }
 
     public PagedBytesIndexFieldData(Index index, @IndexSettings Settings indexSettings, FieldMapper.Names fieldNames,
-                                    FieldDataType fieldDataType, IndexFieldDataCache cache, CircuitBreakerService breakerService) {
-        super(index, indexSettings, fieldNames, fieldDataType, cache);
-        this.breakerService = breakerService;
+                                    FieldDataType fieldDataType, IndexFieldDataCache cache, CircuitBreakerService breakerService,
+                                    GlobalOrdinalsBuilder globalOrdinalsBuilder) {
+        super(index, indexSettings, fieldNames, fieldDataType, cache, globalOrdinalsBuilder, breakerService);
     }
 
     @Override
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/ParentChildIndexFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/ParentChildIndexFieldData.java
index a99bd7f51fb54..2ba8ee201296d 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/ParentChildIndexFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/ParentChildIndexFieldData.java
@@ -33,6 +33,7 @@
 import org.elasticsearch.index.fielddata.*;
 import org.elasticsearch.index.fielddata.fieldcomparator.BytesRefFieldComparatorSource;
 import org.elasticsearch.index.fielddata.fieldcomparator.SortMode;
+import org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsBuilder;
 import org.elasticsearch.index.fielddata.ordinals.Ordinals;
 import org.elasticsearch.index.fielddata.ordinals.OrdinalsBuilder;
 import org.elasticsearch.index.mapper.DocumentMapper;
@@ -199,7 +200,7 @@ public static class Builder implements IndexFieldData.Builder {
         @Override
         public IndexFieldData<?> build(Index index, @IndexSettings Settings indexSettings, FieldMapper<?> mapper,
                                        IndexFieldDataCache cache, CircuitBreakerService breakerService,
-                                       MapperService mapperService) {
+                                       MapperService mapperService, GlobalOrdinalsBuilder globalOrdinalBuilder) {
             return new ParentChildIndexFieldData(index, indexSettings, mapper.names(), mapper.fieldDataType(), cache, mapperService, breakerService);
         }
     }
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/SortedSetDVAtomicFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/SortedSetDVAtomicFieldData.java
index 92d7d27cc3265..184571613d089 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/SortedSetDVAtomicFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/SortedSetDVAtomicFieldData.java
@@ -20,7 +20,9 @@
 package org.elasticsearch.index.fielddata.plain;
 
 import org.apache.lucene.index.AtomicReader;
+import org.apache.lucene.index.FilterAtomicReader;
 import org.apache.lucene.index.SortedSetDocValues;
+import org.apache.lucene.index.TermsEnum;
 import org.apache.lucene.util.ArrayUtil;
 import org.apache.lucene.util.BytesRef;
 import org.apache.lucene.util.LongsRef;
@@ -99,6 +101,23 @@ public org.elasticsearch.index.fielddata.BytesValues.WithOrdinals getHashedBytes
         return new SortedSetHashedValues(reader, field, values, hashes);
     }
 
+    public TermsEnum getTermsEnum() {
+        final TermsEnum termsEnum = getValuesNoException(reader, field).termsEnum();
+        return new FilterAtomicReader.FilterTermsEnum(termsEnum) {
+
+            @Override
+            public void seekExact(long ord) throws IOException {
+                super.seekExact(ord - 1);
+            }
+
+            @Override
+            public long ord() throws IOException {
+                return super.ord() + 1;
+            }
+
+        };
+    }
+
     private static SortedSetDocValues getValuesNoException(AtomicReader reader, String field) {
         try {
             SortedSetDocValues values = reader.getSortedSetDocValues(field);
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/SortedSetDVBytesAtomicFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/SortedSetDVBytesAtomicFieldData.java
index 0a10a7bb1e810..39564af439421 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/SortedSetDVBytesAtomicFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/SortedSetDVBytesAtomicFieldData.java
@@ -46,4 +46,5 @@ public boolean isValuesOrdered() {
     public Strings getScriptValues() {
         return new ScriptDocValues.Strings(getBytesValues(false));
     }
+
 }
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/SortedSetDVBytesIndexFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/SortedSetDVBytesIndexFieldData.java
index 8e2ce5261ed3b..6347822834fd0 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/SortedSetDVBytesIndexFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/SortedSetDVBytesIndexFieldData.java
@@ -20,16 +20,32 @@
 package org.elasticsearch.index.fielddata.plain;
 
 import org.apache.lucene.index.AtomicReaderContext;
+import org.apache.lucene.index.IndexReader;
+import org.elasticsearch.ElasticsearchException;
+import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.index.Index;
+import org.elasticsearch.index.fielddata.FieldDataType;
 import org.elasticsearch.index.fielddata.IndexFieldData;
+import org.elasticsearch.index.fielddata.IndexFieldDataCache;
 import org.elasticsearch.index.fielddata.fieldcomparator.BytesRefFieldComparatorSource;
 import org.elasticsearch.index.fielddata.fieldcomparator.SortMode;
+import org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsBuilder;
 import org.elasticsearch.index.mapper.FieldMapper.Names;
+import org.elasticsearch.indices.fielddata.breaker.CircuitBreakerService;
 
 public class SortedSetDVBytesIndexFieldData extends DocValuesIndexFieldData implements IndexFieldData.WithOrdinals<SortedSetDVBytesAtomicFieldData> {
 
-    public SortedSetDVBytesIndexFieldData(Index index, Names fieldNames) {
-        super(index, fieldNames);
+    private final Settings indexSettings;
+    private final IndexFieldDataCache cache;
+    private final GlobalOrdinalsBuilder globalOrdinalsBuilder;
+    private final CircuitBreakerService breakerService;
+
+    public SortedSetDVBytesIndexFieldData(Index index, IndexFieldDataCache cache, Settings indexSettings, Names fieldNames, GlobalOrdinalsBuilder globalOrdinalBuilder, CircuitBreakerService breakerService, FieldDataType fieldDataType) {
+        super(index, fieldNames, fieldDataType);
+        this.indexSettings = indexSettings;
+        this.cache = cache;
+        this.globalOrdinalsBuilder = globalOrdinalBuilder;
+        this.breakerService = breakerService;
     }
 
     @Override
@@ -50,4 +66,26 @@ public SortedSetDVBytesAtomicFieldData load(AtomicReaderContext context) {
     public SortedSetDVBytesAtomicFieldData loadDirect(AtomicReaderContext context) throws Exception {
         return load(context);
     }
+
+    @Override
+    public WithOrdinals loadGlobal(IndexReader indexReader) {
+        if (indexReader.leaves().size() <= 1) {
+            // ordinals are already global
+            return this;
+        }
+        try {
+            return cache.load(indexReader, this);
+        } catch (Throwable e) {
+            if (e instanceof ElasticsearchException) {
+                throw (ElasticsearchException) e;
+            } else {
+                throw new ElasticsearchException(e.getMessage(), e);
+            }
+        }
+    }
+
+    @Override
+    public WithOrdinals localGlobalDirect(IndexReader indexReader) throws Exception {
+        return globalOrdinalsBuilder.build(indexReader, this, indexSettings, breakerService);
+    }
 }
diff --git a/src/main/java/org/elasticsearch/index/mapper/FieldMapper.java b/src/main/java/org/elasticsearch/index/mapper/FieldMapper.java
index 267640812160e..7aff7dfc5f4ab 100644
--- a/src/main/java/org/elasticsearch/index/mapper/FieldMapper.java
+++ b/src/main/java/org/elasticsearch/index/mapper/FieldMapper.java
@@ -162,15 +162,24 @@ public String toString() {
             public String toString() {
                 return EAGER_VALUE;
             }
+        },
+        EAGER_GLOBAL_ORDINALS {
+            @Override
+            public String toString() {
+                return EAGER_GLOBAL_ORDINALS_VALUE;
+            }
         };
 
         public static final String KEY = "loading";
+        public static final String EAGER_GLOBAL_ORDINALS_VALUE = "eager_global_ordinals";
         public static final String EAGER_VALUE = "eager";
         public static final String LAZY_VALUE = "lazy";
 
         public static Loading parse(String loading, Loading defaultValue) {
             if (Strings.isNullOrEmpty(loading)) {
                 return defaultValue;
+            } else if (EAGER_GLOBAL_ORDINALS_VALUE.equalsIgnoreCase(loading)) {
+                return EAGER_GLOBAL_ORDINALS;
             } else if (EAGER_VALUE.equalsIgnoreCase(loading)) {
                 return EAGER;
             } else if (LAZY_VALUE.equalsIgnoreCase(loading)) {
diff --git a/src/main/java/org/elasticsearch/index/shard/ShardUtils.java b/src/main/java/org/elasticsearch/index/shard/ShardUtils.java
index 811ec4aab2803..5203b376ea262 100644
--- a/src/main/java/org/elasticsearch/index/shard/ShardUtils.java
+++ b/src/main/java/org/elasticsearch/index/shard/ShardUtils.java
@@ -20,6 +20,7 @@
 package org.elasticsearch.index.shard;
 
 import org.apache.lucene.index.AtomicReader;
+import org.apache.lucene.index.IndexReader;
 import org.apache.lucene.index.SegmentReader;
 import org.elasticsearch.common.Nullable;
 import org.elasticsearch.common.lucene.SegmentReaderUtils;
@@ -54,4 +55,11 @@ private static ShardId extractShardId(SegmentReader reader) {
         return null;
     }
 
+    public static ShardId extractShardId(IndexReader reader) {
+        if (!reader.leaves().isEmpty()) {
+            return extractShardId(reader.leaves().get(0).reader());
+        }
+        return null;
+    }
+
 }
diff --git a/src/main/java/org/elasticsearch/indices/fielddata/cache/IndicesFieldDataCache.java b/src/main/java/org/elasticsearch/indices/fielddata/cache/IndicesFieldDataCache.java
index b2910219a8d9b..de58896e78e49 100644
--- a/src/main/java/org/elasticsearch/indices/fielddata/cache/IndicesFieldDataCache.java
+++ b/src/main/java/org/elasticsearch/indices/fielddata/cache/IndicesFieldDataCache.java
@@ -21,6 +21,7 @@
 
 import com.google.common.cache.*;
 import org.apache.lucene.index.AtomicReaderContext;
+import org.apache.lucene.index.IndexReader;
 import org.apache.lucene.index.SegmentReader;
 import org.elasticsearch.common.component.AbstractComponent;
 import org.elasticsearch.common.inject.Inject;
@@ -29,10 +30,8 @@
 import org.elasticsearch.common.unit.ByteSizeValue;
 import org.elasticsearch.common.unit.TimeValue;
 import org.elasticsearch.index.Index;
-import org.elasticsearch.index.fielddata.AtomicFieldData;
-import org.elasticsearch.index.fielddata.FieldDataType;
-import org.elasticsearch.index.fielddata.IndexFieldData;
-import org.elasticsearch.index.fielddata.IndexFieldDataCache;
+import org.elasticsearch.index.fielddata.*;
+import org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsIndexFieldData;
 import org.elasticsearch.index.mapper.FieldMapper;
 import org.elasticsearch.index.service.IndexService;
 import org.elasticsearch.index.shard.ShardId;
@@ -46,10 +45,10 @@
 
 /**
  */
-public class IndicesFieldDataCache extends AbstractComponent implements RemovalListener<IndicesFieldDataCache.Key, AtomicFieldData> {
+public class IndicesFieldDataCache extends AbstractComponent implements RemovalListener<IndicesFieldDataCache.Key, RamUsage> {
 
     private final IndicesFieldDataCacheListener indicesFieldDataCacheListener;
-    private final Cache<Key, AtomicFieldData> cache;
+    private final Cache<Key, RamUsage> cache;
 
     @Inject
     public IndicesFieldDataCache(Settings settings, IndicesFieldDataCacheListener indicesFieldDataCacheListener) {
@@ -58,7 +57,7 @@ public IndicesFieldDataCache(Settings settings, IndicesFieldDataCacheListener in
         final String size = componentSettings.get("size", "-1");
         final long sizeInBytes = componentSettings.getAsMemory("size", "-1").bytes();
         final TimeValue expire = componentSettings.getAsTime("expire", null);
-        CacheBuilder<Key, AtomicFieldData> cacheBuilder = CacheBuilder.newBuilder()
+        CacheBuilder<Key, RamUsage> cacheBuilder = CacheBuilder.newBuilder()
                 .removalListener(this);
         if (sizeInBytes > 0) {
             cacheBuilder.maximumWeight(sizeInBytes).weigher(new FieldDataWeigher());
@@ -80,19 +79,23 @@ public IndexFieldDataCache buildIndexFieldDataCache(IndexService indexService, I
         return new IndexFieldCache(cache, indicesFieldDataCacheListener, indexService, index, fieldNames, fieldDataType);
     }
 
+    public Cache<Key, RamUsage> getCache() {
+        return cache;
+    }
+
     @Override
-    public void onRemoval(RemovalNotification<Key, AtomicFieldData> notification) {
-        final Key key = notification.getKey();
+    public void onRemoval(RemovalNotification<Key, RamUsage> notification) {
+        Key key = notification.getKey();
         assert key != null && key.listeners != null;
         IndexFieldCache indexCache = key.indexCache;
         long sizeInBytes = key.sizeInBytes;
-        final AtomicFieldData value = notification.getValue();
+        final RamUsage value = notification.getValue();
         assert sizeInBytes >= 0 || value != null : "Expected size [" + sizeInBytes + "] to be positive or value [" + value + "] to be non-null";
         if (sizeInBytes == -1 && value != null) {
             sizeInBytes = value.getMemorySizeInBytes();
         }
         for (IndexFieldDataCache.Listener listener : key.listeners) {
-            listener.onUnload(indexCache.fieldNames, indexCache.fieldDataType, notification.wasEvicted(), sizeInBytes, value);
+            listener.onUnload(indexCache.fieldNames, indexCache.fieldDataType, notification.wasEvicted(), sizeInBytes);
         }
     }
 
@@ -108,16 +111,15 @@ public int weigh(Key key, AtomicFieldData fieldData) {
     /**
      * A specific cache instance for the relevant parameters of it (index, fieldNames, fieldType).
      */
-    static class IndexFieldCache implements IndexFieldDataCache, SegmentReader.CoreClosedListener {
+    static class IndexFieldCache implements IndexFieldDataCache, SegmentReader.CoreClosedListener, IndexReader.ReaderClosedListener {
 
         private final IndexService indexService;
         final Index index;
         final FieldMapper.Names fieldNames;
         final FieldDataType fieldDataType;
-        private final Cache<Key, AtomicFieldData> cache;
-        private final IndicesFieldDataCacheListener indicesFieldDataCacheListener;
+        private final Cache<Key, RamUsage> cache;
 
-        IndexFieldCache(final Cache<Key, AtomicFieldData> cache, IndicesFieldDataCacheListener indicesFieldDataCacheListener, IndexService indexService, Index index, FieldMapper.Names fieldNames, FieldDataType fieldDataType) {
+        IndexFieldCache(final Cache<Key, RamUsage> cache, IndicesFieldDataCacheListener indicesFieldDataCacheListener, IndexService indexService, Index index, FieldMapper.Names fieldNames, FieldDataType fieldDataType) {
             this.indexService = indexService;
             this.index = index;
             this.fieldNames = fieldNames;
@@ -127,6 +129,8 @@ static class IndexFieldCache implements IndexFieldDataCache, SegmentReader.CoreC
             assert indexService != null;
         }
 
+        private final IndicesFieldDataCacheListener indicesFieldDataCacheListener;
+
         @Override
         public <FD extends AtomicFieldData, IFD extends IndexFieldData<FD>> FD load(final AtomicReaderContext context, final IFD indexFieldData) throws Exception {
             final Key key = new Key(this, context.reader().getCoreCacheKey());
@@ -153,11 +157,40 @@ public AtomicFieldData call() throws Exception {
             });
         }
 
+        public <IFD extends IndexFieldData.WithOrdinals<?>> IFD load(final IndexReader indexReader, final IFD indexFieldData) throws Exception {
+            final Key key = new Key(this, indexReader.getCoreCacheKey());
+            //noinspection unchecked
+            return (IFD) cache.get(key, new Callable<RamUsage>() {
+                @Override
+                public RamUsage call() throws Exception {
+                    indexReader.addReaderClosedListener(IndexFieldCache.this);
+                    GlobalOrdinalsIndexFieldData ifd = (GlobalOrdinalsIndexFieldData) indexFieldData.localGlobalDirect(indexReader);
+                    key.listeners.add(indicesFieldDataCacheListener);
+                    final ShardId shardId = ShardUtils.extractShardId(indexReader);
+                    if (shardId != null) {
+                        IndexShard shard = indexService.shard(shardId.id());
+                        if (shard != null) {
+                            key.listeners.add(shard.fieldData());
+                        }
+                    }
+                    for (Listener listener : key.listeners) {
+                        listener.onLoad(fieldNames, fieldDataType, ifd);
+                    }
+                    return ifd;
+                }
+            });
+        }
+
         @Override
         public void onClose(Object coreKey) {
             cache.invalidate(new Key(this, coreKey));
         }
 
+        @Override
+        public void onClose(IndexReader reader) {
+            cache.invalidate(new Key(this, reader.getCoreCacheKey()));
+        }
+
         @Override
         public void clear() {
             for (Key key : cache.asMap().keySet()) {
diff --git a/src/main/java/org/elasticsearch/indices/fielddata/cache/IndicesFieldDataCacheListener.java b/src/main/java/org/elasticsearch/indices/fielddata/cache/IndicesFieldDataCacheListener.java
index 51c79de044f4f..17a1dad6025a5 100644
--- a/src/main/java/org/elasticsearch/indices/fielddata/cache/IndicesFieldDataCacheListener.java
+++ b/src/main/java/org/elasticsearch/indices/fielddata/cache/IndicesFieldDataCacheListener.java
@@ -19,11 +19,10 @@
 
 package org.elasticsearch.indices.fielddata.cache;
 
-import org.elasticsearch.common.Nullable;
 import org.elasticsearch.common.inject.Inject;
-import org.elasticsearch.index.fielddata.AtomicFieldData;
 import org.elasticsearch.index.fielddata.FieldDataType;
 import org.elasticsearch.index.fielddata.IndexFieldDataCache;
+import org.elasticsearch.index.fielddata.RamUsage;
 import org.elasticsearch.index.mapper.FieldMapper;
 import org.elasticsearch.indices.fielddata.breaker.CircuitBreakerService;
 
@@ -43,11 +42,11 @@ public IndicesFieldDataCacheListener(CircuitBreakerService circuitBreakerService
     }
 
     @Override
-    public void onLoad(FieldMapper.Names fieldNames, FieldDataType fieldDataType, AtomicFieldData fieldData) {
+    public void onLoad(FieldMapper.Names fieldNames, FieldDataType fieldDataType, RamUsage fieldData) {
     }
 
     @Override
-    public void onUnload(FieldMapper.Names fieldNames, FieldDataType fieldDataType, boolean wasEvicted, long sizeInBytes, @Nullable AtomicFieldData fieldData) {
+    public void onUnload(FieldMapper.Names fieldNames, FieldDataType fieldDataType, boolean wasEvicted, long sizeInBytes) {
         assert sizeInBytes >= 0 : "When reducing circuit breaker, it should be adjusted with a number higher or equal to 0 and not [" + sizeInBytes + "]";
         circuitBreakerService.getBreaker().addWithoutBreaking(-sizeInBytes);
     }
diff --git a/src/main/java/org/elasticsearch/indices/warmer/IndicesWarmer.java b/src/main/java/org/elasticsearch/indices/warmer/IndicesWarmer.java
index 67a6c82e9a96b..8bf24fd294d3f 100644
--- a/src/main/java/org/elasticsearch/indices/warmer/IndicesWarmer.java
+++ b/src/main/java/org/elasticsearch/indices/warmer/IndicesWarmer.java
@@ -19,6 +19,7 @@
 
 package org.elasticsearch.indices.warmer;
 
+import org.apache.lucene.index.IndexReader;
 import org.elasticsearch.cluster.metadata.IndexMetaData;
 import org.elasticsearch.index.engine.Engine;
 import org.elasticsearch.index.shard.ShardId;
@@ -49,6 +50,10 @@ public void awaitTermination() {}
 
         /** Queue tasks to warm-up the given segments and return handles that allow to wait for termination of the execution of those tasks. */
         public abstract TerminationHandle warm(IndexShard indexShard, IndexMetaData indexMetaData, WarmerContext context, ThreadPool threadPool);
+
+        public TerminationHandle warmTop(IndexShard indexShard, IndexMetaData indexMetaData, WarmerContext context, ThreadPool threadPool) {
+            return TerminationHandle.NO_WAIT;
+        }
     }
 
     public static class WarmerContext {
@@ -57,9 +62,18 @@ public static class WarmerContext {
 
         private final Engine.Searcher newSearcher;
 
+        private final IndexReader indexReader;
+
         public WarmerContext(ShardId shardId, Engine.Searcher newSearcher) {
             this.shardId = shardId;
             this.newSearcher = newSearcher;
+            this.indexReader = null;
+        }
+
+        public WarmerContext(ShardId shardId, IndexReader indexReader) {
+            this.shardId = shardId;
+            this.newSearcher = null;
+            this.indexReader = indexReader;
         }
 
         public ShardId shardId() {
@@ -70,6 +84,10 @@ public ShardId shardId() {
         public Engine.Searcher newSearcher() {
             return newSearcher;
         }
+
+        public IndexReader indexReader() {
+            return indexReader;
+        }
     }
 
     void addListener(Listener listener);
diff --git a/src/main/java/org/elasticsearch/indices/warmer/InternalIndicesWarmer.java b/src/main/java/org/elasticsearch/indices/warmer/InternalIndicesWarmer.java
index 2a048f29aac3e..2cd235381ac75 100644
--- a/src/main/java/org/elasticsearch/indices/warmer/InternalIndicesWarmer.java
+++ b/src/main/java/org/elasticsearch/indices/warmer/InternalIndicesWarmer.java
@@ -68,6 +68,14 @@ public void removeListener(Listener listener) {
     }
 
     public void warm(final WarmerContext context) {
+        warmInternal(context, false);
+    }
+
+    public void warmTop(WarmerContext context) {
+        warmInternal(context, true);
+    }
+
+    private void warmInternal(final WarmerContext context, boolean topReader) {
         final IndexMetaData indexMetaData = clusterService.state().metaData().index(context.shardId().index().name());
         if (indexMetaData == null) {
             return;
@@ -84,14 +92,22 @@ public void warm(final WarmerContext context) {
             return;
         }
         if (logger.isTraceEnabled()) {
-            logger.trace("[{}][{}] warming [{}]", context.shardId().index().name(), context.shardId().id(), context.newSearcher().reader());
+            if (topReader) {
+                logger.trace("[{}][{}] warming [{}]", context.shardId().index().name(), context.shardId().id(), context.newSearcher().reader());
+            } else {
+                logger.trace("[{}][{}] top warming [{}]", context.shardId().index().name(), context.shardId().id(), context.indexReader());
+            }
         }
         indexShard.warmerService().onPreWarm();
         long time = System.nanoTime();
         final List<IndicesWarmer.Listener.TerminationHandle> terminationHandles = Lists.newArrayList();
         // get a handle on pending tasks
         for (final Listener listener : listeners) {
-            terminationHandles.add(listener.warm(indexShard, indexMetaData, context, threadPool));
+            if (topReader) {
+                terminationHandles.add(listener.warmTop(indexShard, indexMetaData, context, threadPool));
+            } else {
+                terminationHandles.add(listener.warm(indexShard, indexMetaData, context, threadPool));
+            }
         }
         // wait for termination
         for (IndicesWarmer.Listener.TerminationHandle terminationHandle : terminationHandles) {
@@ -99,14 +115,23 @@ public void warm(final WarmerContext context) {
                 terminationHandle.awaitTermination();
             } catch (InterruptedException e) {
                 Thread.currentThread().interrupt();
-                logger.warn("Warming has been interrupted", e);
+                if (topReader) {
+                    logger.warn("warming has been interrupted", e);
+                } else {
+                    logger.warn("top warming has been interrupted", e);
+                }
                 break;
             }
         }
         long took = System.nanoTime() - time;
         indexShard.warmerService().onPostWarm(took);
         if (indexShard.warmerService().logger().isTraceEnabled()) {
-            indexShard.warmerService().logger().trace("warming took [{}]", new TimeValue(took, TimeUnit.NANOSECONDS));
+            if (topReader) {
+                indexShard.warmerService().logger().trace("top warming took [{}]", new TimeValue(took, TimeUnit.NANOSECONDS));
+            } else {
+                indexShard.warmerService().logger().trace("warming took [{}]", new TimeValue(took, TimeUnit.NANOSECONDS));
+            }
         }
     }
+
 }
diff --git a/src/main/java/org/elasticsearch/percolator/QueryCollector.java b/src/main/java/org/elasticsearch/percolator/QueryCollector.java
index d863e638900f7..18721f9c77c28 100644
--- a/src/main/java/org/elasticsearch/percolator/QueryCollector.java
+++ b/src/main/java/org/elasticsearch/percolator/QueryCollector.java
@@ -107,6 +107,7 @@ abstract class QueryCollector extends Collector {
             if (!aggregatorCollectors.isEmpty()) {
                 facetAggCollectorBuilder.add(new AggregationPhase.AggregationsCollector(aggregatorCollectors, aggregationContext));
             }
+            aggregationContext.setNextReader(context.searcher().getIndexReader().getContext());
         }
         facetAndAggregatorCollector = facetAggCollectorBuilder.build();
     }
diff --git a/src/main/java/org/elasticsearch/search/SearchService.java b/src/main/java/org/elasticsearch/search/SearchService.java
index c91c5a4fe61f6..4747184992960 100644
--- a/src/main/java/org/elasticsearch/search/SearchService.java
+++ b/src/main/java/org/elasticsearch/search/SearchService.java
@@ -49,6 +49,7 @@
 import org.elasticsearch.common.xcontent.XContentParser;
 import org.elasticsearch.index.engine.Engine;
 import org.elasticsearch.index.fielddata.FieldDataType;
+import org.elasticsearch.index.fielddata.IndexFieldData;
 import org.elasticsearch.index.fielddata.IndexFieldDataService;
 import org.elasticsearch.index.mapper.DocumentMapper;
 import org.elasticsearch.index.mapper.FieldMapper;
@@ -768,7 +769,7 @@ public TerminationHandle warm(final IndexShard indexShard, IndexMetaData indexMe
                     if (fieldDataType == null) {
                         continue;
                     }
-                    if (fieldDataType.getLoading() != Loading.EAGER) {
+                    if (fieldDataType.getLoading() == Loading.LAZY) {
                         continue;
                     }
 
@@ -812,6 +813,55 @@ public void awaitTermination() throws InterruptedException {
             };
         }
 
+        @Override
+        public TerminationHandle warmTop(final IndexShard indexShard, IndexMetaData indexMetaData, final WarmerContext context, ThreadPool threadPool) {
+            final MapperService mapperService = indexShard.mapperService();
+            final Map<String, FieldMapper<?>> warmUpGlobalOrdinals = new HashMap<>();
+            for (DocumentMapper docMapper : mapperService) {
+                for (FieldMapper<?> fieldMapper : docMapper.mappers().mappers()) {
+                    final FieldDataType fieldDataType = fieldMapper.fieldDataType();
+                    if (fieldDataType == null) {
+                        continue;
+                    }
+                    if (fieldDataType.getLoading() != Loading.EAGER_GLOBAL_ORDINALS) {
+                        continue;
+                    }
+                    final String indexName = fieldMapper.names().indexName();
+                    if (warmUpGlobalOrdinals.containsKey(indexName)) {
+                        continue;
+                    }
+                    warmUpGlobalOrdinals.put(indexName, fieldMapper);
+                }
+            }
+            final IndexFieldDataService indexFieldDataService = indexShard.indexFieldDataService();
+            final Executor executor = threadPool.executor(executor());
+            final CountDownLatch latch = new CountDownLatch(warmUpGlobalOrdinals.size());
+            for (final FieldMapper<?> fieldMapper : warmUpGlobalOrdinals.values()) {
+                executor.execute(new Runnable() {
+                    @Override
+                    public void run() {
+                        try {
+                            final long start = System.nanoTime();
+                            IndexFieldData ifd = indexFieldDataService.getForField(fieldMapper);
+                            ((IndexFieldData.WithOrdinals) ifd).loadGlobal(context.indexReader());
+                            if (indexShard.warmerService().logger().isTraceEnabled()) {
+                                indexShard.warmerService().logger().trace("warmed global ordinals for [{}], took [{}]", fieldMapper.names().name(), TimeValue.timeValueNanos(System.nanoTime() - start));
+                            }
+                        } catch (Throwable t) {
+                            indexShard.warmerService().logger().warn("failed to warm-up global ordinals for [{}]", t, fieldMapper.names().name());
+                        } finally {
+                            latch.countDown();
+                        }
+                    }
+                });
+            }
+            return new TerminationHandle() {
+                @Override
+                public void awaitTermination() throws InterruptedException {
+                    latch.await();
+                }
+            };
+        }
     }
 
     class SearchWarmer extends IndicesWarmer.Listener {
diff --git a/src/main/java/org/elasticsearch/search/aggregations/AggregationPhase.java b/src/main/java/org/elasticsearch/search/aggregations/AggregationPhase.java
index 5972ef767f794..73b899bc0edf1 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/AggregationPhase.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/AggregationPhase.java
@@ -90,6 +90,7 @@ public void preProcess(SearchContext context) {
             if (!collectors.isEmpty()) {
                 context.searcher().addMainQueryCollector(new AggregationsCollector(collectors, aggregationContext));
             }
+            aggregationContext.setNextReader(context.searcher().getIndexReader().getContext());
         }
     }
 
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/BucketsAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/BucketsAggregator.java
index 578c8f73f0e16..aa33eb05b9f0e 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/BucketsAggregator.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/BucketsAggregator.java
@@ -53,6 +53,13 @@ public BucketsAggregator(String name, BucketAggregationMode bucketAggregationMod
         collectableSugAggregators = collectables.toArray(new Aggregator[collectables.size()]);
     }
 
+    /**
+     * Return an upper bound of the maximum bucket ordinal seen so far.
+     */
+    protected final long maxBucketOrd() {
+        return docCounts.size();
+    }
+
     /**
      * Utility method to collect the given doc in the given bucket (identified by the bucket ordinal)
      */
@@ -110,6 +117,17 @@ protected final InternalAggregations bucketAggregations(long bucketOrd) {
         return new InternalAggregations(Arrays.asList(aggregations));
     }
 
+    /**
+     * Utility method to build empty aggregations of the sub aggregators.
+     */
+    protected final InternalAggregations bucketEmptyAggregations() {
+        final InternalAggregation[] aggregations = new InternalAggregation[subAggregators.length];
+        for (int i = 0; i < subAggregators.length; i++) {
+            aggregations[i] = subAggregators[i].buildEmptyAggregation();
+        }
+        return new InternalAggregations(Arrays.asList(aggregations));
+    }
+
     @Override
     public final boolean release() {
         boolean success = false;
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/AbstractStringTermsAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/AbstractStringTermsAggregator.java
new file mode 100644
index 0000000000000..8c0b67d787539
--- /dev/null
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/AbstractStringTermsAggregator.java
@@ -0,0 +1,57 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.search.aggregations.bucket.terms;
+
+import org.elasticsearch.search.aggregations.Aggregator;
+import org.elasticsearch.search.aggregations.AggregatorFactories;
+import org.elasticsearch.search.aggregations.InternalAggregation;
+import org.elasticsearch.search.aggregations.bucket.BucketsAggregator;
+import org.elasticsearch.search.aggregations.support.AggregationContext;
+
+import java.util.Collections;
+
+abstract class AbstractStringTermsAggregator extends BucketsAggregator {
+
+    protected final InternalOrder order;
+    protected final int requiredSize;
+    protected final int shardSize;
+    protected final long minDocCount;
+
+    public AbstractStringTermsAggregator(String name, AggregatorFactories factories,
+            long estimatedBucketsCount, AggregationContext context, Aggregator parent,
+            InternalOrder order, int requiredSize, int shardSize, long minDocCount) {
+        super(name, BucketAggregationMode.PER_BUCKET, factories, estimatedBucketsCount, context, parent);
+        this.order = InternalOrder.validate(order, this);
+        this.requiredSize = requiredSize;
+        this.shardSize = shardSize;
+        this.minDocCount = minDocCount;
+    }
+
+    @Override
+    public boolean shouldCollect() {
+        return true;
+    }
+
+    @Override
+    public InternalAggregation buildEmptyAggregation() {
+        return new StringTerms(name, order, requiredSize, minDocCount, Collections.<InternalTerms.Bucket>emptyList());
+    }
+
+}
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/GlobalOrdinalsStringTermsAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/GlobalOrdinalsStringTermsAggregator.java
new file mode 100644
index 0000000000000..293e1e25a6021
--- /dev/null
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/GlobalOrdinalsStringTermsAggregator.java
@@ -0,0 +1,168 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.search.aggregations.bucket.terms;
+
+import org.apache.lucene.index.AtomicReaderContext;
+import org.apache.lucene.util.ArrayUtil;
+import org.apache.lucene.util.BytesRef;
+import org.apache.lucene.util.RamUsageEstimator;
+import org.elasticsearch.common.lease.Releasables;
+import org.elasticsearch.common.util.LongHash;
+import org.elasticsearch.index.fielddata.BytesValues;
+import org.elasticsearch.index.fielddata.ordinals.Ordinals;
+import org.elasticsearch.search.aggregations.Aggregator;
+import org.elasticsearch.search.aggregations.AggregatorFactories;
+import org.elasticsearch.search.aggregations.InternalAggregation;
+import org.elasticsearch.search.aggregations.bucket.terms.support.BucketPriorityQueue;
+import org.elasticsearch.search.aggregations.support.AggregationContext;
+import org.elasticsearch.search.aggregations.support.ValuesSource;
+
+import java.io.IOException;
+import java.util.Arrays;
+
+/**
+ * An aggregator of string values that relies on global ordinals in order to build buckets.
+ */
+public class GlobalOrdinalsStringTermsAggregator extends AbstractStringTermsAggregator {
+
+    private final ValuesSource.Bytes.WithOrdinals.FieldData valuesSource;
+    private BytesValues.WithOrdinals globalValues;
+    private Ordinals.Docs globalOrdinals;
+
+    public GlobalOrdinalsStringTermsAggregator(String name, AggregatorFactories factories, ValuesSource.Bytes.WithOrdinals.FieldData valuesSource, long estimatedBucketCount,
+            InternalOrder order, int requiredSize, int shardSize, long minDocCount, AggregationContext aggregationContext, Aggregator parent) {
+        super(name, factories, estimatedBucketCount, aggregationContext, parent, order, requiredSize, shardSize, minDocCount);
+        this.valuesSource = valuesSource;
+    }
+
+    protected long createBucketOrd(long termOrd) {
+        return termOrd;
+    }
+
+    protected long getBucketOrd(long termOrd) {
+        return termOrd;
+    }
+
+    @Override
+    public boolean shouldCollect() {
+        return true;
+    }
+
+    @Override
+    public void setNextReader(AtomicReaderContext reader) {
+        globalValues = valuesSource.globalBytesValues();
+        globalOrdinals = globalValues.ordinals();
+    }
+
+    @Override
+    public void collect(int doc, long owningBucketOrdinal) throws IOException {
+        final int numOrds = globalOrdinals.setDocument(doc);
+        for (int i = 0; i < numOrds; i++) {
+            final long globalOrd = globalOrdinals.nextOrd();
+            collectBucket(doc, createBucketOrd(globalOrd));
+        }
+    }
+
+    private static void copy(BytesRef from, BytesRef to) {
+        if (to.bytes.length < from.length) {
+            to.bytes = new byte[ArrayUtil.oversize(from.length, RamUsageEstimator.NUM_BYTES_BYTE)];
+        }
+        to.offset = 0;
+        to.length = from.length;
+        System.arraycopy(from.bytes, from.offset, to.bytes, 0, from.length);
+    }
+
+    @Override
+    public InternalAggregation buildAggregation(long owningBucketOrdinal) {
+        if (globalOrdinals == null) { // no context in this reader
+            return buildEmptyAggregation();
+        }
+
+        final int size;
+        if (minDocCount == 0) {
+            // if minDocCount == 0 then we can end up with more buckets then maxBucketOrd() returns
+            size = (int) Math.min(globalOrdinals.getMaxOrd(), shardSize);
+        } else {
+            size = (int) Math.min(maxBucketOrd(), shardSize);
+        }
+        BucketPriorityQueue ordered = new BucketPriorityQueue(size, order.comparator(this));
+        StringTerms.Bucket spare = null;
+        for (long termOrd = Ordinals.MIN_ORDINAL; termOrd < globalOrdinals.getMaxOrd(); ++termOrd) {
+            final long bucketOrd = getBucketOrd(termOrd);
+            final long bucketDocCount = bucketOrd < 0 ? 0 : bucketDocCount(bucketOrd);
+            if (minDocCount > 0 && bucketDocCount == 0) {
+                continue;
+            }
+            if (spare == null) {
+                spare = new StringTerms.Bucket(new BytesRef(), 0, null);
+            }
+            spare.bucketOrd = bucketOrd;
+            spare.docCount = bucketDocCount;
+            copy(globalValues.getValueByOrd(termOrd), spare.termBytes);
+            spare = (StringTerms.Bucket) ordered.insertWithOverflow(spare);
+        }
+
+        final InternalTerms.Bucket[] list = new InternalTerms.Bucket[ordered.size()];
+        for (int i = ordered.size() - 1; i >= 0; --i) {
+            final StringTerms.Bucket bucket = (StringTerms.Bucket) ordered.pop();
+            bucket.aggregations = bucket.docCount == 0 ? bucketEmptyAggregations() : bucketAggregations(bucket.bucketOrd);
+            list[i] = bucket;
+        }
+
+        return new StringTerms(name, order, requiredSize, minDocCount, Arrays.asList(list));
+    }
+
+    /**
+     * Variant of {@link GlobalOrdinalsStringTermsAggregator} that rebases hashes in order to make them dense. Might be
+     * useful in case few hashes are visited.
+     */
+    public static class WithHash extends GlobalOrdinalsStringTermsAggregator {
+
+        private final LongHash bucketOrds;
+
+        public WithHash(String name, AggregatorFactories factories, ValuesSource.Bytes.WithOrdinals.FieldData valuesSource, long estimatedBucketCount,
+                InternalOrder order, int requiredSize, int shardSize, long minDocCount, AggregationContext aggregationContext,
+                Aggregator parent) {
+            super(name, factories, valuesSource, estimatedBucketCount, order, requiredSize, shardSize, minDocCount, aggregationContext, parent);
+            bucketOrds = new LongHash(estimatedBucketCount, aggregationContext.bigArrays());
+        }
+
+        @Override
+        protected long createBucketOrd(long termOrd) {
+            long bucketOrd = bucketOrds.add(termOrd);
+            if (bucketOrd < 0) {
+                bucketOrd = -1 - bucketOrd;
+            }
+            return bucketOrd;
+        }
+
+        @Override
+        protected long getBucketOrd(long termOrd) {
+            return bucketOrds.find(termOrd);
+        }
+
+        @Override
+        protected void doRelease() {
+            Releasables.release(bucketOrds);
+        }
+
+    }
+
+}
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/StringTermsAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/StringTermsAggregator.java
index 2951f1358fd5b..465a72a81d3af 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/StringTermsAggregator.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/StringTermsAggregator.java
@@ -34,7 +34,6 @@
 import org.elasticsearch.search.aggregations.Aggregator;
 import org.elasticsearch.search.aggregations.AggregatorFactories;
 import org.elasticsearch.search.aggregations.InternalAggregation;
-import org.elasticsearch.search.aggregations.bucket.BucketsAggregator;
 import org.elasticsearch.search.aggregations.bucket.terms.support.BucketPriorityQueue;
 import org.elasticsearch.search.aggregations.bucket.terms.support.IncludeExclude;
 import org.elasticsearch.search.aggregations.support.AggregationContext;
@@ -46,13 +45,9 @@
 /**
  * An aggregator of string values.
  */
-public class StringTermsAggregator extends BucketsAggregator {
+public class StringTermsAggregator extends AbstractStringTermsAggregator {
 
     private final ValuesSource valuesSource;
-    private final InternalOrder order;
-    protected final int requiredSize;
-    protected final int shardSize;
-    protected final long minDocCount;
     protected final BytesRefHash bucketOrds;
     private final IncludeExclude includeExclude;
     private BytesValues values;
@@ -61,12 +56,8 @@ public StringTermsAggregator(String name, AggregatorFactories factories, ValuesS
                                  InternalOrder order, int requiredSize, int shardSize, long minDocCount,
                                  IncludeExclude includeExclude, AggregationContext aggregationContext, Aggregator parent) {
 
-        super(name, BucketAggregationMode.PER_BUCKET, factories, estimatedBucketCount, aggregationContext, parent);
+        super(name, factories, estimatedBucketCount, aggregationContext, parent, order, requiredSize, shardSize, minDocCount);
         this.valuesSource = valuesSource;
-        this.order = InternalOrder.validate(order, this);
-        this.requiredSize = requiredSize;
-        this.shardSize = shardSize;
-        this.minDocCount = minDocCount;
         this.includeExclude = includeExclude;
         bucketOrds = new BytesRefHash(estimatedBucketCount, aggregationContext.bigArrays());
     }
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/TermsAggregatorFactory.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/TermsAggregatorFactory.java
index 2943ea86107d6..c0ea410cae031 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/TermsAggregatorFactory.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/TermsAggregatorFactory.java
@@ -18,12 +18,10 @@
  */
 package org.elasticsearch.search.aggregations.bucket.terms;
 
-import org.apache.lucene.index.AtomicReaderContext;
 import org.elasticsearch.ElasticsearchIllegalArgumentException;
 import org.elasticsearch.common.ParseField;
 import org.elasticsearch.search.aggregations.*;
 import org.elasticsearch.search.aggregations.Aggregator.BucketAggregationMode;
-import org.elasticsearch.search.aggregations.AggregatorFactories;
 import org.elasticsearch.search.aggregations.bucket.terms.support.IncludeExclude;
 import org.elasticsearch.search.aggregations.support.AggregationContext;
 import org.elasticsearch.search.aggregations.support.ValuesSource;
@@ -38,6 +36,7 @@
 public class TermsAggregatorFactory extends ValuesSourceAggregatorFactory {
 
     public enum ExecutionMode {
+
         MAP(new ParseField("map")) {
 
             @Override
@@ -47,6 +46,11 @@ Aggregator create(String name, AggregatorFactories factories, ValuesSource value
                 return new StringTermsAggregator(name, factories, valuesSource, estimatedBucketCount, order, requiredSize, shardSize, minDocCount, includeExclude, aggregationContext, parent);
             }
 
+            @Override
+            boolean needsGlobalOrdinals() {
+                return false;
+            }
+
         },
         ORDINALS(new ParseField("ordinals")) {
 
@@ -60,6 +64,46 @@ Aggregator create(String name, AggregatorFactories factories, ValuesSource value
                 return new StringTermsAggregator.WithOrdinals(name, factories, (ValuesSource.Bytes.WithOrdinals) valuesSource, estimatedBucketCount, order, requiredSize, shardSize, minDocCount, aggregationContext, parent);
             }
 
+            @Override
+            boolean needsGlobalOrdinals() {
+                return false;
+            }
+
+        },
+        GLOBAL_ORDINALS(new ParseField("global_ordinals")) {
+
+            @Override
+            Aggregator create(String name, AggregatorFactories factories, ValuesSource valuesSource, long estimatedBucketCount,
+                              InternalOrder order, int requiredSize, int shardSize, long minDocCount, IncludeExclude includeExclude,
+                              AggregationContext aggregationContext, Aggregator parent) {
+                if (includeExclude != null) {
+                    throw new ElasticsearchIllegalArgumentException("The `" + this + "` execution mode cannot filter terms.");
+                }
+                return new GlobalOrdinalsStringTermsAggregator(name, factories, (ValuesSource.Bytes.WithOrdinals.FieldData) valuesSource, estimatedBucketCount, order, requiredSize, shardSize, minDocCount, aggregationContext, parent);
+            }
+
+            @Override
+            boolean needsGlobalOrdinals() {
+                return true;
+            }
+
+        },
+        GLOBAL_ORDINALS_HASH(new ParseField("global_ordinals_hash")) {
+
+            @Override
+            Aggregator create(String name, AggregatorFactories factories, ValuesSource valuesSource, long estimatedBucketCount,
+                              InternalOrder order, int requiredSize, int shardSize, long minDocCount, IncludeExclude includeExclude,
+                              AggregationContext aggregationContext, Aggregator parent) {
+                if (includeExclude != null) {
+                    throw new ElasticsearchIllegalArgumentException("The `" + this + "` execution mode cannot filter terms.");
+                }
+                return new GlobalOrdinalsStringTermsAggregator.WithHash(name, factories, (ValuesSource.Bytes.WithOrdinals.FieldData) valuesSource, estimatedBucketCount, order, requiredSize, shardSize, minDocCount, aggregationContext, parent);
+            }
+
+            @Override
+            boolean needsGlobalOrdinals() {
+                return true;
+            }
         };
 
         public static ExecutionMode fromString(String value) {
@@ -81,6 +125,8 @@ abstract Aggregator create(String name, AggregatorFactories factories, ValuesSou
                 InternalOrder order, int requiredSize, int shardSize, long minDocCount,
                 IncludeExclude includeExclude, AggregationContext aggregationContext, Aggregator parent);
 
+        abstract boolean needsGlobalOrdinals();
+
         @Override
         public String toString() {
             return parseField.getPreferredName();
@@ -127,31 +173,6 @@ private static boolean hasParentBucketAggregator(Aggregator parent) {
         }
     }
 
-    private boolean shouldUseOrdinals(Aggregator parent, ValuesSource valuesSource, AggregationContext context) {
-        // if there is a parent bucket aggregator the number of instances of this aggregator is going to be unbounded and most instances
-        // may only aggregate few documents, so don't use ordinals
-        if (hasParentBucketAggregator(parent)) {
-            return false;
-        }
-
-        // be defensive: if the number of unique values is unknown, don't use ordinals
-        final long maxNumUniqueValues = valuesSource.metaData().maxAtomicUniqueValuesCount();
-        if (maxNumUniqueValues == -1) {
-            return false;
-        }
-
-        // if the number of unique values is high compared to the document count, then ordinals are only going to make things slower
-        int maxDoc = 0;
-        for (AtomicReaderContext ctx : context.searchContext().searcher().getTopReaderContext().reader().leaves()) {
-            maxDoc = Math.max(maxDoc, ctx.reader().maxDoc());
-        }
-        if (maxNumUniqueValues > (maxDoc >>> 4)) {
-            return false;
-        }
-
-        return true;
-    }
-
     @Override
     protected Aggregator create(ValuesSource valuesSource, long expectedBucketsCount, AggregationContext aggregationContext, Aggregator parent) {
         long estimatedBucketCount = valuesSource.metaData().maxAtomicUniqueValuesCount();
@@ -181,17 +202,19 @@ protected Aggregator create(ValuesSource valuesSource, long expectedBucketsCount
                 execution = ExecutionMode.MAP;
             }
 
+            // Let's try to use a good default
             if (execution == null) {
-                // Let's try to use a good default
-                if ((valuesSource instanceof ValuesSource.Bytes.WithOrdinals)
-                        && shouldUseOrdinals(parent, valuesSource, aggregationContext)) {
-                    execution = ExecutionMode.ORDINALS;
+                // if there is a parent bucket aggregator the number of instances of this aggregator is going
+                // to be unbounded and most instances may only aggregate few documents, so use hashed based
+                // global ordinals to keep the bucket ords dense.
+                if (hasParentBucketAggregator(parent)) {
+                    execution = ExecutionMode.GLOBAL_ORDINALS_HASH;
                 } else {
-                    execution = ExecutionMode.MAP;
+                    execution = ExecutionMode.GLOBAL_ORDINALS;
                 }
             }
             assert execution != null;
-
+            valuesSource.setNeedsGlobalOrdinals(execution.needsGlobalOrdinals());
             return execution.create(name, factories, valuesSource, estimatedBucketCount, order, requiredSize, shardSize, minDocCount, includeExclude, aggregationContext, parent);
         }
 
diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/AggregationContext.java b/src/main/java/org/elasticsearch/search/aggregations/support/AggregationContext.java
index bbbd0ed83af71..8f36d370a29a8 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/support/AggregationContext.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/support/AggregationContext.java
@@ -20,12 +20,14 @@
 
 import com.carrotsearch.hppc.ObjectObjectOpenHashMap;
 import org.apache.lucene.index.AtomicReaderContext;
+import org.apache.lucene.index.IndexReaderContext;
 import org.apache.lucene.search.Scorer;
 import org.apache.lucene.util.ArrayUtil;
 import org.apache.lucene.util.RamUsageEstimator;
 import org.elasticsearch.cache.recycler.PageCacheRecycler;
 import org.elasticsearch.common.lucene.ReaderContextAware;
 import org.elasticsearch.common.lucene.ScorerAware;
+import org.elasticsearch.common.lucene.TopReaderContextAware;
 import org.elasticsearch.common.util.BigArrays;
 import org.elasticsearch.index.fielddata.IndexFieldData;
 import org.elasticsearch.index.fielddata.IndexGeoPointFieldData;
@@ -47,6 +49,7 @@ public class AggregationContext implements ReaderContextAware, ScorerAware {
 
     private ObjectObjectOpenHashMap<ConfigCacheKey, ValuesSource>[] perDepthFieldDataSources = new ObjectObjectOpenHashMap[4];
     private List<ReaderContextAware> readerAwares = new ArrayList<>();
+    private List<TopReaderContextAware> topReaderAwares = new ArrayList<TopReaderContextAware>();
     private List<ScorerAware> scorerAwares = new ArrayList<>();
 
     private AtomicReaderContext reader;
@@ -76,6 +79,12 @@ public Scorer currentScorer() {
         return scorer;
     }
 
+    public void setNextReader(IndexReaderContext reader) {
+        for (TopReaderContextAware topReaderAware : topReaderAwares) {
+            topReaderAware.setNextReader(reader);
+        }
+    }
+
     public void setNextReader(AtomicReaderContext reader) {
         this.reader = reader;
         for (ReaderContextAware aware : readerAwares) {
@@ -177,6 +186,9 @@ private ValuesSource bytesField(ObjectObjectOpenHashMap<ConfigCacheKey, ValuesSo
             }
             setReaderIfNeeded((ReaderContextAware) dataSource);
             readerAwares.add((ReaderContextAware) dataSource);
+            if (dataSource instanceof TopReaderContextAware) {
+                topReaderAwares.add((TopReaderContextAware) dataSource);
+            }
             fieldDataSources.put(cacheKey, dataSource);
         }
         if (config.script != null) {
diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSource.java b/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSource.java
index f39ef35b7ba0b..c07979a16235c 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSource.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSource.java
@@ -19,11 +19,13 @@
 package org.elasticsearch.search.aggregations.support;
 
 import org.apache.lucene.index.AtomicReaderContext;
+import org.apache.lucene.index.IndexReaderContext;
 import org.apache.lucene.util.ArrayUtil;
 import org.apache.lucene.util.BytesRef;
 import org.apache.lucene.util.BytesRefHash;
 import org.apache.lucene.util.InPlaceMergeSorter;
 import org.elasticsearch.common.lucene.ReaderContextAware;
+import org.elasticsearch.common.lucene.TopReaderContextAware;
 import org.elasticsearch.common.util.CollectionUtils;
 import org.elasticsearch.index.fielddata.*;
 import org.elasticsearch.index.fielddata.AtomicFieldData.Order;
@@ -143,22 +145,34 @@ public MetaData build() {
      */
     public void setNeedsHashes(boolean needsHashes) {}
 
+    public void setNeedsGlobalOrdinals(boolean needsGlobalOrdinals) {}
+
     public abstract MetaData metaData();
 
     public static abstract class Bytes extends ValuesSource {
 
-        public static abstract class WithOrdinals extends Bytes {
+        public static abstract class WithOrdinals extends Bytes implements TopReaderContextAware {
 
             public abstract BytesValues.WithOrdinals bytesValues();
 
+            public abstract void setNextReader(IndexReaderContext reader);
+
+            public abstract BytesValues.WithOrdinals globalBytesValues();
+
             public static class FieldData extends WithOrdinals implements ReaderContextAware {
 
                 protected boolean needsHashes;
                 protected final IndexFieldData.WithOrdinals<?> indexFieldData;
                 protected final MetaData metaData;
+                private boolean needsGlobalOrdinals;
+
                 protected AtomicFieldData.WithOrdinals<?> atomicFieldData;
                 private BytesValues.WithOrdinals bytesValues;
 
+                protected IndexFieldData.WithOrdinals<?> globalFieldData;
+                protected AtomicFieldData.WithOrdinals<?> globalAtomicFieldData;
+                private BytesValues.WithOrdinals globalBytesValues;
+
                 public FieldData(IndexFieldData.WithOrdinals<?> indexFieldData, MetaData metaData) {
                     this.indexFieldData = indexFieldData;
                     this.metaData = metaData;
@@ -174,12 +188,23 @@ public final void setNeedsHashes(boolean needsHashes) {
                     this.needsHashes = needsHashes;
                 }
 
+                @Override
+                public void setNeedsGlobalOrdinals(boolean needsGlobalOrdinals) {
+                    this.needsGlobalOrdinals = needsGlobalOrdinals;
+                }
+
                 @Override
                 public void setNextReader(AtomicReaderContext reader) {
                     atomicFieldData = indexFieldData.load(reader);
                     if (bytesValues != null) {
                         bytesValues = atomicFieldData.getBytesValues(needsHashes);
                     }
+                    if (globalFieldData != null) {
+                        globalAtomicFieldData = globalFieldData.load(reader);
+                        if (globalBytesValues != null) {
+                            globalBytesValues = globalAtomicFieldData.getBytesValues(needsHashes);
+                        }
+                    }
                 }
 
                 @Override
@@ -190,6 +215,20 @@ public BytesValues.WithOrdinals bytesValues() {
                     return bytesValues;
                 }
 
+                @Override
+                public void setNextReader(IndexReaderContext reader) {
+                    if (needsGlobalOrdinals) {
+                        globalFieldData = indexFieldData.loadGlobal(reader.reader());
+                    }
+                }
+
+                @Override
+                public BytesValues.WithOrdinals globalBytesValues() {
+                    if (globalBytesValues == null) {
+                        globalBytesValues = globalAtomicFieldData.getBytesValues(needsHashes);
+                    }
+                    return globalBytesValues;
+                }
             }
 
         }
diff --git a/src/test/java/org/elasticsearch/benchmark/search/aggregations/GlobalOrdinalsBenchmark.java b/src/test/java/org/elasticsearch/benchmark/search/aggregations/GlobalOrdinalsBenchmark.java
new file mode 100644
index 0000000000000..2189409307be6
--- /dev/null
+++ b/src/test/java/org/elasticsearch/benchmark/search/aggregations/GlobalOrdinalsBenchmark.java
@@ -0,0 +1,277 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.elasticsearch.benchmark.search.aggregations;
+
+import com.carrotsearch.hppc.IntIntOpenHashMap;
+import com.carrotsearch.hppc.ObjectOpenHashSet;
+import com.carrotsearch.randomizedtesting.generators.RandomStrings;
+import org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse;
+import org.elasticsearch.action.admin.cluster.stats.ClusterStatsResponse;
+import org.elasticsearch.action.bulk.BulkRequestBuilder;
+import org.elasticsearch.action.search.SearchResponse;
+import org.elasticsearch.action.search.SearchType;
+import org.elasticsearch.benchmark.search.aggregations.TermsAggregationSearchBenchmark.StatsResult;
+import org.elasticsearch.client.Client;
+import org.elasticsearch.common.jna.Natives;
+import org.elasticsearch.common.settings.ImmutableSettings;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.common.unit.ByteSizeValue;
+import org.elasticsearch.common.unit.SizeValue;
+import org.elasticsearch.common.unit.TimeValue;
+import org.elasticsearch.common.xcontent.XContentBuilder;
+import org.elasticsearch.discovery.Discovery;
+import org.elasticsearch.index.fielddata.ordinals.InternalGlobalOrdinalsBuilder;
+import org.elasticsearch.indices.IndexAlreadyExistsException;
+import org.elasticsearch.node.internal.InternalNode;
+import org.elasticsearch.search.aggregations.AggregationBuilders;
+import org.elasticsearch.transport.TransportModule;
+
+import java.io.IOException;
+import java.util.*;
+
+import static org.elasticsearch.cluster.metadata.IndexMetaData.SETTING_NUMBER_OF_REPLICAS;
+import static org.elasticsearch.cluster.metadata.IndexMetaData.SETTING_NUMBER_OF_SHARDS;
+import static org.elasticsearch.common.settings.ImmutableSettings.settingsBuilder;
+import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder;
+import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery;
+import static org.elasticsearch.node.NodeBuilder.nodeBuilder;
+
+/**
+ *
+ */
+public class GlobalOrdinalsBenchmark {
+
+    private static final String INDEX_NAME = "index";
+    private static final String TYPE_NAME = "type";
+    private static final int QUERY_WARMUP = 25;
+    private static final int QUERY_COUNT = 100;
+    private static final int FIELD_START = 1;
+    private static final int FIELD_LIMIT = 1 << 22;
+    private static final boolean USE_DOC_VALUES = false;
+
+    static long COUNT = SizeValue.parseSizeValue("5m").singles();
+    static InternalNode node;
+    static Client client;
+
+    public static void main(String[] args) throws Exception {
+        System.setProperty("es.logger.prefix", "");
+        Natives.tryMlockall();
+        Random random = new Random();
+
+        Settings settings = settingsBuilder()
+                .put("index.refresh_interval", "-1")
+                .put("gateway.type", "local")
+                .put(SETTING_NUMBER_OF_SHARDS, 1)
+                .put(SETTING_NUMBER_OF_REPLICAS, 0)
+                .put(TransportModule.TRANSPORT_TYPE_KEY, "local")
+                .build();
+
+        String clusterName = GlobalOrdinalsBenchmark.class.getSimpleName();
+        node = (InternalNode) nodeBuilder().clusterName(clusterName)
+                    .settings(settingsBuilder().put(settings))
+                    .node();
+
+        client = node.client();
+
+        try {
+            client.admin().indices().prepareCreate(INDEX_NAME)
+                    .addMapping(TYPE_NAME, jsonBuilder().startObject().startObject(TYPE_NAME)
+                            .startArray("dynamic_templates")
+                                .startObject()
+                                    .startObject("default")
+                                        .field("match", "*")
+                                        .field("match_mapping_type", "string")
+                                        .startObject("mapping")
+                                            .field("type", "string")
+                                            .field("index", "not_analyzed")
+                                            .startObject("fields")
+                                                .startObject("doc_values")
+                                                    .field("type", "string")
+                                                    .field("index", "no")
+                                                    .startObject("fielddata")
+                                                        .field("format", "doc_values")
+                                                    .endObject()
+                                                .endObject()
+                                            .endObject()
+                                        .endObject()
+                                    .endObject()
+                                .endObject()
+                            .endArray()
+                    .endObject().endObject())
+                    .get();
+            ObjectOpenHashSet<String> uniqueTerms = ObjectOpenHashSet.newInstance();
+            for (int i = 0; i < FIELD_LIMIT; i++) {
+                boolean added;
+                do {
+                    added = uniqueTerms.add(RandomStrings.randomAsciiOfLength(random, 16));
+                } while (!added);
+            }
+            String[] sValues = uniqueTerms.toArray(String.class);
+            uniqueTerms = null;
+
+            BulkRequestBuilder builder = client.prepareBulk();
+            IntIntOpenHashMap tracker = new IntIntOpenHashMap();
+            for (int i = 0; i < COUNT; i++) {
+                Map<String, Object> fieldValues = new HashMap<>();
+                for (int fieldSuffix = 1; fieldSuffix <= FIELD_LIMIT; fieldSuffix <<= 1) {
+                    int index;
+                    if (tracker.containsKey(fieldSuffix)) {
+                        index = tracker.lget();
+                    } else {
+                        tracker.put(fieldSuffix, index = 0);
+                    }
+                    if (index >= fieldSuffix) {
+                        index = random.nextInt(fieldSuffix);
+                        fieldValues.put("field_" + fieldSuffix, sValues[index]);
+                    } else {
+                        fieldValues.put("field_" + fieldSuffix, sValues[index]);
+                        tracker.put(fieldSuffix, ++index);
+                    }
+                }
+                builder.add(
+                        client.prepareIndex(INDEX_NAME, TYPE_NAME, String.valueOf(i))
+                        .setSource(fieldValues)
+                );
+
+                if (builder.numberOfActions() >= 1000) {
+                    builder.get();
+                    builder = client.prepareBulk();
+                }
+            }
+            if (builder.numberOfActions() > 0) {
+                builder.get();
+            }
+        } catch (IndexAlreadyExistsException e) {
+            System.out.println("--> Index already exists, ignoring indexing phase, waiting for green");
+            ClusterHealthResponse clusterHealthResponse = client.admin().cluster().prepareHealth().setWaitForGreenStatus().setTimeout("10m").execute().actionGet();
+            if (clusterHealthResponse.isTimedOut()) {
+                System.err.println("--> Timed out waiting for cluster health");
+            }
+        }
+
+        client.admin().cluster().prepareUpdateSettings()
+                .setTransientSettings(ImmutableSettings.builder().put("logger.index.fielddata.ordinals", "DEBUG"))
+                .get();
+
+        client.admin().indices().prepareRefresh(INDEX_NAME).execute().actionGet();
+        COUNT = client.prepareCount(INDEX_NAME).setQuery(matchAllQuery()).execute().actionGet().getCount();
+        System.out.println("--> Number of docs in index: " + COUNT);
+
+        List<StatsResult> stats = new ArrayList<>();
+        int[] thresholds = new int[]{2048};
+        for (int threshold : thresholds) {
+            updateThresholdInMapping(threshold);
+
+            for (int fieldSuffix = FIELD_START; fieldSuffix <= FIELD_LIMIT; fieldSuffix <<= 1) {
+                String fieldName = "field_" + fieldSuffix;
+                String name = threshold + "-" + fieldName;
+                if (USE_DOC_VALUES) {
+                    fieldName = fieldName + ".doc_values";
+                    name = name + "_doc_values"; // can't have . in agg name
+                }
+                stats.add(terms(name, fieldName, "global_ordinals"));
+            }
+        }
+
+        for (int fieldSuffix = FIELD_START; fieldSuffix <= FIELD_LIMIT; fieldSuffix <<= 1) {
+            String fieldName = "field_" + fieldSuffix;
+            String name = "segment-ordinals-" + fieldName;
+            if (USE_DOC_VALUES) {
+                fieldName = fieldName + ".doc_values";
+                name = name + "_doc_values"; // can't have . in agg name
+            }
+            stats.add(terms(name, fieldName, "ordinals"));
+        }
+
+        System.out.println("------------------ SUMMARY ----------------------------------------------");
+        System.out.format(Locale.ENGLISH, "%40s%10s%10s%15s\n", "name", "took", "millis", "fieldata size");
+        for (StatsResult stat : stats) {
+            System.out.format(Locale.ENGLISH, "%40s%10s%10d%15s\n", stat.name, TimeValue.timeValueMillis(stat.took), (stat.took / QUERY_COUNT), stat.fieldDataMemoryUsed);
+        }
+        System.out.println("------------------ SUMMARY ----------------------------------------------");
+
+        client.close();
+        node.close();
+    }
+
+    private static void updateThresholdInMapping(int threshold) throws IOException {
+        XContentBuilder builder = jsonBuilder().startObject().startObject(TYPE_NAME).startObject("properties");
+        for (int fieldSuffix = FIELD_START; fieldSuffix <= FIELD_LIMIT; fieldSuffix <<= 1) {
+            builder.startObject("field_" + fieldSuffix)
+                    .field("type", "string")
+                    .field("index", "not_analyzed")
+                    .startObject("fielddata")
+                        .field(InternalGlobalOrdinalsBuilder.ORDINAL_MAPPING_THRESHOLD_KEY, threshold)
+                    .endObject()
+            .endObject();
+        }
+        builder = builder.endObject().endObject().endObject();
+
+        client.admin().indices().preparePutMapping(INDEX_NAME).setType(TYPE_NAME).setSource(builder).get();
+    }
+
+    private static StatsResult terms(String name, String field, String executionHint) {
+        long totalQueryTime;// LM VALUE
+
+        client.admin().indices().prepareClearCache().setFieldDataCache(true).execute().actionGet();
+        System.gc();
+
+        System.out.println("--> Warmup (" + name + ")...");
+        // run just the child query, warm up first
+        for (int j = 0; j < QUERY_WARMUP; j++) {
+            SearchResponse searchResponse = client.prepareSearch(INDEX_NAME)
+                    .setSearchType(SearchType.COUNT)
+                    .setQuery(matchAllQuery())
+                    .addAggregation(AggregationBuilders.terms(name).field(field).executionHint(executionHint))
+                    .get();
+            if (j == 0) {
+                System.out.println("--> Loading (" + field + "): took: " + searchResponse.getTook());
+            }
+            if (searchResponse.getHits().totalHits() != COUNT) {
+                System.err.println("--> mismatch on hits");
+            }
+        }
+        System.out.println("--> Warmup (" + name + ") DONE");
+
+
+        System.out.println("--> Running (" + name + ")...");
+        totalQueryTime = 0;
+        for (int j = 0; j < QUERY_COUNT; j++) {
+            SearchResponse searchResponse = client.prepareSearch(INDEX_NAME)
+                    .setSearchType(SearchType.COUNT)
+                    .setQuery(matchAllQuery())
+                    .addAggregation(AggregationBuilders.terms(name).field(field).executionHint(executionHint))
+                    .get();
+            if (searchResponse.getHits().totalHits() != COUNT) {
+                System.err.println("--> mismatch on hits");
+            }
+            totalQueryTime += searchResponse.getTookInMillis();
+        }
+        System.out.println("--> Terms Agg (" + name + "): " + (totalQueryTime / QUERY_COUNT) + "ms");
+
+        String nodeId = node.injector().getInstance(Discovery.class).localNode().getId();
+        ClusterStatsResponse clusterStateResponse = client.admin().cluster().prepareClusterStats().setNodesIds(nodeId).get();
+        System.out.println("--> Heap used: " + clusterStateResponse.getNodesStats().getJvm().getHeapUsed());
+        ByteSizeValue fieldDataMemoryUsed = clusterStateResponse.getIndicesStats().getFieldData().getMemorySize();
+        System.out.println("--> Fielddata memory size: " + fieldDataMemoryUsed);
+
+        return new StatsResult(name, totalQueryTime, fieldDataMemoryUsed);
+    }
+
+}
diff --git a/src/test/java/org/elasticsearch/benchmark/search/aggregations/TermsAggregationSearchAndIndexingBenchmark.java b/src/test/java/org/elasticsearch/benchmark/search/aggregations/TermsAggregationSearchAndIndexingBenchmark.java
new file mode 100644
index 0000000000000..f88665c47288d
--- /dev/null
+++ b/src/test/java/org/elasticsearch/benchmark/search/aggregations/TermsAggregationSearchAndIndexingBenchmark.java
@@ -0,0 +1,354 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.elasticsearch.benchmark.search.aggregations;
+
+import com.carrotsearch.hppc.ObjectOpenHashSet;
+import com.carrotsearch.randomizedtesting.generators.RandomStrings;
+import jsr166y.ThreadLocalRandom;
+import org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse;
+import org.elasticsearch.action.admin.cluster.stats.ClusterStatsResponse;
+import org.elasticsearch.action.bulk.BulkRequestBuilder;
+import org.elasticsearch.action.bulk.BulkResponse;
+import org.elasticsearch.action.get.GetResponse;
+import org.elasticsearch.action.search.SearchResponse;
+import org.elasticsearch.action.search.SearchType;
+import org.elasticsearch.client.Client;
+import org.elasticsearch.client.Requests;
+import org.elasticsearch.common.jna.Natives;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.common.unit.ByteSizeValue;
+import org.elasticsearch.common.unit.SizeValue;
+import org.elasticsearch.common.unit.TimeValue;
+import org.elasticsearch.common.xcontent.XContentBuilder;
+import org.elasticsearch.discovery.Discovery;
+import org.elasticsearch.indices.IndexAlreadyExistsException;
+import org.elasticsearch.node.internal.InternalNode;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Locale;
+import java.util.Random;
+
+import static org.elasticsearch.benchmark.search.aggregations.TermsAggregationSearchBenchmark.Method;
+import static org.elasticsearch.cluster.metadata.IndexMetaData.SETTING_NUMBER_OF_REPLICAS;
+import static org.elasticsearch.cluster.metadata.IndexMetaData.SETTING_NUMBER_OF_SHARDS;
+import static org.elasticsearch.common.settings.ImmutableSettings.settingsBuilder;
+import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder;
+import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery;
+import static org.elasticsearch.node.NodeBuilder.nodeBuilder;
+
+/**
+ *
+ */
+public class TermsAggregationSearchAndIndexingBenchmark {
+
+    static String indexName = "test";
+    static String typeName = "type1";
+    static Random random = new Random();
+
+    static long COUNT = SizeValue.parseSizeValue("2m").singles();
+    static int BATCH = 1000;
+    static int NUMBER_OF_TERMS = (int) SizeValue.parseSizeValue("100k").singles();
+    static int NUMBER_OF_MULTI_VALUE_TERMS = 10;
+    static int STRING_TERM_SIZE = 5;
+
+    static InternalNode[] nodes;
+
+    public static void main(String[] args) throws Exception {
+        Natives.tryMlockall();
+        Settings settings = settingsBuilder()
+                .put("refresh_interval", "-1")
+                .put(SETTING_NUMBER_OF_SHARDS, 1)
+                .put(SETTING_NUMBER_OF_REPLICAS, 0)
+                .build();
+
+        String clusterName = TermsAggregationSearchAndIndexingBenchmark.class.getSimpleName();
+        nodes = new InternalNode[1];
+        for (int i = 0; i < nodes.length; i++) {
+            nodes[i] = (InternalNode) nodeBuilder().settings(settingsBuilder().put(settings).put("name", "node1"))
+                    .clusterName(clusterName)
+                    .node();
+        }
+        Client client = nodes[0].client();
+
+        client.admin().cluster().prepareHealth(indexName).setWaitForGreenStatus().setTimeout("10s").execute().actionGet();
+        try {
+            client.admin().indices().prepareCreate(indexName)
+                    .addMapping(typeName, generateMapping("eager", "lazy"))
+                    .get();
+            Thread.sleep(5000);
+
+            long startTime = System.currentTimeMillis();
+            ObjectOpenHashSet<String> uniqueTerms = ObjectOpenHashSet.newInstance();
+            for (int i = 0; i < NUMBER_OF_TERMS; i++) {
+                boolean added;
+                do {
+                    added = uniqueTerms.add(RandomStrings.randomAsciiOfLength(random, STRING_TERM_SIZE));
+                } while (!added);
+            }
+            String[] sValues = uniqueTerms.toArray(String.class);
+            long ITERS = COUNT / BATCH;
+            long i = 1;
+            int counter = 0;
+            for (; i <= ITERS; i++) {
+                BulkRequestBuilder request = client.prepareBulk();
+                for (int j = 0; j < BATCH; j++) {
+                    counter++;
+
+                    XContentBuilder builder = jsonBuilder().startObject();
+                    builder.field("id", Integer.toString(counter));
+                    final String sValue = sValues[counter % sValues.length];
+                    builder.field("s_value", sValue);
+                    builder.field("s_value_dv", sValue);
+
+                    for (String field : new String[] {"sm_value", "sm_value_dv"}) {
+                        builder.startArray(field);
+                        for (int k = 0; k < NUMBER_OF_MULTI_VALUE_TERMS; k++) {
+                            builder.value(sValues[ThreadLocalRandom.current().nextInt(sValues.length)]);
+                        }
+                        builder.endArray();
+                    }
+
+                    request.add(Requests.indexRequest(indexName).type("type1").id(Integer.toString(counter))
+                            .source(builder));
+                }
+                BulkResponse response = request.execute().actionGet();
+                if (response.hasFailures()) {
+                    System.err.println("--> failures...");
+                }
+                if (((i * BATCH) % 10000) == 0) {
+                    System.out.println("--> Indexed " + (i * BATCH));
+                }
+            }
+
+            System.out.println("--> Indexing took " + ((System.currentTimeMillis() - startTime) / 1000) + " seconds.");
+        } catch (IndexAlreadyExistsException e) {
+            System.out.println("--> Index already exists, ignoring indexing phase, waiting for green");
+            ClusterHealthResponse clusterHealthResponse = client.admin().cluster().prepareHealth(indexName).setWaitForGreenStatus().setTimeout("10m").execute().actionGet();
+            if (clusterHealthResponse.isTimedOut()) {
+                System.err.println("--> Timed out waiting for cluster health");
+            }
+        }
+        client.admin().indices().preparePutMapping(indexName)
+                .setType(typeName)
+                .setSource(generateMapping("lazy", "lazy"))
+                .get();
+        client.admin().indices().prepareRefresh().execute().actionGet();
+        System.out.println("--> Number of docs in index: " + client.prepareCount().setQuery(matchAllQuery()).execute().actionGet().getCount());
+
+
+        String[] nodeIds = new String[nodes.length];
+        for (int i = 0; i < nodeIds.length; i++) {
+            nodeIds[i] = nodes[i].injector().getInstance(Discovery.class).localNode().getId();
+        }
+
+        List<TestRun> testRuns = new ArrayList<>();
+        testRuns.add(new TestRun("Regular field ordinals", "eager", "lazy", "s_value", "ordinals"));
+        testRuns.add(new TestRun("Docvalues field ordinals", "lazy", "eager", "s_value_dv", "ordinals"));
+        testRuns.add(new TestRun("Regular field global ordinals", "eager_global_ordinals", "lazy", "s_value", null));
+        testRuns.add(new TestRun("Docvalues field global", "lazy", "eager_global_ordinals", "s_value_dv", null));
+
+        List<TestResult> testResults = new ArrayList<>();
+        for (TestRun testRun : testRuns) {
+            client.admin().indices().preparePutMapping(indexName).setType(typeName)
+                    .setSource(generateMapping(testRun.indexedFieldEagerLoading, testRun.docValuesEagerLoading)).get();
+            client.admin().indices().prepareClearCache(indexName).setFieldDataCache(true).get();
+            SearchThread searchThread = new SearchThread(client, testRun.termsAggsField, testRun.termsAggsExecutionHint);
+            RefreshThread refreshThread = new RefreshThread(client);
+            System.out.println("--> Running '" + testRun.name + "' round...");
+            new Thread(refreshThread).start();
+            new Thread(searchThread).start();
+            Thread.sleep(2 * 60 * 1000);
+            refreshThread.stop();
+            searchThread.stop();
+
+            System.out.println("--> Avg refresh time: " + refreshThread.avgRefreshTime + " ms");
+            System.out.println("--> Avg query time: " + searchThread.avgQueryTime + " ms");
+
+            ClusterStatsResponse clusterStateResponse = client.admin().cluster().prepareClusterStats().setNodesIds(nodeIds).get();
+            System.out.println("--> Heap used: " + clusterStateResponse.getNodesStats().getJvm().getHeapUsed());
+            ByteSizeValue fieldDataMemoryUsed = clusterStateResponse.getIndicesStats().getFieldData().getMemorySize();
+            System.out.println("--> Fielddata memory size: " + fieldDataMemoryUsed);
+            testResults.add(new TestResult(testRun.name, refreshThread.avgRefreshTime, searchThread.avgQueryTime, fieldDataMemoryUsed));
+        }
+
+        System.out.println("----------------------------------------- SUMMARY ----------------------------------------------");
+        System.out.format(Locale.ENGLISH, "%30s%18s%15s%15s\n", "name", "avg refresh time", "avg query time", "fieldata size");
+        for (TestResult testResult : testResults) {
+            System.out.format(Locale.ENGLISH, "%30s%18s%15s%15s\n", testResult.name, testResult.avgRefreshTime, testResult.avgQueryTime, testResult.fieldDataSizeInMemory);
+        }
+        System.out.println("----------------------------------------- SUMMARY ----------------------------------------------");
+
+        client.close();
+        for (InternalNode node : nodes) {
+            node.close();
+        }
+    }
+
+    static class RefreshThread implements Runnable {
+
+        private final Client client;
+        private volatile boolean run = true;
+        private volatile boolean stopped = false;
+        private volatile long avgRefreshTime = 0;
+
+        RefreshThread(Client client) throws IOException {
+            this.client = client;
+        }
+
+        @Override
+        public void run() {
+            long totalRefreshTime = 0;
+            int numExecutedRefreshed = 0;
+            while (run) {
+                long docIdLimit = COUNT;
+                for (long docId = 1; run && docId < docIdLimit;) {
+                    try {
+                        for (int j = 0; j < 8; j++) {
+                            GetResponse getResponse = client
+                                    .prepareGet(indexName, "type1", String.valueOf(++docId))
+                                    .get();
+                            client.prepareIndex(indexName, "type1", getResponse.getId())
+                                    .setSource(getResponse.getSource())
+                                    .get();
+                        }
+                        long startTime = System.currentTimeMillis();
+                        client.admin().indices().prepareRefresh(indexName).execute().actionGet();
+                        totalRefreshTime += System.currentTimeMillis() - startTime;
+                        numExecutedRefreshed++;
+                        Thread.sleep(500);
+                    } catch (Throwable e) {
+                        e.printStackTrace();
+                    }
+                }
+            }
+            avgRefreshTime = totalRefreshTime / numExecutedRefreshed;
+            stopped = true;
+        }
+
+        public void stop() throws InterruptedException {
+            run = false;
+            while (!stopped) {
+                Thread.sleep(100);
+            }
+        }
+
+    }
+
+    private static class TestRun {
+
+        final String name;
+        final String indexedFieldEagerLoading;
+        final String docValuesEagerLoading;
+        final String termsAggsField;
+        final String termsAggsExecutionHint;
+
+        private TestRun(String name, String indexedFieldEagerLoading, String docValuesEagerLoading, String termsAggsField, String termsAggsExecutionHint) {
+            this.name = name;
+            this.indexedFieldEagerLoading = indexedFieldEagerLoading;
+            this.docValuesEagerLoading = docValuesEagerLoading;
+            this.termsAggsField = termsAggsField;
+            this.termsAggsExecutionHint = termsAggsExecutionHint;
+        }
+    }
+
+    private static class TestResult {
+
+        final String name;
+        final TimeValue avgRefreshTime;
+        final TimeValue avgQueryTime;
+        final ByteSizeValue fieldDataSizeInMemory;
+
+        private TestResult(String name, long avgRefreshTime, long avgQueryTime, ByteSizeValue fieldDataSizeInMemory) {
+            this.name = name;
+            this.avgRefreshTime = TimeValue.timeValueMillis(avgRefreshTime);
+            this.avgQueryTime = TimeValue.timeValueMillis(avgQueryTime);
+            this.fieldDataSizeInMemory = fieldDataSizeInMemory;
+        }
+    }
+
+    static class SearchThread implements Runnable {
+
+        private final Client client;
+        private final String field;
+        private final String executionHint;
+        private volatile boolean run = true;
+        private volatile boolean stopped = false;
+        private volatile long avgQueryTime = 0;
+
+        SearchThread(Client client, String field, String executionHint) {
+            this.client = client;
+            this.field = field;
+            this.executionHint = executionHint;
+        }
+
+        @Override
+        public void run() {
+            long totalQueryTime = 0;
+            int numExecutedQueries = 0;
+            while (run) {
+                try {
+                    SearchResponse searchResponse = Method.AGGREGATION.addTermsAgg(client.prepareSearch()
+                            .setSearchType(SearchType.COUNT)
+                            .setQuery(matchAllQuery()), "test", field, executionHint)
+                            .execute().actionGet();
+                    if (searchResponse.getHits().totalHits() != COUNT) {
+                        System.err.println("--> mismatch on hits");
+                    }
+                    totalQueryTime += searchResponse.getTookInMillis();
+                    numExecutedQueries++;
+                } catch (Throwable e) {
+                    e.printStackTrace();
+                }
+            }
+            avgQueryTime = totalQueryTime / numExecutedQueries;
+            stopped = true;
+        }
+
+        public void stop() throws InterruptedException {
+            run = false;
+            while (!stopped) {
+                Thread.sleep(100);
+            }
+        }
+
+    }
+
+    private static XContentBuilder generateMapping(String loading1, String loading2) throws IOException {
+        return jsonBuilder().startObject().startObject("type1").startObject("properties")
+                .startObject("s_value")
+                .field("type", "string")
+                .field("index", "not_analyzed")
+                .startObject("fielddata")
+                .field("loading", loading1)
+                .endObject()
+                .endObject()
+                .startObject("s_value_dv")
+                .field("type", "string")
+                .field("index", "no")
+                .startObject("fielddata")
+                .field("loading", loading2)
+                .field("format", "doc_values")
+                .endObject()
+                .endObject()
+                .endObject().endObject().endObject();
+    }
+
+}
diff --git a/src/test/java/org/elasticsearch/benchmark/search/aggregations/TermsAggregationSearchBenchmark.java b/src/test/java/org/elasticsearch/benchmark/search/aggregations/TermsAggregationSearchBenchmark.java
index 53887fb8a7941..76540fc55b2fe 100644
--- a/src/test/java/org/elasticsearch/benchmark/search/aggregations/TermsAggregationSearchBenchmark.java
+++ b/src/test/java/org/elasticsearch/benchmark/search/aggregations/TermsAggregationSearchBenchmark.java
@@ -18,10 +18,12 @@
  */
 package org.elasticsearch.benchmark.search.aggregations;
 
+import com.carrotsearch.hppc.ObjectOpenHashSet;
 import com.carrotsearch.randomizedtesting.generators.RandomStrings;
 import com.google.common.collect.Lists;
 import jsr166y.ThreadLocalRandom;
 import org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse;
+import org.elasticsearch.action.admin.cluster.stats.ClusterStatsResponse;
 import org.elasticsearch.action.bulk.BulkRequestBuilder;
 import org.elasticsearch.action.bulk.BulkResponse;
 import org.elasticsearch.action.search.SearchRequestBuilder;
@@ -30,11 +32,15 @@
 import org.elasticsearch.client.Client;
 import org.elasticsearch.client.Requests;
 import org.elasticsearch.common.StopWatch;
+import org.elasticsearch.common.jna.Natives;
 import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.common.unit.ByteSizeValue;
 import org.elasticsearch.common.unit.SizeValue;
 import org.elasticsearch.common.unit.TimeValue;
 import org.elasticsearch.common.xcontent.XContentBuilder;
+import org.elasticsearch.discovery.Discovery;
 import org.elasticsearch.node.Node;
+import org.elasticsearch.node.internal.InternalNode;
 import org.elasticsearch.search.aggregations.AggregationBuilders;
 
 import java.util.List;
@@ -65,8 +71,9 @@ public class TermsAggregationSearchBenchmark {
     static int STRING_TERM_SIZE = 5;
 
     static Client client;
+    static InternalNode[] nodes;
 
-    private enum Method {
+    public enum Method {
         FACET {
             @Override
             SearchRequestBuilder addTermsAgg(SearchRequestBuilder builder, String name, String field, String executionHint) {
@@ -94,6 +101,7 @@ SearchRequestBuilder addTermsStatsAgg(SearchRequestBuilder builder, String name,
     }
 
     public static void main(String[] args) throws Exception {
+        Natives.tryMlockall();
         Random random = new Random();
 
         Settings settings = settingsBuilder()
@@ -104,9 +112,9 @@ public static void main(String[] args) throws Exception {
                 .build();
 
         String clusterName = TermsAggregationSearchBenchmark.class.getSimpleName();
-        Node[] nodes = new Node[1];
+        nodes = new InternalNode[1];
         for (int i = 0; i < nodes.length; i++) {
-            nodes[i] = nodeBuilder().clusterName(clusterName)
+            nodes[i] = (InternalNode) nodeBuilder().clusterName(clusterName)
                     .settings(settingsBuilder().put(settings).put("name", "node" + i))
                     .node();
         }
@@ -117,15 +125,6 @@ public static void main(String[] args) throws Exception {
 
         client = clientNode.client();
 
-        long[] lValues = new long[NUMBER_OF_TERMS];
-        for (int i = 0; i < NUMBER_OF_TERMS; i++) {
-            lValues[i] = ThreadLocalRandom.current().nextLong();
-        }
-        String[] sValues = new String[NUMBER_OF_TERMS];
-        for (int i = 0; i < NUMBER_OF_TERMS; i++) {
-            sValues[i] = RandomStrings.randomAsciiOfLength(random, STRING_TERM_SIZE);
-        }
-
         Thread.sleep(10000);
         try {
             client.admin().indices().create(createIndexRequest("test").mapping("type1", jsonBuilder()
@@ -164,6 +163,20 @@ public static void main(String[] args) throws Exception {
                 .endObject()
               .endObject())).actionGet();
 
+            long[] lValues = new long[NUMBER_OF_TERMS];
+            for (int i = 0; i < NUMBER_OF_TERMS; i++) {
+                lValues[i] = ThreadLocalRandom.current().nextLong();
+            }
+            ObjectOpenHashSet<String> uniqueTerms = ObjectOpenHashSet.newInstance();
+            for (int i = 0; i < NUMBER_OF_TERMS; i++) {
+                boolean added;
+                do {
+                    added = uniqueTerms.add(RandomStrings.randomAsciiOfLength(random, STRING_TERM_SIZE));
+                } while (!added);
+            }
+            String[] sValues = uniqueTerms.toArray(String.class);
+            uniqueTerms = null;
+
             StopWatch stopWatch = new StopWatch().start();
 
             System.out.println("--> Indexing [" + COUNT + "] ...");
@@ -177,8 +190,8 @@ public static void main(String[] args) throws Exception {
 
                     XContentBuilder builder = jsonBuilder().startObject();
                     builder.field("id", Integer.toString(counter));
-                    final String sValue = sValues[counter % sValues.length];
-                    final long lValue = lValues[counter % lValues.length];
+                    final String sValue = sValues[ThreadLocalRandom.current().nextInt(sValues.length)];
+                    final long lValue = lValues[ThreadLocalRandom.current().nextInt(lValues.length)];
                     builder.field("s_value", sValue);
                     builder.field("l_value", lValue);
                     builder.field("s_value_dv", sValue);
@@ -233,7 +246,9 @@ public static void main(String[] args) throws Exception {
         stats.add(terms("terms_facet_map_s", Method.FACET, "s_value", "map"));
         stats.add(terms("terms_facet_map_s_dv", Method.FACET, "s_value_dv", "map"));
         stats.add(terms("terms_agg_s", Method.AGGREGATION, "s_value", null));
+        stats.add(terms("terms_agg_s_local_ordinals", Method.AGGREGATION, "s_value", "ordinals"));
         stats.add(terms("terms_agg_s_dv", Method.AGGREGATION, "s_value_dv", null));
+        stats.add(terms("terms_agg_s_dv_local_ordinals", Method.AGGREGATION, "s_value_dv", "ordinals"));
         stats.add(terms("terms_agg_map_s", Method.AGGREGATION, "s_value", "map"));
         stats.add(terms("terms_agg_map_s_dv", Method.AGGREGATION, "s_value_dv", "map"));
         stats.add(terms("terms_facet_l", Method.FACET, "l_value", null));
@@ -245,7 +260,9 @@ public static void main(String[] args) throws Exception {
         stats.add(terms("terms_facet_map_sm", Method.FACET, "sm_value", "map"));
         stats.add(terms("terms_facet_map_sm_dv", Method.FACET, "sm_value_dv", "map"));
         stats.add(terms("terms_agg_sm", Method.AGGREGATION, "sm_value", null));
+        stats.add(terms("terms_agg_sm_local_ordinals", Method.AGGREGATION, "sm_value", "ordinals"));
         stats.add(terms("terms_agg_sm_dv", Method.AGGREGATION, "sm_value_dv", null));
+        stats.add(terms("terms_agg_sm_dv_local_ordinals", Method.AGGREGATION, "sm_value_dv", "ordinals"));
         stats.add(terms("terms_agg_map_sm", Method.AGGREGATION, "sm_value", "map"));
         stats.add(terms("terms_agg_map_sm_dv", Method.AGGREGATION, "sm_value_dv", "map"));
         stats.add(terms("terms_facet_lm", Method.FACET, "lm_value", null));
@@ -266,12 +283,25 @@ public static void main(String[] args) throws Exception {
         stats.add(termsStats("terms_stats_agg_sm_l", Method.AGGREGATION, "sm_value", "l_value", null));
         stats.add(termsStats("terms_stats_agg_sm_l_dv", Method.AGGREGATION, "sm_value_dv", "l_value_dv", null));
 
-        System.out.println("------------------ SUMMARY -------------------------------");
-        System.out.format(Locale.ENGLISH, "%25s%10s%10s\n", "name", "took", "millis");
+        stats.add(termsStats("terms_stats_facet_s_l", Method.FACET, "s_value", "l_value", null));
+        stats.add(termsStats("terms_stats_facet_s_l_dv", Method.FACET, "s_value_dv", "l_value_dv", null));
+        stats.add(termsStats("terms_stats_agg_s_l", Method.AGGREGATION, "s_value", "l_value", null));
+        stats.add(termsStats("terms_stats_agg_s_l_dv", Method.AGGREGATION, "s_value_dv", "l_value_dv", null));
+        stats.add(termsStats("terms_stats_facet_s_lm", Method.FACET, "s_value", "lm_value", null));
+        stats.add(termsStats("terms_stats_facet_s_lm_dv", Method.FACET, "s_value_dv", "lm_value_dv", null));
+        stats.add(termsStats("terms_stats_agg_s_lm", Method.AGGREGATION, "s_value", "lm_value", null));
+        stats.add(termsStats("terms_stats_agg_s_lm_dv", Method.AGGREGATION, "s_value_dv", "lm_value_dv", null));
+        stats.add(termsStats("terms_stats_facet_sm_l", Method.FACET, "sm_value", "l_value", null));
+        stats.add(termsStats("terms_stats_facet_sm_l_dv", Method.FACET, "sm_value_dv", "l_value_dv", null));
+        stats.add(termsStats("terms_stats_agg_sm_l", Method.AGGREGATION, "sm_value", "l_value", null));
+        stats.add(termsStats("terms_stats_agg_sm_l_dv", Method.AGGREGATION, "sm_value_dv", "l_value_dv", null));
+        
+        System.out.println("------------------ SUMMARY ----------------------------------------------");
+        System.out.format(Locale.ENGLISH, "%35s%10s%10s%15s\n", "name", "took", "millis", "fieldata size");
         for (StatsResult stat : stats) {
-            System.out.format(Locale.ENGLISH, "%25s%10s%10d\n", stat.name, TimeValue.timeValueMillis(stat.took), (stat.took / QUERY_COUNT));
+            System.out.format(Locale.ENGLISH, "%35s%10s%10d%15s\n", stat.name, TimeValue.timeValueMillis(stat.took), (stat.took / QUERY_COUNT), stat.fieldDataMemoryUsed);
         }
-        System.out.println("------------------ SUMMARY -------------------------------");
+        System.out.println("------------------ SUMMARY ----------------------------------------------");
 
         clientNode.close();
 
@@ -280,13 +310,15 @@ public static void main(String[] args) throws Exception {
         }
     }
 
-    static class StatsResult {
+    public static class StatsResult {
         final String name;
         final long took;
+        final ByteSizeValue fieldDataMemoryUsed;
 
-        StatsResult(String name, long took) {
+        public StatsResult(String name, long took, ByteSizeValue fieldDataMemoryUsed) {
             this.name = name;
             this.took = took;
+            this.fieldDataMemoryUsed = fieldDataMemoryUsed;
         }
     }
 
@@ -294,11 +326,12 @@ private static StatsResult terms(String name, Method method, String field, Strin
         long totalQueryTime;// LM VALUE
 
         client.admin().indices().prepareClearCache().setFieldDataCache(true).execute().actionGet();
+        System.gc();
 
         System.out.println("--> Warmup (" + name + ")...");
         // run just the child query, warm up first
         for (int j = 0; j < QUERY_WARMUP; j++) {
-            SearchResponse searchResponse = method.addTermsAgg(client.prepareSearch()
+            SearchResponse searchResponse = method.addTermsAgg(client.prepareSearch("test")
                     .setSearchType(SearchType.COUNT)
                     .setQuery(matchAllQuery()), name, field, executionHint)
                     .execute().actionGet();
@@ -325,13 +358,25 @@ private static StatsResult terms(String name, Method method, String field, Strin
             totalQueryTime += searchResponse.getTookInMillis();
         }
         System.out.println("--> Terms Agg (" + name + "): " + (totalQueryTime / QUERY_COUNT) + "ms");
-        return new StatsResult(name, totalQueryTime);
+
+        String[] nodeIds = new String[nodes.length];
+        for (int i = 0; i < nodeIds.length; i++) {
+            nodeIds[i] = nodes[i].injector().getInstance(Discovery.class).localNode().getId();
+        }
+
+        ClusterStatsResponse clusterStateResponse = client.admin().cluster().prepareClusterStats().setNodesIds(nodeIds).get();
+        System.out.println("--> Heap used: " + clusterStateResponse.getNodesStats().getJvm().getHeapUsed());
+        ByteSizeValue fieldDataMemoryUsed = clusterStateResponse.getIndicesStats().getFieldData().getMemorySize();
+        System.out.println("--> Fielddata memory size: " + fieldDataMemoryUsed);
+
+        return new StatsResult(name, totalQueryTime, fieldDataMemoryUsed);
     }
 
     private static StatsResult termsStats(String name, Method method, String keyField, String valueField, String executionHint) {
         long totalQueryTime;
 
         client.admin().indices().prepareClearCache().setFieldDataCache(true).execute().actionGet();
+        System.gc();
 
         System.out.println("--> Warmup (" + name + ")...");
         // run just the child query, warm up first
@@ -363,6 +408,6 @@ private static StatsResult termsStats(String name, Method method, String keyFiel
             totalQueryTime += searchResponse.getTookInMillis();
         }
         System.out.println("--> Terms stats agg (" + name + "): " + (totalQueryTime / QUERY_COUNT) + "ms");
-        return new StatsResult(name, totalQueryTime);
+        return new StatsResult(name, totalQueryTime, ByteSizeValue.parseBytesSizeValue("0b"));
     }
 }
diff --git a/src/test/java/org/elasticsearch/index/fielddata/AbstractFieldDataTests.java b/src/test/java/org/elasticsearch/index/fielddata/AbstractFieldDataTests.java
index 23109b370018c..11dd3ef14d852 100644
--- a/src/test/java/org/elasticsearch/index/fielddata/AbstractFieldDataTests.java
+++ b/src/test/java/org/elasticsearch/index/fielddata/AbstractFieldDataTests.java
@@ -27,7 +27,10 @@
 import org.elasticsearch.index.Index;
 import org.elasticsearch.index.mapper.*;
 import org.elasticsearch.index.mapper.Mapper.BuilderContext;
+import org.elasticsearch.indices.fielddata.breaker.CircuitBreakerService;
 import org.elasticsearch.indices.fielddata.breaker.DummyCircuitBreakerService;
+import org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache;
+import org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCacheListener;
 import org.elasticsearch.test.ElasticsearchTestCase;
 import org.elasticsearch.test.index.service.StubIndexService;
 import org.junit.After;
@@ -39,6 +42,8 @@ public abstract class AbstractFieldDataTests extends ElasticsearchTestCase {
     protected IndexFieldDataService ifdService;
     protected IndexWriter writer;
     protected AtomicReaderContext readerContext;
+    protected IndexReader topLevelReader;
+    protected IndicesFieldDataCache indicesFieldDataCache;
 
     protected abstract FieldDataType getFieldDataType();
 
@@ -79,7 +84,12 @@ public <IFD extends IndexFieldData<?>> IFD getForField(FieldDataType type, Strin
 
     @Before
     public void setup() throws Exception {
-        ifdService = new IndexFieldDataService(new Index("test"), new DummyCircuitBreakerService());
+        CircuitBreakerService circuitBreakerService = new DummyCircuitBreakerService();
+        indicesFieldDataCache = new IndicesFieldDataCache(
+                ImmutableSettings.Builder.EMPTY_SETTINGS,
+                new IndicesFieldDataCacheListener(circuitBreakerService)
+        );
+        ifdService = new IndexFieldDataService(new Index("test"), circuitBreakerService, indicesFieldDataCache);
         MapperService mapperService = MapperTestUtils.newMapperService(ifdService.index(), ImmutableSettings.Builder.EMPTY_SETTINGS);
         ifdService.setIndexService(new StubIndexService(mapperService));
         // LogByteSizeMP to preserve doc ID order
@@ -90,7 +100,7 @@ protected AtomicReaderContext refreshReader() throws Exception {
         if (readerContext != null) {
             readerContext.reader().close();
         }
-        AtomicReader reader = SlowCompositeReaderWrapper.wrap(DirectoryReader.open(writer, true));
+        AtomicReader reader = SlowCompositeReaderWrapper.wrap(topLevelReader = DirectoryReader.open(writer, true));
         readerContext = reader.getContext();
         return readerContext;
     }
diff --git a/src/test/java/org/elasticsearch/index/fielddata/AbstractStringFieldDataTests.java b/src/test/java/org/elasticsearch/index/fielddata/AbstractStringFieldDataTests.java
index 7f88026840816..9bccb8cb6b082 100644
--- a/src/test/java/org/elasticsearch/index/fielddata/AbstractStringFieldDataTests.java
+++ b/src/test/java/org/elasticsearch/index/fielddata/AbstractStringFieldDataTests.java
@@ -24,8 +24,10 @@
 import org.apache.lucene.document.Field;
 import org.apache.lucene.document.Field.Store;
 import org.apache.lucene.document.StringField;
+import org.apache.lucene.index.AtomicReaderContext;
 import org.apache.lucene.index.DirectoryReader;
 import org.apache.lucene.index.Term;
+import org.apache.lucene.index.TermsEnum;
 import org.apache.lucene.queries.TermFilter;
 import org.apache.lucene.search.*;
 import org.apache.lucene.search.join.FixedBitSetCachingWrapperFilter;
@@ -37,15 +39,21 @@
 import org.apache.lucene.util._TestUtil;
 import org.elasticsearch.common.lucene.search.NotFilter;
 import org.elasticsearch.common.lucene.search.XFilteredQuery;
+import org.elasticsearch.common.settings.ImmutableSettings;
 import org.elasticsearch.index.fielddata.IndexFieldData.XFieldComparatorSource;
 import org.elasticsearch.index.fielddata.fieldcomparator.BytesRefFieldComparatorSource;
 import org.elasticsearch.index.fielddata.fieldcomparator.SortMode;
+import org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsIndexFieldData;
+import org.elasticsearch.index.fielddata.ordinals.Ordinals;
 import org.elasticsearch.index.search.nested.NestedFieldComparatorSource;
+import org.junit.Test;
 
 import java.io.IOException;
 import java.util.ArrayList;
 import java.util.List;
 
+import static org.hamcrest.Matchers.*;
+
 /**
  */
 public abstract class AbstractStringFieldDataTests extends AbstractFieldDataImplTests {
@@ -414,4 +422,152 @@ public void testNestedSorting(SortMode sortMode) throws IOException {
         }
         searcher.getIndexReader().close();
     }
+
+    @Test
+    public void testGlobalOrdinals() throws Exception {
+        fillExtendedMvSet();
+        refreshReader();
+        FieldDataType fieldDataType = new FieldDataType("string", ImmutableSettings.builder().put("global_ordinals", "fixed"));
+        IndexFieldData.WithOrdinals ifd = getForField(fieldDataType, "value");
+        IndexFieldData.WithOrdinals globalOrdinals = ifd.loadGlobal(topLevelReader);
+        assertThat(topLevelReader.leaves().size(), equalTo(3));
+
+        // First segment
+        assertThat(globalOrdinals, instanceOf(GlobalOrdinalsIndexFieldData.class));
+        AtomicFieldData.WithOrdinals afd = globalOrdinals.load(topLevelReader.leaves().get(0));
+        assertThat(afd.getNumDocs(), equalTo(3));
+        BytesValues.WithOrdinals values = afd.getBytesValues(randomBoolean());
+        Ordinals.Docs ordinals = values.ordinals().ordinals().ordinals();
+        assertThat(ordinals.setDocument(0), equalTo(2));
+        long ord = ordinals.nextOrd();
+        assertThat(ord, equalTo(4l));
+        assertThat(values.getValueByOrd(ord).utf8ToString(), equalTo("02"));
+        ord = ordinals.nextOrd();
+        assertThat(ord, equalTo(6l));
+        assertThat(values.getValueByOrd(ord).utf8ToString(), equalTo("04"));
+        assertThat(ordinals.setDocument(1), equalTo(0));
+        assertThat(ordinals.setDocument(2), equalTo(1));
+        ord = ordinals.nextOrd();
+        assertThat(ord, equalTo(5l));
+        assertThat(values.getValueByOrd(ord).utf8ToString(), equalTo("03"));
+
+        // Second segment
+        afd = globalOrdinals.load(topLevelReader.leaves().get(1));
+        assertThat(afd.getNumDocs(), equalTo(4));
+        values = afd.getBytesValues(randomBoolean());
+        ordinals = values.ordinals().ordinals().ordinals();
+        assertThat(ordinals.setDocument(0), equalTo(3));
+        ord = ordinals.nextOrd();
+        assertThat(ord, equalTo(6l));
+        assertThat(values.getValueByOrd(ord).utf8ToString(), equalTo("04"));
+        ord = ordinals.nextOrd();
+        assertThat(ord, equalTo(7l));
+        assertThat(values.getValueByOrd(ord).utf8ToString(), equalTo("05"));
+        ord = ordinals.nextOrd();
+        assertThat(ord, equalTo(8l));
+        assertThat(values.getValueByOrd(ord).utf8ToString(), equalTo("06"));
+        assertThat(ordinals.setDocument(1), equalTo(3));
+        ord = ordinals.nextOrd();
+        assertThat(ord, equalTo(8l));
+        assertThat(values.getValueByOrd(ord).utf8ToString(), equalTo("06"));
+        ord = ordinals.nextOrd();
+        assertThat(ord, equalTo(9l));
+        assertThat(values.getValueByOrd(ord).utf8ToString(), equalTo("07"));
+        ord = ordinals.nextOrd();
+        assertThat(ord, equalTo(10l));
+        assertThat(values.getValueByOrd(ord).utf8ToString(), equalTo("08"));
+        assertThat(ordinals.setDocument(2), equalTo(0));
+        assertThat(ordinals.setDocument(3), equalTo(3));
+        ord = ordinals.nextOrd();
+        assertThat(ord, equalTo(10l));
+        assertThat(values.getValueByOrd(ord).utf8ToString(), equalTo("08"));
+        ord = ordinals.nextOrd();
+        assertThat(ord, equalTo(11l));
+        assertThat(values.getValueByOrd(ord).utf8ToString(), equalTo("09"));
+        ord = ordinals.nextOrd();
+        assertThat(ord, equalTo(12l));
+        assertThat(values.getValueByOrd(ord).utf8ToString(), equalTo("10"));
+
+        // Third segment
+        afd = globalOrdinals.load(topLevelReader.leaves().get(2));
+        values = afd.getBytesValues(randomBoolean());
+        ordinals = values.ordinals().ordinals().ordinals();
+        assertThat(afd.getNumDocs(), equalTo(1));
+        assertThat(ordinals.setDocument(0), equalTo(3));
+        ord = ordinals.nextOrd();
+        assertThat(ord, equalTo(1l));
+        assertThat(values.getValueByOrd(ord).utf8ToString(), equalTo("!08"));
+        ord = ordinals.nextOrd();
+        assertThat(ord, equalTo(2l));
+        assertThat(values.getValueByOrd(ord).utf8ToString(), equalTo("!09"));
+        ord = ordinals.nextOrd();
+        assertThat(ord, equalTo(3l));
+        assertThat(values.getValueByOrd(ord).utf8ToString(), equalTo("!10"));
+    }
+
+    @Test
+    public void testTermsEnum() throws Exception {
+        fillExtendedMvSet();
+        AtomicReaderContext atomicReaderContext = refreshReader();
+
+        IndexFieldData.WithOrdinals ifd = getForField("value");
+        AtomicFieldData.WithOrdinals afd = ifd.load(atomicReaderContext);
+
+        TermsEnum termsEnum = afd.getTermsEnum();
+        int size = 0;
+        while (termsEnum.next() != null) {
+            size++;
+        }
+        assertThat(size, equalTo(12));
+
+        assertThat(termsEnum.seekExact(new BytesRef("10")), is(true));
+        assertThat(termsEnum.term().utf8ToString(), equalTo("10"));
+        assertThat(termsEnum.next(), nullValue());
+
+        assertThat(termsEnum.seekExact(new BytesRef("08")), is(true));
+        assertThat(termsEnum.term().utf8ToString(), equalTo("08"));
+        size = 0;
+        while (termsEnum.next() != null) {
+            size++;
+        }
+        assertThat(size, equalTo(2));
+
+        termsEnum.seekExact(9);
+        assertThat(termsEnum.term().utf8ToString(), equalTo("07"));
+        size = 0;
+        while (termsEnum.next() != null) {
+            size++;
+        }
+        assertThat(size, equalTo(3));
+    }
+
+    @Test
+    public void testGlobalOrdinalsGetRemovedOnceIndexReaderCloses() throws Exception {
+        fillExtendedMvSet();
+        refreshReader();
+        FieldDataType fieldDataType = new FieldDataType("string", ImmutableSettings.builder().put("global_ordinals", "fixed"));
+        IndexFieldData.WithOrdinals ifd = getForField(fieldDataType, "value");
+        IndexFieldData.WithOrdinals globalOrdinals = ifd.loadGlobal(topLevelReader);
+        assertThat(ifd.loadGlobal(topLevelReader), sameInstance(globalOrdinals));
+        // 3 b/c 1 segment level caches and 1 top level cache
+        assertThat(indicesFieldDataCache.getCache().size(), equalTo(4l));
+
+        IndexFieldData.WithOrdinals cachedInstace = null;
+        for (RamUsage ramUsage : indicesFieldDataCache.getCache().asMap().values()) {
+            if (ramUsage instanceof IndexFieldData.WithOrdinals) {
+                cachedInstace = (IndexFieldData.WithOrdinals) ramUsage;
+                break;
+            }
+        }
+        assertThat(cachedInstace, sameInstance(globalOrdinals));
+        topLevelReader.close();
+        // Now only 3 segment level entries, only the toplevel reader has been closed, but the segment readers are still used by IW
+        assertThat(indicesFieldDataCache.getCache().size(), equalTo(3l));
+
+        refreshReader();
+        assertThat(ifd.loadGlobal(topLevelReader), not(sameInstance(globalOrdinals)));
+
+        ifdService.clear();
+        assertThat(indicesFieldDataCache.getCache().size(), equalTo(0l));
+    }
 }
diff --git a/src/test/java/org/elasticsearch/index/fielddata/DuelFieldDataTests.java b/src/test/java/org/elasticsearch/index/fielddata/DuelFieldDataTests.java
index c8e3b60e296d7..6db8b85e0f77f 100644
--- a/src/test/java/org/elasticsearch/index/fielddata/DuelFieldDataTests.java
+++ b/src/test/java/org/elasticsearch/index/fielddata/DuelFieldDataTests.java
@@ -18,6 +18,8 @@
  */
 package org.elasticsearch.index.fielddata;
 
+import com.carrotsearch.randomizedtesting.generators.RandomPicks;
+import com.carrotsearch.randomizedtesting.generators.RandomStrings;
 import com.google.common.collect.Lists;
 import org.apache.lucene.document.Document;
 import org.apache.lucene.document.Field;
@@ -36,6 +38,8 @@
 import org.elasticsearch.common.unit.DistanceUnit.Distance;
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.common.xcontent.XContentFactory;
+import org.elasticsearch.index.fielddata.ordinals.Ordinals;
+import org.elasticsearch.index.fielddata.ordinals.Ordinals.Docs;
 import org.elasticsearch.index.mapper.DocumentMapper;
 import org.elasticsearch.index.mapper.MapperTestUtils;
 import org.elasticsearch.index.mapper.ParsedDocument;
@@ -362,6 +366,54 @@ public void testDuelStrings() throws Exception {
 
     }
 
+    public void testDuelGlobalOrdinals() throws Exception {
+        Random random = getRandom();
+        final int numDocs = scaledRandomIntBetween(10, 1000);
+        final int numValues = scaledRandomIntBetween(10, 500);
+        final String[] values = new String[numValues];
+        for (int i = 0; i < numValues; ++i) {
+            values[i] = new String(RandomStrings.randomAsciiOfLength(random, 10));
+        }
+        for (int i = 0; i < numDocs; i++) {
+            Document d = new Document();
+            final int numVals = randomInt(3);
+            for (int j = 0; j < numVals; ++j) {
+                final String value = RandomPicks.randomFrom(random, Arrays.asList(values));
+                d.add(new StringField("string", value, Field.Store.NO));
+                if (LuceneTestCase.defaultCodecSupportsSortedSet()) {
+                    d.add(new SortedSetDocValuesField("bytes", new BytesRef(value)));
+                }
+            }
+            writer.addDocument(d);
+            if (randomInt(10) == 0) {
+                refreshReader();
+            }
+        }
+        refreshReader();
+
+        Map<FieldDataType, Type> typeMap = new HashMap<FieldDataType, DuelFieldDataTests.Type>();
+        typeMap.put(new FieldDataType("string", ImmutableSettings.builder().put("format", "fst")), Type.Bytes);
+        typeMap.put(new FieldDataType("string", ImmutableSettings.builder().put("format", "paged_bytes")), Type.Bytes);
+        if (LuceneTestCase.defaultCodecSupportsSortedSet()) {
+            typeMap.put(new FieldDataType("string", ImmutableSettings.builder().put("format", "doc_values")), Type.Bytes);
+        }
+
+        for (Map.Entry<FieldDataType, Type> entry : typeMap.entrySet()) {
+            ifdService.clear();
+            IndexFieldData.WithOrdinals<?> fieldData = getForField(entry.getKey(), entry.getValue().name().toLowerCase(Locale.ROOT));
+            BytesValues.WithOrdinals left = fieldData.load(readerContext).getBytesValues(randomBoolean());
+            fieldData.clear();
+            BytesValues.WithOrdinals right = fieldData.loadGlobal(topLevelReader).load(topLevelReader.leaves().get(0)).getBytesValues(randomBoolean());
+            Docs leftOrds = left.ordinals();
+            Docs rightOrds = right.ordinals();
+            assertEquals(leftOrds.getMaxOrd(), rightOrds.getMaxOrd());
+            assertEquals(leftOrds.getNumOrds(), rightOrds.getNumOrds());
+            for (long ord = Ordinals.MIN_ORDINAL; ord < leftOrds.getMaxOrd(); ++ord) {
+                assertEquals(left.getValueByOrd(ord), right.getValueByOrd(ord));
+            }
+        }
+    }
+
     public void testDuelGeoPoints() throws Exception {
         final String mapping = XContentFactory.jsonBuilder().startObject().startObject("type")
                 .startObject("properties")
diff --git a/src/test/java/org/elasticsearch/index/fielddata/NoOrdinalsStringFieldDataTests.java b/src/test/java/org/elasticsearch/index/fielddata/NoOrdinalsStringFieldDataTests.java
index 0eca185c6246f..aba087184baeb 100644
--- a/src/test/java/org/elasticsearch/index/fielddata/NoOrdinalsStringFieldDataTests.java
+++ b/src/test/java/org/elasticsearch/index/fielddata/NoOrdinalsStringFieldDataTests.java
@@ -25,6 +25,7 @@
 import org.elasticsearch.index.fielddata.fieldcomparator.BytesRefFieldComparatorSource;
 import org.elasticsearch.index.fielddata.fieldcomparator.SortMode;
 import org.elasticsearch.index.mapper.FieldMapper.Names;
+import org.junit.Test;
 
 /** Returns an implementation based on paged bytes which doesn't implement WithOrdinals in order to visit different paths in the code,
  *  eg. BytesRefFieldComparatorSource makes decisions based on whether the field data implements WithOrdinals. */
@@ -46,6 +47,11 @@ public Names getFieldNames() {
                 return in.getFieldNames();
             }
 
+            @Override
+            public FieldDataType getFieldDataType() {
+                return in.getFieldDataType();
+            }
+
             @Override
             public boolean valuesOrdered() {
                 return in.valuesOrdered();
@@ -79,4 +85,9 @@ public void clear(IndexReader reader) {
         };
     }
 
+    @Test
+    @Override
+    public void testTermsEnum() throws Exception {
+        // We can't test this, since the returned IFD instance doesn't implement IndexFieldData.WithOrdinals
+    }
 }
diff --git a/src/test/java/org/elasticsearch/search/aggregations/RandomTests.java b/src/test/java/org/elasticsearch/search/aggregations/RandomTests.java
index 211d57c9a804e..f8cf53b83da5f 100644
--- a/src/test/java/org/elasticsearch/search/aggregations/RandomTests.java
+++ b/src/test/java/org/elasticsearch/search/aggregations/RandomTests.java
@@ -26,7 +26,9 @@
 import org.elasticsearch.action.search.SearchRequestBuilder;
 import org.elasticsearch.action.search.SearchResponse;
 import org.elasticsearch.action.support.IndicesOptions;
+import org.elasticsearch.common.settings.ImmutableSettings;
 import org.elasticsearch.common.xcontent.XContentBuilder;
+import org.elasticsearch.index.fielddata.ordinals.InternalGlobalOrdinalsBuilder;
 import org.elasticsearch.index.query.FilterBuilders;
 import org.elasticsearch.index.query.RangeFilterBuilder;
 import org.elasticsearch.search.aggregations.bucket.filter.Filter;
@@ -151,21 +153,32 @@ public void testDuelTerms() throws Exception {
 
         final IntOpenHashSet valuesSet = new IntOpenHashSet();
         immutableCluster().wipeIndices("idx");
-        prepareCreate("idx").addMapping("type", jsonBuilder().startObject()
-              .startObject("type")
-                .startObject("properties")
-                  .startObject("string_values")
-                    .field("type", "string")
-                    .field("index", "not_analyzed")
-                  .endObject()
-                  .startObject("long_values")
-                    .field("type", "long")
-                  .endObject()
-                  .startObject("double_values")
-                    .field("type", "double")
-                  .endObject()
-                .endObject()
-              .endObject()).execute().actionGet();
+        prepareCreate("idx")
+                .setSettings(ImmutableSettings.builder().put(InternalGlobalOrdinalsBuilder.ORDINAL_MAPPING_THRESHOLD_INDEX_SETTING_KEY, randomIntBetween(1, maxNumTerms)))
+                .addMapping("type", jsonBuilder().startObject()
+                        .startObject("type")
+                        .startObject("properties")
+                        .startObject("string_values")
+                        .field("type", "string")
+                        .field("index", "not_analyzed")
+                        .startObject("fields")
+                            .startObject("doc_values")
+                                .field("type", "string")
+                                .field("index", "no")
+                                .startObject("fielddata")
+                                    .field("format", "doc_values")
+                                .endObject()
+                            .endObject()
+                        .endObject()
+                        .endObject()
+                        .startObject("long_values")
+                        .field("type", "long")
+                        .endObject()
+                        .startObject("double_values")
+                        .field("type", "double")
+                        .endObject()
+                        .endObject()
+                        .endObject()).execute().actionGet();
 
         List<IndexRequestBuilder> indexingRequests = Lists.newArrayList();
         for (int i = 0; i < numDocs; ++i) {
@@ -196,11 +209,19 @@ public void testDuelTerms() throws Exception {
 
         assertNoFailures(client().admin().indices().prepareRefresh("idx").setIndicesOptions(IndicesOptions.lenient()).execute().get());
 
+        TermsAggregatorFactory.ExecutionMode[] globalOrdinalModes = new TermsAggregatorFactory.ExecutionMode[]{
+                TermsAggregatorFactory.ExecutionMode.GLOBAL_ORDINALS_HASH,
+                TermsAggregatorFactory.ExecutionMode.GLOBAL_ORDINALS
+        };
+
         SearchResponse resp = client().prepareSearch("idx")
                 .addAggregation(terms("long").field("long_values").size(maxNumTerms).subAggregation(min("min").field("num")))
                 .addAggregation(terms("double").field("double_values").size(maxNumTerms).subAggregation(max("max").field("num")))
                 .addAggregation(terms("string_map").field("string_values").executionHint(TermsAggregatorFactory.ExecutionMode.MAP.toString()).size(maxNumTerms).subAggregation(stats("stats").field("num")))
-                .addAggregation(terms("string_ordinals").field("string_values").executionHint(TermsAggregatorFactory.ExecutionMode.ORDINALS.toString()).size(maxNumTerms).subAggregation(extendedStats("stats").field("num"))).execute().actionGet();
+                .addAggregation(terms("string_ordinals").field("string_values").executionHint(TermsAggregatorFactory.ExecutionMode.ORDINALS.toString()).size(maxNumTerms).subAggregation(extendedStats("stats").field("num")))
+                .addAggregation(terms("string_global_ordinals").field("string_values").executionHint(globalOrdinalModes[randomInt(globalOrdinalModes.length - 1)].toString()).size(maxNumTerms).subAggregation(extendedStats("stats").field("num")))
+                .addAggregation(terms("string_global_ordinals_doc_values").field("string_values.doc_values").executionHint(globalOrdinalModes[randomInt(globalOrdinalModes.length - 1)].toString()).size(maxNumTerms).subAggregation(extendedStats("stats").field("num")))
+                .execute().actionGet();
         assertAllSuccessful(resp);
         assertEquals(numDocs, resp.getHits().getTotalHits());
 
diff --git a/src/test/java/org/elasticsearch/search/aggregations/bucket/StringTermsTests.java b/src/test/java/org/elasticsearch/search/aggregations/bucket/StringTermsTests.java
index c0c0148dd7b4c..7fe895e35cf18 100644
--- a/src/test/java/org/elasticsearch/search/aggregations/bucket/StringTermsTests.java
+++ b/src/test/java/org/elasticsearch/search/aggregations/bucket/StringTermsTests.java
@@ -26,7 +26,6 @@
 import org.elasticsearch.search.aggregations.bucket.filter.Filter;
 import org.elasticsearch.search.aggregations.bucket.histogram.Histogram;
 import org.elasticsearch.search.aggregations.bucket.terms.Terms;
-import org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory;
 import org.elasticsearch.search.aggregations.metrics.avg.Avg;
 import org.elasticsearch.search.aggregations.metrics.stats.Stats;
 import org.elasticsearch.search.aggregations.metrics.stats.extended.ExtendedStats;
@@ -45,6 +44,7 @@
 import static org.elasticsearch.index.query.FilterBuilders.termFilter;
 import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery;
 import static org.elasticsearch.search.aggregations.AggregationBuilders.*;
+import static org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.ExecutionMode;
 import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertSearchResponse;
 import static org.hamcrest.Matchers.equalTo;
 import static org.hamcrest.Matchers.is;
@@ -60,7 +60,7 @@ public class StringTermsTests extends ElasticsearchIntegrationTest {
     private static final String MULTI_VALUED_FIELD_NAME = "s_values";
 
     public static String randomExecutionHint() {
-        return randomBoolean() ? null : randomFrom(TermsAggregatorFactory.ExecutionMode.values()).toString();
+        return randomBoolean() ? null : randomFrom(ExecutionMode.values()).toString();
     }
 
     @Override
@@ -144,6 +144,34 @@ public void singleValueField() throws Exception {
         }
     }
 
+    @Test
+    public void singleValueField_withGlobalOrdinals() throws Exception {
+        ExecutionMode[] executionModes = new ExecutionMode[] {
+                ExecutionMode.GLOBAL_ORDINALS,
+                ExecutionMode.GLOBAL_ORDINALS_HASH
+        };
+        for (ExecutionMode executionMode : executionModes) {
+            logger.info("Execution mode:" + executionMode);
+            SearchResponse response = client().prepareSearch("idx").setTypes("type")
+                    .addAggregation(terms("terms")
+                            .executionHint(executionMode.toString())
+                            .field(SINGLE_VALUED_FIELD_NAME))
+                    .execute().actionGet();
+            assertSearchResponse(response);
+
+            Terms terms = response.getAggregations().get("terms");
+            assertThat(terms, notNullValue());
+            assertThat(terms.getName(), equalTo("terms"));
+            assertThat(terms.getBuckets().size(), equalTo(5));
+            for (int i = 0; i < 5; i++) {
+                Terms.Bucket bucket = terms.getBucketByKey("val" + i);
+                assertThat(bucket, notNullValue());
+                assertThat(key(bucket), equalTo("val" + i));
+                assertThat(bucket.getDocCount(), equalTo(1l));
+            }
+        }
+    }
+
     @Test
     public void singleValueField_WithRegexFiltering() throws Exception {
 
diff --git a/src/test/java/org/elasticsearch/test/TestCluster.java b/src/test/java/org/elasticsearch/test/TestCluster.java
index 9e45e2be776ef..70e0a22630041 100644
--- a/src/test/java/org/elasticsearch/test/TestCluster.java
+++ b/src/test/java/org/elasticsearch/test/TestCluster.java
@@ -59,6 +59,7 @@
 import org.elasticsearch.env.NodeEnvironment;
 import org.elasticsearch.http.HttpServerTransport;
 import org.elasticsearch.index.engine.IndexEngineModule;
+import org.elasticsearch.index.fielddata.ordinals.InternalGlobalOrdinalsBuilder;
 import org.elasticsearch.node.Node;
 import org.elasticsearch.node.internal.InternalNode;
 import org.elasticsearch.search.SearchService;
@@ -78,12 +79,12 @@
 import java.io.File;
 import java.net.InetSocketAddress;
 import java.util.*;
-import java.util.concurrent.*;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.concurrent.atomic.AtomicInteger;
 
 import static com.carrotsearch.randomizedtesting.RandomizedTest.systemPropertyAsBoolean;
-import static com.google.common.collect.Maps.newTreeMap;
 import static org.apache.lucene.util.LuceneTestCase.rarely;
 import static org.apache.lucene.util.LuceneTestCase.usually;
 import static org.elasticsearch.common.settings.ImmutableSettings.settingsBuilder;
@@ -278,6 +279,7 @@ private static Settings getRandomNodeSettings(long seed) {
             }
         }
         builder.put("plugins.isolation", random.nextBoolean());
+        builder.put(InternalGlobalOrdinalsBuilder.ORDINAL_MAPPING_THRESHOLD_INDEX_SETTING_KEY, 1 + random.nextInt(InternalGlobalOrdinalsBuilder.ORDINAL_MAPPING_THRESHOLD_DEFAULT));
         return builder.build();
     }
 

From 000c33aac3123e4ed22411b5bb8d9063521314f7 Mon Sep 17 00:00:00 2001
From: gabriel-tessier <gabriel-tessier@users.noreply.github.com>
Date: Tue, 1 Apr 2014 18:13:59 +0900
Subject: [PATCH 09/54] fix typo

---
 .../search/aggregations/bucket/global-aggregation.asciidoc    | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/docs/reference/search/aggregations/bucket/global-aggregation.asciidoc b/docs/reference/search/aggregations/bucket/global-aggregation.asciidoc
index 037a9b343130b..da6ddc1af4edc 100644
--- a/docs/reference/search/aggregations/bucket/global-aggregation.asciidoc
+++ b/docs/reference/search/aggregations/bucket/global-aggregation.asciidoc
@@ -28,7 +28,7 @@ Example:
 <1> The `global` aggregation has an empty body
 <2> The sub-aggregations that are registered for this `global` aggregation
 
-The above aggregation demonstrates how one would compute aggregations (`avg_price` in this example) on all the documents in the search context, regardless of the query (in our example, it will compute the the average price over all products in our catalog, not just on the "shirts").
+The above aggregation demonstrates how one would compute aggregations (`avg_price` in this example) on all the documents in the search context, regardless of the query (in our example, it will compute the average price over all products in our catalog, not just on the "shirts").
 
 The response for the above aggreation:
 
@@ -48,4 +48,4 @@ The response for the above aggreation:
 }
 --------------------------------------------------
 
-<1> The number of documents that were aggregated (in our case, all documents within the search context)
\ No newline at end of file
+<1> The number of documents that were aggregated (in our case, all documents within the search context)

From 866c520abb63e95d80d5c67ba834049816e75f4e Mon Sep 17 00:00:00 2001
From: Kevin Wang <kevin807359@gmail.com>
Date: Thu, 3 Apr 2014 10:49:04 +1100
Subject: [PATCH 10/54] Add doc value for binary field.

Close #5669
---
 .../mapping/types/core-types.asciidoc         |   1 +
 .../common/util/CollectionUtils.java          |  57 +++++++++
 .../fielddata/IndexFieldDataService.java      |   5 +
 .../plain/BytesBinaryDVAtomicFieldData.java   | 106 ++++++++++++++++
 .../plain/BytesBinaryDVIndexFieldData.java    |  81 +++++++++++++
 .../index/mapper/core/BinaryFieldMapper.java  | 107 ++++++++++++-----
 .../fielddata/AbstractFieldDataTests.java     |   2 +
 .../fielddata/BinaryDVFieldDataTests.java     | 113 ++++++++++++++++++
 8 files changed, 439 insertions(+), 33 deletions(-)
 create mode 100644 src/main/java/org/elasticsearch/index/fielddata/plain/BytesBinaryDVAtomicFieldData.java
 create mode 100644 src/main/java/org/elasticsearch/index/fielddata/plain/BytesBinaryDVIndexFieldData.java
 create mode 100644 src/test/java/org/elasticsearch/index/fielddata/BinaryDVFieldDataTests.java

diff --git a/docs/reference/mapping/types/core-types.asciidoc b/docs/reference/mapping/types/core-types.asciidoc
index 98f16536adcf7..5b3d5b70b79ce 100644
--- a/docs/reference/mapping/types/core-types.asciidoc
+++ b/docs/reference/mapping/types/core-types.asciidoc
@@ -446,6 +446,7 @@ Defaults to the property/field name.
 |`store` |Set to `true` to store actual field in the index, `false` to not
 store it. Defaults to `false` (note, the JSON document itself is stored,
 and it can be retrieved from it).
+|`doc_values` |Set to `true` to store field values in a column-stride fashion.
 |=======================================================================
 
 [float]
diff --git a/src/main/java/org/elasticsearch/common/util/CollectionUtils.java b/src/main/java/org/elasticsearch/common/util/CollectionUtils.java
index 7bdd54a3e587c..16297d16982b0 100644
--- a/src/main/java/org/elasticsearch/common/util/CollectionUtils.java
+++ b/src/main/java/org/elasticsearch/common/util/CollectionUtils.java
@@ -22,10 +22,12 @@
 import com.carrotsearch.hppc.DoubleArrayList;
 import com.carrotsearch.hppc.FloatArrayList;
 import com.carrotsearch.hppc.LongArrayList;
+import com.carrotsearch.hppc.ObjectArrayList;
 import org.apache.lucene.util.IntroSorter;
 import org.elasticsearch.common.Preconditions;
 
 import java.util.AbstractList;
+import java.util.Arrays;
 import java.util.List;
 import java.util.RandomAccess;
 
@@ -223,6 +225,61 @@ public static <T> List<T> rotate(final List<T> list, int distance) {
         return new RotatedList<>(list, d);
     }
 
+    public static void sortAndDedup(final ObjectArrayList<byte[]> array) {
+        int len = array.size();
+        if (len > 1) {
+            sort(array);
+            int uniqueCount = 1;
+            for (int i = 1; i < len; ++i) {
+                if (!Arrays.equals(array.get(i), array.get(i - 1))) {
+                    array.set(uniqueCount++, array.get(i));
+                }
+            }
+            array.elementsCount = uniqueCount;
+        }
+    }
+
+    public static void sort(final ObjectArrayList<byte[]> array) {
+        new IntroSorter() {
+
+            byte[] pivot;
+
+            @Override
+            protected void swap(int i, int j) {
+                final byte[] tmp = array.get(i);
+                array.set(i, array.get(j));
+                array.set(j, tmp);
+            }
+
+            @Override
+            protected int compare(int i, int j) {
+                return compare(array.get(i), array.get(j));
+            }
+
+            @Override
+            protected void setPivot(int i) {
+                pivot = array.get(i);
+            }
+
+            @Override
+            protected int comparePivot(int j) {
+                return compare(pivot, array.get(j));
+            }
+
+            private int compare(byte[] left, byte[] right) {
+                for (int i = 0, j = 0; i < left.length && j < right.length; i++, j++) {
+                    int a = left[i] & 0xFF;
+                    int b = right[j] & 0xFF;
+                    if (a != b) {
+                        return a - b;
+                    }
+                }
+                return left.length - right.length;
+            }
+
+        }.sort(0, array.size());
+    }
+
     private static class RotatedList<T> extends AbstractList<T> implements RandomAccess {
 
         private final List<T> in;
diff --git a/src/main/java/org/elasticsearch/index/fielddata/IndexFieldDataService.java b/src/main/java/org/elasticsearch/index/fielddata/IndexFieldDataService.java
index 735ff5322d695..56567d411790c 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/IndexFieldDataService.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/IndexFieldDataService.java
@@ -73,6 +73,7 @@ public class IndexFieldDataService extends AbstractIndexComponent {
                 .put("long", new PackedArrayIndexFieldData.Builder().setNumericType(IndexNumericFieldData.NumericType.LONG))
                 .put("geo_point", new GeoPointDoubleArrayIndexFieldData.Builder())
                 .put(ParentFieldMapper.NAME, new ParentChildIndexFieldData.Builder())
+                .put("binary", new DisabledIndexFieldData.Builder())
                 .immutableMap();
 
         docValuesBuildersByType = MapBuilder.<String, IndexFieldData.Builder>newMapBuilder()
@@ -84,6 +85,7 @@ public class IndexFieldDataService extends AbstractIndexComponent {
                 .put("int", new DocValuesIndexFieldData.Builder().numericType(IndexNumericFieldData.NumericType.INT))
                 .put("long", new DocValuesIndexFieldData.Builder().numericType(IndexNumericFieldData.NumericType.LONG))
                 .put("geo_point", new GeoPointBinaryDVIndexFieldData.Builder())
+                .put("binary", new BytesBinaryDVIndexFieldData.Builder())
                 .immutableMap();
 
         buildersByTypeAndFormat = MapBuilder.<Tuple<String, String>, IndexFieldData.Builder>newMapBuilder()
@@ -121,6 +123,9 @@ public class IndexFieldDataService extends AbstractIndexComponent {
                 .put(Tuple.tuple("geo_point", DISABLED_FORMAT), new DisabledIndexFieldData.Builder())
                 .put(Tuple.tuple("geo_point", COMPRESSED_FORMAT), new GeoPointCompressedIndexFieldData.Builder())
 
+                .put(Tuple.tuple("binary", DOC_VALUES_FORMAT), new BytesBinaryDVIndexFieldData.Builder())
+                .put(Tuple.tuple("binary", DISABLED_FORMAT), new DisabledIndexFieldData.Builder())
+
                 .immutableMap();
     }
 
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/BytesBinaryDVAtomicFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/BytesBinaryDVAtomicFieldData.java
new file mode 100644
index 0000000000000..33be9a3af7116
--- /dev/null
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/BytesBinaryDVAtomicFieldData.java
@@ -0,0 +1,106 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.index.fielddata.plain;
+
+import org.apache.lucene.index.AtomicReader;
+import org.apache.lucene.index.BinaryDocValues;
+import org.apache.lucene.store.ByteArrayDataInput;
+import org.apache.lucene.util.BytesRef;
+import org.elasticsearch.index.fielddata.*;
+
+final class BytesBinaryDVAtomicFieldData implements AtomicFieldData<ScriptDocValues> {
+
+    private final AtomicReader reader;
+    private final BinaryDocValues values;
+
+    BytesBinaryDVAtomicFieldData(AtomicReader reader, BinaryDocValues values) {
+        super();
+        this.reader = reader;
+        this.values = values == null ? BinaryDocValues.EMPTY : values;
+    }
+
+    @Override
+    public boolean isMultiValued() {
+        return true;
+    }
+
+    @Override
+    public boolean isValuesOrdered() {
+        return false;
+    }
+
+    @Override
+    public int getNumDocs() {
+        return reader.maxDoc();
+    }
+
+    @Override
+    public long getNumberUniqueValues() {
+        return Long.MAX_VALUE;
+    }
+
+    @Override
+    public long getMemorySizeInBytes() {
+        return -1; // not exposed by Lucene
+    }
+
+    @Override
+    public BytesValues getBytesValues(boolean needsHashes) {
+        return new BytesValues(true) {
+
+            final BytesRef bytes = new BytesRef();
+            final ByteArrayDataInput in = new ByteArrayDataInput();
+
+            @Override
+            public int setDocument(int docId) {
+                values.get(docId, bytes);
+                in.reset(bytes.bytes, bytes.offset, bytes.length);
+                if (bytes.length == 0) {
+                    return 0;
+                } else {
+                    return in.readVInt();
+                }
+            }
+
+            @Override
+            public BytesRef nextValue() {
+                final int length = in.readVInt();
+                scratch.grow(length);
+                in.readBytes(scratch.bytes, 0, length);
+                scratch.length = length;
+                scratch.offset = 0;
+                return scratch;
+            }
+
+        };
+    }
+
+    @Override
+    public ScriptDocValues getScriptValues() {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public void close() {
+        // no-op
+    }
+
+
+}
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/BytesBinaryDVIndexFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/BytesBinaryDVIndexFieldData.java
new file mode 100644
index 0000000000000..69c85cd7f205f
--- /dev/null
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/BytesBinaryDVIndexFieldData.java
@@ -0,0 +1,81 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.index.fielddata.plain;
+
+import org.apache.lucene.index.AtomicReaderContext;
+import org.elasticsearch.ElasticsearchIllegalArgumentException;
+import org.elasticsearch.ElasticsearchIllegalStateException;
+import org.elasticsearch.common.Nullable;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.index.Index;
+import org.elasticsearch.index.fielddata.FieldDataType;
+import org.elasticsearch.index.fielddata.IndexFieldData;
+import org.elasticsearch.index.fielddata.IndexFieldDataCache;
+import org.elasticsearch.index.fielddata.fieldcomparator.SortMode;
+import org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsBuilder;
+import org.elasticsearch.index.mapper.FieldMapper;
+import org.elasticsearch.index.mapper.FieldMapper.Names;
+import org.elasticsearch.index.mapper.MapperService;
+import org.elasticsearch.indices.fielddata.breaker.CircuitBreakerService;
+
+import java.io.IOException;
+
+public class BytesBinaryDVIndexFieldData extends DocValuesIndexFieldData implements IndexFieldData<BytesBinaryDVAtomicFieldData> {
+
+    public BytesBinaryDVIndexFieldData(Index index, Names fieldNames, FieldDataType fieldDataType) {
+        super(index, fieldNames, fieldDataType);
+    }
+
+    @Override
+    public boolean valuesOrdered() {
+        return false;
+    }
+
+    @Override
+    public final XFieldComparatorSource comparatorSource(@Nullable Object missingValue, SortMode sortMode) {
+        throw new ElasticsearchIllegalArgumentException("can't sort on binary field");
+    }
+
+    @Override
+    public BytesBinaryDVAtomicFieldData load(AtomicReaderContext context) {
+        try {
+            return new BytesBinaryDVAtomicFieldData(context.reader(), context.reader().getBinaryDocValues(fieldNames.indexName()));
+        } catch (IOException e) {
+            throw new ElasticsearchIllegalStateException("Cannot load doc values", e);
+        }
+    }
+
+    @Override
+    public BytesBinaryDVAtomicFieldData loadDirect(AtomicReaderContext context) throws Exception {
+        return load(context);
+    }
+
+    public static class Builder implements IndexFieldData.Builder {
+
+        @Override
+        public IndexFieldData<?> build(Index index, Settings indexSettings, FieldMapper<?> mapper, IndexFieldDataCache cache,
+                                       CircuitBreakerService breakerService, MapperService mapperService, GlobalOrdinalsBuilder globalOrdinalBuilder) {
+            // Ignore breaker
+            final Names fieldNames = mapper.names();
+            return new BytesBinaryDVIndexFieldData(index, fieldNames, mapper.fieldDataType());
+        }
+
+    }
+}
diff --git a/src/main/java/org/elasticsearch/index/mapper/core/BinaryFieldMapper.java b/src/main/java/org/elasticsearch/index/mapper/core/BinaryFieldMapper.java
index e64d0f7637eba..f2cf9ce3998e0 100644
--- a/src/main/java/org/elasticsearch/index/mapper/core/BinaryFieldMapper.java
+++ b/src/main/java/org/elasticsearch/index/mapper/core/BinaryFieldMapper.java
@@ -19,18 +19,25 @@
 
 package org.elasticsearch.index.mapper.core;
 
+import com.carrotsearch.hppc.ObjectArrayList;
 import org.apache.lucene.document.Field;
 import org.apache.lucene.document.FieldType;
+import org.apache.lucene.index.FieldInfo;
+import org.apache.lucene.store.ByteArrayDataOutput;
 import org.apache.lucene.util.BytesRef;
+import org.elasticsearch.ElasticsearchException;
 import org.elasticsearch.ElasticsearchParseException;
 import org.elasticsearch.common.Base64;
+import org.elasticsearch.common.Nullable;
 import org.elasticsearch.common.Strings;
 import org.elasticsearch.common.bytes.BytesArray;
 import org.elasticsearch.common.bytes.BytesReference;
 import org.elasticsearch.common.compress.CompressorFactory;
 import org.elasticsearch.common.io.stream.BytesStreamOutput;
 import org.elasticsearch.common.io.stream.StreamOutput;
+import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.unit.ByteSizeValue;
+import org.elasticsearch.common.util.CollectionUtils;
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.common.xcontent.XContentParser;
 import org.elasticsearch.index.codec.docvaluesformat.DocValuesFormatProvider;
@@ -86,8 +93,8 @@ public Builder compressThreshold(long compressThreshold) {
 
         @Override
         public BinaryFieldMapper build(BuilderContext context) {
-            return new BinaryFieldMapper(buildNames(context), fieldType, compress, compressThreshold, postingsProvider,
-                    docValuesProvider, multiFieldsBuilder.build(this, context), copyTo);
+            return new BinaryFieldMapper(buildNames(context), fieldType, docValues, compress, compressThreshold, postingsProvider,
+                    docValuesProvider, fieldDataSettings, multiFieldsBuilder.build(this, context), copyTo);
         }
     }
 
@@ -119,10 +126,10 @@ public Mapper.Builder parse(String name, Map<String, Object> node, ParserContext
 
     private long compressThreshold;
 
-    protected BinaryFieldMapper(Names names, FieldType fieldType, Boolean compress, long compressThreshold,
-                                PostingsFormatProvider postingsProvider, DocValuesFormatProvider docValuesProvider,
+    protected BinaryFieldMapper(Names names, FieldType fieldType, Boolean docValues, Boolean compress, long compressThreshold,
+                                PostingsFormatProvider postingsProvider, DocValuesFormatProvider docValuesProvider, @Nullable Settings fieldDataSettings,
                                 MultiFields multiFields, CopyTo copyTo) {
-        super(names, 1.0f, fieldType, null, null, null, postingsProvider, docValuesProvider, null, null, null, null, multiFields, copyTo);
+        super(names, 1.0f, fieldType, docValues, null, null, postingsProvider, docValuesProvider, null, null, fieldDataSettings, null, multiFields, copyTo);
         this.compress = compress;
         this.compressThreshold = compressThreshold;
     }
@@ -134,7 +141,7 @@ public FieldType defaultFieldType() {
 
     @Override
     public FieldDataType defaultFieldDataType() {
-        return null;
+        return new FieldDataType("binary");
     }
 
     @Override
@@ -171,7 +178,7 @@ public BytesReference value(Object value) {
 
     @Override
     protected void parseCreateField(ParseContext context, List<Field> fields) throws IOException {
-        if (!fieldType().stored()) {
+        if (!fieldType().stored() && !hasDocValues()) {
             return;
         }
         byte[] value = context.parseExternalValue(byte[].class);
@@ -194,7 +201,20 @@ protected void parseCreateField(ParseContext context, List<Field> fields) throws
                 value = bStream.bytes().toBytes();
             }
         }
-        fields.add(new Field(names.indexName(), value, fieldType));
+        if (fieldType().stored()) {
+            fields.add(new Field(names.indexName(), value, fieldType));
+        }
+
+        if (hasDocValues()) {
+            CustomBinaryDocValuesField field = (CustomBinaryDocValuesField) context.doc().getByKey(names().indexName());
+            if (field == null) {
+                field = new CustomBinaryDocValuesField(names().indexName(), value);
+                context.doc().addWithKey(names().indexName(), field);
+            } else {
+                field.add(value);
+            }
+        }
+
     }
 
     @Override
@@ -204,10 +224,7 @@ protected String contentType() {
 
     @Override
     protected void doXContentBody(XContentBuilder builder, boolean includeDefaults, Params params) throws IOException {
-        builder.field("type", contentType());
-        if (includeDefaults || !names.name().equals(names.indexNameClean())) {
-            builder.field("index_name", names.indexNameClean());
-        }
+        super.doXContentBody(builder, includeDefaults, params);
         if (compress != null) {
             builder.field("compress", compress);
         } else if (includeDefaults) {
@@ -218,32 +235,16 @@ protected void doXContentBody(XContentBuilder builder, boolean includeDefaults,
         } else if (includeDefaults) {
             builder.field("compress_threshold", -1);
         }
-        if (includeDefaults || fieldType.stored() != defaultFieldType().stored()) {
-            builder.field("store", fieldType.stored());
-        }
     }
 
     @Override
     public void merge(Mapper mergeWith, MergeContext mergeContext) throws MergeMappingException {
-        if (!(mergeWith instanceof BinaryFieldMapper)) {
-            String mergedType = mergeWith.getClass().getSimpleName();
-            if (mergeWith instanceof AbstractFieldMapper) {
-                mergedType = ((AbstractFieldMapper) mergeWith).contentType();
-            }
-            mergeContext.addConflict("mapper [" + names.fullName() + "] of different type, current_type [" + contentType() + "], merged_type [" + mergedType + "]");
-            // different types, return
+        super.merge(mergeWith, mergeContext);
+        if (!this.getClass().equals(mergeWith.getClass())) {
             return;
         }
 
         BinaryFieldMapper sourceMergeWith = (BinaryFieldMapper) mergeWith;
-
-        if (this.fieldType().stored() != sourceMergeWith.fieldType().stored()) {
-            mergeContext.addConflict("mapper [" + names.fullName() + "] has different store values");
-        }
-        if (!this.names().equals(sourceMergeWith.names())) {
-            mergeContext.addConflict("mapper [" + names.fullName() + "] has different index_name");
-        }
-
         if (!mergeContext.mergeFlags().simulate()) {
             if (sourceMergeWith.compress != null) {
                 this.compress = sourceMergeWith.compress;
@@ -254,8 +255,48 @@ public void merge(Mapper mergeWith, MergeContext mergeContext) throws MergeMappi
         }
     }
 
-    @Override
-    public boolean hasDocValues() {
-        return false;
+    public static class CustomBinaryDocValuesField extends NumberFieldMapper.CustomNumericDocValuesField {
+
+        public static final FieldType TYPE = new FieldType();
+        static {
+            TYPE.setDocValueType(FieldInfo.DocValuesType.BINARY);
+            TYPE.freeze();
+        }
+
+        private final ObjectArrayList<byte[]> bytesList;
+
+        private int totalSize = 0;
+
+        public CustomBinaryDocValuesField(String  name, byte[] bytes) {
+            super(name);
+            bytesList = new ObjectArrayList<>();
+            add(bytes);
+        }
+
+        public void add(byte[] bytes) {
+            bytesList.add(bytes);
+            totalSize += bytes.length;
+        }
+
+        @Override
+        public BytesRef binaryValue() {
+            try {
+                CollectionUtils.sortAndDedup(bytesList);
+                int size = bytesList.size();
+                final byte[] bytes = new byte[totalSize + (size + 1) * 5];
+                ByteArrayDataOutput out = new ByteArrayDataOutput(bytes);
+                out.writeVInt(size);  // write total number of values
+                for (int i = 0; i < size; i ++) {
+                    final byte[] value = bytesList.get(i);
+                    int valueLength = value.length;
+                    out.writeVInt(valueLength);
+                    out.writeBytes(value, 0, valueLength);
+                }
+                return new BytesRef(bytes, 0, out.getPosition());
+            } catch (IOException e) {
+                throw new ElasticsearchException("Failed to get binary value", e);
+            }
+
+        }
     }
 }
diff --git a/src/test/java/org/elasticsearch/index/fielddata/AbstractFieldDataTests.java b/src/test/java/org/elasticsearch/index/fielddata/AbstractFieldDataTests.java
index 11dd3ef14d852..4af68018bf546 100644
--- a/src/test/java/org/elasticsearch/index/fielddata/AbstractFieldDataTests.java
+++ b/src/test/java/org/elasticsearch/index/fielddata/AbstractFieldDataTests.java
@@ -76,6 +76,8 @@ public <IFD extends IndexFieldData<?>> IFD getForField(FieldDataType type, Strin
             mapper = MapperBuilders.geoPointField(fieldName).fieldDataSettings(type.getSettings()).build(context);
         } else if (type.getType().equals("_parent")) {
             mapper = MapperBuilders.parent().type(fieldName).build(context);
+        } else if (type.getType().equals("binary")) {
+            mapper = MapperBuilders.binaryField(fieldName).fieldDataSettings(type.getSettings()).build(context);
         } else {
             throw new UnsupportedOperationException(type.getType());
         }
diff --git a/src/test/java/org/elasticsearch/index/fielddata/BinaryDVFieldDataTests.java b/src/test/java/org/elasticsearch/index/fielddata/BinaryDVFieldDataTests.java
new file mode 100644
index 0000000000000..15bafebf66a28
--- /dev/null
+++ b/src/test/java/org/elasticsearch/index/fielddata/BinaryDVFieldDataTests.java
@@ -0,0 +1,113 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.index.fielddata;
+
+import com.carrotsearch.hppc.ObjectArrayList;
+import org.apache.lucene.index.AtomicReaderContext;
+import org.apache.lucene.util.BytesRef;
+import org.elasticsearch.common.settings.ImmutableSettings;
+import org.elasticsearch.common.util.CollectionUtils;
+import org.elasticsearch.common.xcontent.XContentBuilder;
+import org.elasticsearch.common.xcontent.XContentFactory;
+import org.elasticsearch.index.mapper.DocumentMapper;
+import org.elasticsearch.index.mapper.MapperTestUtils;
+import org.elasticsearch.index.mapper.ParsedDocument;
+import org.junit.Test;
+
+import static org.hamcrest.Matchers.equalTo;
+
+/**
+ *
+ */
+public class BinaryDVFieldDataTests extends AbstractFieldDataTests {
+
+    @Test
+    public void testDocValue() throws Exception {
+        String mapping = XContentFactory.jsonBuilder().startObject().startObject("test")
+                .startObject("properties")
+                .startObject("field")
+                .field("type", "binary")
+                .startObject("fielddata").field("format", "doc_values").endObject()
+                .endObject()
+                .endObject()
+                .endObject().endObject().string();
+
+        final DocumentMapper mapper = MapperTestUtils.newParser().parse(mapping);
+
+
+        ObjectArrayList<byte[]> bytesList1 = new ObjectArrayList<>(2);
+        bytesList1.add(randomBytes());
+        bytesList1.add(randomBytes());
+        XContentBuilder doc = XContentFactory.jsonBuilder().startObject().startArray("field").value(bytesList1.get(0)).value(bytesList1.get(1)).endArray().endObject();
+        ParsedDocument d = mapper.parse("test", "1", doc.bytes());
+        writer.addDocument(d.rootDoc());
+
+        byte[] bytes1 = randomBytes();
+        doc = XContentFactory.jsonBuilder().startObject().field("field", bytes1).endObject();
+        d = mapper.parse("test", "2", doc.bytes());
+        writer.addDocument(d.rootDoc());
+
+        doc = XContentFactory.jsonBuilder().startObject().endObject();
+        d = mapper.parse("test", "3", doc.bytes());
+        writer.addDocument(d.rootDoc());
+
+        // test remove duplicate value
+        ObjectArrayList<byte[]> bytesList2 = new ObjectArrayList<>(2);
+        bytesList2.add(randomBytes());
+        bytesList2.add(randomBytes());
+        doc = XContentFactory.jsonBuilder().startObject().startArray("field").value(bytesList2.get(0)).value(bytesList2.get(1)).value(bytesList2.get(0)).endArray().endObject();
+        d = mapper.parse("test", "4", doc.bytes());
+        writer.addDocument(d.rootDoc());
+
+        AtomicReaderContext reader = refreshReader();
+        IndexFieldData indexFieldData = getForField("field");
+        AtomicFieldData fieldData = indexFieldData.load(reader);
+        assertThat(fieldData.getNumDocs(), equalTo(4));
+
+        BytesValues bytesValues = fieldData.getBytesValues(randomBoolean());
+
+        CollectionUtils.sortAndDedup(bytesList1);
+        assertThat(bytesValues.setDocument(0), equalTo(2));
+        assertThat(bytesValues.nextValue(), equalTo(new BytesRef(bytesList1.get(0))));
+        assertThat(bytesValues.nextValue(), equalTo(new BytesRef(bytesList1.get(1))));
+
+        assertThat(bytesValues.setDocument(1), equalTo(1));
+        assertThat(bytesValues.nextValue(), equalTo(new BytesRef(bytes1)));
+
+        assertThat(bytesValues.setDocument(2), equalTo(0));
+
+        CollectionUtils.sortAndDedup(bytesList2);
+        assertThat(bytesValues.setDocument(3), equalTo(2));
+        assertThat(bytesValues.nextValue(), equalTo(new BytesRef(bytesList2.get(0))));
+        assertThat(bytesValues.nextValue(), equalTo(new BytesRef(bytesList2.get(1))));
+    }
+
+    private byte[] randomBytes() {
+        int size = randomIntBetween(10, 1000);
+        byte[] bytes = new byte[size];
+        getRandom().nextBytes(bytes);
+        return bytes;
+    }
+
+    @Override
+    protected FieldDataType getFieldDataType() {
+        return new FieldDataType("binary", ImmutableSettings.builder().put("format", "doc_values"));
+    }
+}

From 9df655adb234b4c8ef4539572e168b5342f930ad Mon Sep 17 00:00:00 2001
From: Adrien Grand <jpountz@gmail.com>
Date: Fri, 4 Apr 2014 15:46:25 +0200
Subject: [PATCH 11/54] Remove AtomicFieldData.isValuesOrdered.

This method is not used anymore.

Close #5688
---
 .../AbstractAtomicNumericFieldData.java       |  5 -----
 .../index/fielddata/AtomicFieldData.java      |  5 -----
 .../GlobalOrdinalsIndexFieldData.java         |  8 +-------
 .../plain/AbstractGeoPointIndexFieldData.java |  5 -----
 .../plain/BinaryDVAtomicFieldData.java        |  5 -----
 .../plain/BytesBinaryDVAtomicFieldData.java   |  9 +++------
 .../plain/DoubleArrayAtomicFieldData.java     | 20 -------------------
 .../plain/FSTBytesAtomicFieldData.java        | 10 ----------
 .../plain/FloatArrayAtomicFieldData.java      | 20 -------------------
 .../GeoPointBinaryDVAtomicFieldData.java      |  5 -----
 .../GeoPointCompressedAtomicFieldData.java    | 15 --------------
 .../GeoPointDoubleArrayAtomicFieldData.java   | 15 --------------
 .../plain/NumericDVAtomicFieldData.java       |  5 -----
 .../plain/PackedArrayAtomicFieldData.java     | 20 -------------------
 .../plain/PagedBytesAtomicFieldData.java      | 10 ----------
 .../plain/ParentChildAtomicFieldData.java     |  5 -----
 .../SortedSetDVBytesAtomicFieldData.java      |  5 -----
 17 files changed, 4 insertions(+), 163 deletions(-)

diff --git a/src/main/java/org/elasticsearch/index/fielddata/AbstractAtomicNumericFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/AbstractAtomicNumericFieldData.java
index d326857ce70b0..6c7c5c77f1254 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/AbstractAtomicNumericFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/AbstractAtomicNumericFieldData.java
@@ -31,11 +31,6 @@ public AbstractAtomicNumericFieldData(boolean isFloat) {
         this.isFloat = isFloat;
     }
 
-    @Override
-    public boolean isValuesOrdered() {
-        return false;
-    }
-
     @Override
     public ScriptDocValues getScriptValues() {
         if (isFloat) {
diff --git a/src/main/java/org/elasticsearch/index/fielddata/AtomicFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/AtomicFieldData.java
index cc3975ebdfccc..84faf5ff89dd6 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/AtomicFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/AtomicFieldData.java
@@ -33,11 +33,6 @@ public interface AtomicFieldData<Script extends ScriptDocValues> extends RamUsag
      */
     boolean isMultiValued();
 
-    /**
-     * Are the values ordered? (in ascending manner).
-     */
-    boolean isValuesOrdered();
-
     /**
      * The number of docs in this field data.
      */
diff --git a/src/main/java/org/elasticsearch/index/fielddata/ordinals/GlobalOrdinalsIndexFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/ordinals/GlobalOrdinalsIndexFieldData.java
index bcd670f4dac5b..c36e5e54f80fe 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/ordinals/GlobalOrdinalsIndexFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/ordinals/GlobalOrdinalsIndexFieldData.java
@@ -29,11 +29,10 @@
 import org.elasticsearch.index.Index;
 import org.elasticsearch.index.fielddata.*;
 import org.elasticsearch.index.fielddata.fieldcomparator.SortMode;
+import org.elasticsearch.index.fielddata.ordinals.InternalGlobalOrdinalsBuilder.OrdinalMappingSource;
 import org.elasticsearch.index.fielddata.plain.AtomicFieldDataWithOrdinalsTermsEnum;
 import org.elasticsearch.index.mapper.FieldMapper;
 
-import static org.elasticsearch.index.fielddata.ordinals.InternalGlobalOrdinalsBuilder.OrdinalMappingSource;
-
 /**
  * {@link IndexFieldData} impl based on global ordinals.
  */
@@ -162,11 +161,6 @@ public boolean isMultiValued() {
             return afd.isMultiValued();
         }
 
-        @Override
-        public boolean isValuesOrdered() {
-            return afd.isValuesOrdered();
-        }
-
         @Override
         public int getNumDocs() {
             return afd.getNumDocs();
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/AbstractGeoPointIndexFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/AbstractGeoPointIndexFieldData.java
index 886f39bf0fb75..ee62aef40c2ec 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/AbstractGeoPointIndexFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/AbstractGeoPointIndexFieldData.java
@@ -49,11 +49,6 @@ public boolean isMultiValued() {
             return false;
         }
 
-        @Override
-        public boolean isValuesOrdered() {
-            return false;
-        }
-
         @Override
         public long getNumberUniqueValues() {
             return 0;
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/BinaryDVAtomicFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/BinaryDVAtomicFieldData.java
index b550b748f879e..6f30d3d511b28 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/BinaryDVAtomicFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/BinaryDVAtomicFieldData.java
@@ -47,11 +47,6 @@ public boolean isMultiValued() {
         return false;
     }
 
-    @Override
-    public boolean isValuesOrdered() {
-        return true; // single-valued
-    }
-
     @Override
     public int getNumDocs() {
         return reader.maxDoc();
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/BytesBinaryDVAtomicFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/BytesBinaryDVAtomicFieldData.java
index 33be9a3af7116..d25924168b49e 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/BytesBinaryDVAtomicFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/BytesBinaryDVAtomicFieldData.java
@@ -23,7 +23,9 @@
 import org.apache.lucene.index.BinaryDocValues;
 import org.apache.lucene.store.ByteArrayDataInput;
 import org.apache.lucene.util.BytesRef;
-import org.elasticsearch.index.fielddata.*;
+import org.elasticsearch.index.fielddata.AtomicFieldData;
+import org.elasticsearch.index.fielddata.BytesValues;
+import org.elasticsearch.index.fielddata.ScriptDocValues;
 
 final class BytesBinaryDVAtomicFieldData implements AtomicFieldData<ScriptDocValues> {
 
@@ -41,11 +43,6 @@ public boolean isMultiValued() {
         return true;
     }
 
-    @Override
-    public boolean isValuesOrdered() {
-        return false;
-    }
-
     @Override
     public int getNumDocs() {
         return reader.maxDoc();
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/DoubleArrayAtomicFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/DoubleArrayAtomicFieldData.java
index 9ebf9715ec0a0..0b4ec994ae66c 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/DoubleArrayAtomicFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/DoubleArrayAtomicFieldData.java
@@ -72,11 +72,6 @@ public boolean isMultiValued() {
             return false;
         }
 
-        @Override
-        public boolean isValuesOrdered() {
-            return false;
-        }
-
         @Override
         public long getNumberUniqueValues() {
             return 0;
@@ -114,11 +109,6 @@ public boolean isMultiValued() {
             return ordinals.isMultiValued();
         }
 
-        @Override
-        public boolean isValuesOrdered() {
-            return true;
-        }
-
         @Override
         public long getNumberUniqueValues() {
             return ordinals.getNumOrds();
@@ -199,11 +189,6 @@ public boolean isMultiValued() {
             return false;
         }
 
-        @Override
-        public boolean isValuesOrdered() {
-            return false;
-        }
-
         @Override
         public long getNumberUniqueValues() {
             return numOrds;
@@ -297,11 +282,6 @@ public boolean isMultiValued() {
             return false;
         }
 
-        @Override
-        public boolean isValuesOrdered() {
-            return false;
-        }
-
         @Override
         public long getNumberUniqueValues() {
             return numOrds;
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/FSTBytesAtomicFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/FSTBytesAtomicFieldData.java
index 981068adaf2e2..89887a28a55f7 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/FSTBytesAtomicFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/FSTBytesAtomicFieldData.java
@@ -74,11 +74,6 @@ public long getNumberUniqueValues() {
         return ordinals.getNumOrds();
     }
 
-    @Override
-    public boolean isValuesOrdered() {
-        return true;
-    }
-
     @Override
     public long getMemorySizeInBytes() {
         if (size == -1) {
@@ -199,11 +194,6 @@ public int getNumDocs() {
             return ordinals.getNumDocs();
         }
 
-        @Override
-        public boolean isValuesOrdered() {
-            return true;
-        }
-
         @Override
         public BytesValues.WithOrdinals getBytesValues(boolean needsHashes) {
             return new EmptyByteValuesWithOrdinals(ordinals.ordinals());
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/FloatArrayAtomicFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/FloatArrayAtomicFieldData.java
index c0c8c4125a571..db53057c8697d 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/FloatArrayAtomicFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/FloatArrayAtomicFieldData.java
@@ -76,11 +76,6 @@ public long getNumberUniqueValues() {
             return 0;
         }
 
-        @Override
-        public boolean isValuesOrdered() {
-            return false;
-        }
-
         @Override
         public long getMemorySizeInBytes() {
             return 0;
@@ -113,11 +108,6 @@ public boolean isMultiValued() {
             return ordinals.isMultiValued();
         }
 
-        @Override
-        public boolean isValuesOrdered() {
-            return true;
-        }
-
         @Override
         public long getNumberUniqueValues() {
             return ordinals.getNumOrds();
@@ -195,11 +185,6 @@ public boolean isMultiValued() {
             return false;
         }
 
-        @Override
-        public boolean isValuesOrdered() {
-            return false;
-        }
-
         @Override
         public long getNumberUniqueValues() {
             return numOrd;
@@ -295,11 +280,6 @@ public boolean isMultiValued() {
             return false;
         }
 
-        @Override
-        public boolean isValuesOrdered() {
-            return false;
-        }
-
         @Override
         public long getNumberUniqueValues() {
             return numOrd;
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/GeoPointBinaryDVAtomicFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/GeoPointBinaryDVAtomicFieldData.java
index 82605317d9b52..ebcd4e6065cce 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/GeoPointBinaryDVAtomicFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/GeoPointBinaryDVAtomicFieldData.java
@@ -44,11 +44,6 @@ public boolean isMultiValued() {
         return false;
     }
 
-    @Override
-    public boolean isValuesOrdered() {
-        return false;
-    }
-
     @Override
     public int getNumDocs() {
         return reader.maxDoc();
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/GeoPointCompressedAtomicFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/GeoPointCompressedAtomicFieldData.java
index 1b54599e39561..85e99433a8404 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/GeoPointCompressedAtomicFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/GeoPointCompressedAtomicFieldData.java
@@ -74,11 +74,6 @@ public boolean isMultiValued() {
             return ordinals.isMultiValued();
         }
 
-        @Override
-        public boolean isValuesOrdered() {
-            return true;
-        }
-
         @Override
         public long getNumberUniqueValues() {
             return ordinals.getNumOrds();
@@ -152,11 +147,6 @@ public boolean isMultiValued() {
             return false;
         }
 
-        @Override
-        public boolean isValuesOrdered() {
-            return false;
-        }
-
         @Override
         public long getNumberUniqueValues() {
             return numOrds;
@@ -227,11 +217,6 @@ public boolean isMultiValued() {
             return false;
         }
 
-        @Override
-        public boolean isValuesOrdered() {
-            return false;
-        }
-
         @Override
         public long getNumberUniqueValues() {
             return numOrds;
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/GeoPointDoubleArrayAtomicFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/GeoPointDoubleArrayAtomicFieldData.java
index 9c2bd9335b297..f340d7af3b828 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/GeoPointDoubleArrayAtomicFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/GeoPointDoubleArrayAtomicFieldData.java
@@ -70,11 +70,6 @@ public boolean isMultiValued() {
             return ordinals.isMultiValued();
         }
 
-        @Override
-        public boolean isValuesOrdered() {
-            return true;
-        }
-
         @Override
         public long getNumberUniqueValues() {
             return ordinals.getNumOrds();
@@ -144,11 +139,6 @@ public boolean isMultiValued() {
             return false;
         }
 
-        @Override
-        public boolean isValuesOrdered() {
-            return false;
-        }
-
         @Override
         public long getNumberUniqueValues() {
             return numOrds;
@@ -216,11 +206,6 @@ public boolean isMultiValued() {
             return false;
         }
 
-        @Override
-        public boolean isValuesOrdered() {
-            return false;
-        }
-
         @Override
         public long getNumberUniqueValues() {
             return numOrds;
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/NumericDVAtomicFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/NumericDVAtomicFieldData.java
index d315917fb5dda..54573effea6c8 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/NumericDVAtomicFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/NumericDVAtomicFieldData.java
@@ -52,11 +52,6 @@ public boolean isMultiValued() {
         return false;
     }
 
-    @Override
-    public boolean isValuesOrdered() {
-        return true; // single-valued
-    }
-
     @Override
     public int getNumDocs() {
         return reader.maxDoc();
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/PackedArrayAtomicFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/PackedArrayAtomicFieldData.java
index 43d10aaa646ad..59a207e574bd5 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/PackedArrayAtomicFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/PackedArrayAtomicFieldData.java
@@ -73,11 +73,6 @@ public boolean isMultiValued() {
             return false;
         }
 
-        @Override
-        public boolean isValuesOrdered() {
-            return false;
-        }
-
         @Override
         public long getMemorySizeInBytes() {
             return 0;
@@ -115,11 +110,6 @@ public boolean isMultiValued() {
             return ordinals.isMultiValued();
         }
 
-        @Override
-        public boolean isValuesOrdered() {
-            return true;
-        }
-
         @Override
         public long getMemorySizeInBytes() {
             if (size == -1) {
@@ -202,11 +192,6 @@ public boolean isMultiValued() {
             return false;
         }
 
-        @Override
-        public boolean isValuesOrdered() {
-            return false;
-        }
-
         @Override
         public long getNumberUniqueValues() {
             return numOrds;
@@ -306,11 +291,6 @@ public boolean isMultiValued() {
             return false;
         }
 
-        @Override
-        public boolean isValuesOrdered() {
-            return false;
-        }
-
         @Override
         public long getNumberUniqueValues() {
             return numOrds;
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/PagedBytesAtomicFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/PagedBytesAtomicFieldData.java
index a7e3549522220..3949d0d1e64de 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/PagedBytesAtomicFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/PagedBytesAtomicFieldData.java
@@ -74,11 +74,6 @@ public long getNumberUniqueValues() {
         return ordinals.getNumOrds();
     }
 
-    @Override
-    public boolean isValuesOrdered() {
-        return true;
-    }
-
     @Override
     public long getMemorySizeInBytes() {
         if (size == -1) {
@@ -209,11 +204,6 @@ public long getNumberUniqueValues() {
             return 0;
         }
 
-        @Override
-        public boolean isValuesOrdered() {
-            return true;
-        }
-
         @Override
         public BytesValues.WithOrdinals getBytesValues(boolean needsHashes) {
             return new EmptyByteValuesWithOrdinals(ordinals.ordinals());
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/ParentChildAtomicFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/ParentChildAtomicFieldData.java
index 1bf1b83e7f334..f976af49b1160 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/ParentChildAtomicFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/ParentChildAtomicFieldData.java
@@ -55,11 +55,6 @@ public boolean isMultiValued() {
         return true;
     }
 
-    @Override
-    public boolean isValuesOrdered() {
-        return true;
-    }
-
     @Override
     public int getNumDocs() {
         return numDocs;
diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/SortedSetDVBytesAtomicFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/SortedSetDVBytesAtomicFieldData.java
index 39564af439421..73d3ab396b240 100644
--- a/src/main/java/org/elasticsearch/index/fielddata/plain/SortedSetDVBytesAtomicFieldData.java
+++ b/src/main/java/org/elasticsearch/index/fielddata/plain/SortedSetDVBytesAtomicFieldData.java
@@ -37,11 +37,6 @@ public final class SortedSetDVBytesAtomicFieldData extends SortedSetDVAtomicFiel
         super(reader, field);
     }
 
-    @Override
-    public boolean isValuesOrdered() {
-        return true;
-    }
-
     @Override
     public Strings getScriptValues() {
         return new ScriptDocValues.Strings(getBytesValues(false));

From ecab74fe6ca8b135ecf756ffd977322e5df960ed Mon Sep 17 00:00:00 2001
From: Kevin Wang <kevin807359@gmail.com>
Date: Mon, 7 Apr 2014 12:20:46 +1000
Subject: [PATCH 12/54] add lucene language model similarities (Dirichlet &
 JelinekMercer)

---
 .../index-modules/similarity.asciidoc         | 25 +++++++++
 .../LMDirichletSimilarityProvider.java        | 55 +++++++++++++++++++
 .../LMJelinekMercerSimilarityProvider.java    | 55 +++++++++++++++++++
 .../index/similarity/SimilarityTests.java     | 41 ++++++++++++++
 4 files changed, 176 insertions(+)
 create mode 100644 src/main/java/org/elasticsearch/index/similarity/LMDirichletSimilarityProvider.java
 create mode 100644 src/main/java/org/elasticsearch/index/similarity/LMJelinekMercerSimilarityProvider.java

diff --git a/docs/reference/index-modules/similarity.asciidoc b/docs/reference/index-modules/similarity.asciidoc
index ae9f368f3f143..0c95709a860d1 100644
--- a/docs/reference/index-modules/similarity.asciidoc
+++ b/docs/reference/index-modules/similarity.asciidoc
@@ -121,6 +121,31 @@ based model] . This similarity has the following options:
 
 Type name: `IB`
 
+[float]
+[[lm_dirichlet]]
+==== LM Dirichlet similarity.
+
+http://lucene.apache.org/core/4_7_1/core/org/apache/lucene/search/similarities/LMDirichletSimilarity.html[LM
+Dirichlet similarity] . This similarity has the following options:
+
+[horizontal]
+`mu`::  Default to `2000`.
+
+Type name: `LMDirichlet`
+
+[float]
+[[lm_jelinek_mercer]]
+==== LM Jelinek Mercer similarity.
+
+http://lucene.apache.org/core/4_7_1/core/org/apache/lucene/search/similarities/LMJelinekMercerSimilarity.html[LM
+Jelinek Mercer similarity] . This similarity has the following options:
+
+[horizontal]
+`lambda`::  The optimal value depends on both the collection and the query. The optimal value is around `0.1`
+for title queries and `0.7` for long queries. Default to `0.1`.
+
+Type name: `LMJelinekMercer`
+
 [float]
 [[default-base]]
 ==== Default and Base Similarities
diff --git a/src/main/java/org/elasticsearch/index/similarity/LMDirichletSimilarityProvider.java b/src/main/java/org/elasticsearch/index/similarity/LMDirichletSimilarityProvider.java
new file mode 100644
index 0000000000000..797ce6417e81e
--- /dev/null
+++ b/src/main/java/org/elasticsearch/index/similarity/LMDirichletSimilarityProvider.java
@@ -0,0 +1,55 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.index.similarity;
+
+import org.apache.lucene.search.similarities.LMDirichletSimilarity;
+import org.apache.lucene.search.similarities.Similarity;
+import org.elasticsearch.common.inject.Inject;
+import org.elasticsearch.common.inject.assistedinject.Assisted;
+import org.elasticsearch.common.settings.Settings;
+
+/**
+ * {@link SimilarityProvider} for {@link LMDirichletSimilarity}.
+ * <p/>
+ * Configuration options available:
+ * <ul>
+ *     <li>mu</li>
+ * </ul>
+ * @see LMDirichletSimilarity For more information about configuration
+ */
+public class LMDirichletSimilarityProvider extends AbstractSimilarityProvider {
+
+    private final LMDirichletSimilarity similarity;
+
+    @Inject
+    public LMDirichletSimilarityProvider(@Assisted String name, @Assisted Settings settings) {
+        super(name);
+        float mu = settings.getAsFloat("mu", 2000f);
+        this.similarity = new LMDirichletSimilarity(mu);
+    }
+
+    /**
+     * {@inheritDoc}
+     */
+    @Override
+    public Similarity get() {
+        return similarity;
+    }
+}
diff --git a/src/main/java/org/elasticsearch/index/similarity/LMJelinekMercerSimilarityProvider.java b/src/main/java/org/elasticsearch/index/similarity/LMJelinekMercerSimilarityProvider.java
new file mode 100644
index 0000000000000..9be02366b6386
--- /dev/null
+++ b/src/main/java/org/elasticsearch/index/similarity/LMJelinekMercerSimilarityProvider.java
@@ -0,0 +1,55 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.index.similarity;
+
+import org.apache.lucene.search.similarities.LMJelinekMercerSimilarity;
+import org.apache.lucene.search.similarities.Similarity;
+import org.elasticsearch.common.inject.Inject;
+import org.elasticsearch.common.inject.assistedinject.Assisted;
+import org.elasticsearch.common.settings.Settings;
+
+/**
+ * {@link SimilarityProvider} for {@link LMJelinekMercerSimilarity}.
+ * <p/>
+ * Configuration options available:
+ * <ul>
+ *     <li>lambda</li>
+ * </ul>
+ * @see LMJelinekMercerSimilarity For more information about configuration
+ */
+public class LMJelinekMercerSimilarityProvider extends AbstractSimilarityProvider {
+
+    private final LMJelinekMercerSimilarity similarity;
+
+    @Inject
+    public LMJelinekMercerSimilarityProvider(@Assisted String name, @Assisted Settings settings) {
+        super(name);
+        float lambda = settings.getAsFloat("lambda", 0.1f);
+        this.similarity = new LMJelinekMercerSimilarity(lambda);
+    }
+
+    /**
+     * {@inheritDoc}
+     */
+    @Override
+    public Similarity get() {
+        return similarity;
+    }
+}
diff --git a/src/test/java/org/elasticsearch/index/similarity/SimilarityTests.java b/src/test/java/org/elasticsearch/index/similarity/SimilarityTests.java
index 8bf8586a6a84f..a3f9753f4b577 100644
--- a/src/test/java/org/elasticsearch/index/similarity/SimilarityTests.java
+++ b/src/test/java/org/elasticsearch/index/similarity/SimilarityTests.java
@@ -151,6 +151,47 @@ public void testResolveSimilaritiesFromMapping_IB() throws IOException {
         assertThat(((NormalizationH2) similarity.getNormalization()).getC(), equalTo(3f));
     }
 
+    @Test
+    public void testResolveSimilaritiesFromMapping_LMDirichlet() throws IOException {
+        String mapping = XContentFactory.jsonBuilder().startObject().startObject("type")
+                .startObject("properties")
+                .startObject("field1").field("type", "string").field("similarity", "my_similarity").endObject()
+                .endObject()
+                .endObject().endObject().string();
+
+        Settings indexSettings = ImmutableSettings.settingsBuilder()
+                .put("index.similarity.my_similarity.type", "LMDirichlet")
+                .put("index.similarity.my_similarity.mu", 3000f)
+                .build();
+        SimilarityService similarityService = similarityService(indexSettings);
+        DocumentMapper documentMapper = similarityService.mapperService().documentMapperParser().parse(mapping);
+        assertThat(documentMapper.mappers().name("field1").mapper().similarity(), instanceOf(LMDirichletSimilarityProvider.class));
+
+        LMDirichletSimilarity similarity = (LMDirichletSimilarity) documentMapper.mappers().name("field1").mapper().similarity().get();
+        assertThat(similarity.getMu(), equalTo(3000f));
+    }
+
+    @Test
+    public void testResolveSimilaritiesFromMapping_LMJelinekMercer() throws IOException {
+        String mapping = XContentFactory.jsonBuilder().startObject().startObject("type")
+                .startObject("properties")
+                .startObject("field1").field("type", "string").field("similarity", "my_similarity").endObject()
+                .endObject()
+                .endObject().endObject().string();
+
+        Settings indexSettings = ImmutableSettings.settingsBuilder()
+                .put("index.similarity.my_similarity.type", "LMJelinekMercer")
+                .put("index.similarity.my_similarity.lambda", 0.7f)
+                .build();
+        SimilarityService similarityService = similarityService(indexSettings);
+        DocumentMapper documentMapper = similarityService.mapperService().documentMapperParser().parse(mapping);
+        assertThat(documentMapper.mappers().name("field1").mapper().similarity(), instanceOf(LMJelinekMercerSimilarityProvider.class));
+
+        LMJelinekMercerSimilarity similarity = (LMJelinekMercerSimilarity) documentMapper.mappers().name("field1").mapper().similarity().get();
+        assertThat(similarity.getLambda(), equalTo(0.7f));
+    }
+
+
     private static SimilarityService similarityService() {
         return similarityService(ImmutableSettings.Builder.EMPTY_SETTINGS);
     }

From d64d0d6a97ae8dd70ba6018f97dd3c5fc10acd53 Mon Sep 17 00:00:00 2001
From: Shay Banon <kimchy@gmail.com>
Date: Mon, 7 Apr 2014 01:53:37 +0200
Subject: [PATCH 13/54] Remove clear on mock page/array since we use a shared
 cluster, calling clear on the mock array / page recycler can cause removing a
 valid on going reference, and then when its released, the release will fail
 because it can't be found. There is no real reason to call clear, checking if
 pages/arrays have been released takes the snapshot behavior here into
 account. This change also makes sure we don't use the mock classes in places
 where we don't really release.

Note, with this change DoubleTermsTests fails, since it causes failures when creating aggs in the pre process phase, causing obtained arrays not to be released. This needs to be fixed before pulling this change in.
---
 .../common/bytes/PagedBytesReferenceTest.java         |  6 ++----
 .../search/child/ChildrenConstantScoreQueryTests.java |  6 +++---
 .../org/elasticsearch/test/ElasticsearchTestCase.java | 11 -----------
 .../test/cache/recycler/MockBigArrays.java            |  5 +----
 .../test/cache/recycler/MockPageCacheRecycler.java    |  5 +----
 5 files changed, 7 insertions(+), 26 deletions(-)

diff --git a/src/test/java/org/elasticsearch/common/bytes/PagedBytesReferenceTest.java b/src/test/java/org/elasticsearch/common/bytes/PagedBytesReferenceTest.java
index 7bd234bff8a56..499c1c87287d3 100644
--- a/src/test/java/org/elasticsearch/common/bytes/PagedBytesReferenceTest.java
+++ b/src/test/java/org/elasticsearch/common/bytes/PagedBytesReferenceTest.java
@@ -40,18 +40,16 @@ public class PagedBytesReferenceTest extends ElasticsearchTestCase {
 
     private static final int PAGE_SIZE = BigArrays.BYTE_PAGE_SIZE;
 
-    private MockBigArrays bigarrays;
+    private BigArrays bigarrays;
 
     @Before
     public void setUp() throws Exception {
         super.setUp();
-        bigarrays = new MockBigArrays(ImmutableSettings.EMPTY, null);
+        bigarrays = new BigArrays(ImmutableSettings.EMPTY, null);
     }
 
     @After
     public void tearDown() throws Exception {
-        // necessary since we currently never release BigArrays
-        MockBigArrays.reset();
         super.tearDown();
     }
 
diff --git a/src/test/java/org/elasticsearch/index/search/child/ChildrenConstantScoreQueryTests.java b/src/test/java/org/elasticsearch/index/search/child/ChildrenConstantScoreQueryTests.java
index 3bea8d49484d9..cad1ff15b0566 100644
--- a/src/test/java/org/elasticsearch/index/search/child/ChildrenConstantScoreQueryTests.java
+++ b/src/test/java/org/elasticsearch/index/search/child/ChildrenConstantScoreQueryTests.java
@@ -346,8 +346,9 @@ static String reason(FixedBitSet actual, FixedBitSet expected, IndexSearcher ind
     static SearchContext createSearchContext(String indexName, String parentType, String childType) throws IOException {
         final Index index = new Index(indexName);
         final CacheRecycler cacheRecycler = new CacheRecycler(ImmutableSettings.EMPTY);
-        final PageCacheRecycler pageCacheRecycler = new PageCacheRecycler(ImmutableSettings.EMPTY, new ThreadPool());
-        final BigArrays bigArrays = new MockBigArrays(ImmutableSettings.EMPTY, pageCacheRecycler);
+        ThreadPool threadPool = new ThreadPool();
+        final PageCacheRecycler pageCacheRecycler = new PageCacheRecycler(ImmutableSettings.EMPTY, threadPool);
+        final BigArrays bigArrays = new BigArrays(ImmutableSettings.EMPTY, pageCacheRecycler);
         Settings settings = ImmutableSettings.EMPTY;
         MapperService mapperService = MapperTestUtils.newMapperService(index, settings);
         IndexFieldDataService indexFieldDataService = new IndexFieldDataService(index, new DummyCircuitBreakerService());
@@ -358,7 +359,6 @@ static SearchContext createSearchContext(String indexName, String parentType, St
                 childType, new CompressedString(PutMappingRequest.buildFromSimplifiedDef(childType, "_parent", "type=" + parentType).string()), true
         );
 
-        ThreadPool threadPool = new ThreadPool();
         NodeSettingsService nodeSettingsService = new NodeSettingsService(settings);
         IndicesFilterCache indicesFilterCache = new IndicesFilterCache(settings, threadPool, cacheRecycler, nodeSettingsService);
         WeightedFilterCache filterCache = new WeightedFilterCache(index, settings, indicesFilterCache);
diff --git a/src/test/java/org/elasticsearch/test/ElasticsearchTestCase.java b/src/test/java/org/elasticsearch/test/ElasticsearchTestCase.java
index 1d13ebc4b5737..a4c0990c10848 100644
--- a/src/test/java/org/elasticsearch/test/ElasticsearchTestCase.java
+++ b/src/test/java/org/elasticsearch/test/ElasticsearchTestCase.java
@@ -128,22 +128,11 @@ public File getResource(String relativePath) {
         return new File(uri);
     }
 
-    @Before
-    public void resetPageTracking() {
-        MockPageCacheRecycler.reset();
-    }
-
     @After
     public void ensureAllPagesReleased() throws Exception {
         MockPageCacheRecycler.ensureAllPagesAreReleased();
     }
 
-    @Before
-    public void resetArrayTracking() {
-        // useful if there are tests that use MockBigArrays but don't inherit from ElasticsearchTestCase
-        MockBigArrays.reset();
-    }
-
     @After
     public void ensureAllArraysReleased() throws Exception {
         MockBigArrays.ensureAllArraysAreReleased();
diff --git a/src/test/java/org/elasticsearch/test/cache/recycler/MockBigArrays.java b/src/test/java/org/elasticsearch/test/cache/recycler/MockBigArrays.java
index 5d6b4b5b626e0..873990492da11 100644
--- a/src/test/java/org/elasticsearch/test/cache/recycler/MockBigArrays.java
+++ b/src/test/java/org/elasticsearch/test/cache/recycler/MockBigArrays.java
@@ -57,10 +57,6 @@ public static void discardNextCheck() {
         DISCARD = true;
     }
 
-    public static void reset() {
-        ACQUIRED_ARRAYS.clear();
-    }
-
     public static void ensureAllArraysAreReleased() throws Exception {
         if (DISCARD) {
             DISCARD = false;
@@ -82,6 +78,7 @@ public boolean apply(Object input) {
                 return;
             }
             masterCopy.keySet().retainAll(ACQUIRED_ARRAYS.keySet());
+            ACQUIRED_ARRAYS.keySet().removeAll(masterCopy.keySet()); // remove all existing master copy we will report on
             if (!masterCopy.isEmpty()) {
                 final Object cause = masterCopy.entrySet().iterator().next().getValue();
                 throw new RuntimeException(masterCopy.size() + " arrays have not been released", cause instanceof Throwable ? (Throwable) cause : null);
diff --git a/src/test/java/org/elasticsearch/test/cache/recycler/MockPageCacheRecycler.java b/src/test/java/org/elasticsearch/test/cache/recycler/MockPageCacheRecycler.java
index 2866c71e1bdfe..25145bcb4c9ab 100644
--- a/src/test/java/org/elasticsearch/test/cache/recycler/MockPageCacheRecycler.java
+++ b/src/test/java/org/elasticsearch/test/cache/recycler/MockPageCacheRecycler.java
@@ -40,10 +40,6 @@ public class MockPageCacheRecycler extends PageCacheRecycler {
 
     private static final ConcurrentMap<Object, Throwable> ACQUIRED_PAGES = Maps.newConcurrentMap();
 
-    public static void reset() {
-        ACQUIRED_PAGES.clear();
-    }
-
     public static void ensureAllPagesAreReleased() throws Exception {
         final Map<Object, Throwable> masterCopy = Maps.newHashMap(ACQUIRED_PAGES);
         if (masterCopy.isEmpty()) {
@@ -62,6 +58,7 @@ public boolean apply(Object input) {
             return;
         }
         masterCopy.keySet().retainAll(ACQUIRED_PAGES.keySet());
+        ACQUIRED_PAGES.keySet().removeAll(masterCopy.keySet()); // remove all existing master copy we will report on
         if (!masterCopy.isEmpty()) {
             final Throwable t = masterCopy.entrySet().iterator().next().getValue();
             throw new RuntimeException(masterCopy.size() + " pages have not been released", t);

From 37c07ef7659d72dc1e97c0edf7f85c4398477e3a Mon Sep 17 00:00:00 2001
From: Shay Banon <kimchy@gmail.com>
Date: Mon, 7 Apr 2014 11:25:01 +0200
Subject: [PATCH 14/54] disable args tests that cause page/array leak related
 to #5703

---
 .../search/aggregations/bucket/DoubleTermsTests.java     | 9 +++++++++
 .../search/aggregations/bucket/LongTermsTests.java       | 6 ++++++
 .../search/aggregations/bucket/NestedTests.java          | 2 ++
 .../search/aggregations/bucket/StringTermsTests.java     | 5 +++++
 4 files changed, 22 insertions(+)

diff --git a/src/test/java/org/elasticsearch/search/aggregations/bucket/DoubleTermsTests.java b/src/test/java/org/elasticsearch/search/aggregations/bucket/DoubleTermsTests.java
index a1e40b2b0a52a..b964436f863a7 100644
--- a/src/test/java/org/elasticsearch/search/aggregations/bucket/DoubleTermsTests.java
+++ b/src/test/java/org/elasticsearch/search/aggregations/bucket/DoubleTermsTests.java
@@ -18,6 +18,7 @@
  */
 package org.elasticsearch.search.aggregations.bucket;
 
+import org.apache.lucene.util.LuceneTestCase;
 import org.elasticsearch.ElasticsearchException;
 import org.elasticsearch.action.index.IndexRequestBuilder;
 import org.elasticsearch.action.search.SearchResponse;
@@ -216,6 +217,7 @@ public void singleValueField_OrderedByTermDesc() throws Exception {
         }
     }
 
+    @LuceneTestCase.AwaitsFix(bugUrl = "https://github.com/elasticsearch/elasticsearch/issues/5703")
     @Test
     public void singleValuedField_WithSubAggregation() throws Exception {
         SearchResponse response = client().prepareSearch("idx").setTypes("type")
@@ -431,6 +433,7 @@ public void multiValuedField_WithValueScript_WithInheritedSubAggregator() throws
         }
     }
 
+    @LuceneTestCase.AwaitsFix(bugUrl = "https://github.com/elasticsearch/elasticsearch/issues/5703")
     @Test
     public void script_SingleValue() throws Exception {
         SearchResponse response = client().prepareSearch("idx").setTypes("type")
@@ -612,6 +615,7 @@ public void partiallyUnmapped() throws Exception {
         }
     }
 
+    @LuceneTestCase.AwaitsFix(bugUrl = "https://github.com/elasticsearch/elasticsearch/issues/5703")
     @Test
     public void emptyAggregation() throws Exception {
         SearchResponse searchResponse = client().prepareSearch("empty_bucket_idx")
@@ -752,6 +756,7 @@ public void singleValuedField_OrderedBySubAggregationAsc_MultiHierarchyLevels()
         assertThat(max.getValue(), equalTo(asc ? 4.0 : 2.0));
     }
 
+    @LuceneTestCase.AwaitsFix(bugUrl = "https://github.com/elasticsearch/elasticsearch/issues/5703")
     @Test
     public void singleValuedField_OrderedByMissingSubAggregation() throws Exception {
 
@@ -771,6 +776,7 @@ public void singleValuedField_OrderedByMissingSubAggregation() throws Exception
         }
     }
 
+    @LuceneTestCase.AwaitsFix(bugUrl = "https://github.com/elasticsearch/elasticsearch/issues/5703")
     @Test
     public void singleValuedField_OrderedByNonMetricsOrMultiBucketSubAggregation() throws Exception {
 
@@ -791,6 +797,7 @@ public void singleValuedField_OrderedByNonMetricsOrMultiBucketSubAggregation() t
         }
     }
 
+    @LuceneTestCase.AwaitsFix(bugUrl = "https://github.com/elasticsearch/elasticsearch/issues/5703")
     @Test
     public void singleValuedField_OrderedByMultiValuedSubAggregation_WithUknownMetric() throws Exception {
 
@@ -812,6 +819,7 @@ public void singleValuedField_OrderedByMultiValuedSubAggregation_WithUknownMetri
         }
     }
 
+    @LuceneTestCase.AwaitsFix(bugUrl = "https://github.com/elasticsearch/elasticsearch/issues/5703")
     @Test
     public void singleValuedField_OrderedByMultiValuedSubAggregation_WithoutMetric() throws Exception {
 
@@ -833,6 +841,7 @@ public void singleValuedField_OrderedByMultiValuedSubAggregation_WithoutMetric()
         }
     }
 
+    @LuceneTestCase.AwaitsFix(bugUrl = "https://github.com/elasticsearch/elasticsearch/issues/5703")
     @Test
     public void singleValuedField_OrderedBySingleValueSubAggregationDesc() throws Exception {
         boolean asc = false;
diff --git a/src/test/java/org/elasticsearch/search/aggregations/bucket/LongTermsTests.java b/src/test/java/org/elasticsearch/search/aggregations/bucket/LongTermsTests.java
index 27609e840e1a1..7835714925451 100644
--- a/src/test/java/org/elasticsearch/search/aggregations/bucket/LongTermsTests.java
+++ b/src/test/java/org/elasticsearch/search/aggregations/bucket/LongTermsTests.java
@@ -18,6 +18,7 @@
  */
 package org.elasticsearch.search.aggregations.bucket;
 
+import org.apache.lucene.util.LuceneTestCase;
 import org.elasticsearch.ElasticsearchException;
 import org.elasticsearch.action.index.IndexRequestBuilder;
 import org.elasticsearch.action.search.SearchResponse;
@@ -512,6 +513,7 @@ public void script_MultiValued() throws Exception {
         }
     }
 
+    @LuceneTestCase.AwaitsFix(bugUrl = "https://github.com/elasticsearch/elasticsearch/issues/5703")
     @Test
     public void script_MultiValued_WithAggregatorInherited_NoExplicitType() throws Exception {
 
@@ -751,6 +753,7 @@ public void singleValuedField_OrderedBySubAggregationAsc_MultiHierarchyLevels()
         assertThat(max.getValue(), equalTo(asc ? 4.0 : 2.0));
     }
 
+    @LuceneTestCase.AwaitsFix(bugUrl = "https://github.com/elasticsearch/elasticsearch/issues/5703")
     @Test
     public void singleValuedField_OrderedByMissingSubAggregation() throws Exception {
 
@@ -770,6 +773,7 @@ public void singleValuedField_OrderedByMissingSubAggregation() throws Exception
         }
     }
 
+    @LuceneTestCase.AwaitsFix(bugUrl = "https://github.com/elasticsearch/elasticsearch/issues/5703")
     @Test
     public void singleValuedField_OrderedByNonMetricsOrMultiBucketSubAggregation() throws Exception {
 
@@ -790,6 +794,7 @@ public void singleValuedField_OrderedByNonMetricsOrMultiBucketSubAggregation() t
         }
     }
 
+    @LuceneTestCase.AwaitsFix(bugUrl = "https://github.com/elasticsearch/elasticsearch/issues/5703")
     @Test
     public void singleValuedField_OrderedByMultiValuedSubAggregation_WithUknownMetric() throws Exception {
 
@@ -811,6 +816,7 @@ public void singleValuedField_OrderedByMultiValuedSubAggregation_WithUknownMetri
         }
     }
 
+    @LuceneTestCase.AwaitsFix(bugUrl = "https://github.com/elasticsearch/elasticsearch/issues/5703")
     @Test
     public void singleValuedField_OrderedByMultiValuedSubAggregation_WithoutMetric() throws Exception {
 
diff --git a/src/test/java/org/elasticsearch/search/aggregations/bucket/NestedTests.java b/src/test/java/org/elasticsearch/search/aggregations/bucket/NestedTests.java
index 1deb10804546f..e75699a9f5b73 100644
--- a/src/test/java/org/elasticsearch/search/aggregations/bucket/NestedTests.java
+++ b/src/test/java/org/elasticsearch/search/aggregations/bucket/NestedTests.java
@@ -18,6 +18,7 @@
  */
 package org.elasticsearch.search.aggregations.bucket;
 
+import org.apache.lucene.util.LuceneTestCase;
 import org.elasticsearch.ElasticsearchException;
 import org.elasticsearch.action.index.IndexRequestBuilder;
 import org.elasticsearch.action.search.SearchResponse;
@@ -145,6 +146,7 @@ public void simple() throws Exception {
         assertThat(stats.getAvg(), equalTo((double) sum / count));
     }
 
+    @LuceneTestCase.AwaitsFix(bugUrl = "https://github.com/elasticsearch/elasticsearch/issues/5703")
     @Test
     public void onNonNestedField() throws Exception {
         MockBigArrays.discardNextCheck();
diff --git a/src/test/java/org/elasticsearch/search/aggregations/bucket/StringTermsTests.java b/src/test/java/org/elasticsearch/search/aggregations/bucket/StringTermsTests.java
index 7fe895e35cf18..a511e734dd082 100644
--- a/src/test/java/org/elasticsearch/search/aggregations/bucket/StringTermsTests.java
+++ b/src/test/java/org/elasticsearch/search/aggregations/bucket/StringTermsTests.java
@@ -19,6 +19,7 @@
 package org.elasticsearch.search.aggregations.bucket;
 
 import com.google.common.base.Strings;
+import org.apache.lucene.util.LuceneTestCase;
 import org.elasticsearch.ElasticsearchException;
 import org.elasticsearch.action.index.IndexRequestBuilder;
 import org.elasticsearch.action.search.SearchResponse;
@@ -942,6 +943,7 @@ public void singleValuedField_OrderedBySubAggregationAsc_MultiHierarchyLevels()
     }
 
 
+    @LuceneTestCase.AwaitsFix(bugUrl = "https://github.com/elasticsearch/elasticsearch/issues/5703")
     @Test
     public void singleValuedField_OrderedByMissingSubAggregation() throws Exception {
 
@@ -962,6 +964,7 @@ public void singleValuedField_OrderedByMissingSubAggregation() throws Exception
         }
     }
 
+    @LuceneTestCase.AwaitsFix(bugUrl = "https://github.com/elasticsearch/elasticsearch/issues/5703")
     @Test
     public void singleValuedField_OrderedByNonMetricsOrMultiBucketSubAggregation() throws Exception {
 
@@ -983,6 +986,7 @@ public void singleValuedField_OrderedByNonMetricsOrMultiBucketSubAggregation() t
         }
     }
 
+    @LuceneTestCase.AwaitsFix(bugUrl = "https://github.com/elasticsearch/elasticsearch/issues/5703")
     @Test
     public void singleValuedField_OrderedByMultiValuedSubAggregation_WithUknownMetric() throws Exception {
 
@@ -1004,6 +1008,7 @@ public void singleValuedField_OrderedByMultiValuedSubAggregation_WithUknownMetri
         }
     }
 
+    @LuceneTestCase.AwaitsFix(bugUrl = "https://github.com/elasticsearch/elasticsearch/issues/5703")
     @Test
     public void singleValuedField_OrderedByMultiValuedSubAggregation_WithoutMetric() throws Exception {
 

From 40a1ac82ef8d8177bfaf59529067761e4e35d71d Mon Sep 17 00:00:00 2001
From: Miltos Allamanis <m.allamanis@ed.ac.uk>
Date: Mon, 10 Feb 2014 19:19:06 +0000
Subject: [PATCH 15/54] Renamed XContentParser.Token named "t" to "token".

The name "token" was declared 191 times for XContentParser.Token
objects, while "t" was used only 6 times.
---
 .../action/update/UpdateRequest.java          |  8 ++--
 .../cluster/metadata/MappingMetaData.java     | 18 ++++-----
 .../common/xcontent/XContentHelper.java       |  8 ++--
 .../support/AbstractXContentParser.java       | 40 +++++++++----------
 4 files changed, 37 insertions(+), 37 deletions(-)

diff --git a/src/main/java/org/elasticsearch/action/update/UpdateRequest.java b/src/main/java/org/elasticsearch/action/update/UpdateRequest.java
index 590af784cb932..87e11c6231635 100644
--- a/src/main/java/org/elasticsearch/action/update/UpdateRequest.java
+++ b/src/main/java/org/elasticsearch/action/update/UpdateRequest.java
@@ -537,13 +537,13 @@ public UpdateRequest source(byte[] source, int offset, int length) throws Except
     public UpdateRequest source(BytesReference source) throws Exception {
         XContentType xContentType = XContentFactory.xContentType(source);
         try (XContentParser parser = XContentFactory.xContent(xContentType).createParser(source)) {
-            XContentParser.Token t = parser.nextToken();
-            if (t == null) {
+            XContentParser.Token token = parser.nextToken();
+            if (token == null) {
                 return this;
             }
             String currentFieldName = null;
-            while ((t = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
-                if (t == XContentParser.Token.FIELD_NAME) {
+            while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
+                if (token == XContentParser.Token.FIELD_NAME) {
                     currentFieldName = parser.currentName();
                 } else if ("script".equals(currentFieldName)) {
                     script = parser.textOrNull();
diff --git a/src/main/java/org/elasticsearch/cluster/metadata/MappingMetaData.java b/src/main/java/org/elasticsearch/cluster/metadata/MappingMetaData.java
index 1f7cbc18fa1bd..b31bd52b9839f 100644
--- a/src/main/java/org/elasticsearch/cluster/metadata/MappingMetaData.java
+++ b/src/main/java/org/elasticsearch/cluster/metadata/MappingMetaData.java
@@ -434,28 +434,28 @@ private void innerParse(XContentParser parser, ParseContext context) throws IOEx
             return;
         }
 
-        XContentParser.Token t = parser.currentToken();
-        if (t == null) {
-            t = parser.nextToken();
+        XContentParser.Token token = parser.currentToken();
+        if (token == null) {
+            token = parser.nextToken();
         }
-        if (t == XContentParser.Token.START_OBJECT) {
-            t = parser.nextToken();
+        if (token == XContentParser.Token.START_OBJECT) {
+            token = parser.nextToken();
         }
         String idPart = context.idParsingStillNeeded() ? id().pathElements()[context.locationId] : null;
         String routingPart = context.routingParsingStillNeeded() ? routing().pathElements()[context.locationRouting] : null;
         String timestampPart = context.timestampParsingStillNeeded() ? timestamp().pathElements()[context.locationTimestamp] : null;
 
-        for (; t == XContentParser.Token.FIELD_NAME; t = parser.nextToken()) {
+        for (; token == XContentParser.Token.FIELD_NAME; token = parser.nextToken()) {
             // Must point to field name
             String fieldName = parser.currentName();
             // And then the value...
-            t = parser.nextToken();
+            token = parser.nextToken();
             boolean incLocationId = false;
             boolean incLocationRouting = false;
             boolean incLocationTimestamp = false;
             if (context.idParsingStillNeeded() && fieldName.equals(idPart)) {
                 if (context.locationId + 1 == id.pathElements().length) {
-                    if (!t.isValue()) {
+                    if (!token.isValue()) {
                         throw new MapperParsingException("id field must be a value but was either an object or an array");
                     }
                     context.id = parser.textOrNull();
@@ -482,7 +482,7 @@ private void innerParse(XContentParser parser, ParseContext context) throws IOEx
             }
 
             if (incLocationId || incLocationRouting || incLocationTimestamp) {
-                if (t == XContentParser.Token.START_OBJECT) {
+                if (token == XContentParser.Token.START_OBJECT) {
                     context.locationId += incLocationId ? 1 : 0;
                     context.locationRouting += incLocationRouting ? 1 : 0;
                     context.locationTimestamp += incLocationTimestamp ? 1 : 0;
diff --git a/src/main/java/org/elasticsearch/common/xcontent/XContentHelper.java b/src/main/java/org/elasticsearch/common/xcontent/XContentHelper.java
index f8dfea5d251db..34a15327986b3 100644
--- a/src/main/java/org/elasticsearch/common/xcontent/XContentHelper.java
+++ b/src/main/java/org/elasticsearch/common/xcontent/XContentHelper.java
@@ -271,16 +271,16 @@ private static boolean allListValuesAreMapsOfOne(List list) {
     }
 
     public static void copyCurrentStructure(XContentGenerator generator, XContentParser parser) throws IOException {
-        XContentParser.Token t = parser.currentToken();
+        XContentParser.Token token = parser.currentToken();
 
         // Let's handle field-name separately first
-        if (t == XContentParser.Token.FIELD_NAME) {
+        if (token == XContentParser.Token.FIELD_NAME) {
             generator.writeFieldName(parser.currentName());
-            t = parser.nextToken();
+            token = parser.nextToken();
             // fall-through to copy the associated value
         }
 
-        switch (t) {
+        switch (token) {
             case START_ARRAY:
                 generator.writeStartArray();
                 while (parser.nextToken() != XContentParser.Token.END_ARRAY) {
diff --git a/src/main/java/org/elasticsearch/common/xcontent/support/AbstractXContentParser.java b/src/main/java/org/elasticsearch/common/xcontent/support/AbstractXContentParser.java
index fc9a23354d906..b92006d082bdc 100644
--- a/src/main/java/org/elasticsearch/common/xcontent/support/AbstractXContentParser.java
+++ b/src/main/java/org/elasticsearch/common/xcontent/support/AbstractXContentParser.java
@@ -260,38 +260,38 @@ static Map<String, Object> readOrderedMap(XContentParser parser) throws IOExcept
 
     static Map<String, Object> readMap(XContentParser parser, MapFactory mapFactory) throws IOException {
         Map<String, Object> map = mapFactory.newMap();
-        XContentParser.Token t = parser.currentToken();
-        if (t == null) {
-            t = parser.nextToken();
+        XContentParser.Token token = parser.currentToken();
+        if (token == null) {
+            token = parser.nextToken();
         }
-        if (t == XContentParser.Token.START_OBJECT) {
-            t = parser.nextToken();
+        if (token == XContentParser.Token.START_OBJECT) {
+            token = parser.nextToken();
         }
-        for (; t == XContentParser.Token.FIELD_NAME; t = parser.nextToken()) {
+        for (; token == XContentParser.Token.FIELD_NAME; token = parser.nextToken()) {
             // Must point to field name
             String fieldName = parser.currentName();
             // And then the value...
-            t = parser.nextToken();
-            Object value = readValue(parser, mapFactory, t);
+            token = parser.nextToken();
+            Object value = readValue(parser, mapFactory, token);
             map.put(fieldName, value);
         }
         return map;
     }
 
-    private static List<Object> readList(XContentParser parser, MapFactory mapFactory, XContentParser.Token t) throws IOException {
+    private static List<Object> readList(XContentParser parser, MapFactory mapFactory, XContentParser.Token token) throws IOException {
         ArrayList<Object> list = new ArrayList<>();
-        while ((t = parser.nextToken()) != XContentParser.Token.END_ARRAY) {
-            list.add(readValue(parser, mapFactory, t));
+        while ((token = parser.nextToken()) != XContentParser.Token.END_ARRAY) {
+            list.add(readValue(parser, mapFactory, token));
         }
         return list;
     }
 
-    private static Object readValue(XContentParser parser, MapFactory mapFactory, XContentParser.Token t) throws IOException {
-        if (t == XContentParser.Token.VALUE_NULL) {
+    private static Object readValue(XContentParser parser, MapFactory mapFactory, XContentParser.Token token) throws IOException {
+        if (token == XContentParser.Token.VALUE_NULL) {
             return null;
-        } else if (t == XContentParser.Token.VALUE_STRING) {
+        } else if (token == XContentParser.Token.VALUE_STRING) {
             return parser.text();
-        } else if (t == XContentParser.Token.VALUE_NUMBER) {
+        } else if (token == XContentParser.Token.VALUE_NUMBER) {
             XContentParser.NumberType numberType = parser.numberType();
             if (numberType == XContentParser.NumberType.INT) {
                 return parser.intValue();
@@ -302,13 +302,13 @@ private static Object readValue(XContentParser parser, MapFactory mapFactory, XC
             } else if (numberType == XContentParser.NumberType.DOUBLE) {
                 return parser.doubleValue();
             }
-        } else if (t == XContentParser.Token.VALUE_BOOLEAN) {
+        } else if (token == XContentParser.Token.VALUE_BOOLEAN) {
             return parser.booleanValue();
-        } else if (t == XContentParser.Token.START_OBJECT) {
+        } else if (token == XContentParser.Token.START_OBJECT) {
             return readMap(parser, mapFactory);
-        } else if (t == XContentParser.Token.START_ARRAY) {
-            return readList(parser, mapFactory, t);
-        } else if (t == XContentParser.Token.VALUE_EMBEDDED_OBJECT) {
+        } else if (token == XContentParser.Token.START_ARRAY) {
+            return readList(parser, mapFactory, token);
+        } else if (token == XContentParser.Token.VALUE_EMBEDDED_OBJECT) {
             return parser.binaryValue();
         }
         return null;

From dc15dee323800c1fae468227072cdbf6d13f37bb Mon Sep 17 00:00:00 2001
From: Miltos Allamanis <m.allamanis@ed.ac.uk>
Date: Mon, 10 Feb 2014 19:20:24 +0000
Subject: [PATCH 16/54] Renamed ClusterBlocks variable named "block" to
 "blocks".

"blocks" has been used 9 times for variable names of type
ClusterBlocks but block has been used only this case. It
seems to be a typo.
---
 src/main/java/org/elasticsearch/cluster/ClusterState.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/main/java/org/elasticsearch/cluster/ClusterState.java b/src/main/java/org/elasticsearch/cluster/ClusterState.java
index 802503098e8f8..8ceaccf9a1b51 100644
--- a/src/main/java/org/elasticsearch/cluster/ClusterState.java
+++ b/src/main/java/org/elasticsearch/cluster/ClusterState.java
@@ -497,8 +497,8 @@ public Builder blocks(ClusterBlocks.Builder blocksBuilder) {
             return blocks(blocksBuilder.build());
         }
 
-        public Builder blocks(ClusterBlocks block) {
-            this.blocks = block;
+        public Builder blocks(ClusterBlocks blocks) {
+            this.blocks = blocks;
             return this;
         }
 

From 043d78565fa605bcad570217a048544badb1c820 Mon Sep 17 00:00:00 2001
From: Richard Pijnenburg <richard@ispavailability.com>
Date: Wed, 20 Nov 2013 10:09:35 +0100
Subject: [PATCH 17/54] Removing EOL client rubberband and adding official php
 client

---
 docs/community/clients.asciidoc | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/docs/community/clients.asciidoc b/docs/community/clients.asciidoc
index 307f08228a3c9..2c9fd0ebcc686 100644
--- a/docs/community/clients.asciidoc
+++ b/docs/community/clients.asciidoc
@@ -39,9 +39,6 @@ See the {client}/ruby-api/current/index.html[official Elasticsearch Ruby client]
 * http://github.com/karmi/tire[Tire]:
   Ruby API & DSL, with ActiveRecord/ActiveModel integration.
 
-* http://github.com/grantr/rubberband[rubberband]:
-  Ruby client.
-
 * https://github.com/PoseBiz/stretcher[stretcher]:
   Ruby client.
 

From d8364e89a7bcb21f3df78f9f84323b4c8a8bcb90 Mon Sep 17 00:00:00 2001
From: Richard Pijnenburg <richard@ispavailability.com>
Date: Wed, 11 Dec 2013 11:36:45 +0100
Subject: [PATCH 18/54] Fix typo and add more clients

---
 docs/community/clients.asciidoc | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/docs/community/clients.asciidoc b/docs/community/clients.asciidoc
index 2c9fd0ebcc686..d9427bd64b2f7 100644
--- a/docs/community/clients.asciidoc
+++ b/docs/community/clients.asciidoc
@@ -45,6 +45,9 @@ See the {client}/ruby-api/current/index.html[official Elasticsearch Ruby client]
 * https://github.com/wireframe/elastic_searchable/[elastic_searchable]:
   Ruby client + Rails integration.
 
+* https://github.com/ddnexus/flex[Flex]:
+  Ruby Client.
+
 
 [[community-php]]
 === PHP
@@ -59,6 +62,8 @@ See the {client}/php-api/current/index.html[official Elasticsearch PHP client].
 * http://github.com/polyfractal/Sherlock[Sherlock]:
   PHP client, one-to-one mapping with query DSL, fluid interface.
 
+* https://github.com/nervetattoo/elasticsearch[elasticsearch]
+  PHP 5.3 client
 
 [[community-java]]
 === Java
@@ -181,3 +186,7 @@ See the {client}/javascript-api/current/index.html[official Elasticsearch JavaSc
 * https://github.com/jasonfill/ColdFusion-ElasticSearch-Client[ColdFusion-Elasticsearch-Client]
   Cold Fusion client for Elasticsearch
 
+[[community-nodejs]]
+=== NodeJS
+* https://github.com/phillro/node-elasticsearch-client[Node-Elasticsearch-Client]
+  A node.js client for elasticsearch

From c6caeea8871c374255b1f2848f80d2c1c44ce17e Mon Sep 17 00:00:00 2001
From: Richard Pijnenburg <richard@ispavailability.com>
Date: Thu, 20 Feb 2014 12:07:34 +0100
Subject: [PATCH 19/54] Update link to puppet module and remove link to other
 RPM repo as we have our own.

---
 docs/community/misc.asciidoc | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/docs/community/misc.asciidoc b/docs/community/misc.asciidoc
index 62c8938c1409c..7ba49c6fdca3d 100644
--- a/docs/community/misc.asciidoc
+++ b/docs/community/misc.asciidoc
@@ -1,15 +1,12 @@
 [[misc]]
 == Misc
 
-* https://github.com/electrical/puppet-elasticsearch[Puppet]:
+* https://github.com/elasticsearch/puppet-elasticsearch[Puppet]:
   Elasticsearch puppet module.
 
 * http://github.com/elasticsearch/cookbook-elasticsearch[Chef]:
   Chef cookbook for Elasticsearch
 
-* https://github.com/tavisto/elasticsearch-rpms[elasticsearch-rpms]:
-  RPMs for elasticsearch.
-
 * http://www.github.com/neogenix/daikon[daikon]:
   Daikon Elasticsearch CLI
 

From c9b0b04f55c3a5d5502ed15e95114ff6b8a01d6b Mon Sep 17 00:00:00 2001
From: uboness <uboness@gmail.com>
Date: Thu, 3 Apr 2014 12:00:35 +0200
Subject: [PATCH 20/54] Aggregation cleanup

- consolidated value source parsing under a single parser that is reused in all the value source aggs parsers
- consolidated include/exclude parsing under a single parser
- cleaned up value format handling, to have consistent behaviour across all values source aggs
---
 .../common/geo/GeoHashUtils.java              |  12 +-
 .../aggregations/AggregatorFactories.java     |   2 +-
 .../aggregations/InternalAggregation.java     |   5 +
 .../bucket/geogrid/GeoHashGridParser.java     |  36 +--
 .../bucket/histogram/DateHistogramParser.java |  86 +-----
 .../bucket/histogram/ExtendedBounds.java      |   5 +-
 .../bucket/histogram/HistogramAggregator.java |  17 +-
 .../bucket/histogram/HistogramParser.java     |  68 +----
 .../histogram/InternalDateHistogram.java      |  15 +-
 .../bucket/histogram/InternalHistogram.java   |  66 +++--
 .../bucket/missing/MissingParser.java         |  32 +--
 .../bucket/range/InternalRange.java           |  22 +-
 .../bucket/range/RangeAggregator.java         |  54 ++--
 .../bucket/range/RangeParser.java             |  69 +----
 .../bucket/range/date/DateRangeParser.java    |  79 +-----
 .../bucket/range/date/InternalDateRange.java  |   3 +-
 .../range/geodistance/GeoDistanceParser.java  |  89 ++----
 .../geodistance/InternalGeoDistance.java      |   7 +-
 .../bucket/range/ipv4/InternalIPv4Range.java  |  19 +-
 .../bucket/range/ipv4/IpRangeParser.java      |  69 +----
 .../significant/SignificantLongTerms.java     |  17 +-
 .../SignificantLongTermsAggregator.java       |  10 +-
 .../SignificantTermsAggregatorFactory.java    |  17 +-
 .../significant/SignificantTermsParser.java   | 134 ++-------
 .../bucket/terms/DoubleTerms.java             |  19 +-
 .../bucket/terms/DoubleTermsAggregator.java   |   6 +-
 .../aggregations/bucket/terms/LongTerms.java  |  15 +-
 .../bucket/terms/LongTermsAggregator.java     |   8 +-
 .../aggregations/bucket/terms/Terms.java      |  11 +-
 .../bucket/terms/TermsAggregatorFactory.java  |  12 +-
 .../bucket/terms/TermsParser.java             | 192 ++-----------
 .../bucket/terms/support/IncludeExclude.java  |  88 ++++++
 ...icValuesSourceMetricsAggregatorParser.java |  74 +++++
 .../ValuesSourceMetricsAggregatorParser.java  | 111 --------
 .../aggregations/metrics/avg/AvgParser.java   |   9 +-
 .../cardinality/CardinalityParser.java        |  67 +----
 .../aggregations/metrics/max/MaxParser.java   |   9 +-
 .../aggregations/metrics/min/MinParser.java   |   9 +-
 .../percentiles/PercentilesParser.java        |  71 +----
 .../metrics/stats/StatsParser.java            |   9 +-
 .../stats/extended/ExtendedStatsParser.java   |   9 +-
 .../aggregations/metrics/sum/SumParser.java   |   9 +-
 .../valuecount/ValueCountAggregator.java      |  12 +-
 .../metrics/valuecount/ValueCountParser.java  |  61 +---
 .../aggregations/support/FieldContext.java    |   9 +-
 .../aggregations/support/GeoPointParser.java  | 103 +++++++
 .../aggregations/support/ScriptValueType.java |  48 ----
 .../aggregations/support/ValueType.java       | 151 ++++++++++
 .../aggregations/support/ValuesSource.java    |   4 +-
 .../ValuesSourceAggregatorFactory.java        |  23 +-
 .../support/ValuesSourceConfig.java           |  34 +--
 .../support/ValuesSourceParser.java           | 266 ++++++++++++++++++
 .../support/format/ValueFormat.java           | 103 +++++++
 .../support/format/ValueFormatter.java        |   4 +
 .../support/format/ValueParser.java           |   7 +
 .../bucket/DateHistogramTests.java            |   3 +-
 .../bucket/SignificantTermsTests.java         |   2 -
 57 files changed, 1200 insertions(+), 1291 deletions(-)
 create mode 100644 src/main/java/org/elasticsearch/search/aggregations/metrics/NumericValuesSourceMetricsAggregatorParser.java
 delete mode 100644 src/main/java/org/elasticsearch/search/aggregations/metrics/ValuesSourceMetricsAggregatorParser.java
 create mode 100644 src/main/java/org/elasticsearch/search/aggregations/support/GeoPointParser.java
 delete mode 100644 src/main/java/org/elasticsearch/search/aggregations/support/ScriptValueType.java
 create mode 100644 src/main/java/org/elasticsearch/search/aggregations/support/ValueType.java
 create mode 100644 src/main/java/org/elasticsearch/search/aggregations/support/ValuesSourceParser.java
 create mode 100644 src/main/java/org/elasticsearch/search/aggregations/support/format/ValueFormat.java

diff --git a/src/main/java/org/elasticsearch/common/geo/GeoHashUtils.java b/src/main/java/org/elasticsearch/common/geo/GeoHashUtils.java
index 7c97e0614b5a9..17d163913495f 100644
--- a/src/main/java/org/elasticsearch/common/geo/GeoHashUtils.java
+++ b/src/main/java/org/elasticsearch/common/geo/GeoHashUtils.java
@@ -449,12 +449,12 @@ public static long encodeAsLong(double latitude, double longitude, int precision
      */
     public static String toString(long geohashAsLong)
     {
-        int precision= (int) (geohashAsLong&15);
-        char[] chars=new char[precision];
-        geohashAsLong>>=4;                    
-        for (int i = precision-1; i >=0 ; i--) {
-            chars[i]=  BASE_32[(int) (geohashAsLong&31)];
-            geohashAsLong>>=5;                    
+        int precision = (int) (geohashAsLong&15);
+        char[] chars = new char[precision];
+        geohashAsLong >>= 4;
+        for (int i = precision - 1; i >= 0 ; i--) {
+            chars[i] =  BASE_32[(int) (geohashAsLong & 31)];
+            geohashAsLong >>= 5;
         }
         return new String(chars);        
     }
diff --git a/src/main/java/org/elasticsearch/search/aggregations/AggregatorFactories.java b/src/main/java/org/elasticsearch/search/aggregations/AggregatorFactories.java
index 5e5553f9297e1..40d8f53fd5ee3 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/AggregatorFactories.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/AggregatorFactories.java
@@ -39,7 +39,7 @@ public class AggregatorFactories {
 
     public static final AggregatorFactories EMPTY = new Empty();
 
-    private final AggregatorFactory[] factories;
+    private AggregatorFactory[] factories;
 
     public static Builder builder() {
         return new Builder();
diff --git a/src/main/java/org/elasticsearch/search/aggregations/InternalAggregation.java b/src/main/java/org/elasticsearch/search/aggregations/InternalAggregation.java
index a2e61516659f4..30a5215c2da89 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/InternalAggregation.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/InternalAggregation.java
@@ -74,6 +74,11 @@ public String name() {
         public BytesReference stream() {
             return stream;
         }
+
+        @Override
+        public String toString() {
+            return name;
+        }
     }
 
     protected static class ReduceContext {
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/geogrid/GeoHashGridParser.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/geogrid/GeoHashGridParser.java
index 9fb24fdf8270e..e6ea8a81b7f17 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/geogrid/GeoHashGridParser.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/geogrid/GeoHashGridParser.java
@@ -21,8 +21,10 @@
 import org.elasticsearch.common.geo.GeoHashUtils;
 import org.elasticsearch.common.geo.GeoPoint;
 import org.elasticsearch.common.xcontent.XContentParser;
-import org.elasticsearch.index.fielddata.*;
-import org.elasticsearch.index.mapper.FieldMapper;
+import org.elasticsearch.index.fielddata.BytesValues;
+import org.elasticsearch.index.fielddata.DoubleValues;
+import org.elasticsearch.index.fielddata.GeoPointValues;
+import org.elasticsearch.index.fielddata.LongValues;
 import org.elasticsearch.index.query.GeoBoundingBoxFilterBuilder;
 import org.elasticsearch.search.aggregations.Aggregator;
 import org.elasticsearch.search.aggregations.AggregatorFactory;
@@ -53,21 +55,19 @@ public String type() {
     @Override
     public AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException {
 
-        String field = null;
+        ValuesSourceParser vsParser = ValuesSourceParser.geoPoint(aggregationName, InternalGeoHashGrid.TYPE, context).build();
+
         int precision = DEFAULT_PRECISION;
         int requiredSize = DEFAULT_MAX_NUM_CELLS;
         int shardSize = -1;
 
-
         XContentParser.Token token;
         String currentFieldName = null;
         while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
             if (token == XContentParser.Token.FIELD_NAME) {
                 currentFieldName = parser.currentName();
-            } else if (token == XContentParser.Token.VALUE_STRING) {
-                if ("field".equals(currentFieldName)) {
-                    field = parser.text();
-                }
+            } else if (vsParser.token(currentFieldName, token, parser)) {
+                continue;
             } else if (token == XContentParser.Token.VALUE_NUMBER) {
                 if ("precision".equals(currentFieldName)) {
                     precision = parser.intValue();
@@ -76,7 +76,6 @@ public AggregatorFactory parse(String aggregationName, XContentParser parser, Se
                 } else if ("shard_size".equals(currentFieldName) || "shardSize".equals(currentFieldName)) {
                     shardSize = parser.intValue();
                 }
-
             }
         }
 
@@ -97,20 +96,8 @@ public AggregatorFactory parse(String aggregationName, XContentParser parser, Se
             shardSize = requiredSize;
         }
 
-        ValuesSourceConfig<ValuesSource.GeoPoint> config = new ValuesSourceConfig<>(ValuesSource.GeoPoint.class);
-        if (field == null) {
-            return new GeoGridFactory(aggregationName, config, precision, requiredSize, shardSize);
-        }
-
-        FieldMapper<?> mapper = context.smartNameFieldMapper(field);
-        if (mapper == null) {
-            config.unmapped(true);
-            return new GeoGridFactory(aggregationName, config, precision, requiredSize, shardSize);
-        }
+        return new GeoGridFactory(aggregationName, vsParser.config(), precision, requiredSize, shardSize);
 
-        IndexFieldData<?> indexFieldData = context.fieldData().getForField(mapper);
-        config.fieldContext(new FieldContext(field, indexFieldData));
-        return new GeoGridFactory(aggregationName, config, precision, requiredSize, shardSize);
     }
 
 
@@ -120,9 +107,8 @@ private static class GeoGridFactory extends ValuesSourceAggregatorFactory<Values
         private int requiredSize;
         private int shardSize;
 
-        public GeoGridFactory(String name, ValuesSourceConfig<ValuesSource.GeoPoint> valueSourceConfig,
-                              int precision, int requiredSize, int shardSize) {
-            super(name, InternalGeoHashGrid.TYPE.name(), valueSourceConfig);
+        public GeoGridFactory(String name, ValuesSourceConfig<ValuesSource.GeoPoint> config, int precision, int requiredSize, int shardSize) {
+            super(name, InternalGeoHashGrid.TYPE.name(), config);
             this.precision = precision;
             this.requiredSize = requiredSize;
             this.shardSize = shardSize;
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/DateHistogramParser.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/DateHistogramParser.java
index a2101e98a3878..3148382c0a809 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/DateHistogramParser.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/DateHistogramParser.java
@@ -21,28 +21,19 @@
 import com.google.common.collect.ImmutableMap;
 import org.elasticsearch.common.ParseField;
 import org.elasticsearch.common.collect.MapBuilder;
-import org.elasticsearch.common.joda.DateMathParser;
 import org.elasticsearch.common.rounding.DateTimeUnit;
 import org.elasticsearch.common.rounding.TimeZoneRounding;
 import org.elasticsearch.common.unit.TimeValue;
 import org.elasticsearch.common.xcontent.XContentParser;
-import org.elasticsearch.index.fielddata.IndexFieldData;
-import org.elasticsearch.index.mapper.FieldMapper;
-import org.elasticsearch.index.mapper.core.DateFieldMapper;
-import org.elasticsearch.script.SearchScript;
 import org.elasticsearch.search.SearchParseException;
 import org.elasticsearch.search.aggregations.Aggregator;
 import org.elasticsearch.search.aggregations.AggregatorFactory;
-import org.elasticsearch.search.aggregations.support.ValuesSource;
-import org.elasticsearch.search.aggregations.support.FieldContext;
-import org.elasticsearch.search.aggregations.support.ValuesSourceConfig;
-import org.elasticsearch.search.aggregations.support.format.ValueFormatter;
-import org.elasticsearch.search.aggregations.support.format.ValueParser;
+import org.elasticsearch.search.aggregations.support.ValueType;
+import org.elasticsearch.search.aggregations.support.ValuesSourceParser;
 import org.elasticsearch.search.internal.SearchContext;
 import org.joda.time.DateTimeZone;
 
 import java.io.IOException;
-import java.util.Map;
 
 /**
  *
@@ -82,12 +73,12 @@ public String type() {
     @Override
     public AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException {
 
-        ValuesSourceConfig<ValuesSource.Numeric> config = new ValuesSourceConfig<>(ValuesSource.Numeric.class);
+        ValuesSourceParser vsParser = ValuesSourceParser.numeric(aggregationName, InternalDateHistogram.TYPE, context)
+                .targetValueType(ValueType.DATE)
+                .requiresSortedValues(true)
+                .formattable(true)
+                .build();
 
-        String field = null;
-        String script = null;
-        String scriptLang = null;
-        Map<String, Object> scriptParams = null;
         boolean keyed = false;
         long minDocCount = 1;
         ExtendedBounds extendedBounds = null;
@@ -96,24 +87,18 @@ public AggregatorFactory parse(String aggregationName, XContentParser parser, Se
         boolean preZoneAdjustLargeInterval = false;
         DateTimeZone preZone = DateTimeZone.UTC;
         DateTimeZone postZone = DateTimeZone.UTC;
-        String format = null;
         long preOffset = 0;
         long postOffset = 0;
-        boolean assumeSorted = false;
 
         XContentParser.Token token;
         String currentFieldName = null;
         while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
             if (token == XContentParser.Token.FIELD_NAME) {
                 currentFieldName = parser.currentName();
+            } else if (vsParser.token(currentFieldName, token, parser)) {
+                continue;
             } else if (token == XContentParser.Token.VALUE_STRING) {
-                if ("field".equals(currentFieldName)) {
-                    field = parser.text();
-                } else if ("script".equals(currentFieldName)) {
-                    script = parser.text();
-                } else if ("lang".equals(currentFieldName)) {
-                    scriptLang = parser.text();
-                } else if ("time_zone".equals(currentFieldName) || "timeZone".equals(currentFieldName)) {
+                if ("time_zone".equals(currentFieldName) || "timeZone".equals(currentFieldName)) {
                     preZone = parseZone(parser.text());
                 } else if ("pre_zone".equals(currentFieldName) || "preZone".equals(currentFieldName)) {
                     preZone = parseZone(parser.text());
@@ -125,16 +110,12 @@ public AggregatorFactory parse(String aggregationName, XContentParser parser, Se
                     postOffset = parseOffset(parser.text());
                 } else if ("interval".equals(currentFieldName)) {
                     interval = parser.text();
-                } else if ("format".equals(currentFieldName)) {
-                    format = parser.text();
                 } else {
                     throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
                 }
             } else if (token == XContentParser.Token.VALUE_BOOLEAN) {
                 if ("keyed".equals(currentFieldName)) {
                     keyed = parser.booleanValue();
-                } else if ("script_values_sorted".equals(currentFieldName) || "scriptValuesSorted".equals(currentFieldName)) {
-                    assumeSorted = parser.booleanValue();
                 } else if ("pre_zone_adjust_large_interval".equals(currentFieldName) || "preZoneAdjustLargeInterval".equals(currentFieldName)) {
                     preZoneAdjustLargeInterval = parser.booleanValue();
                 } else {
@@ -153,9 +134,7 @@ public AggregatorFactory parse(String aggregationName, XContentParser parser, Se
                     throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
                 }
             } else if (token == XContentParser.Token.START_OBJECT) {
-                if ("params".equals(currentFieldName)) {
-                    scriptParams = parser.map();
-                } else if ("order".equals(currentFieldName)) {
+                if ("order".equals(currentFieldName)) {
                     while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
                         if (token == XContentParser.Token.FIELD_NAME) {
                             currentFieldName = parser.currentName();
@@ -204,17 +183,6 @@ public AggregatorFactory parse(String aggregationName, XContentParser parser, Se
             throw new SearchParseException(context, "Missing required field [interval] for histogram aggregation [" + aggregationName + "]");
         }
 
-        SearchScript searchScript = null;
-        if (script != null) {
-            searchScript = context.scriptService().search(context.lookup(), scriptLang, script, scriptParams);
-            config.script(searchScript);
-        }
-
-        if (!assumeSorted) {
-            // we need values to be sorted and unique for efficiency
-            config.ensureSorted(true);
-        }
-
         TimeZoneRounding.Builder tzRoundingBuilder;
         DateTimeUnit dateTimeUnit = dateFieldUnits.get(interval);
         if (dateTimeUnit != null) {
@@ -230,38 +198,8 @@ public AggregatorFactory parse(String aggregationName, XContentParser parser, Se
                 .preOffset(preOffset).postOffset(postOffset)
                 .build();
 
-        ValueFormatter valueFormatter = format != null ? new ValueFormatter.DateTime(format) : null;
-
-        if (field == null) {
-
-            if (searchScript != null) {
-                ValueParser valueParser = new ValueParser.DateMath(new DateMathParser(DateFieldMapper.Defaults.DATE_TIME_FORMATTER, DateFieldMapper.Defaults.TIME_UNIT));
-                return new HistogramAggregator.Factory(aggregationName, config, valueFormatter, valueParser, rounding, order, keyed, minDocCount, extendedBounds, InternalDateHistogram.FACTORY);
-            }
-
-            // falling back on the get field data context
-            return new HistogramAggregator.Factory(aggregationName, config, valueFormatter, null, rounding, order, keyed, minDocCount, extendedBounds, InternalDateHistogram.FACTORY);
-        }
-
-        FieldMapper<?> mapper = context.smartNameFieldMapper(field);
-        if (mapper == null) {
-            config.unmapped(true);
-            valueFormatter = format == null ? new ValueFormatter.DateTime(DateFieldMapper.Defaults.DATE_TIME_FORMATTER) : null;
-            ValueParser valueParser = new ValueParser.DateMath(new DateMathParser(DateFieldMapper.Defaults.DATE_TIME_FORMATTER, DateFieldMapper.Defaults.TIME_UNIT));
-            return new HistogramAggregator.Factory(aggregationName, config, valueFormatter, valueParser, rounding, order, keyed, minDocCount, extendedBounds, InternalDateHistogram.FACTORY);
-        }
-
-        if (!(mapper instanceof DateFieldMapper)) {
-            throw new SearchParseException(context, "date histogram can only be aggregated on date fields but [" + field + "] is not a date field");
-        }
+        return new HistogramAggregator.Factory(aggregationName, vsParser.config(), rounding, order, keyed, minDocCount, extendedBounds, InternalDateHistogram.FACTORY);
 
-        IndexFieldData<?> indexFieldData = context.fieldData().getForField(mapper);
-        config.fieldContext(new FieldContext(field, indexFieldData));
-        if (format == null) {
-            valueFormatter = new ValueFormatter.DateTime(((DateFieldMapper) mapper).dateTimeFormatter());
-        }
-        ValueParser valueParser = new ValueParser.DateMath(new DateMathParser(((DateFieldMapper) mapper).dateTimeFormatter(), DateFieldMapper.Defaults.TIME_UNIT));
-        return new HistogramAggregator.Factory(aggregationName, config, valueFormatter, valueParser, rounding, order, keyed, minDocCount, extendedBounds, InternalDateHistogram.FACTORY);
     }
 
     private static InternalOrder resolveOrder(String key, boolean asc) {
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/ExtendedBounds.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/ExtendedBounds.java
index 303254606860e..b041ef34fdb5e 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/ExtendedBounds.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/ExtendedBounds.java
@@ -47,11 +47,12 @@ public class ExtendedBounds {
     }
 
     void processAndValidate(String aggName, SearchContext context, ValueParser parser) {
+        assert parser != null;
         if (minAsStr != null) {
-            min = parser != null ? parser.parseLong(minAsStr, context) : Long.parseLong(minAsStr);
+            min = parser.parseLong(minAsStr, context);
         }
         if (maxAsStr != null) {
-            max = parser != null ? parser.parseLong(maxAsStr, context) : Long.parseLong(maxAsStr);
+            max = parser.parseLong(maxAsStr, context);
         }
         if (min != null && max != null && min.compareTo(max) > 0) {
             throw new SearchParseException(context, "[extended_bounds.min][" + min + "] cannot be greater than " +
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/HistogramAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/HistogramAggregator.java
index ebee7e705a02b..fab705e25f09d 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/HistogramAggregator.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/HistogramAggregator.java
@@ -44,11 +44,11 @@
 public class HistogramAggregator extends BucketsAggregator {
 
     private final ValuesSource.Numeric valuesSource;
-    private final ValueFormatter formatter;
-    private final ValueParser parser;
+    private final @Nullable ValueFormatter formatter;
     private final Rounding rounding;
     private final InternalOrder order;
     private final boolean keyed;
+
     private final long minDocCount;
     private final ExtendedBounds extendedBounds;
     private final InternalHistogram.Factory histogramFactory;
@@ -58,7 +58,7 @@ public class HistogramAggregator extends BucketsAggregator {
 
     public HistogramAggregator(String name, AggregatorFactories factories, Rounding rounding, InternalOrder order,
                                boolean keyed, long minDocCount, @Nullable ExtendedBounds extendedBounds,
-                               @Nullable ValuesSource.Numeric valuesSource, ValueFormatter formatter, ValueParser parser,
+                               @Nullable ValuesSource.Numeric valuesSource, @Nullable ValueFormatter formatter,
                                long initialCapacity, InternalHistogram.Factory<?> histogramFactory,
                                AggregationContext aggregationContext, Aggregator parent) {
 
@@ -70,7 +70,6 @@ public HistogramAggregator(String name, AggregatorFactories factories, Rounding
         this.extendedBounds = extendedBounds;
         this.valuesSource = valuesSource;
         this.formatter = formatter;
-        this.parser = parser;
         this.histogramFactory = histogramFactory;
 
         bucketOrds = new LongHash(initialCapacity, aggregationContext.bigArrays());
@@ -147,11 +146,11 @@ public static class Factory extends ValuesSourceAggregatorFactory<ValuesSource.N
         private final ExtendedBounds extendedBounds;
         private final InternalHistogram.Factory<?> histogramFactory;
 
-        public Factory(String name, ValuesSourceConfig<ValuesSource.Numeric> config, ValueFormatter formatter, ValueParser parser,
+        public Factory(String name, ValuesSourceConfig<ValuesSource.Numeric> config,
                        Rounding rounding, InternalOrder order, boolean keyed, long minDocCount,
                        ExtendedBounds extendedBounds, InternalHistogram.Factory<?> histogramFactory) {
 
-            super(name, histogramFactory.type(), config, formatter, parser);
+            super(name, histogramFactory.type(), config);
             this.rounding = rounding;
             this.order = order;
             this.keyed = keyed;
@@ -162,7 +161,7 @@ public Factory(String name, ValuesSourceConfig<ValuesSource.Numeric> config, Val
 
         @Override
         protected Aggregator createUnmapped(AggregationContext aggregationContext, Aggregator parent) {
-            return new HistogramAggregator(name, factories, rounding, order, keyed, minDocCount, null, null, null, null, 0, histogramFactory, aggregationContext, parent);
+            return new HistogramAggregator(name, factories, rounding, order, keyed, minDocCount, null, null, config.formatter(), 0, histogramFactory, aggregationContext, parent);
         }
 
         @Override
@@ -175,10 +174,10 @@ protected Aggregator create(ValuesSource.Numeric valuesSource, long expectedBuck
             ExtendedBounds roundedBounds = null;
             if (extendedBounds != null) {
                 // we need to process & validate here using the parser
-                extendedBounds.processAndValidate(name, aggregationContext.searchContext(), parser);
+                extendedBounds.processAndValidate(name, aggregationContext.searchContext(), config.parser());
                 roundedBounds = extendedBounds.round(rounding);
             }
-            return new HistogramAggregator(name, factories, rounding, order, keyed, minDocCount, roundedBounds, valuesSource, formatter, parser, 50, histogramFactory, aggregationContext, parent);
+            return new HistogramAggregator(name, factories, rounding, order, keyed, minDocCount, roundedBounds, valuesSource, config.formatter(), 50, histogramFactory, aggregationContext, parent);
         }
 
     }
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/HistogramParser.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/HistogramParser.java
index 6159dd95191e2..98e010ff215eb 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/HistogramParser.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/HistogramParser.java
@@ -21,20 +21,15 @@
 import org.elasticsearch.common.ParseField;
 import org.elasticsearch.common.rounding.Rounding;
 import org.elasticsearch.common.xcontent.XContentParser;
-import org.elasticsearch.index.fielddata.IndexFieldData;
-import org.elasticsearch.index.mapper.FieldMapper;
 import org.elasticsearch.search.SearchParseException;
 import org.elasticsearch.search.aggregations.Aggregator;
 import org.elasticsearch.search.aggregations.AggregatorFactory;
-import org.elasticsearch.search.aggregations.support.ValuesSource;
-import org.elasticsearch.search.aggregations.support.FieldContext;
-import org.elasticsearch.search.aggregations.support.ValuesSourceConfig;
-import org.elasticsearch.search.aggregations.support.format.ValueFormatter;
+import org.elasticsearch.search.aggregations.support.ValueType;
+import org.elasticsearch.search.aggregations.support.ValuesSourceParser;
 import org.elasticsearch.search.aggregations.support.format.ValueParser;
 import org.elasticsearch.search.internal.SearchContext;
 
 import java.io.IOException;
-import java.util.Map;
 
 /**
  * Parses the histogram request
@@ -51,18 +46,16 @@ public String type() {
     @Override
     public AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException {
 
-        ValuesSourceConfig<ValuesSource.Numeric> config = new ValuesSourceConfig<>(ValuesSource.Numeric.class);
+        ValuesSourceParser vsParser = ValuesSourceParser.numeric(aggregationName, InternalHistogram.TYPE, context)
+                .requiresSortedValues(true)
+                .targetValueType(ValueType.NUMERIC)
+                .formattable(true)
+                .build();
 
-        String field = null;
-        String script = null;
-        String scriptLang = null;
-        Map<String, Object> scriptParams = null;
         boolean keyed = false;
         long minDocCount = 1;
         InternalOrder order = (InternalOrder) InternalOrder.KEY_ASC;
         long interval = -1;
-        boolean assumeSorted = false;
-        String format = null;
         ExtendedBounds extendedBounds = null;
 
         XContentParser.Token token;
@@ -70,16 +63,8 @@ public AggregatorFactory parse(String aggregationName, XContentParser parser, Se
         while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
             if (token == XContentParser.Token.FIELD_NAME) {
                 currentFieldName = parser.currentName();
-            } else if (token == XContentParser.Token.VALUE_STRING) {
-                if ("field".equals(currentFieldName)) {
-                    field = parser.text();
-                } else if ("script".equals(currentFieldName)) {
-                    script = parser.text();
-                } else if ("lang".equals(currentFieldName)) {
-                    scriptLang = parser.text();
-                } else {
-                    throw new SearchParseException(context, "Unknown key for a " + token + " in aggregation [" + aggregationName + "]: [" + currentFieldName + "].");
-                }
+            } else if (vsParser.token(currentFieldName, token, parser)) {
+                continue;
             } else if (token == XContentParser.Token.VALUE_NUMBER) {
                 if ("interval".equals(currentFieldName)) {
                     interval = parser.longValue();
@@ -91,15 +76,11 @@ public AggregatorFactory parse(String aggregationName, XContentParser parser, Se
             } else if (token == XContentParser.Token.VALUE_BOOLEAN) {
                 if ("keyed".equals(currentFieldName)) {
                     keyed = parser.booleanValue();
-                } else if ("script_values_sorted".equals(currentFieldName) || "scriptValuesSorted".equals(currentFieldName)) {
-                    assumeSorted = parser.booleanValue();
                 } else {
                     throw new SearchParseException(context, "Unknown key for a " + token + " in aggregation [" + aggregationName + "]: [" + currentFieldName + "].");
                 }
             } else if (token == XContentParser.Token.START_OBJECT) {
-                if ("params".equals(currentFieldName)) {
-                    scriptParams = parser.map();
-                } else if ("order".equals(currentFieldName)) {
+                if ("order".equals(currentFieldName)) {
                     while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
                         if (token == XContentParser.Token.FIELD_NAME) {
                             currentFieldName = parser.currentName();
@@ -143,35 +124,10 @@ public AggregatorFactory parse(String aggregationName, XContentParser parser, Se
 
         if (extendedBounds != null) {
             // with numeric histogram, we can process here and fail fast if the bounds are invalid
-            extendedBounds.processAndValidate(aggregationName, context, null);
-        }
-
-        if (script != null) {
-            config.script(context.scriptService().search(context.lookup(), scriptLang, script, scriptParams));
-        }
-
-        if (!assumeSorted) {
-            // we need values to be sorted and unique for efficiency
-            config.ensureSorted(true);
+            extendedBounds.processAndValidate(aggregationName, context, ValueParser.RAW);
         }
 
-        if (field == null) {
-            return new HistogramAggregator.Factory(aggregationName, config, null, null, rounding, order, keyed, minDocCount, extendedBounds, InternalHistogram.FACTORY);
-        }
-
-        FieldMapper<?> mapper = context.smartNameFieldMapper(field);
-        if (mapper == null) {
-            config.unmapped(true);
-            return new HistogramAggregator.Factory(aggregationName, config, null, null, rounding, order, keyed, minDocCount, extendedBounds, InternalHistogram.FACTORY);
-        }
-
-        IndexFieldData<?> indexFieldData = context.fieldData().getForField(mapper);
-        config.fieldContext(new FieldContext(field, indexFieldData));
-
-        ValueFormatter valueFormatter = format == null ? ValueFormatter.RAW : new ValueFormatter.Number.Pattern(format);
-        ValueParser valueParser = format == null ? ValueParser.RAW : new ValueParser.Number.Pattern(format);
-
-        return new HistogramAggregator.Factory(aggregationName, config, valueFormatter, valueParser, rounding, order, keyed, minDocCount, extendedBounds, InternalHistogram.FACTORY);
+        return new HistogramAggregator.Factory(aggregationName, vsParser.config(), rounding, order, keyed, minDocCount, extendedBounds, InternalHistogram.FACTORY);
 
     }
 
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/InternalDateHistogram.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/InternalDateHistogram.java
index 4859f748c2ddd..bb093261e914a 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/InternalDateHistogram.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/InternalDateHistogram.java
@@ -19,8 +19,8 @@
 package org.elasticsearch.search.aggregations.bucket.histogram;
 
 import com.carrotsearch.hppc.ObjectObjectOpenHashMap;
+import org.elasticsearch.common.Nullable;
 import org.elasticsearch.common.io.stream.StreamInput;
-import org.elasticsearch.index.mapper.core.DateFieldMapper;
 import org.elasticsearch.search.aggregations.AggregationStreams;
 import org.elasticsearch.search.aggregations.InternalAggregations;
 import org.elasticsearch.search.aggregations.support.format.ValueFormatter;
@@ -53,11 +53,8 @@ public static void registerStream() {
 
     static class Bucket extends InternalHistogram.Bucket implements DateHistogram.Bucket {
 
-        private final ValueFormatter formatter;
-
-        Bucket(long key, long docCount, InternalAggregations aggregations, ValueFormatter formatter) {
-            super(key, docCount, aggregations);
-            this.formatter = formatter;
+        Bucket(long key, long docCount, InternalAggregations aggregations, @Nullable ValueFormatter formatter) {
+            super(key, docCount, formatter, aggregations);
         }
 
         @Override
@@ -88,12 +85,12 @@ public String type() {
 
         @Override
         public InternalDateHistogram create(String name, List<InternalDateHistogram.Bucket> buckets, InternalOrder order,
-                                            long minDocCount, EmptyBucketInfo emptyBucketInfo, ValueFormatter formatter, boolean keyed) {
+                                            long minDocCount, EmptyBucketInfo emptyBucketInfo, @Nullable ValueFormatter formatter, boolean keyed) {
             return new InternalDateHistogram(name, buckets, order, minDocCount, emptyBucketInfo, formatter, keyed);
         }
 
         @Override
-        public InternalDateHistogram.Bucket createBucket(long key, long docCount, InternalAggregations aggregations, ValueFormatter formatter) {
+        public InternalDateHistogram.Bucket createBucket(long key, long docCount, InternalAggregations aggregations, @Nullable ValueFormatter formatter) {
             return new Bucket(key, docCount, aggregations, formatter);
         }
     }
@@ -103,7 +100,7 @@ public InternalDateHistogram.Bucket createBucket(long key, long docCount, Intern
     InternalDateHistogram() {} // for serialization
 
     InternalDateHistogram(String name, List<InternalDateHistogram.Bucket> buckets, InternalOrder order, long minDocCount,
-                          EmptyBucketInfo emptyBucketInfo, ValueFormatter formatter, boolean keyed) {
+                          EmptyBucketInfo emptyBucketInfo, @Nullable ValueFormatter formatter, boolean keyed) {
         super(name, buckets, order, minDocCount, emptyBucketInfo, formatter, keyed);
     }
 
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/InternalHistogram.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/InternalHistogram.java
index b23f25c32d735..c64554d14a227 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/InternalHistogram.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/InternalHistogram.java
@@ -22,6 +22,7 @@
 import com.google.common.collect.Lists;
 import org.apache.lucene.util.CollectionUtil;
 import org.elasticsearch.Version;
+import org.elasticsearch.common.Nullable;
 import org.elasticsearch.common.io.stream.StreamInput;
 import org.elasticsearch.common.io.stream.StreamOutput;
 import org.elasticsearch.common.rounding.Rounding;
@@ -68,17 +69,19 @@ public static class Bucket implements Histogram.Bucket {
 
         long key;
         long docCount;
+        protected transient final @Nullable ValueFormatter formatter;
         InternalAggregations aggregations;
 
-        public Bucket(long key, long docCount, InternalAggregations aggregations) {
+        public Bucket(long key, long docCount, @Nullable ValueFormatter formatter, InternalAggregations aggregations) {
             this.key = key;
             this.docCount = docCount;
+            this.formatter = formatter;
             this.aggregations = aggregations;
         }
 
         @Override
         public String getKey() {
-            return String.valueOf(key);
+            return formatter != null ? formatter.format(key) : ValueFormatter.RAW.format(key);
         }
 
         @Override
@@ -121,6 +124,28 @@ <B extends Bucket> B reduce(List<B> buckets, BigArrays bigArrays) {
             reduced.aggregations = InternalAggregations.reduce(aggregations, bigArrays);
             return (B) reduced;
         }
+
+        void toXContent(XContentBuilder builder, Params params, boolean keyed, @Nullable ValueFormatter formatter) throws IOException {
+            if (formatter != null) {
+                Text keyTxt = new StringText(formatter.format(key));
+                if (keyed) {
+                    builder.startObject(keyTxt.string());
+                } else {
+                    builder.startObject();
+                }
+                builder.field(CommonFields.KEY_AS_STRING, keyTxt);
+            } else {
+                if (keyed) {
+                    builder.startObject(String.valueOf(getKeyAsNumber()));
+                } else {
+                    builder.startObject();
+                }
+            }
+            builder.field(CommonFields.KEY, key);
+            builder.field(CommonFields.DOC_COUNT, docCount);
+            aggregations.toXContentInternal(builder, params);
+            builder.endObject();
+        }
     }
 
     static class EmptyBucketInfo {
@@ -173,12 +198,12 @@ public String type() {
         }
 
         public InternalHistogram<B> create(String name, List<B> buckets, InternalOrder order, long minDocCount,
-                                           EmptyBucketInfo emptyBucketInfo, ValueFormatter formatter, boolean keyed) {
+                                           EmptyBucketInfo emptyBucketInfo, @Nullable ValueFormatter formatter, boolean keyed) {
             return new InternalHistogram<>(name, buckets, order, minDocCount, emptyBucketInfo, formatter, keyed);
         }
 
-        public B createBucket(long key, long docCount, InternalAggregations aggregations, ValueFormatter formatter) {
-            return (B) new Bucket(key, docCount, aggregations);
+        public B createBucket(long key, long docCount, InternalAggregations aggregations, @Nullable ValueFormatter formatter) {
+            return (B) new Bucket(key, docCount, formatter, aggregations);
         }
 
     }
@@ -186,14 +211,15 @@ public B createBucket(long key, long docCount, InternalAggregations aggregations
     protected List<B> buckets;
     private LongObjectOpenHashMap<B> bucketsMap;
     private InternalOrder order;
-    private ValueFormatter formatter;
+    private @Nullable ValueFormatter formatter;
     private boolean keyed;
     private long minDocCount;
     private EmptyBucketInfo emptyBucketInfo;
 
     InternalHistogram() {} // for serialization
 
-    InternalHistogram(String name, List<B> buckets, InternalOrder order, long minDocCount, EmptyBucketInfo emptyBucketInfo, ValueFormatter formatter, boolean keyed) {
+    InternalHistogram(String name, List<B> buckets, InternalOrder order, long minDocCount,
+                      EmptyBucketInfo emptyBucketInfo, @Nullable ValueFormatter formatter, boolean keyed) {
         super(name);
         this.buckets = buckets;
         this.order = order;
@@ -416,8 +442,8 @@ public InternalAggregation reduce(ReduceContext reduceContext) {
         return reduced;
     }
 
-    protected B createBucket(long key, long docCount, InternalAggregations aggregations, ValueFormatter formatter) {
-        return (B) new InternalHistogram.Bucket(key, docCount, aggregations);
+    protected B createBucket(long key, long docCount, InternalAggregations aggregations, @Nullable ValueFormatter formatter) {
+        return (B) new InternalHistogram.Bucket(key, docCount, formatter, aggregations);
     }
 
     @Override
@@ -465,29 +491,9 @@ public XContentBuilder toXContent(XContentBuilder builder, Params params) throws
         } else {
             builder.startArray(CommonFields.BUCKETS);
         }
-
         for (B bucket : buckets) {
-            if (formatter != null) {
-                Text keyTxt = new StringText(formatter.format(bucket.key));
-                if (keyed) {
-                    builder.startObject(keyTxt.string());
-                } else {
-                    builder.startObject();
-                }
-                builder.field(CommonFields.KEY_AS_STRING, keyTxt);
-            } else {
-                if (keyed) {
-                    builder.startObject(String.valueOf(bucket.getKeyAsNumber()));
-                } else {
-                    builder.startObject();
-                }
-            }
-            builder.field(CommonFields.KEY, bucket.key);
-            builder.field(CommonFields.DOC_COUNT, bucket.docCount);
-            bucket.aggregations.toXContentInternal(builder, params);
-            builder.endObject();
+            bucket.toXContent(builder, params, keyed, formatter);
         }
-
         if (keyed) {
             builder.endObject();
         } else {
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/missing/MissingParser.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/missing/MissingParser.java
index 44c7539375c23..b37de4c743cc6 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/missing/MissingParser.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/missing/MissingParser.java
@@ -19,13 +19,10 @@
 package org.elasticsearch.search.aggregations.bucket.missing;
 
 import org.elasticsearch.common.xcontent.XContentParser;
-import org.elasticsearch.index.mapper.FieldMapper;
 import org.elasticsearch.search.SearchParseException;
 import org.elasticsearch.search.aggregations.Aggregator;
 import org.elasticsearch.search.aggregations.AggregatorFactory;
-import org.elasticsearch.search.aggregations.support.ValuesSource;
-import org.elasticsearch.search.aggregations.support.FieldContext;
-import org.elasticsearch.search.aggregations.support.ValuesSourceConfig;
+import org.elasticsearch.search.aggregations.support.ValuesSourceParser;
 import org.elasticsearch.search.internal.SearchContext;
 
 import java.io.IOException;
@@ -43,37 +40,22 @@ public String type() {
     @Override
     public AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException {
 
-        ValuesSourceConfig<ValuesSource> config = new ValuesSourceConfig<>(ValuesSource.class);
-
-        String field = null;
+        ValuesSourceParser vsParser = ValuesSourceParser.any(aggregationName, InternalMissing.TYPE, context)
+                .scriptable(false)
+                .build();
 
         XContentParser.Token token;
         String currentFieldName = null;
         while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
             if (token == XContentParser.Token.FIELD_NAME) {
                 currentFieldName = parser.currentName();
-            } else if (token == XContentParser.Token.VALUE_STRING) {
-                if ("field".equals(currentFieldName)) {
-                    field = parser.text();
-                } else {
-                    throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
-                }
+            } else if (vsParser.token(currentFieldName, token, parser)) {
+                continue;
             } else {
                 throw new SearchParseException(context, "Unexpected token " + token + " in [" + aggregationName + "].");
             }
         }
 
-        if (field == null) {
-            return new MissingAggregator.Factory(aggregationName, config);
-        }
-
-        FieldMapper<?> mapper = context.smartNameFieldMapper(field);
-        if (mapper == null) {
-            config.unmapped(true);
-            return new MissingAggregator.Factory(aggregationName, config);
-        }
-
-        config.fieldContext(new FieldContext(field, context.fieldData().getForField(mapper)));
-        return new MissingAggregator.Factory(aggregationName, config);
+        return new MissingAggregator.Factory(aggregationName, vsParser.config());
     }
 }
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/InternalRange.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/InternalRange.java
index b9859e6cf98f7..cea46cd8585d0 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/InternalRange.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/InternalRange.java
@@ -19,6 +19,7 @@
 package org.elasticsearch.search.aggregations.bucket.range;
 
 import com.google.common.collect.Lists;
+import org.elasticsearch.common.Nullable;
 import org.elasticsearch.common.io.stream.StreamInput;
 import org.elasticsearch.common.io.stream.StreamOutput;
 import org.elasticsearch.common.text.StringText;
@@ -65,8 +66,8 @@ public static class Bucket implements Range.Bucket {
         InternalAggregations aggregations;
         private String key;
 
-        public Bucket(String key, double from, double to, long docCount, InternalAggregations aggregations, ValueFormatter formatter) {
-            this.key = key != null ? key : key(from, to, formatter);
+        public Bucket(String key, double from, double to, long docCount, InternalAggregations aggregations, @Nullable ValueFormatter formatter) {
+            this.key = key != null ? key : generateKey(from, to, formatter);
             this.from = from;
             this.to = to;
             this.docCount = docCount;
@@ -123,7 +124,7 @@ Bucket reduce(List<Bucket> ranges, BigArrays bigArrays) {
             return reduced;
         }
 
-        void toXContent(XContentBuilder builder, Params params, ValueFormatter formatter, boolean keyed) throws IOException {
+        void toXContent(XContentBuilder builder, Params params, @Nullable ValueFormatter formatter, boolean keyed) throws IOException {
             if (keyed) {
                 builder.startObject(key);
             } else {
@@ -147,11 +148,11 @@ void toXContent(XContentBuilder builder, Params params, ValueFormatter formatter
             builder.endObject();
         }
 
-        private static String key(double from, double to, ValueFormatter formatter) {
+        protected String generateKey(double from, double to, @Nullable ValueFormatter formatter) {
             StringBuilder sb = new StringBuilder();
-            sb.append(Double.isInfinite(from) ? "*" : formatter != null ? formatter.format(from) : from);
+            sb.append(Double.isInfinite(from) ? "*" : formatter != null ? formatter.format(from) : ValueFormatter.RAW.format(from));
             sb.append("-");
-            sb.append(Double.isInfinite(to) ? "*" : formatter != null ? formatter.format(to) : to);
+            sb.append(Double.isInfinite(to) ? "*" : formatter != null ? formatter.format(to) : ValueFormatter.RAW.format(to));
             return sb.toString();
         }
 
@@ -163,26 +164,25 @@ public String type() {
             return TYPE.name();
         }
 
-        public R create(String name, List<B> ranges, ValueFormatter formatter, boolean keyed, boolean unmapped) {
+        public R create(String name, List<B> ranges, @Nullable ValueFormatter formatter, boolean keyed, boolean unmapped) {
             return (R) new InternalRange<>(name, ranges, formatter, keyed, unmapped);
         }
 
 
-        public B createBucket(String key, double from, double to, long docCount, InternalAggregations aggregations, ValueFormatter formatter) {
+        public B createBucket(String key, double from, double to, long docCount, InternalAggregations aggregations, @Nullable ValueFormatter formatter) {
             return (B) new Bucket(key, from, to, docCount, aggregations, formatter);
         }
     }
 
     private List<B> ranges;
     private Map<String, B> rangeMap;
-    private ValueFormatter formatter;
+    private @Nullable ValueFormatter formatter;
     private boolean keyed;
-
     private boolean unmapped;
 
     public InternalRange() {} // for serialization
 
-    public InternalRange(String name, List<B> ranges, ValueFormatter formatter, boolean keyed, boolean unmapped) {
+    public InternalRange(String name, List<B> ranges, @Nullable ValueFormatter formatter, boolean keyed, boolean unmapped) {
         super(name);
         this.ranges = ranges;
         this.formatter = formatter;
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/RangeAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/RangeAggregator.java
index 3ba307220ea31..d9c8af6c679ea 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/RangeAggregator.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/RangeAggregator.java
@@ -21,6 +21,7 @@
 import com.google.common.collect.Lists;
 import org.apache.lucene.index.AtomicReaderContext;
 import org.apache.lucene.util.InPlaceMergeSorter;
+import org.elasticsearch.common.Nullable;
 import org.elasticsearch.index.fielddata.DoubleValues;
 import org.elasticsearch.search.aggregations.*;
 import org.elasticsearch.search.aggregations.bucket.BucketsAggregator;
@@ -28,8 +29,10 @@
 import org.elasticsearch.search.aggregations.support.ValuesSource;
 import org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory;
 import org.elasticsearch.search.aggregations.support.ValuesSourceConfig;
+import org.elasticsearch.search.aggregations.support.format.ValueFormat;
 import org.elasticsearch.search.aggregations.support.format.ValueFormatter;
 import org.elasticsearch.search.aggregations.support.format.ValueParser;
+import org.elasticsearch.search.internal.SearchContext;
 
 import java.io.IOException;
 import java.util.ArrayList;
@@ -65,18 +68,19 @@ public String toString() {
             return "[" + from + " to " + to + ")";
         }
 
-        public void process(ValueParser parser, AggregationContext aggregationContext) {
+        public void process(ValueParser parser, SearchContext context) {
+            assert parser != null;
             if (fromAsStr != null) {
-                from = parser != null ? parser.parseDouble(fromAsStr, aggregationContext.searchContext()) : Double.valueOf(fromAsStr);
+                from = parser.parseDouble(fromAsStr, context);
             }
             if (toAsStr != null) {
-                to = parser != null ? parser.parseDouble(toAsStr, aggregationContext.searchContext()) : Double.valueOf(toAsStr);
+                to = parser.parseDouble(toAsStr, context);
             }
         }
     }
 
     private final ValuesSource.Numeric valuesSource;
-    private final ValueFormatter formatter;
+    private final @Nullable ValueFormatter formatter;
     private final Range[] ranges;
     private final boolean keyed;
     private final InternalRange.Factory rangeFactory;
@@ -87,8 +91,7 @@ public void process(ValueParser parser, AggregationContext aggregationContext) {
     public RangeAggregator(String name,
                            AggregatorFactories factories,
                            ValuesSource.Numeric valuesSource,
-                           ValueFormatter formatter,
-                           ValueParser parser,
+                           @Nullable ValueFormat format,
                            InternalRange.Factory rangeFactory,
                            List<Range> ranges,
                            boolean keyed,
@@ -98,12 +101,14 @@ public RangeAggregator(String name,
         super(name, BucketAggregationMode.MULTI_BUCKETS, factories, ranges.size() * (parent == null ? 1 : parent.estimatedBucketCount()), aggregationContext, parent);
         assert valuesSource != null;
         this.valuesSource = valuesSource;
-        this.formatter = formatter;
+        this.formatter = format != null ? format.formatter() : null;
         this.keyed = keyed;
         this.rangeFactory = rangeFactory;
         this.ranges = ranges.toArray(new Range[ranges.size()]);
+
+        ValueParser parser = format != null ? format.parser() : ValueParser.RAW;
         for (int i = 0; i < this.ranges.length; i++) {
-            this.ranges[i].process(parser, context);
+            this.ranges[i].process(parser, context.searchContext());
         }
         sortRanges(this.ranges);
 
@@ -193,8 +198,8 @@ public InternalAggregation buildAggregation(long owningBucketOrdinal) {
         for (int i = 0; i < ranges.length; i++) {
             Range range = ranges[i];
             final long bucketOrd = subBucketOrdinal(owningBucketOrdinal, i);
-            org.elasticsearch.search.aggregations.bucket.range.Range.Bucket bucket = rangeFactory.createBucket(
-                    range.key, range.from, range.to, bucketDocCount(bucketOrd),bucketAggregations(bucketOrd), formatter);
+            org.elasticsearch.search.aggregations.bucket.range.Range.Bucket bucket =
+                    rangeFactory.createBucket(range.key, range.from, range.to, bucketDocCount(bucketOrd),bucketAggregations(bucketOrd), formatter);
             buckets.add(bucket);
         }
         // value source can be null in the case of unmapped fields
@@ -207,8 +212,8 @@ public InternalAggregation buildEmptyAggregation() {
         List<org.elasticsearch.search.aggregations.bucket.range.Range.Bucket> buckets = Lists.newArrayListWithCapacity(ranges.length);
         for (int i = 0; i < ranges.length; i++) {
             Range range = ranges[i];
-            org.elasticsearch.search.aggregations.bucket.range.Range.Bucket bucket = rangeFactory.createBucket(
-                    range.key, range.from, range.to, 0, subAggs, formatter);
+            org.elasticsearch.search.aggregations.bucket.range.Range.Bucket bucket =
+                    rangeFactory.createBucket(range.key, range.from, range.to, 0, subAggs, formatter);
             buckets.add(bucket);
         }
         // value source can be null in the case of unmapped fields
@@ -242,33 +247,30 @@ public static class Unmapped extends NonCollectingAggregator {
         private final boolean keyed;
         private final InternalRange.Factory factory;
         private final ValueFormatter formatter;
-        private final ValueParser parser;
 
         public Unmapped(String name,
                         List<RangeAggregator.Range> ranges,
                         boolean keyed,
-                        ValueFormatter formatter,
-                        ValueParser parser,
-                        AggregationContext aggregationContext,
+                        ValueFormat format,
+                        AggregationContext context,
                         Aggregator parent,
                         InternalRange.Factory factory) {
 
-            super(name, aggregationContext, parent);
+            super(name, context, parent);
             this.ranges = ranges;
+            ValueParser parser = format != null ? format.parser() : ValueParser.RAW;
             for (Range range : this.ranges) {
-                range.process(parser, context);
+                range.process(parser, context.searchContext());
             }
             this.keyed = keyed;
-            this.formatter = formatter;
-            this.parser = parser;
+            this.formatter = format != null ? format.formatter() : null;
             this.factory = factory;
         }
 
         @Override
         public InternalAggregation buildEmptyAggregation() {
             InternalAggregations subAggs = buildEmptySubAggregations();
-            List<org.elasticsearch.search.aggregations.bucket.range.Range.Bucket> buckets =
-                    new ArrayList<>(ranges.size());
+            List<org.elasticsearch.search.aggregations.bucket.range.Range.Bucket> buckets = new ArrayList<>(ranges.size());
             for (RangeAggregator.Range range : ranges) {
                 buckets.add(factory.createBucket(range.key, range.from, range.to, 0, subAggs, formatter));
             }
@@ -282,8 +284,8 @@ public static class Factory extends ValuesSourceAggregatorFactory<ValuesSource.N
         private final List<Range> ranges;
         private final boolean keyed;
 
-        public Factory(String name, ValuesSourceConfig<ValuesSource.Numeric> valueSourceConfig, ValueFormatter formatter, ValueParser parser, InternalRange.Factory rangeFactory, List<Range> ranges, boolean keyed) {
-            super(name, rangeFactory.type(), valueSourceConfig, formatter, parser);
+        public Factory(String name, ValuesSourceConfig<ValuesSource.Numeric> valueSourceConfig, InternalRange.Factory rangeFactory, List<Range> ranges, boolean keyed) {
+            super(name, rangeFactory.type(), valueSourceConfig);
             this.rangeFactory = rangeFactory;
             this.ranges = ranges;
             this.keyed = keyed;
@@ -291,12 +293,12 @@ public Factory(String name, ValuesSourceConfig<ValuesSource.Numeric> valueSource
 
         @Override
         protected Aggregator createUnmapped(AggregationContext aggregationContext, Aggregator parent) {
-            return new Unmapped(name, ranges, keyed, formatter, parser, aggregationContext, parent, rangeFactory);
+            return new Unmapped(name, ranges, keyed, config.format(), aggregationContext, parent, rangeFactory);
         }
 
         @Override
         protected Aggregator create(ValuesSource.Numeric valuesSource, long expectedBucketsCount, AggregationContext aggregationContext, Aggregator parent) {
-            return new RangeAggregator(name, factories, valuesSource, formatter, parser, rangeFactory, ranges, keyed, aggregationContext, parent);
+            return new RangeAggregator(name, factories, valuesSource, config.format(), rangeFactory, ranges, keyed, aggregationContext, parent);
         }
     }
 
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/RangeParser.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/RangeParser.java
index 73b8332fa9dae..55edd4a41fbf2 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/RangeParser.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/RangeParser.java
@@ -19,22 +19,16 @@
 package org.elasticsearch.search.aggregations.bucket.range;
 
 import org.elasticsearch.common.xcontent.XContentParser;
-import org.elasticsearch.index.fielddata.IndexFieldData;
-import org.elasticsearch.index.mapper.FieldMapper;
 import org.elasticsearch.search.SearchParseException;
 import org.elasticsearch.search.aggregations.Aggregator;
 import org.elasticsearch.search.aggregations.AggregatorFactory;
-import org.elasticsearch.search.aggregations.support.FieldContext;
 import org.elasticsearch.search.aggregations.support.ValuesSource;
-import org.elasticsearch.search.aggregations.support.ValuesSourceConfig;
-import org.elasticsearch.search.aggregations.support.format.ValueFormatter;
-import org.elasticsearch.search.aggregations.support.format.ValueParser;
+import org.elasticsearch.search.aggregations.support.ValuesSourceParser;
 import org.elasticsearch.search.internal.SearchContext;
 
 import java.io.IOException;
 import java.util.ArrayList;
 import java.util.List;
-import java.util.Map;
 
 /**
  *
@@ -49,34 +43,21 @@ public String type() {
     @Override
     public AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException {
 
-        ValuesSourceConfig<ValuesSource.Numeric> config = new ValuesSourceConfig<>(ValuesSource.Numeric.class);
-
-        String field = null;
         List<RangeAggregator.Range> ranges = null;
-        String script = null;
-        String scriptLang = null;
-        Map<String, Object> scriptParams = null;
         boolean keyed = false;
-        boolean assumeSorted = false;
-        String format = null;
+
+        ValuesSourceParser<ValuesSource.Numeric> vsParser = ValuesSourceParser.numeric(aggregationName, InternalRange.TYPE, context)
+                .requiresSortedValues(true)
+                .formattable(true)
+                .build();
 
         XContentParser.Token token;
         String currentFieldName = null;
         while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
             if (token == XContentParser.Token.FIELD_NAME) {
                 currentFieldName = parser.currentName();
-            } else if (token == XContentParser.Token.VALUE_STRING) {
-                if ("field".equals(currentFieldName)) {
-                    field = parser.text();
-                } else if ("script".equals(currentFieldName)) {
-                    script = parser.text();
-                } else if ("lang".equals(currentFieldName)) {
-                    scriptLang = parser.text();
-                } else if ("format".equals(currentFieldName)) {
-                    format = parser.text();
-                } else {
-                    throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
-                }
+            } else if (vsParser.token(currentFieldName, token, parser)) {
+                continue;
             } else if (token == XContentParser.Token.START_ARRAY) {
                 if ("ranges".equals(currentFieldName)) {
                     ranges = new ArrayList<>();
@@ -111,17 +92,9 @@ public AggregatorFactory parse(String aggregationName, XContentParser parser, Se
                 } else {
                     throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
                 }
-            } else if (token == XContentParser.Token.START_OBJECT) {
-                if ("params".equals(currentFieldName)) {
-                    scriptParams = parser.map();
-                } else {
-                    throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
-                }
             } else if (token == XContentParser.Token.VALUE_BOOLEAN) {
                 if ("keyed".equals(currentFieldName)) {
                     keyed = parser.booleanValue();
-                } else if ("script_values_sorted".equals(currentFieldName) || "scriptValuesSorted".equals(currentFieldName)) {
-                    assumeSorted = parser.booleanValue();
                 } else {
                     throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
                 }
@@ -134,30 +107,6 @@ public AggregatorFactory parse(String aggregationName, XContentParser parser, Se
             throw new SearchParseException(context, "Missing [ranges] in ranges aggregator [" + aggregationName + "]");
         }
 
-        if (script != null) {
-            config.script(context.scriptService().search(context.lookup(), scriptLang, script, scriptParams));
-        }
-
-        if (!assumeSorted) {
-            // we need values to be sorted and unique for efficiency
-            config.ensureSorted(true);
-        }
-
-        if (field == null) {
-            return new RangeAggregator.Factory(aggregationName, config, null, null, InternalRange.FACTORY, ranges, keyed);
-        }
-
-        ValueFormatter valueFormatter = format == null ? ValueFormatter.RAW : new ValueFormatter.Number.Pattern(format);
-        ValueParser valueParser = format == null ? ValueParser.RAW : new ValueParser.Number.Pattern(format);
-
-        FieldMapper<?> mapper = context.smartNameFieldMapper(field);
-        if (mapper == null) {
-            config.unmapped(true);
-            return new RangeAggregator.Factory(aggregationName, config, valueFormatter, valueParser, InternalRange.FACTORY, ranges, keyed);
-        }
-
-        IndexFieldData<?> indexFieldData = context.fieldData().getForField(mapper);
-        config.fieldContext(new FieldContext(field, indexFieldData));
-        return new RangeAggregator.Factory(aggregationName, config, valueFormatter, valueParser, InternalRange.FACTORY, ranges, keyed);
+        return new RangeAggregator.Factory(aggregationName, vsParser.config(), InternalRange.FACTORY, ranges, keyed);
     }
 }
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/date/DateRangeParser.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/date/DateRangeParser.java
index 6d3ce8d5a8723..31a9c5d1472d0 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/date/DateRangeParser.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/date/DateRangeParser.java
@@ -19,25 +19,18 @@
 package org.elasticsearch.search.aggregations.bucket.range.date;
 
 import org.elasticsearch.common.xcontent.XContentParser;
-import org.elasticsearch.index.fielddata.IndexFieldData;
-import org.elasticsearch.index.mapper.FieldMapper;
-import org.elasticsearch.index.mapper.core.DateFieldMapper;
 import org.elasticsearch.search.SearchParseException;
-import org.elasticsearch.search.aggregations.AggregationExecutionException;
 import org.elasticsearch.search.aggregations.Aggregator;
 import org.elasticsearch.search.aggregations.AggregatorFactory;
 import org.elasticsearch.search.aggregations.bucket.range.RangeAggregator;
+import org.elasticsearch.search.aggregations.support.ValueType;
 import org.elasticsearch.search.aggregations.support.ValuesSource;
-import org.elasticsearch.search.aggregations.support.FieldContext;
-import org.elasticsearch.search.aggregations.support.ValuesSourceConfig;
-import org.elasticsearch.search.aggregations.support.format.ValueFormatter;
-import org.elasticsearch.search.aggregations.support.format.ValueParser;
+import org.elasticsearch.search.aggregations.support.ValuesSourceParser;
 import org.elasticsearch.search.internal.SearchContext;
 
 import java.io.IOException;
 import java.util.ArrayList;
 import java.util.List;
-import java.util.Map;
 
 /**
  *
@@ -52,34 +45,22 @@ public String type() {
     @Override
     public AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException {
 
-        ValuesSourceConfig<ValuesSource.Numeric> config = new ValuesSourceConfig<>(ValuesSource.Numeric.class);
+        ValuesSourceParser<ValuesSource.Numeric> vsParser = ValuesSourceParser.numeric(aggregationName, InternalDateRange.TYPE, context)
+                .targetValueType(ValueType.DATE)
+                .requiresSortedValues(true)
+                .formattable(true)
+                .build();
 
-        String field = null;
         List<RangeAggregator.Range> ranges = null;
-        String script = null;
-        String scriptLang = null;
-        Map<String, Object> scriptParams = null;
         boolean keyed = false;
-        String format = null;
-        boolean assumeSorted = false;
 
         XContentParser.Token token;
         String currentFieldName = null;
         while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
             if (token == XContentParser.Token.FIELD_NAME) {
                 currentFieldName = parser.currentName();
-            } else if (token == XContentParser.Token.VALUE_STRING) {
-                if ("field".equals(currentFieldName)) {
-                    field = parser.text();
-                } else if ("script".equals(currentFieldName)) {
-                    script = parser.text();
-                } else if ("lang".equals(currentFieldName)) {
-                    scriptLang = parser.text();
-                } else if ("format".equals(currentFieldName)) {
-                    format = parser.text();
-                } else {
-                    throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
-                }
+            } else if (vsParser.token(currentFieldName, token, parser)) {
+                continue;
             } else if (token == XContentParser.Token.START_ARRAY) {
                 if ("ranges".equals(currentFieldName)) {
                     ranges = new ArrayList<>();
@@ -116,17 +97,9 @@ public AggregatorFactory parse(String aggregationName, XContentParser parser, Se
                         ranges.add(new RangeAggregator.Range(key, from, fromAsStr, to, toAsStr));
                     }
                 }
-            } else if (token == XContentParser.Token.START_OBJECT) {
-                if ("params".equals(currentFieldName)) {
-                    scriptParams = parser.map();
-                } else {
-                    throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
-                }
             } else if (token == XContentParser.Token.VALUE_BOOLEAN) {
                 if ("keyed".equals(currentFieldName)) {
                     keyed = parser.booleanValue();
-                } else if ("script_values_sorted".equals(currentFieldName) || "scriptValuesSorted".equals(currentFieldName)) {
-                    assumeSorted = parser.booleanValue();
                 } else {
                     throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
                 }
@@ -139,38 +112,6 @@ public AggregatorFactory parse(String aggregationName, XContentParser parser, Se
             throw new SearchParseException(context, "Missing [ranges] in ranges aggregator [" + aggregationName + "]");
         }
 
-        if (script != null) {
-            config.script(context.scriptService().search(context.lookup(), scriptLang, script, scriptParams));
-        }
-
-        if (!assumeSorted) {
-            // we need values to be sorted and unique for efficiency
-            config.ensureSorted(true);
-        }
-
-        ValueFormatter valueFormatter = format != null ? new ValueFormatter.DateTime(format) : ValueFormatter.DateTime.DEFAULT;
-        ValueParser valueParser = ValueParser.DateMath.DEFAULT;
-
-        if (field == null) {
-            return new RangeAggregator.Factory(aggregationName, config, valueFormatter, valueParser, InternalDateRange.FACTORY, ranges, keyed);
-        }
-
-        FieldMapper<?> mapper = context.smartNameFieldMapper(field);
-        if (mapper == null) {
-            config.unmapped(true);
-            return new RangeAggregator.Factory(aggregationName, config, valueFormatter, valueParser, InternalDateRange.FACTORY, ranges, keyed);
-        }
-
-        if (!(mapper instanceof DateFieldMapper)) {
-            throw new AggregationExecutionException("date_range aggregation can only be applied to date fields which is not the case with field [" + field + "]");
-        }
-
-        IndexFieldData<?> indexFieldData = context.fieldData().getForField(mapper);
-        config.fieldContext(new FieldContext(field, indexFieldData));
-        if (format == null) {
-            valueFormatter = new ValueFormatter.DateTime(((DateFieldMapper) mapper).dateTimeFormatter());
-        }
-        valueParser = new ValueParser.DateMath(((DateFieldMapper) mapper).dateMathParser());
-        return new RangeAggregator.Factory(aggregationName, config, valueFormatter, valueParser, InternalDateRange.FACTORY, ranges, keyed);
+        return new RangeAggregator.Factory(aggregationName, vsParser.config(), InternalDateRange.FACTORY, ranges, keyed);
     }
 }
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/date/InternalDateRange.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/date/InternalDateRange.java
index 25e6829365c13..622d250fc2853 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/date/InternalDateRange.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/date/InternalDateRange.java
@@ -18,6 +18,7 @@
  */
 package org.elasticsearch.search.aggregations.bucket.range.date;
 
+import org.elasticsearch.common.Nullable;
 import org.elasticsearch.common.io.stream.StreamInput;
 import org.elasticsearch.search.aggregations.AggregationStreams;
 import org.elasticsearch.search.aggregations.InternalAggregation;
@@ -93,7 +94,7 @@ public Bucket createBucket(String key, double from, double to, long docCount, In
 
     InternalDateRange() {} // for serialization
 
-    InternalDateRange(String name, List<InternalDateRange.Bucket> ranges, ValueFormatter formatter, boolean keyed, boolean unmapped) {
+    InternalDateRange(String name, List<InternalDateRange.Bucket> ranges, @Nullable ValueFormatter formatter, boolean keyed, boolean unmapped) {
         super(name, ranges, formatter, keyed, unmapped);
     }
 
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/geodistance/GeoDistanceParser.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/geodistance/GeoDistanceParser.java
index f74e1d4012e8f..4276690655dfc 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/geodistance/GeoDistanceParser.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/geodistance/GeoDistanceParser.java
@@ -18,12 +18,15 @@
  */
 package org.elasticsearch.search.aggregations.bucket.range.geodistance;
 
+import org.elasticsearch.common.ParseField;
 import org.elasticsearch.common.geo.GeoDistance;
 import org.elasticsearch.common.geo.GeoPoint;
 import org.elasticsearch.common.unit.DistanceUnit;
 import org.elasticsearch.common.xcontent.XContentParser;
-import org.elasticsearch.index.fielddata.*;
-import org.elasticsearch.index.mapper.FieldMapper;
+import org.elasticsearch.index.fielddata.BytesValues;
+import org.elasticsearch.index.fielddata.DoubleValues;
+import org.elasticsearch.index.fielddata.GeoPointValues;
+import org.elasticsearch.index.fielddata.LongValues;
 import org.elasticsearch.search.SearchParseException;
 import org.elasticsearch.search.aggregations.Aggregator;
 import org.elasticsearch.search.aggregations.AggregatorFactory;
@@ -31,7 +34,6 @@
 import org.elasticsearch.search.aggregations.bucket.range.RangeAggregator;
 import org.elasticsearch.search.aggregations.bucket.range.RangeAggregator.Unmapped;
 import org.elasticsearch.search.aggregations.support.*;
-import org.elasticsearch.search.aggregations.support.format.ValueFormatter;
 import org.elasticsearch.search.internal.SearchContext;
 
 import java.io.IOException;
@@ -43,6 +45,8 @@
  */
 public class GeoDistanceParser implements Aggregator.Parser {
 
+    private static final ParseField ORIGIN_FIELD = new ParseField("origin", "center", "point", "por");
+
     @Override
     public String type() {
         return InternalGeoDistance.TYPE.name();
@@ -62,9 +66,13 @@ private static String key(String key, double from, double to) {
     @Override
     public AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException {
 
-        String field = null;
+        ValuesSourceParser<ValuesSource.GeoPoint> vsParser = ValuesSourceParser.geoPoint(aggregationName, InternalGeoDistance.TYPE, context)
+                .requiresSortedValues(true)
+                .build();
+
+        GeoPointParser geoPointParser = new GeoPointParser(aggregationName, InternalGeoDistance.TYPE, context, ORIGIN_FIELD);
+
         List<RangeAggregator.Range> ranges = null;
-        GeoPoint origin = null;
         DistanceUnit unit = DistanceUnit.DEFAULT;
         GeoDistance distanceType = GeoDistance.DEFAULT;
         boolean keyed = false;
@@ -74,16 +82,15 @@ public AggregatorFactory parse(String aggregationName, XContentParser parser, Se
         while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
             if (token == XContentParser.Token.FIELD_NAME) {
                 currentFieldName = parser.currentName();
+            } else if (vsParser.token(currentFieldName, token, parser)) {
+                continue;
+            } else if (geoPointParser.token(currentFieldName, token, parser)) {
+                continue;
             } else if (token == XContentParser.Token.VALUE_STRING) {
-                if ("field".equals(currentFieldName)) {
-                    field = parser.text();
-                } else if ("unit".equals(currentFieldName)) {
+                if ("unit".equals(currentFieldName)) {
                     unit = DistanceUnit.fromString(parser.text());
                 } else if ("distance_type".equals(currentFieldName) || "distanceType".equals(currentFieldName)) {
                     distanceType = GeoDistance.fromString(parser.text());
-                } else if ("point".equals(currentFieldName) || "origin".equals(currentFieldName) || "center".equals(currentFieldName)) {
-                    origin = new GeoPoint();
-                    origin.resetFromString(parser.text());
                 } else {
                     throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
                 }
@@ -124,44 +131,7 @@ public AggregatorFactory parse(String aggregationName, XContentParser parser, Se
                         }
                         ranges.add(new RangeAggregator.Range(key(key, from, to), from, fromAsStr, to, toAsStr));
                     }
-                } else if ("point".equals(currentFieldName) || "origin".equals(currentFieldName) || "center".equals(currentFieldName)) {
-                    double lat = Double.NaN;
-                    double lon = Double.NaN;
-                    while ((token = parser.nextToken()) != XContentParser.Token.END_ARRAY) {
-                        if (Double.isNaN(lon)) {
-                            lon = parser.doubleValue();
-                        } else if (Double.isNaN(lat)) {
-                            lat = parser.doubleValue();
-                        } else {
-                            throw new SearchParseException(context, "malformed [origin] geo point array in geo_distance aggregator [" + aggregationName + "]. " +
-                                    "a geo point array must be of the form [lon, lat]");
-                        }
-                    }
-                    origin = new GeoPoint(lat, lon);
-                } else {
-                    throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
-                }
-            } else if (token == XContentParser.Token.START_OBJECT) {
-                if ("point".equals(currentFieldName) || "origin".equals(currentFieldName) || "center".equals(currentFieldName)) {
-                    double lat = Double.NaN;
-                    double lon = Double.NaN;
-                    while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
-                        if (token == XContentParser.Token.FIELD_NAME) {
-                            currentFieldName = parser.currentName();
-                        } else if (token == XContentParser.Token.VALUE_NUMBER) {
-                            if ("lat".equals(currentFieldName)) {
-                                lat = parser.doubleValue();
-                            } else if ("lon".equals(currentFieldName)) {
-                                lon = parser.doubleValue();
-                            }
-                        }
-                    }
-                    if (Double.isNaN(lat) || Double.isNaN(lon)) {
-                        throw new SearchParseException(context, "malformed [origin] geo point object. either [lat] or [lon] (or both) are " +
-                                "missing in geo_distance aggregator [" + aggregationName + "]");
-                    }
-                    origin = new GeoPoint(lat, lon);
-                } else {
+                } else  {
                     throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
                 }
             } else {
@@ -173,25 +143,12 @@ public AggregatorFactory parse(String aggregationName, XContentParser parser, Se
             throw new SearchParseException(context, "Missing [ranges] in geo_distance aggregator [" + aggregationName + "]");
         }
 
+        GeoPoint origin = geoPointParser.geoPoint();
         if (origin == null) {
             throw new SearchParseException(context, "Missing [origin] in geo_distance aggregator [" + aggregationName + "]");
         }
 
-        ValuesSourceConfig<ValuesSource.GeoPoint> config = new ValuesSourceConfig<>(ValuesSource.GeoPoint.class);
-
-        if (field == null) {
-            return new GeoDistanceFactory(aggregationName, config, InternalGeoDistance.FACTORY, origin, unit, distanceType, ranges, keyed);
-        }
-
-        FieldMapper<?> mapper = context.smartNameFieldMapper(field);
-        if (mapper == null) {
-            config.unmapped(true);
-            return new GeoDistanceFactory(aggregationName, config, InternalGeoDistance.FACTORY, origin, unit, distanceType, ranges, keyed);
-        }
-
-        IndexFieldData<?> indexFieldData = context.fieldData().getForField(mapper);
-        config.fieldContext(new FieldContext(field, indexFieldData));
-        return new GeoDistanceFactory(aggregationName, config, InternalGeoDistance.FACTORY, origin, unit, distanceType, ranges, keyed);
+        return new GeoDistanceFactory(aggregationName, vsParser.config(), InternalGeoDistance.FACTORY, origin, unit, distanceType, ranges, keyed);
     }
 
     private static class GeoDistanceFactory extends ValuesSourceAggregatorFactory<ValuesSource.GeoPoint> {
@@ -217,7 +174,7 @@ public GeoDistanceFactory(String name, ValuesSourceConfig<ValuesSource.GeoPoint>
 
         @Override
         protected Aggregator createUnmapped(AggregationContext aggregationContext, Aggregator parent) {
-            return new Unmapped(name, ranges, keyed, null, null, aggregationContext, parent, rangeFactory);
+            return new Unmapped(name, ranges, keyed, null, aggregationContext, parent, rangeFactory);
         }
 
         @Override
@@ -228,7 +185,7 @@ protected Aggregator create(final ValuesSource.GeoPoint valuesSource, long expec
                 // we need to ensure uniqueness
                 distanceSource = new ValuesSource.Numeric.SortedAndUnique(distanceSource);
             }
-            return new RangeAggregator(name, factories, distanceSource, null, null, rangeFactory, ranges, keyed, aggregationContext, parent);
+            return new RangeAggregator(name, factories, distanceSource, null, rangeFactory, ranges, keyed, aggregationContext, parent);
         }
 
         private static class DistanceValues extends DoubleValues {
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/geodistance/InternalGeoDistance.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/geodistance/InternalGeoDistance.java
index 63ce76f82fa00..102486d26a176 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/geodistance/InternalGeoDistance.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/geodistance/InternalGeoDistance.java
@@ -18,6 +18,7 @@
  */
 package org.elasticsearch.search.aggregations.bucket.range.geodistance;
 
+import org.elasticsearch.common.Nullable;
 import org.elasticsearch.common.io.stream.StreamInput;
 import org.elasticsearch.search.aggregations.AggregationStreams;
 import org.elasticsearch.search.aggregations.InternalAggregation;
@@ -70,19 +71,19 @@ public String type() {
         }
 
         @Override
-        public InternalGeoDistance create(String name, List<Bucket> ranges, ValueFormatter formatter, boolean keyed, boolean unmapped) {
+        public InternalGeoDistance create(String name, List<Bucket> ranges, @Nullable ValueFormatter formatter, boolean keyed, boolean unmapped) {
             return new InternalGeoDistance(name, ranges, formatter, keyed, unmapped);
         }
 
         @Override
-        public Bucket createBucket(String key, double from, double to, long docCount, InternalAggregations aggregations, ValueFormatter formatter) {
+        public Bucket createBucket(String key, double from, double to, long docCount, InternalAggregations aggregations, @Nullable ValueFormatter formatter) {
             return new Bucket(key, from, to, docCount, aggregations, formatter);
         }
     }
 
     InternalGeoDistance() {} // for serialization
 
-    public InternalGeoDistance(String name, List<Bucket> ranges, ValueFormatter formatter, boolean keyed, boolean unmapped) {
+    public InternalGeoDistance(String name, List<Bucket> ranges, @Nullable ValueFormatter formatter, boolean keyed, boolean unmapped) {
         super(name, ranges, formatter, keyed, unmapped);
     }
 
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/ipv4/InternalIPv4Range.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/ipv4/InternalIPv4Range.java
index ad41a90319bdb..bd0141e291fe6 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/ipv4/InternalIPv4Range.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/ipv4/InternalIPv4Range.java
@@ -18,6 +18,7 @@
  */
 package org.elasticsearch.search.aggregations.bucket.range.ipv4;
 
+import org.elasticsearch.common.inject.internal.Nullable;
 import org.elasticsearch.common.io.stream.StreamInput;
 import org.elasticsearch.search.aggregations.AggregationStreams;
 import org.elasticsearch.search.aggregations.InternalAggregation;
@@ -54,12 +55,12 @@ public static void registerStream() {
 
     public static class Bucket extends InternalRange.Bucket implements IPv4Range.Bucket {
 
-        public Bucket(String key, double from, double to, long docCount, List<InternalAggregation> aggregations, ValueFormatter formatter) {
-            super(key, from, to, docCount, new InternalAggregations(aggregations), formatter);
+        public Bucket(String key, double from, double to, long docCount, List<InternalAggregation> aggregations) {
+            super(key, from, to, docCount, new InternalAggregations(aggregations), ValueFormatter.IPv4);
         }
 
-        public Bucket(String key, double from, double to, long docCount, InternalAggregations aggregations, ValueFormatter formatter) {
-            super(key, from, to, docCount, aggregations, formatter);
+        public Bucket(String key, double from, double to, long docCount, InternalAggregations aggregations) {
+            super(key, from, to, docCount, aggregations, ValueFormatter.IPv4);
         }
 
         @Override
@@ -83,13 +84,13 @@ public String type() {
         }
 
         @Override
-        public InternalIPv4Range create(String name, List<Bucket> ranges, ValueFormatter formatter, boolean keyed, boolean unmapped) {
+        public InternalIPv4Range create(String name, List<Bucket> ranges, @Nullable ValueFormatter formatter, boolean keyed, boolean unmapped) {
             return new InternalIPv4Range(name, ranges, keyed, unmapped);
         }
 
         @Override
-        public Bucket createBucket(String key, double from, double to, long docCount, InternalAggregations aggregations, ValueFormatter formatter) {
-            return new Bucket(key, from, to, docCount, aggregations, formatter);
+        public Bucket createBucket(String key, double from, double to, long docCount, InternalAggregations aggregations, @Nullable ValueFormatter formatter) {
+            return new Bucket(key, from, to, docCount, aggregations);
         }
     }
 
@@ -105,8 +106,8 @@ public Type type() {
     }
 
     @Override
-    protected Bucket createBucket(String key, double from, double to, long docCount, InternalAggregations aggregations, ValueFormatter formatter) {
-        return new Bucket(key, from, to, docCount, aggregations, formatter);
+    protected Bucket createBucket(String key, double from, double to, long docCount, InternalAggregations aggregations, @Nullable ValueFormatter formatter ) {
+        return new Bucket(key, from, to, docCount, aggregations);
     }
 
 }
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/ipv4/IpRangeParser.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/ipv4/IpRangeParser.java
index f476227eecd86..d411d17d12ea3 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/ipv4/IpRangeParser.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/ipv4/IpRangeParser.java
@@ -19,25 +19,18 @@
 package org.elasticsearch.search.aggregations.bucket.range.ipv4;
 
 import org.elasticsearch.common.xcontent.XContentParser;
-import org.elasticsearch.index.fielddata.IndexFieldData;
-import org.elasticsearch.index.mapper.FieldMapper;
-import org.elasticsearch.index.mapper.ip.IpFieldMapper;
 import org.elasticsearch.search.SearchParseException;
-import org.elasticsearch.search.aggregations.AggregationExecutionException;
 import org.elasticsearch.search.aggregations.Aggregator;
 import org.elasticsearch.search.aggregations.AggregatorFactory;
 import org.elasticsearch.search.aggregations.bucket.range.RangeAggregator;
+import org.elasticsearch.search.aggregations.support.ValueType;
 import org.elasticsearch.search.aggregations.support.ValuesSource;
-import org.elasticsearch.search.aggregations.support.FieldContext;
-import org.elasticsearch.search.aggregations.support.ValuesSourceConfig;
-import org.elasticsearch.search.aggregations.support.format.ValueFormatter;
-import org.elasticsearch.search.aggregations.support.format.ValueParser;
+import org.elasticsearch.search.aggregations.support.ValuesSourceParser;
 import org.elasticsearch.search.internal.SearchContext;
 
 import java.io.IOException;
 import java.util.ArrayList;
 import java.util.List;
-import java.util.Map;
 
 /**
  *
@@ -52,31 +45,22 @@ public String type() {
     @Override
     public AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException {
 
-        ValuesSourceConfig<ValuesSource.Numeric> config = new ValuesSourceConfig<>(ValuesSource.Numeric.class);
+        ValuesSourceParser<ValuesSource.Numeric> vsParser = ValuesSourceParser.numeric(aggregationName, InternalIPv4Range.TYPE, context)
+                .targetValueType(ValueType.IP)
+                .requiresSortedValues(true)
+                .formattable(false)
+                .build();
 
-        String field = null;
         List<RangeAggregator.Range> ranges = null;
-        String script = null;
-        String scriptLang = null;
-        Map<String, Object> scriptParams = null;
         boolean keyed = false;
-        boolean assumeSorted = false;
 
         XContentParser.Token token;
         String currentFieldName = null;
         while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
             if (token == XContentParser.Token.FIELD_NAME) {
                 currentFieldName = parser.currentName();
-            } else if (token == XContentParser.Token.VALUE_STRING) {
-                if ("field".equals(currentFieldName)) {
-                    field = parser.text();
-                } else if ("script".equals(currentFieldName)) {
-                    script = parser.text();
-                } else if ("lang".equals(currentFieldName)) {
-                    scriptLang = parser.text();
-                } else {
-                    throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
-                }
+            } else if (vsParser.token(currentFieldName, token, parser)) {
+                continue;
             } else if (token == XContentParser.Token.START_ARRAY) {
                 if ("ranges".equals(currentFieldName)) {
                     ranges = new ArrayList<>();
@@ -118,17 +102,9 @@ public AggregatorFactory parse(String aggregationName, XContentParser parser, Se
                 } else {
                     throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
                 }
-            } else if (token == XContentParser.Token.START_OBJECT) {
-                if ("params".equals(currentFieldName)) {
-                    scriptParams = parser.map();
-                } else {
-                    throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
-                }
             } else if (token == XContentParser.Token.VALUE_BOOLEAN) {
                 if ("keyed".equals(currentFieldName)) {
                     keyed = parser.booleanValue();
-                } else if ("script_values_sorted".equals(currentFieldName) || "scriptValuesSorted".equals(currentFieldName)) {
-                    assumeSorted = parser.booleanValue();
                 } else {
                     throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
                 }
@@ -141,32 +117,7 @@ public AggregatorFactory parse(String aggregationName, XContentParser parser, Se
             throw new SearchParseException(context, "Missing [ranges] in ranges aggregator [" + aggregationName + "]");
         }
 
-        if (script != null) {
-            config.script(context.scriptService().search(context.lookup(), scriptLang, script, scriptParams));
-        }
-
-        if (!assumeSorted) {
-            // we need values to be sorted and unique for efficiency
-            config.ensureSorted(true);
-        }
-
-        if (field == null) {
-            return new RangeAggregator.Factory(aggregationName, config, ValueFormatter.IPv4, ValueParser.IPv4, InternalIPv4Range.FACTORY, ranges, keyed);
-        }
-
-        FieldMapper<?> mapper = context.smartNameFieldMapper(field);
-        if (mapper == null) {
-            config.unmapped(true);
-            return new RangeAggregator.Factory(aggregationName, config, ValueFormatter.IPv4, ValueParser.IPv4, InternalIPv4Range.FACTORY, ranges, keyed);
-        }
-
-        if (!(mapper instanceof IpFieldMapper)) {
-            throw new AggregationExecutionException("ip_range aggregation can only be applied to ip fields which is not the case with field [" + field + "]");
-        }
-
-        IndexFieldData<?> indexFieldData = context.fieldData().getForField(mapper);
-        config.fieldContext(new FieldContext(field, indexFieldData));
-        return new RangeAggregator.Factory(aggregationName, config, ValueFormatter.IPv4, ValueParser.IPv4, InternalIPv4Range.FACTORY, ranges, keyed);
+        return new RangeAggregator.Factory(aggregationName, vsParser.config(), InternalIPv4Range.FACTORY, ranges, keyed);
     }
 
     private static void parseMaskRange(String cidr, RangeAggregator.Range range, String aggregationName, SearchContext ctx) {
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/SignificantLongTerms.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/SignificantLongTerms.java
index 151ad5d96d410..371ea4e16397f 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/SignificantLongTerms.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/SignificantLongTerms.java
@@ -18,6 +18,7 @@
  */
 package org.elasticsearch.search.aggregations.bucket.significant;
 
+import org.elasticsearch.common.Nullable;
 import org.elasticsearch.common.io.stream.StreamInput;
 import org.elasticsearch.common.io.stream.StreamOutput;
 import org.elasticsearch.common.text.StringText;
@@ -84,15 +85,15 @@ public String getKey() {
 
     }
 
-    private ValueFormatter valueFormatter;
+    private ValueFormatter formatter;
 
     SignificantLongTerms() {} // for serialization
 
-    public SignificantLongTerms(long subsetSize, long supersetSize, String name, ValueFormatter valueFormatter,
-            int requiredSize, long minDocCount, Collection<InternalSignificantTerms.Bucket> buckets) {
+    public SignificantLongTerms(long subsetSize, long supersetSize, String name, @Nullable ValueFormatter formatter,
+                                int requiredSize, long minDocCount, Collection<InternalSignificantTerms.Bucket> buckets) {
 
         super(subsetSize, supersetSize, name, requiredSize, minDocCount, buckets);
-        this.valueFormatter = valueFormatter;
+        this.formatter = formatter;
     }
 
     @Override
@@ -103,7 +104,7 @@ public Type type() {
     @Override
     public void readFrom(StreamInput in) throws IOException {
         this.name = in.readString();
-        this.valueFormatter = ValueFormatterStreams.readOptional(in);
+        this.formatter = ValueFormatterStreams.readOptional(in);
         this.requiredSize = readSize(in);
         this.minDocCount = in.readVLong();
         this.subsetSize = in.readVLong();
@@ -124,7 +125,7 @@ public void readFrom(StreamInput in) throws IOException {
     @Override
     public void writeTo(StreamOutput out) throws IOException {
         out.writeString(name);
-        ValueFormatterStreams.writeOptional(valueFormatter, out);
+        ValueFormatterStreams.writeOptional(formatter, out);
         writeSize(requiredSize, out);
         out.writeVLong(minDocCount);
         out.writeVLong(subsetSize);
@@ -146,8 +147,8 @@ public XContentBuilder toXContent(XContentBuilder builder, Params params) throws
         for (InternalSignificantTerms.Bucket bucket : buckets) {
             builder.startObject();
             builder.field(CommonFields.KEY, ((Bucket) bucket).term);
-            if (valueFormatter != null) {
-                builder.field(CommonFields.KEY_AS_STRING, valueFormatter.format(((Bucket) bucket).term));
+            if (formatter != null) {
+                builder.field(CommonFields.KEY_AS_STRING, formatter.format(((Bucket) bucket).term));
             }
             builder.field(CommonFields.DOC_COUNT, bucket.getDocCount());
             builder.field("score", bucket.score);
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/SignificantLongTermsAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/SignificantLongTermsAggregator.java
index a1c915368f058..8fed8059a00ef 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/SignificantLongTermsAggregator.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/SignificantLongTermsAggregator.java
@@ -19,13 +19,14 @@
 package org.elasticsearch.search.aggregations.bucket.significant;
 
 import org.apache.lucene.index.IndexReader;
+import org.elasticsearch.common.Nullable;
 import org.elasticsearch.common.lease.Releasables;
 import org.elasticsearch.search.aggregations.Aggregator;
 import org.elasticsearch.search.aggregations.AggregatorFactories;
 import org.elasticsearch.search.aggregations.bucket.terms.LongTermsAggregator;
 import org.elasticsearch.search.aggregations.support.AggregationContext;
 import org.elasticsearch.search.aggregations.support.ValuesSource;
-import org.elasticsearch.search.aggregations.support.format.ValueFormatter;
+import org.elasticsearch.search.aggregations.support.format.ValueFormat;
 import org.elasticsearch.search.internal.ContextIndexSearcher;
 
 import java.io.IOException;
@@ -37,11 +38,11 @@
  */
 public class SignificantLongTermsAggregator extends LongTermsAggregator {
 
-    public SignificantLongTermsAggregator(String name, AggregatorFactories factories, ValuesSource.Numeric valuesSource, ValueFormatter formatter,
+    public SignificantLongTermsAggregator(String name, AggregatorFactories factories, ValuesSource.Numeric valuesSource, @Nullable ValueFormat format,
               long estimatedBucketCount, int requiredSize, int shardSize, long minDocCount,
               AggregationContext aggregationContext, Aggregator parent, SignificantTermsAggregatorFactory termsAggFactory) {
 
-        super(name, factories, valuesSource, formatter, estimatedBucketCount, null, requiredSize, shardSize, minDocCount, aggregationContext, parent);
+        super(name, factories, valuesSource, format, estimatedBucketCount, null, requiredSize, shardSize, minDocCount, aggregationContext, parent);
         this.termsAggFactory = termsAggFactory;
     }
 
@@ -95,8 +96,7 @@ public SignificantLongTerms buildAggregation(long owningBucketOrdinal) {
             bucket.aggregations = bucketAggregations(bucket.bucketOrd);
             list[i] = bucket;
         }
-        return new SignificantLongTerms(subsetSize, supersetSize, name, formatter, requiredSize, minDocCount,
-                Arrays.asList(list));
+        return new SignificantLongTerms(subsetSize, supersetSize, name, formatter, requiredSize, minDocCount, Arrays.asList(list));
     }
 
     @Override
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/SignificantTermsAggregatorFactory.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/SignificantTermsAggregatorFactory.java
index 7e3d7e6ce5902..387cfc0c4d2be 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/SignificantTermsAggregatorFactory.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/SignificantTermsAggregatorFactory.java
@@ -28,15 +28,16 @@
 import org.elasticsearch.common.lucene.index.FilterableTermsEnum;
 import org.elasticsearch.common.lucene.index.FreqTermsEnum;
 import org.elasticsearch.index.mapper.FieldMapper;
-import org.elasticsearch.search.aggregations.*;
+import org.elasticsearch.search.aggregations.AggregationExecutionException;
+import org.elasticsearch.search.aggregations.Aggregator;
 import org.elasticsearch.search.aggregations.Aggregator.BucketAggregationMode;
+import org.elasticsearch.search.aggregations.InternalAggregation;
+import org.elasticsearch.search.aggregations.NonCollectingAggregator;
 import org.elasticsearch.search.aggregations.bucket.terms.support.IncludeExclude;
 import org.elasticsearch.search.aggregations.support.AggregationContext;
 import org.elasticsearch.search.aggregations.support.ValuesSource;
 import org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory;
 import org.elasticsearch.search.aggregations.support.ValuesSourceConfig;
-import org.elasticsearch.search.aggregations.support.format.ValueFormatter;
-import org.elasticsearch.search.aggregations.support.format.ValueParser;
 import org.elasticsearch.search.internal.SearchContext;
 
 import java.io.IOException;
@@ -60,9 +61,11 @@ public class SignificantTermsAggregatorFactory extends ValuesSourceAggregatorFac
     private int numberOfAggregatorsCreated = 0;
     private Filter filter;
 
-    public SignificantTermsAggregatorFactory(String name, ValuesSourceConfig valueSourceConfig, ValueFormatter formatter, ValueParser parser,
-            int requiredSize, int shardSize, long minDocCount, IncludeExclude includeExclude, String executionHint, Filter filter) {
-        super(name, SignificantStringTerms.TYPE.name(), valueSourceConfig, formatter, parser);
+    public SignificantTermsAggregatorFactory(String name, ValuesSourceConfig valueSourceConfig, int requiredSize,
+                                             int shardSize, long minDocCount, IncludeExclude includeExclude,
+                                             String executionHint, Filter filter) {
+
+        super(name, SignificantStringTerms.TYPE.name(), valueSourceConfig);
         this.requiredSize = requiredSize;
         this.shardSize = shardSize;
         this.minDocCount = minDocCount;
@@ -150,7 +153,7 @@ protected Aggregator create(ValuesSource valuesSource, long expectedBucketsCount
             if (((ValuesSource.Numeric) valuesSource).isFloatingPoint()) {
                 throw new UnsupportedOperationException("No support for examining floating point numerics");
             }
-            return new SignificantLongTermsAggregator(name, factories, (ValuesSource.Numeric) valuesSource, formatter, estimatedBucketCount, requiredSize, shardSize, minDocCount, aggregationContext, parent, this);
+            return new SignificantLongTermsAggregator(name, factories, (ValuesSource.Numeric) valuesSource, config.format(), estimatedBucketCount, requiredSize, shardSize, minDocCount, aggregationContext, parent, this);
         }
 
         throw new AggregationExecutionException("sigfnificant_terms aggregation cannot be applied to field [" + config.fieldContext().field() +
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/SignificantTermsParser.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/SignificantTermsParser.java
index de3688c9e316f..d8f2ef80e58a8 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/SignificantTermsParser.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/SignificantTermsParser.java
@@ -19,27 +19,16 @@
 package org.elasticsearch.search.aggregations.bucket.significant;
 
 import org.apache.lucene.search.Filter;
-import org.elasticsearch.common.regex.Regex;
 import org.elasticsearch.common.xcontent.XContentParser;
-import org.elasticsearch.index.fielddata.IndexFieldData;
-import org.elasticsearch.index.fielddata.IndexNumericFieldData;
-import org.elasticsearch.index.mapper.FieldMapper;
-import org.elasticsearch.index.mapper.core.DateFieldMapper;
-import org.elasticsearch.index.mapper.ip.IpFieldMapper;
 import org.elasticsearch.search.SearchParseException;
 import org.elasticsearch.search.aggregations.Aggregator;
 import org.elasticsearch.search.aggregations.AggregatorFactory;
 import org.elasticsearch.search.aggregations.bucket.BucketUtils;
 import org.elasticsearch.search.aggregations.bucket.terms.support.IncludeExclude;
-import org.elasticsearch.search.aggregations.support.FieldContext;
-import org.elasticsearch.search.aggregations.support.ValuesSource;
-import org.elasticsearch.search.aggregations.support.ValuesSourceConfig;
-import org.elasticsearch.search.aggregations.support.format.ValueFormatter;
-import org.elasticsearch.search.aggregations.support.format.ValueParser;
+import org.elasticsearch.search.aggregations.support.ValuesSourceParser;
 import org.elasticsearch.search.internal.SearchContext;
 
 import java.io.IOException;
-import java.util.regex.Pattern;
 
 /**
  *
@@ -60,33 +49,32 @@ public String type() {
     @Override
     public AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException {
 
-        String field = null;
+        ValuesSourceParser vsParser = ValuesSourceParser.any(aggregationName, SignificantStringTerms.TYPE, context)
+                .scriptable(false)
+                .formattable(true)
+                .requiresSortedValues(true)
+                .requiresUniqueValues(true)
+                .build();
+
+        IncludeExclude.Parser incExcParser = new IncludeExclude.Parser(aggregationName, SignificantStringTerms.TYPE, context);
+
         Filter filter = null;
         int requiredSize = DEFAULT_REQUIRED_SIZE;
         int shardSize = DEFAULT_SHARD_SIZE;
-        String format = null;
-        String include = null;
-        int includeFlags = 0; // 0 means no flags
-        String exclude = null;
-        int excludeFlags = 0; // 0 means no flags
-        String executionHint = null;
         long minDocCount = DEFAULT_MIN_DOC_COUNT;
+        String executionHint = null;
 
         XContentParser.Token token;
         String currentFieldName = null;
         while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
             if (token == XContentParser.Token.FIELD_NAME) {
                 currentFieldName = parser.currentName();
+            } else if (vsParser.token(currentFieldName, token, parser)) {
+                continue;
+            } else if (incExcParser.token(currentFieldName, token, parser)) {
+                continue;
             } else if (token == XContentParser.Token.VALUE_STRING) {
-                if ("field".equals(currentFieldName)) {
-                    field = parser.text();
-                } else if ("format".equals(currentFieldName)) {
-                    format = parser.text();
-                } else if ("include".equals(currentFieldName)) {
-                    include = parser.text();
-                } else if ("exclude".equals(currentFieldName)) {
-                    exclude = parser.text();
-                } else if ("execution_hint".equals(currentFieldName) || "executionHint".equals(currentFieldName)) {
+                if ("execution_hint".equals(currentFieldName) || "executionHint".equals(currentFieldName)) {
                     executionHint = parser.text();
                 } else {
                     throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
@@ -112,43 +100,8 @@ public AggregatorFactory parse(String aggregationName, XContentParser parser, Se
                 // filters defined by end users and parsed below are not.
 //                if ("background_context".equals(currentFieldName)) {
 //                    filter = context.queryParserService().parseInnerFilter(parser).filter();
-//                } else
-
-                if ("include".equals(currentFieldName)) {
-                    while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
-                        if (token == XContentParser.Token.FIELD_NAME) {
-                            currentFieldName = parser.currentName();
-                        } else if (token == XContentParser.Token.VALUE_STRING) {
-                            if ("pattern".equals(currentFieldName)) {
-                                include = parser.text();
-                            } else if ("flags".equals(currentFieldName)) {
-                                includeFlags = Regex.flagsFromString(parser.text());
-                            }
-                        } else if (token == XContentParser.Token.VALUE_NUMBER) {
-                            if ("flags".equals(currentFieldName)) {
-                                includeFlags = parser.intValue();
-                            }
-                        }
-                    }
-                } else if ("exclude".equals(currentFieldName)) {
-                    while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
-                        if (token == XContentParser.Token.FIELD_NAME) {
-                            currentFieldName = parser.currentName();
-                        } else if (token == XContentParser.Token.VALUE_STRING) {
-                            if ("pattern".equals(currentFieldName)) {
-                                exclude = parser.text();
-                            } else if ("flags".equals(currentFieldName)) {
-                                excludeFlags = Regex.flagsFromString(parser.text());
-                            }
-                        } else if (token == XContentParser.Token.VALUE_NUMBER) {
-                            if ("flags".equals(currentFieldName)) {
-                                excludeFlags = parser.intValue();
-                            }
-                        }
-                    }
-                } else {
-                    throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
-                }
+//                }
+
             } else {
                 throw new SearchParseException(context, "Unexpected token " + token + " in [" + aggregationName + "].");
             }
@@ -171,55 +124,8 @@ public AggregatorFactory parse(String aggregationName, XContentParser parser, Se
             shardSize = requiredSize;
         }
 
-        IncludeExclude includeExclude = null;
-        if (include != null || exclude != null) {
-            Pattern includePattern = include != null ? Pattern.compile(include, includeFlags) : null;
-            Pattern excludePattern = exclude != null ? Pattern.compile(exclude, excludeFlags) : null;
-            includeExclude = new IncludeExclude(includePattern, excludePattern);
-        }
-
-        FieldMapper<?> mapper = context.smartNameFieldMapper(field);
-        if (mapper == null) {
-            ValuesSourceConfig<?> config = new ValuesSourceConfig<>(ValuesSource.Bytes.class);
-            config.unmapped(true);
-            return new SignificantTermsAggregatorFactory(aggregationName, config, null, null, requiredSize, shardSize, minDocCount, includeExclude, executionHint, filter);
-        }
-        IndexFieldData<?> indexFieldData = context.fieldData().getForField(mapper);
-
-        ValuesSourceConfig<?> config;
-        ValueFormatter valueFormatter = null;
-        ValueParser valueParser = null;
-        if (mapper instanceof DateFieldMapper) {
-            DateFieldMapper dateMapper = (DateFieldMapper) mapper;
-            config = new ValuesSourceConfig<>(ValuesSource.Numeric.class);
-            valueFormatter = format == null ?
-                    new ValueFormatter.DateTime(dateMapper.dateTimeFormatter()) :
-                    new ValueFormatter.DateTime(format);
-            valueParser = new ValueParser.DateMath(dateMapper.dateMathParser());
-
-        } else if (mapper instanceof IpFieldMapper) {
-            config = new ValuesSourceConfig<>(ValuesSource.Numeric.class);
-            valueFormatter = ValueFormatter.IPv4;
-            valueParser = ValueParser.IPv4;
-
-        } else if (indexFieldData instanceof IndexNumericFieldData) {
-            config = new ValuesSourceConfig<>(ValuesSource.Numeric.class);
-            if (format != null) {
-                valueFormatter = new ValueFormatter.Number.Pattern(format);
-            }
-
-        } else {
-            config = new ValuesSourceConfig<>(ValuesSource.Bytes.class);
-            // TODO: it will make sense to set false instead here if the aggregator factory uses
-            // ordinals instead of hash tables
-            config.needsHashes(true);
-        }
-
-        config.fieldContext(new FieldContext(field, indexFieldData));
-        // We need values to be unique to be able to run terms aggs efficiently
-        config.ensureUnique(true);
-
-        return new SignificantTermsAggregatorFactory(aggregationName, config, valueFormatter, valueParser, requiredSize, shardSize, minDocCount, includeExclude, executionHint, filter);
+        IncludeExclude includeExclude = incExcParser.includeExclude();
+        return new SignificantTermsAggregatorFactory(aggregationName, vsParser.config(), requiredSize, shardSize, minDocCount, includeExclude, executionHint, filter);
     }
 
 }
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/DoubleTerms.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/DoubleTerms.java
index 7943007e4e7b7..d81e8bba91ef6 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/DoubleTerms.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/DoubleTerms.java
@@ -18,6 +18,7 @@
  */
 package org.elasticsearch.search.aggregations.bucket.terms;
 
+import org.elasticsearch.common.Nullable;
 import org.elasticsearch.common.io.stream.StreamInput;
 import org.elasticsearch.common.io.stream.StreamOutput;
 import org.elasticsearch.common.text.StringText;
@@ -88,17 +89,13 @@ int compareTerm(Terms.Bucket other) {
 
     }
 
-    private ValueFormatter valueFormatter;
+    private @Nullable ValueFormatter formatter;
 
     DoubleTerms() {} // for serialization
 
-    public DoubleTerms(String name, InternalOrder order, int requiredSize, long minDocCount, Collection<InternalTerms.Bucket> buckets) {
-        this(name, order, null, requiredSize, minDocCount, buckets);
-    }
-
-    public DoubleTerms(String name, InternalOrder order, ValueFormatter valueFormatter, int requiredSize, long minDocCount, Collection<InternalTerms.Bucket> buckets) {
+    public DoubleTerms(String name, InternalOrder order, @Nullable ValueFormatter formatter, int requiredSize, long minDocCount, Collection<InternalTerms.Bucket> buckets) {
         super(name, order, requiredSize, minDocCount, buckets);
-        this.valueFormatter = valueFormatter;
+        this.formatter = formatter;
     }
 
     @Override
@@ -166,7 +163,7 @@ public InternalTerms reduce(ReduceContext reduceContext) {
     public void readFrom(StreamInput in) throws IOException {
         this.name = in.readString();
         this.order = InternalOrder.Streams.readOrder(in);
-        this.valueFormatter = ValueFormatterStreams.readOptional(in);
+        this.formatter = ValueFormatterStreams.readOptional(in);
         this.requiredSize = readSize(in);
         this.minDocCount = in.readVLong();
         int size = in.readVInt();
@@ -182,7 +179,7 @@ public void readFrom(StreamInput in) throws IOException {
     public void writeTo(StreamOutput out) throws IOException {
         out.writeString(name);
         InternalOrder.Streams.writeOrder(order, out);
-        ValueFormatterStreams.writeOptional(valueFormatter, out);
+        ValueFormatterStreams.writeOptional(formatter, out);
         writeSize(requiredSize, out);
         out.writeVLong(minDocCount);
         out.writeVInt(buckets.size());
@@ -200,8 +197,8 @@ public XContentBuilder toXContent(XContentBuilder builder, Params params) throws
         for (InternalTerms.Bucket bucket : buckets) {
             builder.startObject();
             builder.field(CommonFields.KEY, ((Bucket) bucket).term);
-            if (valueFormatter != null) {
-                builder.field(CommonFields.KEY_AS_STRING, valueFormatter.format(((Bucket) bucket).term));
+            if (formatter != null) {
+                builder.field(CommonFields.KEY_AS_STRING, formatter.format(((Bucket) bucket).term));
             }
             builder.field(CommonFields.DOC_COUNT, bucket.getDocCount());
             ((InternalAggregations) bucket.getAggregations()).toXContentInternal(builder, params);
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/DoubleTermsAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/DoubleTermsAggregator.java
index 25c83762422bb..c9a44ea24e22c 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/DoubleTermsAggregator.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/DoubleTermsAggregator.java
@@ -19,6 +19,7 @@
 package org.elasticsearch.search.aggregations.bucket.terms;
 
 import org.apache.lucene.index.AtomicReaderContext;
+import org.elasticsearch.common.Nullable;
 import org.elasticsearch.common.lease.Releasables;
 import org.elasticsearch.common.util.LongHash;
 import org.elasticsearch.index.fielddata.DoubleValues;
@@ -28,6 +29,7 @@
 import org.elasticsearch.search.aggregations.bucket.terms.support.BucketPriorityQueue;
 import org.elasticsearch.search.aggregations.support.AggregationContext;
 import org.elasticsearch.search.aggregations.support.ValuesSource;
+import org.elasticsearch.search.aggregations.support.format.ValueFormat;
 import org.elasticsearch.search.aggregations.support.format.ValueFormatter;
 
 import java.io.IOException;
@@ -48,11 +50,11 @@ public class DoubleTermsAggregator extends BucketsAggregator {
     private final LongHash bucketOrds;
     private DoubleValues values;
 
-    public DoubleTermsAggregator(String name, AggregatorFactories factories, ValuesSource.Numeric valuesSource, ValueFormatter formatter, long estimatedBucketCount,
+    public DoubleTermsAggregator(String name, AggregatorFactories factories, ValuesSource.Numeric valuesSource, @Nullable ValueFormat format, long estimatedBucketCount,
                                InternalOrder order, int requiredSize, int shardSize, long minDocCount, AggregationContext aggregationContext, Aggregator parent) {
         super(name, BucketAggregationMode.PER_BUCKET, factories, estimatedBucketCount, aggregationContext, parent);
         this.valuesSource = valuesSource;
-        this.formatter = formatter;
+        this.formatter = format != null ? format.formatter() : null;
         this.order = InternalOrder.validate(order, this);
         this.requiredSize = requiredSize;
         this.shardSize = shardSize;
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/LongTerms.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/LongTerms.java
index b26d3558cc9cd..65e626dbea359 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/LongTerms.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/LongTerms.java
@@ -18,6 +18,7 @@
  */
 package org.elasticsearch.search.aggregations.bucket.terms;
 
+import org.elasticsearch.common.Nullable;
 import org.elasticsearch.common.io.stream.StreamInput;
 import org.elasticsearch.common.io.stream.StreamOutput;
 import org.elasticsearch.common.text.StringText;
@@ -88,13 +89,13 @@ int compareTerm(Terms.Bucket other) {
         }
     }
 
-    private ValueFormatter valueFormatter;
+    private @Nullable ValueFormatter formatter;
 
     LongTerms() {} // for serialization
 
-    public LongTerms(String name, InternalOrder order, ValueFormatter valueFormatter, int requiredSize, long minDocCount, Collection<InternalTerms.Bucket> buckets) {
+    public LongTerms(String name, InternalOrder order, @Nullable ValueFormatter formatter, int requiredSize, long minDocCount, Collection<InternalTerms.Bucket> buckets) {
         super(name, order, requiredSize, minDocCount, buckets);
-        this.valueFormatter = valueFormatter;
+        this.formatter = formatter;
     }
 
     @Override
@@ -162,7 +163,7 @@ public InternalTerms reduce(ReduceContext reduceContext) {
     public void readFrom(StreamInput in) throws IOException {
         this.name = in.readString();
         this.order = InternalOrder.Streams.readOrder(in);
-        this.valueFormatter = ValueFormatterStreams.readOptional(in);
+        this.formatter = ValueFormatterStreams.readOptional(in);
         this.requiredSize = readSize(in);
         this.minDocCount = in.readVLong();
         int size = in.readVInt();
@@ -178,7 +179,7 @@ public void readFrom(StreamInput in) throws IOException {
     public void writeTo(StreamOutput out) throws IOException {
         out.writeString(name);
         InternalOrder.Streams.writeOrder(order, out);
-        ValueFormatterStreams.writeOptional(valueFormatter, out);
+        ValueFormatterStreams.writeOptional(formatter, out);
         writeSize(requiredSize, out);
         out.writeVLong(minDocCount);
         out.writeVInt(buckets.size());
@@ -196,8 +197,8 @@ public XContentBuilder toXContent(XContentBuilder builder, Params params) throws
         for (InternalTerms.Bucket bucket : buckets) {
             builder.startObject();
             builder.field(CommonFields.KEY, ((Bucket) bucket).term);
-            if (valueFormatter != null) {
-                builder.field(CommonFields.KEY_AS_STRING, valueFormatter.format(((Bucket) bucket).term));
+            if (formatter != null) {
+                builder.field(CommonFields.KEY_AS_STRING, formatter.format(((Bucket) bucket).term));
             }
             builder.field(CommonFields.DOC_COUNT, bucket.getDocCount());
             ((InternalAggregations) bucket.getAggregations()).toXContentInternal(builder, params);
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/LongTermsAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/LongTermsAggregator.java
index 9e8d0b925338b..cb1a222429314 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/LongTermsAggregator.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/LongTermsAggregator.java
@@ -19,6 +19,7 @@
 package org.elasticsearch.search.aggregations.bucket.terms;
 
 import org.apache.lucene.index.AtomicReaderContext;
+import org.elasticsearch.common.Nullable;
 import org.elasticsearch.common.lease.Releasables;
 import org.elasticsearch.common.util.LongHash;
 import org.elasticsearch.index.fielddata.LongValues;
@@ -29,6 +30,7 @@
 import org.elasticsearch.search.aggregations.bucket.terms.support.BucketPriorityQueue;
 import org.elasticsearch.search.aggregations.support.AggregationContext;
 import org.elasticsearch.search.aggregations.support.ValuesSource;
+import org.elasticsearch.search.aggregations.support.format.ValueFormat;
 import org.elasticsearch.search.aggregations.support.format.ValueFormatter;
 
 import java.io.IOException;
@@ -45,15 +47,15 @@ public class LongTermsAggregator extends BucketsAggregator {
     protected final int shardSize;
     protected final long minDocCount;
     protected final ValuesSource.Numeric valuesSource;
-    protected final ValueFormatter formatter;
+    protected final @Nullable ValueFormatter formatter;
     protected final LongHash bucketOrds;
     private LongValues values;
 
-    public LongTermsAggregator(String name, AggregatorFactories factories, ValuesSource.Numeric valuesSource, ValueFormatter formatter, long estimatedBucketCount,
+    public LongTermsAggregator(String name, AggregatorFactories factories, ValuesSource.Numeric valuesSource, @Nullable ValueFormat format, long estimatedBucketCount,
                                InternalOrder order, int requiredSize, int shardSize, long minDocCount, AggregationContext aggregationContext, Aggregator parent) {
         super(name, BucketAggregationMode.PER_BUCKET, factories, estimatedBucketCount, aggregationContext, parent);
         this.valuesSource = valuesSource;
-        this.formatter = formatter;
+        this.formatter = format != null ? format.formatter() : null;
         this.order = InternalOrder.validate(order, this);
         this.requiredSize = requiredSize;
         this.shardSize = shardSize;
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/Terms.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/Terms.java
index 3c7b9b273642a..f1663fd153a7b 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/Terms.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/Terms.java
@@ -21,7 +21,6 @@
 import org.elasticsearch.common.xcontent.ToXContent;
 import org.elasticsearch.search.aggregations.Aggregator;
 import org.elasticsearch.search.aggregations.bucket.MultiBucketsAggregation;
-import org.elasticsearch.search.aggregations.support.ScriptValueType;
 
 import java.util.Collection;
 import java.util.Comparator;
@@ -34,13 +33,13 @@ public interface Terms extends MultiBucketsAggregation {
 
     static enum ValueType {
 
-        STRING(ScriptValueType.STRING),
-        LONG(ScriptValueType.LONG),
-        DOUBLE(ScriptValueType.DOUBLE);
+        STRING(org.elasticsearch.search.aggregations.support.ValueType.STRING),
+        LONG(org.elasticsearch.search.aggregations.support.ValueType.LONG),
+        DOUBLE(org.elasticsearch.search.aggregations.support.ValueType.DOUBLE);
 
-        final ScriptValueType scriptValueType;
+        final org.elasticsearch.search.aggregations.support.ValueType scriptValueType;
 
-        private ValueType(ScriptValueType scriptValueType) {
+        private ValueType(org.elasticsearch.search.aggregations.support.ValueType scriptValueType) {
             this.scriptValueType = scriptValueType;
         }
 
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/TermsAggregatorFactory.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/TermsAggregatorFactory.java
index c0ea410cae031..9dd52435cbe16 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/TermsAggregatorFactory.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/TermsAggregatorFactory.java
@@ -27,8 +27,6 @@
 import org.elasticsearch.search.aggregations.support.ValuesSource;
 import org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory;
 import org.elasticsearch.search.aggregations.support.ValuesSourceConfig;
-import org.elasticsearch.search.aggregations.support.format.ValueFormatter;
-import org.elasticsearch.search.aggregations.support.format.ValueParser;
 
 /**
  *
@@ -140,10 +138,10 @@ public String toString() {
     private final IncludeExclude includeExclude;
     private final String executionHint;
 
-    public TermsAggregatorFactory(String name, ValuesSourceConfig config, ValueFormatter formatter, ValueParser parser,
-            InternalOrder order, int requiredSize, int shardSize, long minDocCount, IncludeExclude includeExclude, String executionHint) {
+    public TermsAggregatorFactory(String name, ValuesSourceConfig config, InternalOrder order, int requiredSize,
+                                  int shardSize, long minDocCount, IncludeExclude includeExclude, String executionHint) {
 
-        super(name, StringTerms.TYPE.name(), config, formatter, parser);
+        super(name, StringTerms.TYPE.name(), config);
         this.order = order;
         this.requiredSize = requiredSize;
         this.shardSize = shardSize;
@@ -225,9 +223,9 @@ protected Aggregator create(ValuesSource valuesSource, long expectedBucketsCount
 
         if (valuesSource instanceof ValuesSource.Numeric) {
             if (((ValuesSource.Numeric) valuesSource).isFloatingPoint()) {
-                return new DoubleTermsAggregator(name, factories, (ValuesSource.Numeric) valuesSource, formatter, estimatedBucketCount, order, requiredSize, shardSize, minDocCount, aggregationContext, parent);
+                return new DoubleTermsAggregator(name, factories, (ValuesSource.Numeric) valuesSource, config.format(), estimatedBucketCount, order, requiredSize, shardSize, minDocCount, aggregationContext, parent);
             }
-            return new LongTermsAggregator(name, factories, (ValuesSource.Numeric) valuesSource, formatter, estimatedBucketCount, order, requiredSize, shardSize, minDocCount, aggregationContext, parent);
+            return new LongTermsAggregator(name, factories, (ValuesSource.Numeric) valuesSource, config.format(), estimatedBucketCount, order, requiredSize, shardSize, minDocCount, aggregationContext, parent);
         }
 
         throw new AggregationExecutionException("terms aggregation cannot be applied to field [" + config.fieldContext().field() +
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/TermsParser.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/TermsParser.java
index a7baae01fbe61..be379ada8a238 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/TermsParser.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/TermsParser.java
@@ -18,28 +18,15 @@
  */
 package org.elasticsearch.search.aggregations.bucket.terms;
 
-import org.elasticsearch.common.regex.Regex;
 import org.elasticsearch.common.xcontent.XContentParser;
-import org.elasticsearch.index.fielddata.IndexFieldData;
-import org.elasticsearch.index.fielddata.IndexNumericFieldData;
-import org.elasticsearch.index.mapper.FieldMapper;
-import org.elasticsearch.index.mapper.core.DateFieldMapper;
-import org.elasticsearch.index.mapper.ip.IpFieldMapper;
-import org.elasticsearch.script.SearchScript;
 import org.elasticsearch.search.SearchParseException;
 import org.elasticsearch.search.aggregations.Aggregator;
 import org.elasticsearch.search.aggregations.AggregatorFactory;
 import org.elasticsearch.search.aggregations.bucket.terms.support.IncludeExclude;
-import org.elasticsearch.search.aggregations.support.ValuesSource;
-import org.elasticsearch.search.aggregations.support.FieldContext;
-import org.elasticsearch.search.aggregations.support.ValuesSourceConfig;
-import org.elasticsearch.search.aggregations.support.format.ValueFormatter;
-import org.elasticsearch.search.aggregations.support.format.ValueParser;
+import org.elasticsearch.search.aggregations.support.ValuesSourceParser;
 import org.elasticsearch.search.internal.SearchContext;
 
 import java.io.IOException;
-import java.util.Map;
-import java.util.regex.Pattern;
 
 /**
  *
@@ -54,56 +41,37 @@ public String type() {
     @Override
     public AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException {
 
-        String field = null;
-        String script = null;
-        String scriptLang = null;
-        Map<String, Object> scriptParams = null;
-        Terms.ValueType valueType = null;
         int requiredSize = 10;
         int shardSize = -1;
         String orderKey = "_count";
         boolean orderAsc = false;
-        String format = null;
-        boolean assumeUnique = false;
-        String include = null;
-        int includeFlags = 0; // 0 means no flags
-        String exclude = null;
-        int excludeFlags = 0; // 0 means no flags
+
         String executionHint = null;
         long minDocCount = 1;
 
+        ValuesSourceParser vsParser = ValuesSourceParser.any(aggregationName, StringTerms.TYPE, context)
+                .requiresSortedValues(true)
+                .requiresUniqueValues(true)
+                .formattable(true)
+                .build();
+
+        IncludeExclude.Parser incExcParser = new IncludeExclude.Parser(aggregationName, StringTerms.TYPE, context);
 
         XContentParser.Token token;
         String currentFieldName = null;
         while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
             if (token == XContentParser.Token.FIELD_NAME) {
                 currentFieldName = parser.currentName();
+            } else if (vsParser.token(currentFieldName, token, parser)) {
+                continue;
+            } else if (incExcParser.token(currentFieldName, token, parser)) {
+                continue;
             } else if (token == XContentParser.Token.VALUE_STRING) {
-                if ("field".equals(currentFieldName)) {
-                    field = parser.text();
-                } else if ("script".equals(currentFieldName)) {
-                    script = parser.text();
-                } else if ("lang".equals(currentFieldName)) {
-                    scriptLang = parser.text();
-                } else if ("value_type".equals(currentFieldName) || "valueType".equals(currentFieldName)) {
-                    valueType = Terms.ValueType.resolveType(parser.text());
-                } else if ("format".equals(currentFieldName)) {
-                    format = parser.text();
-                } else if ("include".equals(currentFieldName)) {
-                    include = parser.text();
-                } else if ("exclude".equals(currentFieldName)) {
-                    exclude = parser.text();
-                } else if ("execution_hint".equals(currentFieldName) || "executionHint".equals(currentFieldName)) {
+                if ("execution_hint".equals(currentFieldName) || "executionHint".equals(currentFieldName)) {
                     executionHint = parser.text();
                 } else {
                     throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
                 }
-            } else if (token == XContentParser.Token.VALUE_BOOLEAN) {
-                if ("script_values_unique".equals(currentFieldName) || "scriptValuesUnique".equals(currentFieldName)) {
-                    assumeUnique = parser.booleanValue();
-                } else {
-                    throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
-                }
             } else if (token == XContentParser.Token.VALUE_NUMBER) {
                 if ("size".equals(currentFieldName)) {
                     requiredSize = parser.intValue();
@@ -115,9 +83,7 @@ public AggregatorFactory parse(String aggregationName, XContentParser parser, Se
                     throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
                 }
             } else if (token == XContentParser.Token.START_OBJECT) {
-                if ("params".equals(currentFieldName)) {
-                    scriptParams = parser.map();
-                } else if ("order".equals(currentFieldName)) {
+                if ("order".equals(currentFieldName)) {
                     while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
                         if (token == XContentParser.Token.FIELD_NAME) {
                             orderKey = parser.currentName();
@@ -134,38 +100,6 @@ public AggregatorFactory parse(String aggregationName, XContentParser parser, Se
                             throw new SearchParseException(context, "Unexpected token " + token + " for [order] in [" + aggregationName + "].");
                         }
                     }
-                } else if ("include".equals(currentFieldName)) {
-                    while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
-                        if (token == XContentParser.Token.FIELD_NAME) {
-                            currentFieldName = parser.currentName();
-                        } else if (token == XContentParser.Token.VALUE_STRING) {
-                            if ("pattern".equals(currentFieldName)) {
-                                include = parser.text();
-                            } else if ("flags".equals(currentFieldName)) {
-                                includeFlags = Regex.flagsFromString(parser.text());
-                            }
-                        } else if (token == XContentParser.Token.VALUE_NUMBER) {
-                            if ("flags".equals(currentFieldName)) {
-                                includeFlags = parser.intValue();
-                            }
-                        }
-                    }
-                } else if ("exclude".equals(currentFieldName)) {
-                    while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
-                        if (token == XContentParser.Token.FIELD_NAME) {
-                            currentFieldName = parser.currentName();
-                        } else if (token == XContentParser.Token.VALUE_STRING) {
-                            if ("pattern".equals(currentFieldName)) {
-                                exclude = parser.text();
-                            } else if ("flags".equals(currentFieldName)) {
-                                excludeFlags = Regex.flagsFromString(parser.text());
-                            }
-                        } else if (token == XContentParser.Token.VALUE_NUMBER) {
-                            if ("flags".equals(currentFieldName)) {
-                                excludeFlags = parser.intValue();
-                            }
-                        }
-                    }
                 } else {
                     throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
                 }
@@ -187,101 +121,9 @@ public AggregatorFactory parse(String aggregationName, XContentParser parser, Se
             shardSize = requiredSize;
         }
 
-        IncludeExclude includeExclude = null;
-        if (include != null || exclude != null) {
-            Pattern includePattern =  include != null ? Pattern.compile(include, includeFlags) : null;
-            Pattern excludePattern = exclude != null ? Pattern.compile(exclude, excludeFlags) : null;
-            includeExclude = new IncludeExclude(includePattern, excludePattern);
-        }
-
+        IncludeExclude includeExclude = incExcParser.includeExclude();
         InternalOrder order = resolveOrder(orderKey, orderAsc);
-        SearchScript searchScript = null;
-        if (script != null) {
-            searchScript = context.scriptService().search(context.lookup(), scriptLang, script, scriptParams);
-        }
-
-        if (field == null) {
-
-            Class<? extends ValuesSource> valueSourceType = script == null ?
-                    ValuesSource.class : // unknown, will inherit whatever is in the context
-                    valueType != null ? valueType.scriptValueType.getValuesSourceType() : // the user explicitly defined a value type
-                    ValuesSource.Bytes.class; // defaulting to bytes
-
-            ValuesSourceConfig<?> config = new ValuesSourceConfig(valueSourceType);
-            ValueFormatter valueFormatter = null;
-            ValueParser valueParser = null;
-            if (valueType != null) {
-                config.scriptValueType(valueType.scriptValueType);
-                if (valueType != Terms.ValueType.STRING && format != null) {
-                    valueFormatter = new ValueFormatter.Number.Pattern(format);
-                    valueParser = new ValueParser.Number.Pattern(format);
-                }
-            }
-            config.script(searchScript);
-            if (!assumeUnique) {
-                config.ensureUnique(true);
-            }
-            return new TermsAggregatorFactory(aggregationName, config, valueFormatter, valueParser, order, requiredSize, shardSize, minDocCount, includeExclude, executionHint);
-        }
-
-        FieldMapper<?> mapper = context.smartNameFieldMapper(field);
-        if (mapper == null) {
-            ValuesSourceConfig<?> config = new ValuesSourceConfig<>(ValuesSource.Bytes.class);
-            ValueFormatter valueFormatter = null;
-            ValueParser valueParser = null;
-            config.unmapped(true);
-            if (valueType != null) {
-                config.scriptValueType(valueType.scriptValueType);
-                if (valueType != Terms.ValueType.STRING && format != null) {
-                    valueFormatter = new ValueFormatter.Number.Pattern(format);
-                    valueParser = new ValueParser.Number.Pattern(format);
-                }
-            }
-            return new TermsAggregatorFactory(aggregationName, config, valueFormatter, valueParser, order, requiredSize, shardSize, minDocCount, includeExclude, executionHint);
-        }
-        IndexFieldData<?> indexFieldData = context.fieldData().getForField(mapper);
-
-        ValuesSourceConfig<?> config;
-        ValueFormatter valueFormatter = null;
-        ValueParser valueParser = null;
-
-        if (mapper instanceof DateFieldMapper) {
-            DateFieldMapper dateMapper = (DateFieldMapper) mapper;
-            config = new ValuesSourceConfig<>(ValuesSource.Numeric.class);
-            valueFormatter = format == null ?
-                    new ValueFormatter.DateTime(dateMapper.dateTimeFormatter()) :
-                    new ValueFormatter.DateTime(format);
-            valueParser = new ValueParser.DateMath(dateMapper.dateMathParser());
-
-        } else if (mapper instanceof IpFieldMapper) {
-            config = new ValuesSourceConfig<>(ValuesSource.Numeric.class);
-            valueFormatter = ValueFormatter.IPv4;
-            valueParser = ValueParser.IPv4;
-
-        } else if (indexFieldData instanceof IndexNumericFieldData) {
-            config = new ValuesSourceConfig<>(ValuesSource.Numeric.class);
-            if (format != null) {
-                valueFormatter = new ValueFormatter.Number.Pattern(format);
-                valueParser = new ValueParser.Number.Pattern(format);
-            }
-
-        } else {
-            config = new ValuesSourceConfig<>(ValuesSource.Bytes.class);
-            // TODO: it will make sense to set false instead here if the aggregator factory uses
-            // ordinals instead of hash tables
-            config.needsHashes(true);
-        }
-
-        config.script(searchScript);
-
-        config.fieldContext(new FieldContext(field, indexFieldData));
-
-        // We need values to be unique to be able to run terms aggs efficiently
-        if (!assumeUnique) {
-            config.ensureUnique(true);
-        }
-
-        return new TermsAggregatorFactory(aggregationName, config, valueFormatter, valueParser, order, requiredSize, shardSize, minDocCount, includeExclude, executionHint);
+        return new TermsAggregatorFactory(aggregationName, vsParser.config(), order, requiredSize, shardSize, minDocCount, includeExclude, executionHint);
     }
 
     static InternalOrder resolveOrder(String key, boolean asc) {
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/support/IncludeExclude.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/support/IncludeExclude.java
index 936efca63745a..27c012c6a5824 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/support/IncludeExclude.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/support/IncludeExclude.java
@@ -21,7 +21,13 @@
 import org.apache.lucene.util.BytesRef;
 import org.apache.lucene.util.CharsRef;
 import org.apache.lucene.util.UnicodeUtil;
+import org.elasticsearch.common.regex.Regex;
+import org.elasticsearch.common.xcontent.XContentParser;
+import org.elasticsearch.search.SearchParseException;
+import org.elasticsearch.search.aggregations.InternalAggregation;
+import org.elasticsearch.search.internal.SearchContext;
 
+import java.io.IOException;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
 
@@ -64,4 +70,86 @@ public boolean accept(BytesRef value) {
         }
         return !exclude.reset(scratch).matches();
     }
+
+    public static class Parser {
+
+        private final String aggName;
+        private final InternalAggregation.Type aggType;
+        private final SearchContext context;
+
+        String include = null;
+        int includeFlags = 0; // 0 means no flags
+        String exclude = null;
+        int excludeFlags = 0; // 0 means no flags
+
+        public Parser(String aggName, InternalAggregation.Type aggType, SearchContext context) {
+            this.aggName = aggName;
+            this.aggType = aggType;
+            this.context = context;
+        }
+
+        public boolean token(String currentFieldName, XContentParser.Token token, XContentParser parser) throws IOException {
+
+            if (token == XContentParser.Token.VALUE_STRING) {
+                if ("include".equals(currentFieldName)) {
+                    include = parser.text();
+                } else if ("exclude".equals(currentFieldName)) {
+                    exclude = parser.text();
+                } else {
+                    return false;
+                }
+                return true;
+            }
+
+            if (token == XContentParser.Token.START_OBJECT) {
+                if ("include".equals(currentFieldName)) {
+                    while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
+                        if (token == XContentParser.Token.FIELD_NAME) {
+                            currentFieldName = parser.currentName();
+                        } else if (token == XContentParser.Token.VALUE_STRING) {
+                            if ("pattern".equals(currentFieldName)) {
+                                include = parser.text();
+                            } else if ("flags".equals(currentFieldName)) {
+                                includeFlags = Regex.flagsFromString(parser.text());
+                            }
+                        } else if (token == XContentParser.Token.VALUE_NUMBER) {
+                            if ("flags".equals(currentFieldName)) {
+                                includeFlags = parser.intValue();
+                            }
+                        }
+                    }
+                } else if ("exclude".equals(currentFieldName)) {
+                    while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
+                        if (token == XContentParser.Token.FIELD_NAME) {
+                            currentFieldName = parser.currentName();
+                        } else if (token == XContentParser.Token.VALUE_STRING) {
+                            if ("pattern".equals(currentFieldName)) {
+                                exclude = parser.text();
+                            } else if ("flags".equals(currentFieldName)) {
+                                excludeFlags = Regex.flagsFromString(parser.text());
+                            }
+                        } else if (token == XContentParser.Token.VALUE_NUMBER) {
+                            if ("flags".equals(currentFieldName)) {
+                                excludeFlags = parser.intValue();
+                            }
+                        }
+                    }
+                } else {
+                    return false;
+                }
+                return true;
+            }
+
+            return false;
+        }
+
+        public IncludeExclude includeExclude() {
+            if (include == null && exclude == null) {
+                return null;
+            }
+            Pattern includePattern =  include != null ? Pattern.compile(include, includeFlags) : null;
+            Pattern excludePattern = exclude != null ? Pattern.compile(exclude, excludeFlags) : null;
+            return new IncludeExclude(includePattern, excludePattern);
+        }
+    }
 }
diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/NumericValuesSourceMetricsAggregatorParser.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/NumericValuesSourceMetricsAggregatorParser.java
new file mode 100644
index 0000000000000..44f76f3224a1b
--- /dev/null
+++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/NumericValuesSourceMetricsAggregatorParser.java
@@ -0,0 +1,74 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.elasticsearch.search.aggregations.metrics;
+
+import org.elasticsearch.common.xcontent.XContentParser;
+import org.elasticsearch.search.SearchParseException;
+import org.elasticsearch.search.aggregations.Aggregator;
+import org.elasticsearch.search.aggregations.AggregatorFactory;
+import org.elasticsearch.search.aggregations.InternalAggregation;
+import org.elasticsearch.search.aggregations.support.ValuesSource;
+import org.elasticsearch.search.aggregations.support.ValuesSourceConfig;
+import org.elasticsearch.search.aggregations.support.ValuesSourceParser;
+import org.elasticsearch.search.internal.SearchContext;
+
+import java.io.IOException;
+
+/**
+ *
+ */
+public abstract class NumericValuesSourceMetricsAggregatorParser<S extends MetricsAggregation> implements Aggregator.Parser {
+
+    protected final InternalAggregation.Type aggType;
+
+    protected NumericValuesSourceMetricsAggregatorParser(InternalAggregation.Type aggType) {
+        this.aggType = aggType;
+    }
+
+    @Override
+    public String type() {
+        return aggType.name();
+    }
+
+    protected boolean requiresSortedValues() {
+        return false;
+    }
+
+    @Override
+    public AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException {
+
+        ValuesSourceParser<ValuesSource.Numeric> vsParser = ValuesSourceParser.numeric(aggregationName, aggType, context)
+                .requiresSortedValues(requiresSortedValues())
+                .build();
+
+        XContentParser.Token token;
+        String currentFieldName = null;
+        while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
+            if (token == XContentParser.Token.FIELD_NAME) {
+                currentFieldName = parser.currentName();
+            } else if (!vsParser.token(currentFieldName, token, parser)) {
+                throw new SearchParseException(context, "Unexpected token " + token + " in [" + aggregationName + "].");
+            }
+        }
+
+        return createFactory(aggregationName, vsParser.config());
+    }
+
+    protected abstract AggregatorFactory createFactory(String aggregationName, ValuesSourceConfig<ValuesSource.Numeric> config);
+}
diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/ValuesSourceMetricsAggregatorParser.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/ValuesSourceMetricsAggregatorParser.java
deleted file mode 100644
index 0d1018d7e1819..0000000000000
--- a/src/main/java/org/elasticsearch/search/aggregations/metrics/ValuesSourceMetricsAggregatorParser.java
+++ /dev/null
@@ -1,111 +0,0 @@
-/*
- * Licensed to Elasticsearch under one or more contributor
- * license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright
- * ownership. Elasticsearch licenses this file to you under
- * the Apache License, Version 2.0 (the "License"); you may
- * not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *    http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.elasticsearch.search.aggregations.metrics;
-
-import org.elasticsearch.common.xcontent.XContentParser;
-import org.elasticsearch.index.fielddata.IndexFieldData;
-import org.elasticsearch.index.mapper.FieldMapper;
-import org.elasticsearch.search.SearchParseException;
-import org.elasticsearch.search.aggregations.Aggregator;
-import org.elasticsearch.search.aggregations.AggregatorFactory;
-import org.elasticsearch.search.aggregations.support.ValuesSource;
-import org.elasticsearch.search.aggregations.support.FieldContext;
-import org.elasticsearch.search.aggregations.support.ValuesSourceConfig;
-import org.elasticsearch.search.internal.SearchContext;
-
-import java.io.IOException;
-import java.util.Map;
-
-/**
- *
- */
-public abstract class ValuesSourceMetricsAggregatorParser<S extends MetricsAggregation> implements Aggregator.Parser {
-
-    protected boolean requiresSortedValues() {
-        return false;
-    }
-
-    @Override
-    public AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException {
-
-        ValuesSourceConfig<ValuesSource.Numeric> config = new ValuesSourceConfig<>(ValuesSource.Numeric.class);
-
-        String field = null;
-        String script = null;
-        String scriptLang = null;
-        Map<String, Object> scriptParams = null;
-        boolean assumeSorted = false;
-
-        XContentParser.Token token;
-        String currentFieldName = null;
-        while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
-            if (token == XContentParser.Token.FIELD_NAME) {
-                currentFieldName = parser.currentName();
-            } else if (token == XContentParser.Token.VALUE_STRING) {
-                if ("field".equals(currentFieldName)) {
-                    field = parser.text();
-                } else if ("script".equals(currentFieldName)) {
-                    script = parser.text();
-                } else if ("lang".equals(currentFieldName)) {
-                    scriptLang = parser.text();
-                } else {
-                    throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
-                }
-            } else if (token == XContentParser.Token.START_OBJECT) {
-                if ("params".equals(currentFieldName)) {
-                    scriptParams = parser.map();
-                } else {
-                    throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
-                }
-            } else if (token == XContentParser.Token.VALUE_BOOLEAN) {
-                if ("script_values_sorted".equals(currentFieldName) || "scriptValuesSorted".equals(currentFieldName)) {
-                    assumeSorted = parser.booleanValue();
-                } else {
-                    throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
-                }
-            } else {
-                throw new SearchParseException(context, "Unexpected token " + token + " in [" + aggregationName + "].");
-            }
-        }
-
-        if (script != null) {
-            config.script(context.scriptService().search(context.lookup(), scriptLang, script, scriptParams));
-        }
-
-        if (!assumeSorted && requiresSortedValues()) {
-            config.ensureSorted(true);
-        }
-
-        if (field == null) {
-            return createFactory(aggregationName, config);
-        }
-
-        FieldMapper<?> mapper = context.smartNameFieldMapper(field);
-        if (mapper == null) {
-            config.unmapped(true);
-            return createFactory(aggregationName, config);
-        }
-
-        IndexFieldData<?> indexFieldData = context.fieldData().getForField(mapper);
-        config.fieldContext(new FieldContext(field, indexFieldData));
-        return createFactory(aggregationName, config);
-    }
-
-    protected abstract AggregatorFactory createFactory(String aggregationName, ValuesSourceConfig<ValuesSource.Numeric> config);
-}
diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/avg/AvgParser.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/avg/AvgParser.java
index fcaa441c77161..1c2b2be53451d 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/metrics/avg/AvgParser.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/avg/AvgParser.java
@@ -19,18 +19,17 @@
 package org.elasticsearch.search.aggregations.metrics.avg;
 
 import org.elasticsearch.search.aggregations.AggregatorFactory;
-import org.elasticsearch.search.aggregations.metrics.ValuesSourceMetricsAggregatorParser;
+import org.elasticsearch.search.aggregations.metrics.NumericValuesSourceMetricsAggregatorParser;
 import org.elasticsearch.search.aggregations.support.ValuesSource;
 import org.elasticsearch.search.aggregations.support.ValuesSourceConfig;
 
 /**
  *
  */
-public class AvgParser extends ValuesSourceMetricsAggregatorParser<InternalAvg> {
+public class AvgParser extends NumericValuesSourceMetricsAggregatorParser<InternalAvg> {
 
-    @Override
-    public String type() {
-        return InternalAvg.TYPE.name();
+    public AvgParser() {
+        super(InternalAvg.TYPE);
     }
 
     @Override
diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/cardinality/CardinalityParser.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/cardinality/CardinalityParser.java
index efd166a86c068..8b176b5ee2548 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/metrics/cardinality/CardinalityParser.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/cardinality/CardinalityParser.java
@@ -21,20 +21,15 @@
 
 import org.elasticsearch.common.ParseField;
 import org.elasticsearch.common.xcontent.XContentParser;
-import org.elasticsearch.index.fielddata.IndexFieldData;
-import org.elasticsearch.index.mapper.FieldMapper;
 import org.elasticsearch.index.mapper.core.Murmur3FieldMapper;
-import org.elasticsearch.index.mapper.core.NumberFieldMapper;
 import org.elasticsearch.search.SearchParseException;
 import org.elasticsearch.search.aggregations.Aggregator;
 import org.elasticsearch.search.aggregations.AggregatorFactory;
-import org.elasticsearch.search.aggregations.support.ValuesSource;
-import org.elasticsearch.search.aggregations.support.FieldContext;
 import org.elasticsearch.search.aggregations.support.ValuesSourceConfig;
+import org.elasticsearch.search.aggregations.support.ValuesSourceParser;
 import org.elasticsearch.search.internal.SearchContext;
 
 import java.io.IOException;
-import java.util.Map;
 
 
 public class CardinalityParser implements Aggregator.Parser {
@@ -48,34 +43,19 @@ public String type() {
 
     @Override
     public AggregatorFactory parse(String name, XContentParser parser, SearchContext context) throws IOException {
+
+        ValuesSourceParser vsParser = ValuesSourceParser.any(name, InternalCardinality.TYPE, context).build();
+
         long precisionThreshold = -1;
         Boolean rehash = null;
-        String field = null;
-        String script = null;
-        String scriptLang = null;
-        Map<String, Object> scriptParams = null;
 
         XContentParser.Token token;
         String currentFieldName = null;
         while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
             if (token == XContentParser.Token.FIELD_NAME) {
                 currentFieldName = parser.currentName();
-            } else if (token == XContentParser.Token.VALUE_STRING) {
-                if ("field".equals(currentFieldName)) {
-                    field = parser.text();
-                } else if ("script".equals(currentFieldName)) {
-                    script = parser.text();
-                } else if ("lang".equals(currentFieldName)) {
-                    scriptLang = parser.text();
-                } else {
-                    throw new SearchParseException(context, "Unknown key for a " + token + " in [" + name + "]: [" + currentFieldName + "].");
-                }
-            } else if (token == XContentParser.Token.START_OBJECT) {
-                if ("params".equals(currentFieldName)) {
-                    scriptParams = parser.map();
-                } else {
-                    throw new SearchParseException(context, "Unknown key for a " + token + " in [" + name + "]: [" + currentFieldName + "].");
-                }
+            } else if (vsParser.token(currentFieldName, token, parser)) {
+                continue;
             } else if (token == XContentParser.Token.VALUE_BOOLEAN) {
                 if ("rehash".equals(currentFieldName)) {
                     rehash = parser.booleanValue();
@@ -93,41 +73,16 @@ public AggregatorFactory parse(String name, XContentParser parser, SearchContext
             }
         }
 
-        ValuesSourceConfig<?> config = null;
-
-        if (script != null) {
-            config = new ValuesSourceConfig<>(ValuesSource.Bytes.class);
-            config.script(context.scriptService().search(context.lookup(), scriptLang, script, scriptParams));
-        }
-
-        if (field != null) {
-            FieldMapper<?> mapper = context.smartNameFieldMapper(field);
-            if (config == null) {
-                if (mapper instanceof NumberFieldMapper) {
-                    config = new ValuesSourceConfig<>(ValuesSource.Numeric.class);
-                } else {
-                    config = new ValuesSourceConfig<>(ValuesSource.Bytes.class);
-                    if (mapper == null) {
-                        config.unmapped(true);
-                    }
-                }
-                if (rehash == null && mapper instanceof Murmur3FieldMapper) {
-                    rehash = false;
-                }
-            }
-            if (mapper != null) {
-                IndexFieldData<?> indexFieldData = context.fieldData().getForField(mapper);
-                config.fieldContext(new FieldContext(field, indexFieldData));
-            }
-        } else if (config == null) {
-            config = new ValuesSourceConfig<>(ValuesSource.Bytes.class);
-        }
+        ValuesSourceConfig<?> config = vsParser.config();
 
-        if (rehash == null) {
+        if (rehash == null && config.fieldContext() != null && config.fieldContext().mapper() instanceof Murmur3FieldMapper) {
+            rehash = false;
+        } else if (rehash == null) {
             rehash = true;
         }
 
         return new CardinalityAggregatorFactory(name, config, precisionThreshold, rehash);
+
     }
 
 }
diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/max/MaxParser.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/max/MaxParser.java
index 4c3c8d3587c9e..a5cdac5f638d5 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/metrics/max/MaxParser.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/max/MaxParser.java
@@ -19,18 +19,17 @@
 package org.elasticsearch.search.aggregations.metrics.max;
 
 import org.elasticsearch.search.aggregations.AggregatorFactory;
-import org.elasticsearch.search.aggregations.metrics.ValuesSourceMetricsAggregatorParser;
+import org.elasticsearch.search.aggregations.metrics.NumericValuesSourceMetricsAggregatorParser;
 import org.elasticsearch.search.aggregations.support.ValuesSource;
 import org.elasticsearch.search.aggregations.support.ValuesSourceConfig;
 
 /**
  *
  */
-public class MaxParser extends ValuesSourceMetricsAggregatorParser<InternalMax> {
+public class MaxParser extends NumericValuesSourceMetricsAggregatorParser<InternalMax> {
 
-    @Override
-    public String type() {
-        return InternalMax.TYPE.name();
+    public MaxParser() {
+        super(InternalMax.TYPE);
     }
 
     @Override
diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/min/MinParser.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/min/MinParser.java
index de55a0297997a..8f25127f73622 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/metrics/min/MinParser.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/min/MinParser.java
@@ -19,18 +19,17 @@
 package org.elasticsearch.search.aggregations.metrics.min;
 
 import org.elasticsearch.search.aggregations.AggregatorFactory;
-import org.elasticsearch.search.aggregations.metrics.ValuesSourceMetricsAggregatorParser;
+import org.elasticsearch.search.aggregations.metrics.NumericValuesSourceMetricsAggregatorParser;
 import org.elasticsearch.search.aggregations.support.ValuesSource;
 import org.elasticsearch.search.aggregations.support.ValuesSourceConfig;
 
 /**
  *
  */
-public class MinParser extends ValuesSourceMetricsAggregatorParser<InternalMin> {
+public class MinParser extends NumericValuesSourceMetricsAggregatorParser<InternalMin> {
 
-    @Override
-    public String type() {
-        return InternalMin.TYPE.name();
+    public MinParser() {
+        super(InternalMin.TYPE);
     }
 
     @Override
diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/percentiles/PercentilesParser.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/percentiles/PercentilesParser.java
index 7529dc9a85933..1e3c21627c487 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/metrics/percentiles/PercentilesParser.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/percentiles/PercentilesParser.java
@@ -20,15 +20,12 @@
 
 import com.carrotsearch.hppc.DoubleArrayList;
 import org.elasticsearch.common.xcontent.XContentParser;
-import org.elasticsearch.index.fielddata.IndexFieldData;
-import org.elasticsearch.index.mapper.FieldMapper;
 import org.elasticsearch.search.SearchParseException;
 import org.elasticsearch.search.aggregations.Aggregator;
 import org.elasticsearch.search.aggregations.AggregatorFactory;
 import org.elasticsearch.search.aggregations.metrics.percentiles.tdigest.TDigest;
 import org.elasticsearch.search.aggregations.support.ValuesSource;
-import org.elasticsearch.search.aggregations.support.FieldContext;
-import org.elasticsearch.search.aggregations.support.ValuesSourceConfig;
+import org.elasticsearch.search.aggregations.support.ValuesSourceParser;
 import org.elasticsearch.search.internal.SearchContext;
 
 import java.io.IOException;
@@ -55,14 +52,11 @@ public String type() {
     @Override
     public AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException {
 
-        ValuesSourceConfig<ValuesSource.Numeric> config = new ValuesSourceConfig<>(ValuesSource.Numeric.class);
+        ValuesSourceParser<ValuesSource.Numeric> vsParser = ValuesSourceParser.numeric(aggregationName, InternalPercentiles.TYPE, context)
+                .requiresSortedValues(true)
+                .build();
 
-        String field = null;
-        String script = null;
-        String scriptLang = null;
         double[] percents = DEFAULT_PERCENTS;
-        Map<String, Object> scriptParams = null;
-        boolean assumeSorted = false;
         boolean keyed = true;
         Map<String, Object> settings = null;
 
@@ -71,19 +65,8 @@ public AggregatorFactory parse(String aggregationName, XContentParser parser, Se
         while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
             if (token == XContentParser.Token.FIELD_NAME) {
                 currentFieldName = parser.currentName();
-            } else if (token == XContentParser.Token.VALUE_STRING) {
-                if ("field".equals(currentFieldName)) {
-                    field = parser.text();
-                } else if ("script".equals(currentFieldName)) {
-                    script = parser.text();
-                } else if ("lang".equals(currentFieldName)) {
-                    scriptLang = parser.text();
-                } else {
-                    if (settings == null) {
-                        settings = new HashMap<>();
-                    }
-                    settings.put(currentFieldName, parser.text());
-                }
+            } else if (vsParser.token(currentFieldName, token, parser)) {
+                continue;
             } else if (token == XContentParser.Token.START_ARRAY) {
                 if ("percents".equals(currentFieldName)) {
                     DoubleArrayList values = new DoubleArrayList(10);
@@ -101,56 +84,22 @@ public AggregatorFactory parse(String aggregationName, XContentParser parser, Se
                 } else {
                     throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
                 }
-            } else if (token == XContentParser.Token.START_OBJECT) {
-                if ("params".equals(currentFieldName)) {
-                    scriptParams = parser.map();
-                } else {
-                    throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
-                }
-            } else if (token == XContentParser.Token.VALUE_BOOLEAN) {
-                if ("script_values_sorted".equals(currentFieldName) || "scriptValuesSorted".equals(currentFieldName)) {
-                    assumeSorted = parser.booleanValue();
-                } if ("keyed".equals(currentFieldName)) {
+            } else if (token.isValue()) {
+                if (token == XContentParser.Token.VALUE_BOOLEAN && "keyed".equals(currentFieldName)) {
                     keyed = parser.booleanValue();
                 } else {
                     if (settings == null) {
                         settings = new HashMap<>();
                     }
-                    settings.put(currentFieldName, parser.booleanValue());
-                }
-            } else if (token == XContentParser.Token.VALUE_NUMBER) {
-                if (settings == null) {
-                    settings = new HashMap<>();
+                    settings.put(currentFieldName, parser.objectText());
                 }
-                settings.put(currentFieldName, parser.numberValue());
             } else {
                 throw new SearchParseException(context, "Unexpected token " + token + " in [" + aggregationName + "].");
             }
         }
 
         PercentilesEstimator.Factory estimatorFactory = EstimatorType.TDIGEST.estimatorFactory(settings);
-
-        if (script != null) {
-            config.script(context.scriptService().search(context.lookup(), scriptLang, script, scriptParams));
-        }
-
-        if (!assumeSorted) {
-            config.ensureSorted(true);
-        }
-
-        if (field == null) {
-            return new PercentilesAggregator.Factory(aggregationName, config, percents, estimatorFactory, keyed);
-        }
-
-        FieldMapper<?> mapper = context.smartNameFieldMapper(field);
-        if (mapper == null) {
-            config.unmapped(true);
-            return new PercentilesAggregator.Factory(aggregationName, config, percents, estimatorFactory, keyed);
-        }
-
-        IndexFieldData<?> indexFieldData = context.fieldData().getForField(mapper);
-        config.fieldContext(new FieldContext(field, indexFieldData));
-        return new PercentilesAggregator.Factory(aggregationName, config, percents, estimatorFactory, keyed);
+        return new PercentilesAggregator.Factory(aggregationName, vsParser.config(), percents, estimatorFactory, keyed);
     }
 
     /**
diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/StatsParser.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/StatsParser.java
index 42722928f1028..5ec9b2a59a7af 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/StatsParser.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/StatsParser.java
@@ -19,18 +19,17 @@
 package org.elasticsearch.search.aggregations.metrics.stats;
 
 import org.elasticsearch.search.aggregations.AggregatorFactory;
-import org.elasticsearch.search.aggregations.metrics.ValuesSourceMetricsAggregatorParser;
+import org.elasticsearch.search.aggregations.metrics.NumericValuesSourceMetricsAggregatorParser;
 import org.elasticsearch.search.aggregations.support.ValuesSource;
 import org.elasticsearch.search.aggregations.support.ValuesSourceConfig;
 
 /**
  *
  */
-public class StatsParser extends ValuesSourceMetricsAggregatorParser<InternalStats> {
+public class StatsParser extends NumericValuesSourceMetricsAggregatorParser<InternalStats> {
 
-    @Override
-    public String type() {
-        return InternalStats.TYPE.name();
+    public StatsParser() {
+        super(InternalStats.TYPE);
     }
 
     @Override
diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/extended/ExtendedStatsParser.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/extended/ExtendedStatsParser.java
index 3b66fa5bc2140..ee70c3d5b861b 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/extended/ExtendedStatsParser.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/extended/ExtendedStatsParser.java
@@ -19,18 +19,17 @@
 package org.elasticsearch.search.aggregations.metrics.stats.extended;
 
 import org.elasticsearch.search.aggregations.AggregatorFactory;
-import org.elasticsearch.search.aggregations.metrics.ValuesSourceMetricsAggregatorParser;
+import org.elasticsearch.search.aggregations.metrics.NumericValuesSourceMetricsAggregatorParser;
 import org.elasticsearch.search.aggregations.support.ValuesSource;
 import org.elasticsearch.search.aggregations.support.ValuesSourceConfig;
 
 /**
  *
  */
-public class ExtendedStatsParser extends ValuesSourceMetricsAggregatorParser<InternalExtendedStats> {
+public class ExtendedStatsParser extends NumericValuesSourceMetricsAggregatorParser<InternalExtendedStats> {
 
-    @Override
-    public String type() {
-        return InternalExtendedStats.TYPE.name();
+    public ExtendedStatsParser() {
+        super(InternalExtendedStats.TYPE);
     }
 
     @Override
diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/sum/SumParser.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/sum/SumParser.java
index c67d3ce9144c1..b43285a7bb516 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/metrics/sum/SumParser.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/sum/SumParser.java
@@ -19,18 +19,17 @@
 package org.elasticsearch.search.aggregations.metrics.sum;
 
 import org.elasticsearch.search.aggregations.AggregatorFactory;
-import org.elasticsearch.search.aggregations.metrics.ValuesSourceMetricsAggregatorParser;
+import org.elasticsearch.search.aggregations.metrics.NumericValuesSourceMetricsAggregatorParser;
 import org.elasticsearch.search.aggregations.support.ValuesSource;
 import org.elasticsearch.search.aggregations.support.ValuesSourceConfig;
 
 /**
  *
  */
-public class SumParser extends ValuesSourceMetricsAggregatorParser<InternalSum> {
+public class SumParser extends NumericValuesSourceMetricsAggregatorParser<InternalSum> {
 
-    @Override
-    public String type() {
-        return InternalSum.TYPE.name();
+    public SumParser() {
+        super(InternalSum.TYPE);
     }
 
     @Override
diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/valuecount/ValueCountAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/valuecount/ValueCountAggregator.java
index bb5a8cd0bcb76..a9d37fd293d86 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/metrics/valuecount/ValueCountAggregator.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/valuecount/ValueCountAggregator.java
@@ -40,13 +40,13 @@
  */
 public class ValueCountAggregator extends MetricsAggregator.SingleValue {
 
-    private final ValuesSource.Bytes valuesSource;
+    private final ValuesSource valuesSource;
     private BytesValues values;
 
     // a count per bucket
     LongArray counts;
 
-    public ValueCountAggregator(String name, long expectedBucketsCount, ValuesSource.Bytes valuesSource, AggregationContext aggregationContext, Aggregator parent) {
+    public ValueCountAggregator(String name, long expectedBucketsCount, ValuesSource valuesSource, AggregationContext aggregationContext, Aggregator parent) {
         super(name, 0, aggregationContext, parent);
         this.valuesSource = valuesSource;
         if (valuesSource != null) {
@@ -96,10 +96,10 @@ public void doRelease() {
         Releasables.release(counts);
     }
 
-    public static class Factory extends ValuesSourceAggregatorFactory.LeafOnly<ValuesSource.Bytes> {
+    public static class Factory<VS extends ValuesSource> extends ValuesSourceAggregatorFactory.LeafOnly<VS> {
 
-        public Factory(String name, ValuesSourceConfig<ValuesSource.Bytes> valuesSourceBuilder) {
-            super(name, InternalValueCount.TYPE.name(), valuesSourceBuilder);
+        public Factory(String name, ValuesSourceConfig<VS> config) {
+            super(name, InternalValueCount.TYPE.name(), config);
         }
 
         @Override
@@ -108,7 +108,7 @@ protected Aggregator createUnmapped(AggregationContext aggregationContext, Aggre
         }
 
         @Override
-        protected Aggregator create(ValuesSource.Bytes valuesSource, long expectedBucketsCount, AggregationContext aggregationContext, Aggregator parent) {
+        protected Aggregator create(ValuesSource valuesSource, long expectedBucketsCount, AggregationContext aggregationContext, Aggregator parent) {
             return new ValueCountAggregator(name, expectedBucketsCount, valuesSource, aggregationContext, parent);
         }
 
diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/valuecount/ValueCountParser.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/valuecount/ValueCountParser.java
index 60098c18fd219..e3ddfc51328b4 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/metrics/valuecount/ValueCountParser.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/valuecount/ValueCountParser.java
@@ -19,18 +19,13 @@
 package org.elasticsearch.search.aggregations.metrics.valuecount;
 
 import org.elasticsearch.common.xcontent.XContentParser;
-import org.elasticsearch.index.fielddata.IndexFieldData;
-import org.elasticsearch.index.mapper.FieldMapper;
 import org.elasticsearch.search.SearchParseException;
 import org.elasticsearch.search.aggregations.Aggregator;
 import org.elasticsearch.search.aggregations.AggregatorFactory;
-import org.elasticsearch.search.aggregations.support.ValuesSource;
-import org.elasticsearch.search.aggregations.support.FieldContext;
-import org.elasticsearch.search.aggregations.support.ValuesSourceConfig;
+import org.elasticsearch.search.aggregations.support.ValuesSourceParser;
 import org.elasticsearch.search.internal.SearchContext;
 
 import java.io.IOException;
-import java.util.Map;
 
 /**
  *
@@ -45,64 +40,20 @@ public String type() {
     @Override
     public AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException {
 
-        ValuesSourceConfig<ValuesSource.Bytes> config = new ValuesSourceConfig<>(ValuesSource.Bytes.class);
-
-        String field = null;
-        String script = null;
-        String scriptLang = null;
-        Map<String, Object> scriptParams = null;
-        boolean assumeUnique = false;
+        ValuesSourceParser vsParser = ValuesSourceParser.any(aggregationName, InternalValueCount.TYPE, context)
+                .requiresUniqueValues(true)
+                .build();
 
         XContentParser.Token token;
         String currentFieldName = null;
         while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
             if (token == XContentParser.Token.FIELD_NAME) {
                 currentFieldName = parser.currentName();
-            } else if (token == XContentParser.Token.VALUE_STRING) {
-                if ("field".equals(currentFieldName)) {
-                    field = parser.text();
-                } else if ("script".equals(currentFieldName)) {
-                    script = parser.text();
-                } else if ("lang".equals(currentFieldName)) {
-                    scriptLang = parser.text();
-                } else {
-                    throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
-                }
-            } else if (token == XContentParser.Token.VALUE_BOOLEAN) {
-                if ("script_values_unique".equals(currentFieldName) || "scriptValuesUnique".equals(currentFieldName)) {
-                    assumeUnique = parser.booleanValue();
-                } else {
-                    throw new SearchParseException(context, "Unknown key for a " + token + " in [" + aggregationName + "]: [" + currentFieldName + "].");
-                }
-            } else if (token == XContentParser.Token.START_OBJECT) {
-                if ("params".equals(currentFieldName)) {
-                    scriptParams = parser.map();
-                }
-            } else {
+            } else if (!vsParser.token(currentFieldName, token, parser)) {
                 throw new SearchParseException(context, "Unexpected token " + token + " in [" + aggregationName + "].");
             }
         }
 
-        if (script != null) {
-            config.script(context.scriptService().search(context.lookup(), scriptLang, script, scriptParams));
-        }
-
-        if (!assumeUnique) {
-            config.ensureUnique(true);
-        }
-
-        if (field == null) {
-            return new ValueCountAggregator.Factory(aggregationName, config);
-        }
-
-        FieldMapper<?> mapper = context.smartNameFieldMapper(field);
-        if (mapper == null) {
-            config.unmapped(true);
-            return new ValueCountAggregator.Factory(aggregationName, config);
-        }
-
-        IndexFieldData<?> indexFieldData = context.fieldData().getForField(mapper);
-        config.fieldContext(new FieldContext(field, indexFieldData));
-        return new ValueCountAggregator.Factory(aggregationName, config);
+        return new ValueCountAggregator.Factory(aggregationName, vsParser.config());
     }
 }
diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/FieldContext.java b/src/main/java/org/elasticsearch/search/aggregations/support/FieldContext.java
index 9a84f4a69d638..169d87c40c60b 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/support/FieldContext.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/support/FieldContext.java
@@ -19,6 +19,7 @@
 package org.elasticsearch.search.aggregations.support;
 
 import org.elasticsearch.index.fielddata.IndexFieldData;
+import org.elasticsearch.index.mapper.FieldMapper;
 
 /**
  * Used by all field data based aggregators. This determine the context of the field data the aggregators are operating
@@ -28,6 +29,7 @@ public class FieldContext {
 
     private final String field;
     private final IndexFieldData<?> indexFieldData;
+    private final FieldMapper<?> mapper;
 
     /**
      * Constructs a field data context for the given field and its index field data
@@ -35,9 +37,10 @@ public class FieldContext {
      * @param field             The name of the field
      * @param indexFieldData    The index field data of the field
      */
-    public FieldContext(String field, IndexFieldData<?> indexFieldData) {
+    public FieldContext(String field, IndexFieldData<?> indexFieldData, FieldMapper mapper) {
         this.field = field;
         this.indexFieldData = indexFieldData;
+        this.mapper = mapper;
     }
 
     public String field() {
@@ -51,4 +54,8 @@ public IndexFieldData<?> indexFieldData() {
         return indexFieldData;
     }
 
+    public FieldMapper mapper() {
+        return mapper;
+    }
+
 }
diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/GeoPointParser.java b/src/main/java/org/elasticsearch/search/aggregations/support/GeoPointParser.java
new file mode 100644
index 0000000000000..35c381ec2a197
--- /dev/null
+++ b/src/main/java/org/elasticsearch/search/aggregations/support/GeoPointParser.java
@@ -0,0 +1,103 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.search.aggregations.support;
+
+import org.elasticsearch.common.ParseField;
+import org.elasticsearch.common.geo.GeoPoint;
+import org.elasticsearch.common.xcontent.XContentParser;
+import org.elasticsearch.search.SearchParseException;
+import org.elasticsearch.search.aggregations.InternalAggregation;
+import org.elasticsearch.search.internal.SearchContext;
+
+import java.io.IOException;
+
+/**
+ *
+ */
+public class GeoPointParser {
+
+    private final String aggName;
+    private final InternalAggregation.Type aggType;
+    private final SearchContext context;
+    private final ParseField field;
+
+    GeoPoint point;
+
+    public GeoPointParser(String aggName, InternalAggregation.Type aggType, SearchContext context, ParseField field) {
+        this.aggName = aggName;
+        this.aggType = aggType;
+        this.context = context;
+        this.field = field;
+    }
+
+    public boolean token(String currentFieldName, XContentParser.Token token, XContentParser parser) throws IOException {
+        if (!field.match(currentFieldName)) {
+            return false;
+        }
+        if (token == XContentParser.Token.VALUE_STRING) {
+            point = new GeoPoint();
+            point.resetFromString(parser.text());
+            return true;
+        }
+        if (token == XContentParser.Token.START_ARRAY) {
+            double lat = Double.NaN;
+            double lon = Double.NaN;
+            while ((token = parser.nextToken()) != XContentParser.Token.END_ARRAY) {
+                if (Double.isNaN(lon)) {
+                    lon = parser.doubleValue();
+                } else if (Double.isNaN(lat)) {
+                    lat = parser.doubleValue();
+                } else {
+                    throw new SearchParseException(context, "malformed [" + currentFieldName + "] geo point array in [" +
+                            aggName + "] " + aggType + " aggregation. a geo point array must be of the form [lon, lat]");
+                }
+            }
+            point = new GeoPoint(lat, lon);
+            return true;
+        }
+        if (token == XContentParser.Token.START_OBJECT) {
+            double lat = Double.NaN;
+            double lon = Double.NaN;
+            while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
+                if (token == XContentParser.Token.FIELD_NAME) {
+                    currentFieldName = parser.currentName();
+                } else if (token == XContentParser.Token.VALUE_NUMBER) {
+                    if ("lat".equals(currentFieldName)) {
+                        lat = parser.doubleValue();
+                    } else if ("lon".equals(currentFieldName)) {
+                        lon = parser.doubleValue();
+                    }
+                }
+            }
+            if (Double.isNaN(lat) || Double.isNaN(lon)) {
+                throw new SearchParseException(context, "malformed [" + currentFieldName + "] geo point object. either [lat] or [lon] (or both) are " +
+                        "missing in [" + aggName + "] " + aggType + " aggregation");
+            }
+            point = new GeoPoint(lat, lon);
+            return true;
+        }
+        return false;
+    }
+
+    public GeoPoint geoPoint() {
+        return point;
+    }
+
+}
diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/ScriptValueType.java b/src/main/java/org/elasticsearch/search/aggregations/support/ScriptValueType.java
deleted file mode 100644
index 9e6082eaeb809..0000000000000
--- a/src/main/java/org/elasticsearch/search/aggregations/support/ScriptValueType.java
+++ /dev/null
@@ -1,48 +0,0 @@
-/*
- * Licensed to Elasticsearch under one or more contributor
- * license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright
- * ownership. Elasticsearch licenses this file to you under
- * the Apache License, Version 2.0 (the "License"); you may
- * not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *    http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.elasticsearch.search.aggregations.support;
-
-/**
- *
- */
-public enum ScriptValueType {
-
-    STRING(ValuesSource.Bytes.class),
-    LONG(ValuesSource.Numeric.class),
-    DOUBLE(ValuesSource.Numeric.class);
-
-    final Class<? extends ValuesSource> valuesSourceType;
-
-    private ScriptValueType(Class<? extends ValuesSource> valuesSourceType) {
-        this.valuesSourceType = valuesSourceType;
-    }
-
-    public Class<? extends ValuesSource> getValuesSourceType() {
-        return valuesSourceType;
-    }
-
-    public boolean isNumeric() {
-        return this != STRING;
-    }
-
-    public boolean isFloatingPoint() {
-        return this == DOUBLE;
-    }
-}
diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/ValueType.java b/src/main/java/org/elasticsearch/search/aggregations/support/ValueType.java
new file mode 100644
index 0000000000000..9d2ac2f46686f
--- /dev/null
+++ b/src/main/java/org/elasticsearch/search/aggregations/support/ValueType.java
@@ -0,0 +1,151 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.search.aggregations.support;
+
+import org.elasticsearch.index.fielddata.IndexFieldData;
+import org.elasticsearch.index.fielddata.IndexGeoPointFieldData;
+import org.elasticsearch.index.fielddata.IndexNumericFieldData;
+import org.elasticsearch.search.aggregations.support.format.ValueFormat;
+
+/**
+ *
+ */
+public enum ValueType {
+
+    @Deprecated ANY("any", ValuesSource.class, IndexFieldData.class, null),
+    STRING("string", ValuesSource.Bytes.class, IndexFieldData.class, null),
+    LONG("byte|short|integer|long", ValuesSource.Numeric.class, IndexNumericFieldData.class, ValueFormat.RAW) {
+        @Override
+        public boolean isNumeric() {
+            return true;
+        }
+    },
+    DOUBLE("float|double", ValuesSource.Numeric.class, IndexNumericFieldData.class, ValueFormat.RAW) {
+        @Override
+        public boolean isNumeric() {
+            return true;
+        }
+
+        @Override
+        public boolean isFloatingPoint() {
+            return true;
+        }
+    },
+    NUMBER("number", ValuesSource.Numeric.class, IndexNumericFieldData.class, ValueFormat.RAW) {
+        @Override
+        public boolean isNumeric() {
+            return true;
+        }
+    },
+    DATE("date", ValuesSource.Numeric.class, IndexNumericFieldData.class, ValueFormat.DateTime.DEFAULT) {
+        @Override
+        public boolean isNumeric() {
+            return true;
+        }
+    },
+    IP("ip", ValuesSource.Numeric.class, IndexNumericFieldData.class, ValueFormat.IPv4) {
+        @Override
+        public boolean isNumeric() {
+            return true;
+        }
+    },
+    NUMERIC("numeric", ValuesSource.Numeric.class, IndexNumericFieldData.class, ValueFormat.RAW) {
+        @Override
+        public boolean isNumeric() {
+            return true;
+        }
+    },
+    GEOPOINT("geo_point", ValuesSource.GeoPoint.class, IndexGeoPointFieldData.class, null) {
+        @Override
+        public boolean isGeoPoint() {
+            return true;
+        }
+    };
+
+    final String description;
+    final Class<? extends ValuesSource> valuesSourceType;
+    final Class<? extends IndexFieldData> fieldDataType;
+    final ValueFormat defaultFormat;
+
+    private ValueType(String description, Class<? extends ValuesSource> valuesSourceType, Class<? extends IndexFieldData> fieldDataType, ValueFormat defaultFormat) {
+        this.description = description;
+        this.valuesSourceType = valuesSourceType;
+        this.fieldDataType = fieldDataType;
+        this.defaultFormat = defaultFormat;
+    }
+
+    public String description() {
+        return description;
+    }
+
+    public Class<? extends ValuesSource> getValuesSourceType() {
+        return valuesSourceType;
+    }
+
+    public boolean compatibleWith(IndexFieldData fieldData) {
+        return fieldDataType.isInstance(fieldData);
+    }
+
+    public boolean isA(ValueType valueType) {
+        return valueType.valuesSourceType.isAssignableFrom(valuesSourceType) &&
+                valueType.fieldDataType.isAssignableFrom(fieldDataType);
+    }
+
+    public boolean isNotA(ValueType valueType) {
+        return !isA(valueType);
+    }
+
+    public ValueFormat defaultFormat() {
+        return defaultFormat;
+    }
+
+    public boolean isNumeric() {
+        return false;
+    }
+
+    public boolean isFloatingPoint() {
+        return false;
+    }
+
+    public boolean isGeoPoint() {
+        return false;
+    }
+
+    public static ValueType resolveForScript(String type) {
+        switch (type) {
+            case "string":  return STRING;
+            case "double":
+            case "float":   return DOUBLE;
+            case "long":
+            case "integer":
+            case "short":
+            case "byte":    return LONG;
+            case "date":    return DATE;
+            case "ip":      return IP;
+            default:
+                return null;
+        }
+    }
+
+    @Override
+    public String toString() {
+        return description;
+    }
+}
diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSource.java b/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSource.java
index c07979a16235c..c9be5521caddb 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSource.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSource.java
@@ -545,13 +545,13 @@ public org.elasticsearch.index.fielddata.DoubleValues doubleValues() {
         }
 
         public static class Script extends Numeric {
-            private final ScriptValueType scriptValueType;
+            private final ValueType scriptValueType;
 
             private final ScriptDoubleValues doubleValues;
             private final ScriptLongValues longValues;
             private final ScriptBytesValues bytesValues;
 
-            public Script(SearchScript script, ScriptValueType scriptValueType) {
+            public Script(SearchScript script, ValueType scriptValueType) {
                 this.scriptValueType = scriptValueType;
                 longValues = new ScriptLongValues(script);
                 doubleValues = new ScriptDoubleValues(script);
diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSourceAggregatorFactory.java b/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSourceAggregatorFactory.java
index 81b098ffa2dac..13cac61ea6035 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSourceAggregatorFactory.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSourceAggregatorFactory.java
@@ -19,8 +19,7 @@
 package org.elasticsearch.search.aggregations.support;
 
 import org.elasticsearch.search.aggregations.*;
-import org.elasticsearch.search.aggregations.support.format.ValueFormatter;
-import org.elasticsearch.search.aggregations.support.format.ValueParser;
+import org.elasticsearch.search.aggregations.support.format.ValueFormat;
 
 /**
  *
@@ -33,10 +32,6 @@ protected LeafOnly(String name, String type, ValuesSourceConfig<VS> valuesSource
             super(name, type, valuesSourceConfig);
         }
 
-        protected LeafOnly(String name, String type, ValuesSourceConfig<VS> valuesSourceConfig, ValueFormatter formatter, ValueParser parser) {
-            super(name, type, valuesSourceConfig, formatter, parser);
-        }
-
         @Override
         public AggregatorFactory subFactories(AggregatorFactories subFactories) {
             throw new AggregationInitializationException("Aggregator [" + name + "] of type [" + type + "] cannot accept sub-aggregations");
@@ -44,18 +39,10 @@ public AggregatorFactory subFactories(AggregatorFactories subFactories) {
     }
 
     protected ValuesSourceConfig<VS> config;
-    protected ValueFormatter formatter;
-    protected ValueParser parser;
 
     protected ValuesSourceAggregatorFactory(String name, String type, ValuesSourceConfig<VS> config) {
-        this(name, type, config, null, null);
-    }
-
-    protected ValuesSourceAggregatorFactory(String name, String type, ValuesSourceConfig<VS> config, ValueFormatter formatter, ValueParser parser) {
         super(name, type);
         this.config = config;
-        this.formatter = formatter;
-        this.parser = parser;
     }
 
     @Override
@@ -85,9 +72,13 @@ private void resolveValuesSourceConfigFromAncestors(String aggName, AggregatorFa
                 config = ((ValuesSourceAggregatorFactory) parent).config;
                 if (config != null && config.valid()) {
                     if (requiredValuesSourceType == null || requiredValuesSourceType.isAssignableFrom(config.valueSourceType)) {
+                        ValueFormat format = config.format;
                         this.config = config;
-                        this.formatter = ((ValuesSourceAggregatorFactory) parent).formatter;
-                        this.parser = ((ValuesSourceAggregatorFactory) parent).parser;
+                        // if the user explicitly defined a format pattern, we'll do our best to keep it even when we inherit the
+                        // value source form one of the ancestor aggregations
+                        if (this.config.formatPattern != null && format != null && format instanceof ValueFormat.Patternable) {
+                            this.config.format = ((ValueFormat.Patternable) format).create(this.config.formatPattern);
+                        }
                         return;
                     }
                 }
diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSourceConfig.java b/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSourceConfig.java
index ac9eaf0508c45..1ae409bce2d31 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSourceConfig.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSourceConfig.java
@@ -19,6 +19,9 @@
 package org.elasticsearch.search.aggregations.support;
 
 import org.elasticsearch.script.SearchScript;
+import org.elasticsearch.search.aggregations.support.format.ValueFormat;
+import org.elasticsearch.search.aggregations.support.format.ValueFormatter;
+import org.elasticsearch.search.aggregations.support.format.ValueParser;
 
 /**
  *
@@ -28,11 +31,13 @@ public class ValuesSourceConfig<VS extends ValuesSource> {
     final Class<VS> valueSourceType;
     FieldContext fieldContext;
     SearchScript script;
-    ScriptValueType scriptValueType;
+    ValueType scriptValueType;
     boolean unmapped = false;
     boolean needsHashes = false;
     boolean ensureUnique = false;
     boolean ensureSorted = false;
+    String formatPattern;
+    ValueFormat format;
 
     public ValuesSourceConfig(Class<VS> valueSourceType) {
         this.valueSourceType = valueSourceType;
@@ -46,6 +51,10 @@ public FieldContext fieldContext() {
         return fieldContext;
     }
 
+    public SearchScript script() {
+        return script;
+    }
+
     public boolean unmapped() {
         return unmapped;
     }
@@ -64,15 +73,6 @@ public ValuesSourceConfig<VS> script(SearchScript script) {
         return this;
     }
 
-    public ValuesSourceConfig<VS> scriptValueType(ScriptValueType scriptValueType) {
-        this.scriptValueType = scriptValueType;
-        return this;
-    }
-
-    public ScriptValueType scriptValueType() {
-        return scriptValueType;
-    }
-
     public ValuesSourceConfig<VS> unmapped(boolean unmapped) {
         this.unmapped = unmapped;
         return this;
@@ -83,13 +83,15 @@ public ValuesSourceConfig<VS> needsHashes(boolean needsHashes) {
         return this;
     }
 
-    public ValuesSourceConfig<VS> ensureUnique(boolean unique) {
-        this.ensureUnique = unique;
-        return this;
+    public ValueFormat format() {
+        return format;
     }
 
-    public ValuesSourceConfig<VS> ensureSorted(boolean sorted) {
-        this.ensureSorted = sorted;
-        return this;
+    public ValueFormatter formatter() {
+        return format != null ? format.formatter() : null;
+    }
+
+    public ValueParser parser() {
+        return format != null ? format.parser() : null;
     }
 }
diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSourceParser.java b/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSourceParser.java
new file mode 100644
index 0000000000000..e5575864bd036
--- /dev/null
+++ b/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSourceParser.java
@@ -0,0 +1,266 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.search.aggregations.support;
+
+import org.elasticsearch.common.Nullable;
+import org.elasticsearch.common.xcontent.XContentParser;
+import org.elasticsearch.index.fielddata.IndexFieldData;
+import org.elasticsearch.index.fielddata.IndexGeoPointFieldData;
+import org.elasticsearch.index.fielddata.IndexNumericFieldData;
+import org.elasticsearch.index.mapper.FieldMapper;
+import org.elasticsearch.index.mapper.core.DateFieldMapper;
+import org.elasticsearch.index.mapper.core.NumberFieldMapper;
+import org.elasticsearch.index.mapper.ip.IpFieldMapper;
+import org.elasticsearch.script.SearchScript;
+import org.elasticsearch.search.SearchParseException;
+import org.elasticsearch.search.aggregations.InternalAggregation;
+import org.elasticsearch.search.aggregations.support.format.ValueFormat;
+import org.elasticsearch.search.internal.SearchContext;
+
+import java.io.IOException;
+import java.util.Map;
+
+/**
+ *
+ */
+public class ValuesSourceParser<VS extends ValuesSource> {
+
+    public static Builder any(String aggName, InternalAggregation.Type aggType, SearchContext context) {
+        return new Builder<>(aggName, aggType, context, ValuesSource.class);
+    }
+
+    public static Builder<ValuesSource.Numeric> numeric(String aggName, InternalAggregation.Type aggType, SearchContext context) {
+        return new Builder<>(aggName, aggType, context, ValuesSource.Numeric.class).targetValueType(ValueType.NUMERIC);
+    }
+
+    public static Builder<ValuesSource.Bytes> bytes(String aggName, InternalAggregation.Type aggType, SearchContext context) {
+        return new Builder<>(aggName, aggType, context, ValuesSource.Bytes.class).targetValueType(ValueType.STRING);
+    }
+
+    public static Builder<ValuesSource.GeoPoint> geoPoint(String aggName, InternalAggregation.Type aggType, SearchContext context) {
+        return new Builder<>(aggName, aggType, context, ValuesSource.GeoPoint.class).targetValueType(ValueType.GEOPOINT).scriptable(false);
+    }
+
+    private static class Input {
+        String field = null;
+        String script = null;
+        String lang = null;
+        Map<String, Object> params = null;
+        ValueType valueType = null;
+        boolean assumeUnique = false;
+        boolean assumeSorted = false;
+        String format = null;
+    }
+
+    private final String aggName;
+    private final InternalAggregation.Type aggType;
+    private final SearchContext context;
+    private final Class<VS> valuesSourceType;
+
+    private boolean scriptable = true;
+    private boolean formattable = false;
+    private ValueType targetValueType = null;
+    private boolean requiresSortedValues = false;
+    private boolean requiresUniqueValues = false;
+
+    private Input input = new Input();
+
+    private ValuesSourceParser(String aggName, InternalAggregation.Type aggType, SearchContext context, Class<VS> valuesSourceType) {
+        this.aggName = aggName;
+        this.aggType = aggType;
+        this.context = context;
+        this.valuesSourceType = valuesSourceType;
+    }
+
+    public boolean token(String currentFieldName, XContentParser.Token token, XContentParser parser) throws IOException {
+        if (token == XContentParser.Token.VALUE_STRING) {
+            if ("field".equals(currentFieldName)) {
+                input.field = parser.text();
+            } else if (formattable && "format".equals(currentFieldName)) {
+                input.format = parser.text();
+            } else if (scriptable) {
+                if ("script".equals(currentFieldName)) {
+                    input.script = parser.text();
+                } else if ("lang".equals(currentFieldName)) {
+                    input.lang = parser.text();
+                } else if ("value_type".equals(currentFieldName) || "valueType".equals(currentFieldName)) {
+                    input.valueType = ValueType.resolveForScript(parser.text());
+                    if (targetValueType != null && input.valueType.isNotA(targetValueType)) {
+                        throw new SearchParseException(context, aggType.name() + " aggregation [" + aggName +
+                                "] was configured with an incompatible value type [" + input.valueType + "]. [" + aggType +
+                                "] aggregation can only work on value of type [" + targetValueType + "]");
+                    }
+                } else {
+                    return false;
+                }
+                return true;
+            } else {
+                return false;
+            }
+            return true;
+        }
+        if (scriptable && token == XContentParser.Token.VALUE_BOOLEAN) {
+            if ("script_values_unique".equals(currentFieldName) || "scriptValuesUnique".equals(currentFieldName)) {
+                input.assumeUnique = parser.booleanValue();
+            } else if ("script_values_sorted".equals(currentFieldName) || "scriptValuesSorted".equals(currentFieldName)) {
+                input.assumeSorted = parser.booleanValue();
+            } else {
+                return false;
+            }
+            return true;
+        }
+        if (scriptable && token == XContentParser.Token.START_OBJECT) {
+            if ("params".equals(currentFieldName)) {
+                input.params = parser.map();
+                return true;
+            }
+            return false;
+        }
+
+        return false;
+    }
+
+    public ValuesSourceConfig<VS> config() {
+
+        ValueType valueType = input.valueType != null ? input.valueType : targetValueType;
+
+        if (input.field == null) {
+            if (input.script == null) {
+                return new ValuesSourceConfig(ValuesSource.class);
+            }
+            Class valuesSourceType = valueType != null ? (Class<VS>) valueType.getValuesSourceType() : this.valuesSourceType;
+            if (valuesSourceType == null || valuesSourceType == ValuesSource.class) {
+                // the specific value source type is undefined, but for scripts, we need to have a specific value source
+                // type to know how to handle the script values, so we fallback on Bytes
+                valuesSourceType = ValuesSource.Bytes.class;
+            }
+            ValuesSourceConfig<VS> config = new ValuesSourceConfig<VS>(valuesSourceType);
+            config.format = resolveFormat(input.format, valueType);
+            config.script = createScript();
+            config.scriptValueType = valueType;
+            config.ensureUnique = requiresUniqueValues && !input.assumeUnique;
+            config.ensureSorted = requiresSortedValues && !input.assumeSorted;
+            return config;
+        }
+
+        FieldMapper<?> mapper = context.smartNameFieldMapper(input.field);
+        if (mapper == null) {
+            Class<VS> valuesSourceType = valueType != null ? (Class<VS>) valueType.getValuesSourceType() : this.valuesSourceType;
+            ValuesSourceConfig<VS> config = new ValuesSourceConfig<>(valuesSourceType);
+            config.format = resolveFormat(input.format, valueType);
+            config.unmapped = true;
+            if (valueType != null) {
+                // todo do we really need this for unmapped?
+                config.scriptValueType = valueType;
+            }
+            return config;
+        }
+
+        IndexFieldData<?> indexFieldData = context.fieldData().getForField(mapper);
+
+        ValuesSourceConfig config;
+        if (valuesSourceType == ValuesSource.class) {
+            if (indexFieldData instanceof IndexNumericFieldData) {
+                config = new ValuesSourceConfig<>(ValuesSource.Numeric.class);
+            } else if (indexFieldData instanceof IndexGeoPointFieldData) {
+                config = new ValuesSourceConfig<>(ValuesSource.GeoPoint.class);
+            } else {
+                config = new ValuesSourceConfig<>(ValuesSource.Bytes.class);
+            }
+        } else {
+            config = new ValuesSourceConfig(valuesSourceType);
+        }
+
+        config.fieldContext = new FieldContext(input.field, indexFieldData, mapper);
+        config.script = createScript();
+        if (config.script != null) {
+            config.ensureUnique = requiresUniqueValues && !input.assumeUnique;
+            config.ensureSorted = requiresSortedValues && !input.assumeSorted;
+        }
+        config.format = resolveFormat(input.format, mapper);
+        return config;
+    }
+
+    private SearchScript createScript() {
+        return input.script == null ? null : context.scriptService().search(context.lookup(), input.lang, input.script, input.params);
+    }
+
+    private static ValueFormat resolveFormat(@Nullable String format, @Nullable ValueType valueType) {
+        if (valueType == null) {
+            return null; // we can't figure it out
+        }
+        ValueFormat valueFormat = valueType.defaultFormat;
+        if (valueFormat != null && valueFormat instanceof ValueFormat.Patternable && format != null) {
+            return ((ValueFormat.Patternable) valueFormat).create(format);
+        }
+        return valueFormat;
+    }
+
+    private static ValueFormat resolveFormat(@Nullable String format, FieldMapper mapper) {
+        if (mapper instanceof  DateFieldMapper) {
+            return format != null ? ValueFormat.DateTime.format(format) : ValueFormat.DateTime.mapper((DateFieldMapper) mapper);
+        }
+        if (mapper instanceof IpFieldMapper) {
+            return ValueFormat.IPv4;
+        }
+        if (mapper instanceof NumberFieldMapper) {
+            return format != null ? ValueFormat.Number.format(format) : ValueFormat.RAW;
+        }
+        return null;
+    }
+
+    public static class Builder<VS extends ValuesSource> {
+
+        private final ValuesSourceParser<VS> parser;
+
+        private Builder(String aggName, InternalAggregation.Type aggType, SearchContext context, Class<VS> valuesSourceType) {
+            parser = new ValuesSourceParser<>(aggName, aggType, context, valuesSourceType);
+        }
+
+        public Builder<VS> scriptable(boolean scriptable) {
+            parser.scriptable = scriptable;
+            return this;
+        }
+
+        public Builder<VS> formattable(boolean formattable) {
+            parser.formattable = formattable;
+            return this;
+        }
+
+        public Builder<VS> targetValueType(ValueType valueType) {
+            parser.targetValueType = valueType;
+            return this;
+        }
+
+        public Builder<VS> requiresSortedValues(boolean requiresSortedValues) {
+            parser.requiresSortedValues = requiresSortedValues;
+            return this;
+        }
+
+        public Builder<VS> requiresUniqueValues(boolean requiresUniqueValues) {
+            parser.requiresUniqueValues = requiresUniqueValues;
+            return this;
+        }
+
+        public ValuesSourceParser<VS> build() {
+            return parser;
+        }
+    }
+}
diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/format/ValueFormat.java b/src/main/java/org/elasticsearch/search/aggregations/support/format/ValueFormat.java
new file mode 100644
index 0000000000000..54c69ec914ae8
--- /dev/null
+++ b/src/main/java/org/elasticsearch/search/aggregations/support/format/ValueFormat.java
@@ -0,0 +1,103 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.search.aggregations.support.format;
+
+import org.elasticsearch.index.mapper.core.DateFieldMapper;
+
+/**
+ *
+ */
+public class ValueFormat {
+
+    public static final ValueFormat RAW = new ValueFormat(ValueFormatter.RAW, ValueParser.RAW);
+    public static final ValueFormat IPv4 = new ValueFormat(ValueFormatter.IPv4, ValueParser.IPv4);
+
+    private final ValueFormatter formatter;
+    private final ValueParser parser;
+
+    public ValueFormat(ValueFormatter formatter, ValueParser parser) {
+        assert formatter != null && parser != null;
+        this.formatter = formatter;
+        this.parser = parser;
+    }
+
+    public ValueFormatter formatter() {
+        return formatter;
+    }
+
+    public ValueParser parser() {
+        return parser;
+    }
+
+    public abstract static class Patternable<VF extends Patternable<VF>> extends ValueFormat {
+
+        private final String pattern;
+
+        public Patternable(String pattern, ValueFormatter formatter, ValueParser parser) {
+            super(formatter, parser);
+            this.pattern = pattern;
+        }
+
+        public String pattern() {
+            return pattern;
+        }
+
+        public abstract VF create(String pattern);
+    }
+
+    public static class DateTime extends Patternable<DateTime> {
+
+        public static final DateTime DEFAULT = new DateTime(DateFieldMapper.Defaults.DATE_TIME_FORMATTER.format(), ValueFormatter.DateTime.DEFAULT, ValueParser.DateMath.DEFAULT);
+
+        public static DateTime format(String format) {
+            return new DateTime(format, new ValueFormatter.DateTime(format), new ValueParser.DateMath(format, DateFieldMapper.Defaults.TIME_UNIT));
+        }
+
+        public static DateTime mapper(DateFieldMapper mapper) {
+            return new DateTime(mapper.dateTimeFormatter().format(), ValueFormatter.DateTime.mapper(mapper), ValueParser.DateMath.mapper(mapper));
+        }
+
+        public DateTime(String pattern, ValueFormatter formatter, ValueParser parser) {
+            super(pattern, formatter, parser);
+        }
+
+        @Override
+        public DateTime create(String pattern) {
+            return format(pattern);
+        }
+    }
+
+    public static class Number extends Patternable<Number> {
+
+        public static Number format(String format) {
+            return new Number(format, new ValueFormatter.Number.Pattern(format), new ValueParser.Number.Pattern(format));
+        }
+
+        public Number(String pattern, ValueFormatter formatter, ValueParser parser) {
+            super(pattern, formatter, parser);
+        }
+
+        @Override
+        public Number create(String pattern) {
+            return format(pattern);
+        }
+    }
+
+}
diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/format/ValueFormatter.java b/src/main/java/org/elasticsearch/search/aggregations/support/format/ValueFormatter.java
index a5738a6da47bd..565d0328f45be 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/support/format/ValueFormatter.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/support/format/ValueFormatter.java
@@ -102,6 +102,10 @@ public static class DateTime implements ValueFormatter {
 
         public static final ValueFormatter DEFAULT = new ValueFormatter.DateTime(DateFieldMapper.Defaults.DATE_TIME_FORMATTER);
 
+        public static DateTime mapper(DateFieldMapper mapper) {
+            return new DateTime(mapper.dateTimeFormatter());
+        }
+
         static final byte ID = 2;
 
         FormatDateTimeFormatter formatter;
diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/format/ValueParser.java b/src/main/java/org/elasticsearch/search/aggregations/support/format/ValueParser.java
index f60eb79fe8301..94646e5cd6908 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/support/format/ValueParser.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/support/format/ValueParser.java
@@ -51,6 +51,8 @@ public interface ValueParser {
      */
     static class DateTime implements ValueParser {
 
+        public static final DateTime DEFAULT = new DateTime(DateFieldMapper.Defaults.DATE_TIME_FORMATTER);
+
         private FormatDateTimeFormatter formatter;
 
         public DateTime(String format) {
@@ -98,6 +100,10 @@ public long parseLong(String value, SearchContext searchContext) {
         public double parseDouble(String value, SearchContext searchContext) {
             return parseLong(value, searchContext);
         }
+
+        public static DateMath mapper(DateFieldMapper mapper) {
+            return new DateMath(new DateMathParser(mapper.dateTimeFormatter(), DateFieldMapper.Defaults.TIME_UNIT));
+        }
     }
 
     /**
@@ -170,4 +176,5 @@ public double parseDouble(String value, SearchContext searchContext) {
             }
         }
     }
+
 }
diff --git a/src/test/java/org/elasticsearch/search/aggregations/bucket/DateHistogramTests.java b/src/test/java/org/elasticsearch/search/aggregations/bucket/DateHistogramTests.java
index 391bfd3e3a788..8e28216f07698 100644
--- a/src/test/java/org/elasticsearch/search/aggregations/bucket/DateHistogramTests.java
+++ b/src/test/java/org/elasticsearch/search/aggregations/bucket/DateHistogramTests.java
@@ -1166,7 +1166,8 @@ public void singleValueField_WithExtendedBounds() throws Exception {
                             .field("date")
                             .interval(DateHistogram.Interval.days(interval))
                             .minDocCount(0)
-                            .extendedBounds(boundsMin, boundsMax)
+                            // when explicitly specifying a format, the extended bounds should be defined by the same format
+                            .extendedBounds(format(boundsMin, pattern), format(boundsMax, pattern))
                             .format(pattern))
                     .execute().actionGet();
 
diff --git a/src/test/java/org/elasticsearch/search/aggregations/bucket/SignificantTermsTests.java b/src/test/java/org/elasticsearch/search/aggregations/bucket/SignificantTermsTests.java
index f807bd94aa0ae..23d35bbf349de 100644
--- a/src/test/java/org/elasticsearch/search/aggregations/bucket/SignificantTermsTests.java
+++ b/src/test/java/org/elasticsearch/search/aggregations/bucket/SignificantTermsTests.java
@@ -197,7 +197,5 @@ private void checkExpectedStringTermsFound(SignificantTerms topTerms) {
         assertEquals(3, kellyTerm.getSubsetDf());
         assertEquals(4, kellyTerm.getSupersetDf());
     }
-    
-    
 
 }

From 94278d81e32951c88075d7eb4eced643bad27c9b Mon Sep 17 00:00:00 2001
From: wittyameta <wittyameta@gmail.com>
Date: Thu, 3 Apr 2014 16:04:44 +0530
Subject: [PATCH 21/54] Update advanced-scripting.asciidoc

---
 docs/reference/modules/advanced-scripting.asciidoc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/reference/modules/advanced-scripting.asciidoc b/docs/reference/modules/advanced-scripting.asciidoc
index d215661ae3e0e..13adef44c33bd 100644
--- a/docs/reference/modules/advanced-scripting.asciidoc
+++ b/docs/reference/modules/advanced-scripting.asciidoc
@@ -177,7 +177,7 @@ return score;
 === Term vectors:
 
 The `_index` variable can only be used to gather statistics for single terms. If you want to use information on all terms in a field, you must store the term vectors (set `term_vector` in the mapping as described in the <<mapping-core-types,mapping documentation>>). To access them, call
-`_index.getTermVectors()` to get a
+`_index.termVectors()` to get a
 https://lucene.apache.org/core/4_0_0/core/org/apache/lucene/index/Fields.html[Fields]
 instance. This object can then be used as described in https://lucene.apache.org/core/4_0_0/core/org/apache/lucene/index/Fields.html[lucene doc] to iterate over fields and then for each field iterate over each term in the field.
 The method will return null if the term vectors were not stored.

From 705c7e2469546fcb241f119570265a76262eac75 Mon Sep 17 00:00:00 2001
From: Shay Banon <kimchy@gmail.com>
Date: Sun, 6 Apr 2014 23:39:51 +0200
Subject: [PATCH 22/54] Recycled bytes in http + rest layer refactoring phase 2
 Refactor the rest layer handlers to simplify common code paths (like
 handling) failures, and introduce optional (enabled for netty) rest channel
 bytes recycling

---
 .../action/get/MultiGetResponse.java          |   4 +-
 .../percolate/MultiPercolateResponse.java     |   4 +-
 .../action/percolate/PercolateResponse.java   |   4 -
 .../termvector/MultiTermVectorsResponse.java  |   4 +-
 .../action/termvector/TermVectorResponse.java |   2 -
 .../org/elasticsearch/http/HttpChannel.java   |   6 +-
 .../http/netty/NettyHttpChannel.java          |  13 +-
 .../http/netty/NettyHttpServerTransport.java  |   5 +-
 .../elasticsearch/index/get/GetResult.java    |   8 +-
 .../elasticsearch/rest/BytesRestResponse.java |  24 ++--
 .../org/elasticsearch/rest/RestChannel.java   |  78 ++++++++++-
 .../elasticsearch/rest/RestController.java    |  14 +-
 .../org/elasticsearch/rest/RestFilter.java    |   2 +-
 .../org/elasticsearch/rest/RestHandler.java   |   2 +-
 .../health/RestClusterHealthAction.java       |  68 ++--------
 .../hotthreads/RestNodesHotThreadsAction.java |  33 ++---
 .../node/info/RestNodesInfoAction.java        |  33 ++---
 .../node/restart/RestNodesRestartAction.java  |  42 ++----
 .../shutdown/RestNodesShutdownAction.java     |  42 ++----
 .../node/stats/RestNodesStatsAction.java      |  34 +----
 .../delete/RestDeleteRepositoryAction.java    |   3 +-
 .../get/RestGetRepositoriesAction.java        |  31 ++---
 .../put/RestPutRepositoryAction.java          |   3 +-
 .../reroute/RestClusterRerouteAction.java     |  24 +---
 .../RestClusterGetSettingsAction.java         |  39 ++----
 .../RestClusterUpdateSettingsAction.java      |  35 ++---
 .../shards/RestClusterSearchShardsAction.java |  26 +---
 .../create/RestCreateSnapshotAction.java      |  36 +-----
 .../delete/RestDeleteSnapshotAction.java      |   9 +-
 .../snapshots/get/RestGetSnapshotsAction.java |  36 +-----
 .../restore/RestRestoreSnapshotAction.java    |  36 +-----
 .../status/RestSnapshotsStatusAction.java     |  27 +---
 .../cluster/state/RestClusterStateAction.java |  34 ++---
 .../cluster/stats/RestClusterStatsAction.java |  35 +----
 .../tasks/RestPendingClusterTasksAction.java  |  39 +-----
 .../alias/RestIndicesAliasesAction.java       |  13 +-
 .../delete/RestIndexDeleteAliasesAction.java  |   3 +-
 .../alias/get/RestGetAliasesAction.java       |  65 ++++------
 .../get/RestGetIndicesAliasesAction.java      |  53 +++-----
 .../alias/put/RestIndexPutAliasAction.java    |  27 +---
 .../indices/analyze/RestAnalyzeAction.java    |  43 +-----
 .../clear/RestClearIndicesCacheAction.java    |  99 +++++---------
 .../indices/close/RestCloseIndexAction.java   |   3 +-
 .../indices/create/RestCreateIndexAction.java |  23 +---
 .../indices/delete/RestDeleteIndexAction.java |   3 +-
 .../indices/RestIndicesExistsAction.java      |  27 +---
 .../exists/types/RestTypesExistsAction.java   |  28 +---
 .../admin/indices/flush/RestFlushAction.java  |  33 +----
 .../delete/RestDeleteMappingAction.java       |   3 +-
 .../get/RestGetFieldMappingAction.java        |  47 ++-----
 .../mapping/get/RestGetMappingAction.java     |  75 ++++-------
 .../mapping/put/RestPutMappingAction.java     |   3 +-
 .../indices/open/RestOpenIndexAction.java     |   3 +-
 .../indices/optimize/RestOptimizeAction.java  |  63 +++------
 .../indices/recovery/RestRecoveryAction.java  |  40 ++----
 .../indices/refresh/RestRefreshAction.java    |  31 +----
 .../segments/RestIndicesSegmentsAction.java   |  33 ++---
 .../settings/RestGetSettingsAction.java       |  45 ++-----
 .../settings/RestUpdateSettingsAction.java    |  31 ++---
 .../indices/stats/RestIndicesStatsAction.java |  31 ++---
 .../status/RestIndicesStatusAction.java       |  31 ++---
 .../delete/RestDeleteIndexTemplateAction.java |   3 +-
 .../get/RestGetIndexTemplateAction.java       |  43 ++----
 .../head/RestHeadIndexTemplateAction.java     |  31 ++---
 .../put/RestPutIndexTemplateAction.java       |  20 +--
 .../query/RestValidateQueryAction.java        | 116 +++++++----------
 .../warmer/delete/RestDeleteWarmerAction.java |   9 +-
 .../warmer/get/RestGetWarmerAction.java       |  48 +++----
 .../warmer/put/RestPutWarmerAction.java       |  15 ++-
 .../rest/action/bulk/RestBulkAction.java      | 122 +++++++-----------
 .../rest/action/cat/AbstractCatAction.java    |  14 +-
 .../rest/action/cat/RestAliasAction.java      |  26 +---
 .../rest/action/cat/RestAllocationAction.java |  41 ++----
 .../rest/action/cat/RestCatAction.java        |  10 +-
 .../rest/action/cat/RestCountAction.java      |  24 +---
 .../rest/action/cat/RestHealthAction.java     |  24 +---
 .../rest/action/cat/RestIndicesAction.java    |  57 ++------
 .../rest/action/cat/RestMasterAction.java     |  25 +---
 .../rest/action/cat/RestNodesAction.java      |  51 ++------
 .../cat/RestPendingClusterTasksAction.java    |  30 ++---
 .../rest/action/cat/RestPluginsAction.java    |  18 ++-
 .../rest/action/cat/RestRecoveryAction.java   |  42 ++----
 .../rest/action/cat/RestSegmentsAction.java   |  38 ++----
 .../rest/action/cat/RestShardsAction.java     |  21 +--
 .../rest/action/cat/RestThreadPoolAction.java |  28 ++--
 .../rest/action/count/RestCountAction.java    |  82 ++++--------
 .../rest/action/delete/RestDeleteAction.java  |  45 ++-----
 .../RestDeleteByQueryAction.java              | 122 +++++++-----------
 .../action/explain/RestExplainAction.java     |  56 +++-----
 .../rest/action/get/RestGetAction.java        |  36 ++----
 .../rest/action/get/RestGetSourceAction.java  |  37 ++----
 .../rest/action/get/RestHeadAction.java       |  28 +---
 .../rest/action/get/RestMultiGetAction.java   |  39 +-----
 .../rest/action/index/RestIndexAction.java    |  45 +++----
 .../rest/action/main/RestMainAction.java      |  71 ++++------
 .../action/mlt/RestMoreLikeThisAction.java    |  95 +++++---------
 .../percolate/RestMultiPercolateAction.java   |  39 +-----
 .../action/percolate/RestPercolateAction.java |  24 +---
 .../action/search/RestClearScrollAction.java  |  29 +----
 .../action/search/RestMultiSearchAction.java  |  41 +-----
 .../rest/action/search/RestSearchAction.java  |  80 +++---------
 .../action/search/RestSearchScrollAction.java |  58 ++-------
 .../action/suggest/RestSuggestAction.java     |  78 ++++-------
 .../support/AcknowledgedRestListener.java}    |  31 +++--
 .../action/support/RestActionListener.java    |  65 ++++++++++
 .../action/support/RestBuilderListener.java   |  44 +++++++
 .../support/RestResponseListener.java}        |  38 +++---
 .../rest/action/support/RestTable.java        |  29 +++--
 .../support/RestToXContentListener.java       |  50 +++++++
 .../action/support/RestXContentBuilder.java   |  84 ------------
 .../RestMultiTermVectorsAction.java           |  39 +-----
 .../termvector/RestTermVectorAction.java      |  48 +------
 .../rest/action/update/RestUpdateAction.java  |  99 ++++++--------
 .../search/internal/InternalSearchHit.java    |   3 +-
 .../GetTermVectorCheckDocFreqTests.java       |  12 +-
 115 files changed, 1357 insertions(+), 2737 deletions(-)
 rename src/main/java/org/elasticsearch/rest/{AcknowledgedRestResponseActionListener.java => action/support/AcknowledgedRestListener.java} (64%)
 create mode 100644 src/main/java/org/elasticsearch/rest/action/support/RestActionListener.java
 create mode 100644 src/main/java/org/elasticsearch/rest/action/support/RestBuilderListener.java
 rename src/main/java/org/elasticsearch/rest/{AbstractRestResponseActionListener.java => action/support/RestResponseListener.java} (52%)
 create mode 100644 src/main/java/org/elasticsearch/rest/action/support/RestToXContentListener.java
 delete mode 100644 src/main/java/org/elasticsearch/rest/action/support/RestXContentBuilder.java

diff --git a/src/main/java/org/elasticsearch/action/get/MultiGetResponse.java b/src/main/java/org/elasticsearch/action/get/MultiGetResponse.java
index 0c367ef009211..2efb90bc3a1a9 100644
--- a/src/main/java/org/elasticsearch/action/get/MultiGetResponse.java
+++ b/src/main/java/org/elasticsearch/action/get/MultiGetResponse.java
@@ -124,7 +124,6 @@ public Iterator<MultiGetItemResponse> iterator() {
 
     @Override
     public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
-        builder.startObject();
         builder.startArray(Fields.DOCS);
         for (MultiGetItemResponse response : responses) {
             if (response.isFailed()) {
@@ -137,11 +136,12 @@ public XContentBuilder toXContent(XContentBuilder builder, Params params) throws
                 builder.endObject();
             } else {
                 GetResponse getResponse = response.getResponse();
+                builder.startObject();
                 getResponse.toXContent(builder, params);
+                builder.endObject();
             }
         }
         builder.endArray();
-        builder.endObject();
         return builder;
     }
 
diff --git a/src/main/java/org/elasticsearch/action/percolate/MultiPercolateResponse.java b/src/main/java/org/elasticsearch/action/percolate/MultiPercolateResponse.java
index 87e17eba0bd00..27e9ce44f6560 100644
--- a/src/main/java/org/elasticsearch/action/percolate/MultiPercolateResponse.java
+++ b/src/main/java/org/elasticsearch/action/percolate/MultiPercolateResponse.java
@@ -59,7 +59,6 @@ public Item[] getItems() {
 
     @Override
     public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
-        builder.startObject();
         builder.startArray(Fields.RESPONSES);
         for (MultiPercolateResponse.Item item : items) {
             if (item.isFailure()) {
@@ -67,11 +66,12 @@ public XContentBuilder toXContent(XContentBuilder builder, Params params) throws
                 builder.field(Fields.ERROR, item.getErrorMessage());
                 builder.endObject();
             } else {
+                builder.startObject();
                 item.getResponse().toXContent(builder, params);
+                builder.endObject();
             }
         }
         builder.endArray();
-        builder.endObject();
         return builder;
     }
 
diff --git a/src/main/java/org/elasticsearch/action/percolate/PercolateResponse.java b/src/main/java/org/elasticsearch/action/percolate/PercolateResponse.java
index 87cf5cc1423e3..284bca38168a8 100644
--- a/src/main/java/org/elasticsearch/action/percolate/PercolateResponse.java
+++ b/src/main/java/org/elasticsearch/action/percolate/PercolateResponse.java
@@ -122,8 +122,6 @@ public Iterator<Match> iterator() {
 
     @Override
     public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
-        builder.startObject();
-
         builder.field(Fields.TOOK, tookInMillis);
         RestActions.buildBroadcastShardsHeader(builder, this);
 
@@ -172,8 +170,6 @@ public XContentBuilder toXContent(XContentBuilder builder, Params params) throws
         if (aggregations != null) {
             aggregations.toXContent(builder, params);
         }
-
-        builder.endObject();
         return builder;
     }
 
diff --git a/src/main/java/org/elasticsearch/action/termvector/MultiTermVectorsResponse.java b/src/main/java/org/elasticsearch/action/termvector/MultiTermVectorsResponse.java
index e4814d90cd7c9..3313c05236ae1 100644
--- a/src/main/java/org/elasticsearch/action/termvector/MultiTermVectorsResponse.java
+++ b/src/main/java/org/elasticsearch/action/termvector/MultiTermVectorsResponse.java
@@ -124,7 +124,6 @@ public Iterator<MultiTermVectorsItemResponse> iterator() {
 
     @Override
     public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
-        builder.startObject();
         builder.startArray(Fields.DOCS);
         for (MultiTermVectorsItemResponse response : responses) {
             if (response.isFailed()) {
@@ -137,11 +136,12 @@ public XContentBuilder toXContent(XContentBuilder builder, Params params) throws
                 builder.endObject();
             } else {
                 TermVectorResponse getResponse = response.getResponse();
+                builder.startObject();
                 getResponse.toXContent(builder, params);
+                builder.endObject();
             }
         }
         builder.endArray();
-        builder.endObject();
         return builder;
     }
 
diff --git a/src/main/java/org/elasticsearch/action/termvector/TermVectorResponse.java b/src/main/java/org/elasticsearch/action/termvector/TermVectorResponse.java
index f662cbbc0fdfe..706d285a9f0b9 100644
--- a/src/main/java/org/elasticsearch/action/termvector/TermVectorResponse.java
+++ b/src/main/java/org/elasticsearch/action/termvector/TermVectorResponse.java
@@ -164,7 +164,6 @@ public XContentBuilder toXContent(XContentBuilder builder, Params params) throws
         assert index != null;
         assert type != null;
         assert id != null;
-        builder.startObject();
         builder.field(FieldStrings._INDEX, index);
         builder.field(FieldStrings._TYPE, type);
         builder.field(FieldStrings._ID, id);
@@ -182,7 +181,6 @@ public XContentBuilder toXContent(XContentBuilder builder, Params params) throws
             buildField(builder, spare, theFields, fieldIter);
         }
         builder.endObject();
-        builder.endObject();
         return builder;
 
     }
diff --git a/src/main/java/org/elasticsearch/http/HttpChannel.java b/src/main/java/org/elasticsearch/http/HttpChannel.java
index 569426f6acbfc..2194cfee4661f 100644
--- a/src/main/java/org/elasticsearch/http/HttpChannel.java
+++ b/src/main/java/org/elasticsearch/http/HttpChannel.java
@@ -20,10 +20,14 @@
 package org.elasticsearch.http;
 
 import org.elasticsearch.rest.RestChannel;
+import org.elasticsearch.rest.RestRequest;
 
 /**
  *
  */
-public interface HttpChannel extends RestChannel {
+public abstract class HttpChannel extends RestChannel {
 
+    protected HttpChannel(RestRequest request) {
+        super(request);
+    }
 }
diff --git a/src/main/java/org/elasticsearch/http/netty/NettyHttpChannel.java b/src/main/java/org/elasticsearch/http/netty/NettyHttpChannel.java
index d940343f6592d..bcfabb4507b81 100644
--- a/src/main/java/org/elasticsearch/http/netty/NettyHttpChannel.java
+++ b/src/main/java/org/elasticsearch/http/netty/NettyHttpChannel.java
@@ -22,6 +22,8 @@
 import org.apache.lucene.util.BytesRef;
 import org.apache.lucene.util.UnicodeUtil;
 import org.elasticsearch.common.bytes.BytesReference;
+import org.elasticsearch.common.io.stream.BytesStreamOutput;
+import org.elasticsearch.common.io.stream.ReleasableBytesStreamOutput;
 import org.elasticsearch.common.lease.Releasable;
 import org.elasticsearch.common.netty.ReleaseChannelFutureListener;
 import org.elasticsearch.http.HttpChannel;
@@ -42,7 +44,7 @@
 /**
  *
  */
-public class NettyHttpChannel implements HttpChannel {
+public class NettyHttpChannel extends HttpChannel {
 
     private static final ChannelBuffer END_JSONP;
 
@@ -54,16 +56,21 @@ public class NettyHttpChannel implements HttpChannel {
 
     private final NettyHttpServerTransport transport;
     private final Channel channel;
-    private final NettyHttpRequest request;
     private final org.jboss.netty.handler.codec.http.HttpRequest nettyRequest;
 
     public NettyHttpChannel(NettyHttpServerTransport transport, Channel channel, NettyHttpRequest request) {
+        super(request);
         this.transport = transport;
         this.channel = channel;
-        this.request = request;
         this.nettyRequest = request.request();
     }
 
+    @Override
+    public BytesStreamOutput newBytesOutput() {
+        return new ReleasableBytesStreamOutput(transport.bigArrays);
+    }
+
+
     @Override
     public void sendResponse(RestResponse response) {
         // Decide whether to close the connection or not.
diff --git a/src/main/java/org/elasticsearch/http/netty/NettyHttpServerTransport.java b/src/main/java/org/elasticsearch/http/netty/NettyHttpServerTransport.java
index a29690f6f7bf8..5f275460f82b7 100644
--- a/src/main/java/org/elasticsearch/http/netty/NettyHttpServerTransport.java
+++ b/src/main/java/org/elasticsearch/http/netty/NettyHttpServerTransport.java
@@ -33,6 +33,7 @@
 import org.elasticsearch.common.transport.PortsRange;
 import org.elasticsearch.common.unit.ByteSizeUnit;
 import org.elasticsearch.common.unit.ByteSizeValue;
+import org.elasticsearch.common.util.BigArrays;
 import org.elasticsearch.common.util.concurrent.EsExecutors;
 import org.elasticsearch.http.*;
 import org.elasticsearch.http.HttpRequest;
@@ -64,6 +65,7 @@ public class NettyHttpServerTransport extends AbstractLifecycleComponent<HttpSer
     }
 
     private final NetworkService networkService;
+    final BigArrays bigArrays;
 
     final ByteSizeValue maxContentLength;
     final ByteSizeValue maxInitialLineLength;
@@ -110,9 +112,10 @@ public class NettyHttpServerTransport extends AbstractLifecycleComponent<HttpSer
     private volatile HttpServerAdapter httpServerAdapter;
 
     @Inject
-    public NettyHttpServerTransport(Settings settings, NetworkService networkService) {
+    public NettyHttpServerTransport(Settings settings, NetworkService networkService, BigArrays bigArrays) {
         super(settings);
         this.networkService = networkService;
+        this.bigArrays = bigArrays;
 
         if (settings.getAsBoolean("netty.epollBugWorkaround", false)) {
             System.setProperty("org.jboss.netty.epollBugWorkaround", "true");
diff --git a/src/main/java/org/elasticsearch/index/get/GetResult.java b/src/main/java/org/elasticsearch/index/get/GetResult.java
index 6d0742924ef27..445da7b0ac053 100644
--- a/src/main/java/org/elasticsearch/index/get/GetResult.java
+++ b/src/main/java/org/elasticsearch/index/get/GetResult.java
@@ -30,7 +30,6 @@
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.common.xcontent.XContentBuilderString;
 import org.elasticsearch.common.xcontent.XContentHelper;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
 import org.elasticsearch.search.lookup.SourceLookup;
 
 import java.io.IOException;
@@ -210,7 +209,7 @@ public XContentBuilder toXContentEmbedded(XContentBuilder builder, Params params
         builder.field(Fields.FOUND, exists);
 
         if (source != null) {
-            RestXContentBuilder.restDocumentSource(source, builder, params);
+            XContentHelper.writeRawField("_source", source, builder, params);
         }
 
         if (fields != null && !fields.isEmpty()) {
@@ -238,14 +237,11 @@ public XContentBuilder toXContentEmbedded(XContentBuilder builder, Params params
     @Override
     public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
         if (!isExists()) {
-            builder.startObject();
             builder.field(Fields._INDEX, index);
             builder.field(Fields._TYPE, type);
             builder.field(Fields._ID, id);
             builder.field(Fields.FOUND, false);
-            builder.endObject();
         } else {
-            builder.startObject();
             builder.field(Fields._INDEX, index);
             builder.field(Fields._TYPE, type);
             builder.field(Fields._ID, id);
@@ -253,8 +249,6 @@ public XContentBuilder toXContent(XContentBuilder builder, Params params) throws
                 builder.field(Fields._VERSION, version);
             }
             toXContentEmbedded(builder, params);
-
-            builder.endObject();
         }
         return builder;
     }
diff --git a/src/main/java/org/elasticsearch/rest/BytesRestResponse.java b/src/main/java/org/elasticsearch/rest/BytesRestResponse.java
index 822cb3a7388e4..5143db8f580d5 100644
--- a/src/main/java/org/elasticsearch/rest/BytesRestResponse.java
+++ b/src/main/java/org/elasticsearch/rest/BytesRestResponse.java
@@ -27,7 +27,6 @@
 import java.io.IOException;
 
 import static org.elasticsearch.ExceptionsHelper.detailedMessage;
-import static org.elasticsearch.rest.action.support.RestXContentBuilder.restContentBuilder;
 
 public class BytesRestResponse extends RestResponse {
 
@@ -80,12 +79,21 @@ public BytesRestResponse(RestStatus status, String contentType, BytesReference c
         this.contentType = contentType;
     }
 
-    public BytesRestResponse(RestRequest request, Throwable t) throws IOException {
-        this(request, ((t instanceof ElasticsearchException) ? ((ElasticsearchException) t).status() : RestStatus.INTERNAL_SERVER_ERROR), t);
+    public BytesRestResponse(RestChannel channel, Throwable t) throws IOException {
+        this(channel, ((t instanceof ElasticsearchException) ? ((ElasticsearchException) t).status() : RestStatus.INTERNAL_SERVER_ERROR), t);
     }
 
-    public BytesRestResponse(RestRequest request, RestStatus status, Throwable t) throws IOException {
-        this(status, convert(request, status, t));
+    public BytesRestResponse(RestChannel channel, RestStatus status, Throwable t) throws IOException {
+        this.status = status;
+        if (channel.request().method() == RestRequest.Method.HEAD) {
+            this.content = BytesArray.EMPTY;
+            this.contentType = TEXT_CONTENT_TYPE;
+        } else {
+            XContentBuilder builder = convert(channel, status, t);
+            this.content = builder.bytes();
+            this.contentType = builder.contentType().restContentType();
+        }
+        this.contentThreadSafe = true;
     }
 
     @Override
@@ -108,11 +116,11 @@ public RestStatus status() {
         return this.status;
     }
 
-    private static XContentBuilder convert(RestRequest request, RestStatus status, Throwable t) throws IOException {
-        XContentBuilder builder = restContentBuilder(request).startObject()
+    private static XContentBuilder convert(RestChannel channel, RestStatus status, Throwable t) throws IOException {
+        XContentBuilder builder = channel.newBuilder().startObject()
                 .field("error", detailedMessage(t))
                 .field("status", status.getStatus());
-        if (t != null && request.paramAsBoolean("error_trace", false)) {
+        if (t != null && channel.request().paramAsBoolean("error_trace", false)) {
             builder.startObject("error_trace");
             boolean first = true;
             int counter = 0;
diff --git a/src/main/java/org/elasticsearch/rest/RestChannel.java b/src/main/java/org/elasticsearch/rest/RestChannel.java
index 85345db510c6b..e3aaeaaab4cf6 100644
--- a/src/main/java/org/elasticsearch/rest/RestChannel.java
+++ b/src/main/java/org/elasticsearch/rest/RestChannel.java
@@ -19,10 +19,82 @@
 
 package org.elasticsearch.rest;
 
+import org.elasticsearch.common.Nullable;
+import org.elasticsearch.common.bytes.BytesReference;
+import org.elasticsearch.common.io.stream.BytesStreamOutput;
+import org.elasticsearch.common.xcontent.XContentBuilder;
+import org.elasticsearch.common.xcontent.XContentFactory;
+import org.elasticsearch.common.xcontent.XContentType;
+
+import java.io.IOException;
+
 /**
- *
+ * A channel used to construct bytes / builder based outputs, and send responses.
  */
-public interface RestChannel {
+public abstract class RestChannel {
+
+    protected final RestRequest request;
+
+    private BytesStreamOutput bytesOut;
+
+    protected RestChannel(RestRequest request) {
+        this.request = request;
+    }
+
+    public XContentBuilder newBuilder() throws IOException {
+        return newBuilder(request.hasContent() ? request.content() : null);
+    }
+
+    public XContentBuilder newBuilder(@Nullable BytesReference autoDetectSource) throws IOException {
+        XContentType contentType = XContentType.fromRestContentType(request.param("format", request.header("Content-Type")));
+        if (contentType == null) {
+            // try and guess it from the auto detect source
+            if (autoDetectSource != null) {
+                contentType = XContentFactory.xContentType(autoDetectSource);
+            }
+        }
+        if (contentType == null) {
+            // default to JSON
+            contentType = XContentType.JSON;
+        }
+        XContentBuilder builder = new XContentBuilder(XContentFactory.xContent(contentType), bytesOutput());
+        if (request.paramAsBoolean("pretty", false)) {
+            builder.prettyPrint().lfAtEnd();
+        }
+
+        builder.humanReadable(request.paramAsBoolean("human", builder.humanReadable()));
+
+        String casing = request.param("case");
+        if (casing != null && "camelCase".equals(casing)) {
+            builder.fieldCaseConversion(XContentBuilder.FieldCaseConversion.CAMELCASE);
+        } else {
+            // we expect all REST interfaces to write results in underscore casing, so
+            // no need for double casing
+            builder.fieldCaseConversion(XContentBuilder.FieldCaseConversion.NONE);
+        }
+        return builder;
+    }
+
+    /**
+     * A channel level bytes output that can be reused. It gets reset on each call to this
+     * method.
+     */
+    public final BytesStreamOutput bytesOutput() {
+        if (bytesOut == null) {
+            bytesOut = newBytesOutput();
+        } else {
+            bytesOut.reset();
+        }
+        return bytesOut;
+    }
+
+    protected BytesStreamOutput newBytesOutput() {
+        return new BytesStreamOutput();
+    }
+
+    public RestRequest request() {
+        return this.request;
+    }
 
-    void sendResponse(RestResponse response);
+    public abstract void sendResponse(RestResponse response);
 }
\ No newline at end of file
diff --git a/src/main/java/org/elasticsearch/rest/RestController.java b/src/main/java/org/elasticsearch/rest/RestController.java
index e57528b4df52b..19a4521479cbd 100644
--- a/src/main/java/org/elasticsearch/rest/RestController.java
+++ b/src/main/java/org/elasticsearch/rest/RestController.java
@@ -140,11 +140,11 @@ public void dispatchRequest(final RestRequest request, final RestChannel channel
         if (filters.length == 0) {
             try {
                 executeHandler(request, channel);
-            } catch (Exception e) {
+            } catch (Throwable e) {
                 try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response for uri [" + request.uri() + "]", e1);
+                    channel.sendResponse(new BytesRestResponse(channel, e));
+                } catch (Throwable e1) {
+                    logger.error("failed to send failure response for uri [" + request.uri() + "]", e1);
                 }
             }
         } else {
@@ -153,7 +153,7 @@ public void dispatchRequest(final RestRequest request, final RestChannel channel
         }
     }
 
-    void executeHandler(RestRequest request, RestChannel channel) {
+    void executeHandler(RestRequest request, RestChannel channel) throws Exception {
         final RestHandler handler = getHandler(request);
         if (handler != null) {
             handler.handleRequest(request, channel);
@@ -219,7 +219,7 @@ public void continueProcessing(RestRequest request, RestChannel channel) {
                 }
             } catch (Exception e) {
                 try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
+                    channel.sendResponse(new BytesRestResponse(channel, e));
                 } catch (IOException e1) {
                     logger.error("Failed to send failure response for uri [" + request.uri() + "]", e1);
                 }
@@ -230,7 +230,7 @@ public void continueProcessing(RestRequest request, RestChannel channel) {
     class RestHandlerFilter extends RestFilter {
 
         @Override
-        public void process(RestRequest request, RestChannel channel, RestFilterChain filterChain) {
+        public void process(RestRequest request, RestChannel channel, RestFilterChain filterChain) throws Exception {
             executeHandler(request, channel);
         }
     }
diff --git a/src/main/java/org/elasticsearch/rest/RestFilter.java b/src/main/java/org/elasticsearch/rest/RestFilter.java
index d9066f446119f..846a39c222dde 100644
--- a/src/main/java/org/elasticsearch/rest/RestFilter.java
+++ b/src/main/java/org/elasticsearch/rest/RestFilter.java
@@ -44,5 +44,5 @@ public void close() throws ElasticsearchException {
      * Process the rest request. Using the channel to send a response, or the filter chain to continue
      * processing the request.
      */
-    public abstract void process(RestRequest request, RestChannel channel, RestFilterChain filterChain);
+    public abstract void process(RestRequest request, RestChannel channel, RestFilterChain filterChain) throws Exception;
 }
diff --git a/src/main/java/org/elasticsearch/rest/RestHandler.java b/src/main/java/org/elasticsearch/rest/RestHandler.java
index c68e6b63a8167..3f53dcbc8c4fb 100644
--- a/src/main/java/org/elasticsearch/rest/RestHandler.java
+++ b/src/main/java/org/elasticsearch/rest/RestHandler.java
@@ -24,5 +24,5 @@
  */
 public interface RestHandler {
 
-    void handleRequest(RestRequest request, RestChannel channel);
+    void handleRequest(RestRequest request, RestChannel channel) throws Exception;
 }
\ No newline at end of file
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/cluster/health/RestClusterHealthAction.java b/src/main/java/org/elasticsearch/rest/action/admin/cluster/health/RestClusterHealthAction.java
index de961faea6908..e2978744c8096 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/cluster/health/RestClusterHealthAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/cluster/health/RestClusterHealthAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.admin.cluster.health;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.cluster.health.ClusterHealthRequest;
 import org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse;
 import org.elasticsearch.action.admin.cluster.health.ClusterHealthStatus;
@@ -27,15 +26,15 @@
 import org.elasticsearch.common.Strings;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.common.xcontent.XContentBuilder;
-import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
+import org.elasticsearch.rest.BaseRestHandler;
+import org.elasticsearch.rest.RestChannel;
+import org.elasticsearch.rest.RestController;
+import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.action.support.RestToXContentListener;
 
-import java.io.IOException;
 import java.util.Locale;
 
 import static org.elasticsearch.client.Requests.clusterHealthRequest;
-import static org.elasticsearch.rest.RestStatus.PRECONDITION_FAILED;
 
 /**
  *
@@ -55,55 +54,16 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         ClusterHealthRequest clusterHealthRequest = clusterHealthRequest(Strings.splitStringByCommaToArray(request.param("index")));
         clusterHealthRequest.local(request.paramAsBoolean("local", clusterHealthRequest.local()));
         clusterHealthRequest.listenerThreaded(false);
-        try {
-            clusterHealthRequest.masterNodeTimeout(request.paramAsTime("master_timeout", clusterHealthRequest.masterNodeTimeout()));
-            clusterHealthRequest.timeout(request.paramAsTime("timeout", clusterHealthRequest.timeout()));
-            String waitForStatus = request.param("wait_for_status");
-            if (waitForStatus != null) {
-                clusterHealthRequest.waitForStatus(ClusterHealthStatus.valueOf(waitForStatus.toUpperCase(Locale.ROOT)));
-            }
-            clusterHealthRequest.waitForRelocatingShards(request.paramAsInt("wait_for_relocating_shards", clusterHealthRequest.waitForRelocatingShards()));
-            clusterHealthRequest.waitForActiveShards(request.paramAsInt("wait_for_active_shards", clusterHealthRequest.waitForActiveShards()));
-            clusterHealthRequest.waitForNodes(request.param("wait_for_nodes", clusterHealthRequest.waitForNodes()));
-        } catch (Exception e) {
-            try {
-                XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                channel.sendResponse(new BytesRestResponse(PRECONDITION_FAILED, builder.startObject().field("error", e.getMessage()).endObject()));
-            } catch (IOException e1) {
-                logger.error("Failed to send failure response", e1);
-            }
-            return;
+        clusterHealthRequest.masterNodeTimeout(request.paramAsTime("master_timeout", clusterHealthRequest.masterNodeTimeout()));
+        clusterHealthRequest.timeout(request.paramAsTime("timeout", clusterHealthRequest.timeout()));
+        String waitForStatus = request.param("wait_for_status");
+        if (waitForStatus != null) {
+            clusterHealthRequest.waitForStatus(ClusterHealthStatus.valueOf(waitForStatus.toUpperCase(Locale.ROOT)));
         }
+        clusterHealthRequest.waitForRelocatingShards(request.paramAsInt("wait_for_relocating_shards", clusterHealthRequest.waitForRelocatingShards()));
+        clusterHealthRequest.waitForActiveShards(request.paramAsInt("wait_for_active_shards", clusterHealthRequest.waitForActiveShards()));
+        clusterHealthRequest.waitForNodes(request.param("wait_for_nodes", clusterHealthRequest.waitForNodes()));
 
-        client.admin().cluster().health(clusterHealthRequest, new ActionListener<ClusterHealthResponse>() {
-            @Override
-            public void onResponse(ClusterHealthResponse response) {
-                try {
-                    RestStatus status = RestStatus.OK;
-                    // not sure..., we handle the health API, so we are not unavailable
-                    // in any case, "/" should be used for
-                    //if (response.status() == ClusterHealthStatus.RED) {
-                    //    status = RestStatus.SERVICE_UNAVAILABLE;
-                    //}
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
-                    response.toXContent(builder, request);
-                    builder.endObject();
-
-                    channel.sendResponse(new BytesRestResponse(status, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
-            }
-        });
+        client.admin().cluster().health(clusterHealthRequest, new RestToXContentListener<ClusterHealthResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/cluster/node/hotthreads/RestNodesHotThreadsAction.java b/src/main/java/org/elasticsearch/rest/action/admin/cluster/node/hotthreads/RestNodesHotThreadsAction.java
index 85f7a8a20e37b..ab6dd5969b25e 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/cluster/node/hotthreads/RestNodesHotThreadsAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/cluster/node/hotthreads/RestNodesHotThreadsAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.admin.cluster.node.hotthreads;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.cluster.node.hotthreads.NodeHotThreads;
 import org.elasticsearch.action.admin.cluster.node.hotthreads.NodesHotThreadsRequest;
 import org.elasticsearch.action.admin.cluster.node.hotthreads.NodesHotThreadsResponse;
@@ -29,8 +28,7 @@
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.unit.TimeValue;
 import org.elasticsearch.rest.*;
-
-import java.io.IOException;
+import org.elasticsearch.rest.action.support.RestResponseListener;
 
 
 /**
@@ -59,29 +57,16 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         nodesHotThreadsRequest.type(request.param("type", nodesHotThreadsRequest.type()));
         nodesHotThreadsRequest.interval(TimeValue.parseTimeValue(request.param("interval"), nodesHotThreadsRequest.interval()));
         nodesHotThreadsRequest.snapshots(request.paramAsInt("snapshots", nodesHotThreadsRequest.snapshots()));
-        client.admin().cluster().nodesHotThreads(nodesHotThreadsRequest, new ActionListener<NodesHotThreadsResponse>() {
-            @Override
-            public void onResponse(NodesHotThreadsResponse response) {
-                try {
-                    StringBuilder sb = new StringBuilder();
-                    for (NodeHotThreads node : response) {
-                        sb.append("::: ").append(node.getNode().toString()).append("\n");
-                        Strings.spaceify(3, node.getHotThreads(), sb);
-                        sb.append('\n');
-                    }
-                    channel.sendResponse(new BytesRestResponse(RestStatus.OK, sb.toString()));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
+        client.admin().cluster().nodesHotThreads(nodesHotThreadsRequest, new RestResponseListener<NodesHotThreadsResponse>(channel) {
             @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
+            public RestResponse buildResponse(NodesHotThreadsResponse response) throws Exception {
+                StringBuilder sb = new StringBuilder();
+                for (NodeHotThreads node : response) {
+                    sb.append("::: ").append(node.getNode().toString()).append("\n");
+                    Strings.spaceify(3, node.getHotThreads(), sb);
+                    sb.append('\n');
                 }
+                return new BytesRestResponse(RestStatus.OK, sb.toString());
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/cluster/node/info/RestNodesInfoAction.java b/src/main/java/org/elasticsearch/rest/action/admin/cluster/node/info/RestNodesInfoAction.java
index 38eed71fa6890..2a9c086d72da1 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/cluster/node/info/RestNodesInfoAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/cluster/node/info/RestNodesInfoAction.java
@@ -20,7 +20,6 @@
 package org.elasticsearch.rest.action.admin.cluster.node.info;
 
 import com.google.common.collect.Sets;
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.cluster.node.info.NodesInfoRequest;
 import org.elasticsearch.action.admin.cluster.node.info.NodesInfoResponse;
 import org.elasticsearch.client.Client;
@@ -30,9 +29,8 @@
 import org.elasticsearch.common.settings.SettingsFilter;
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
-import java.io.IOException;
 import java.util.Set;
 
 import static org.elasticsearch.rest.RestRequest.Method.GET;
@@ -72,7 +70,7 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
             Set<String> metricsOrNodeIds = Strings.splitStringByCommaToSet(request.param("nodeId", "_all"));
             boolean isMetricsOnly = ALLOWED_METRICS.containsAll(metricsOrNodeIds);
             if (isMetricsOnly) {
-                nodeIds = new String[] { "_all" };
+                nodeIds = new String[]{"_all"};
                 metrics = metricsOrNodeIds;
             } else {
                 nodeIds = metricsOrNodeIds.toArray(new String[]{});
@@ -102,28 +100,15 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
             nodesInfoRequest.plugins(metrics.contains("plugins"));
         }
 
-        client.admin().cluster().nodesInfo(nodesInfoRequest, new ActionListener<NodesInfoResponse>() {
-            @Override
-            public void onResponse(NodesInfoResponse response) {
-                try {
-                    response.settingsFilter(settingsFilter);
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
-                    response.toXContent(builder, request);
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(RestStatus.OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
+        client.admin().cluster().nodesInfo(nodesInfoRequest, new RestBuilderListener<NodesInfoResponse>(channel) {
 
             @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
+            public RestResponse buildResponse(NodesInfoResponse response, XContentBuilder builder) throws Exception {
+                response.settingsFilter(settingsFilter);
+                builder.startObject();
+                response.toXContent(builder, request);
+                builder.endObject();
+                return new BytesRestResponse(RestStatus.OK, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/cluster/node/restart/RestNodesRestartAction.java b/src/main/java/org/elasticsearch/rest/action/admin/cluster/node/restart/RestNodesRestartAction.java
index ec37a1062cdd4..62f5bb8d310d7 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/cluster/node/restart/RestNodesRestartAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/cluster/node/restart/RestNodesRestartAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.admin.cluster.node.restart;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.cluster.node.restart.NodesRestartRequest;
 import org.elasticsearch.action.admin.cluster.node.restart.NodesRestartResponse;
 import org.elasticsearch.client.Client;
@@ -28,10 +27,7 @@
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
-
-import java.io.IOException;
-
-import static org.elasticsearch.rest.action.support.RestXContentBuilder.restContentBuilder;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
 /**
  *
@@ -52,36 +48,22 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         NodesRestartRequest nodesRestartRequest = new NodesRestartRequest(nodesIds);
         nodesRestartRequest.listenerThreaded(false);
         nodesRestartRequest.delay(request.paramAsTime("delay", nodesRestartRequest.delay()));
-        client.admin().cluster().nodesRestart(nodesRestartRequest, new ActionListener<NodesRestartResponse>() {
+        client.admin().cluster().nodesRestart(nodesRestartRequest, new RestBuilderListener<NodesRestartResponse>(channel) {
             @Override
-            public void onResponse(NodesRestartResponse result) {
-                try {
-                    XContentBuilder builder = restContentBuilder(request);
-                    builder.startObject();
-                    builder.field("cluster_name", result.getClusterName().value());
-
-                    builder.startObject("nodes");
-                    for (NodesRestartResponse.NodeRestartResponse nodeInfo : result) {
-                        builder.startObject(nodeInfo.getNode().id());
-                        builder.field("name", nodeInfo.getNode().name());
-                        builder.endObject();
-                    }
-                    builder.endObject();
+            public RestResponse buildResponse(NodesRestartResponse result, XContentBuilder builder) throws Exception {
+                builder.startObject();
+                builder.field("cluster_name", result.getClusterName().value());
 
+                builder.startObject("nodes");
+                for (NodesRestartResponse.NodeRestartResponse nodeInfo : result) {
+                    builder.startObject(nodeInfo.getNode().id());
+                    builder.field("name", nodeInfo.getNode().name());
                     builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(RestStatus.OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
                 }
-            }
+                builder.endObject();
 
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
+                builder.endObject();
+                return new BytesRestResponse(RestStatus.OK, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/cluster/node/shutdown/RestNodesShutdownAction.java b/src/main/java/org/elasticsearch/rest/action/admin/cluster/node/shutdown/RestNodesShutdownAction.java
index aafcfb37e2469..a98180a0d19a6 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/cluster/node/shutdown/RestNodesShutdownAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/cluster/node/shutdown/RestNodesShutdownAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.admin.cluster.node.shutdown;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.cluster.node.shutdown.NodesShutdownRequest;
 import org.elasticsearch.action.admin.cluster.node.shutdown.NodesShutdownResponse;
 import org.elasticsearch.client.Client;
@@ -29,10 +28,7 @@
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
-
-import java.io.IOException;
-
-import static org.elasticsearch.rest.action.support.RestXContentBuilder.restContentBuilder;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
 /**
  *
@@ -55,36 +51,22 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         nodesShutdownRequest.listenerThreaded(false);
         nodesShutdownRequest.delay(request.paramAsTime("delay", nodesShutdownRequest.delay()));
         nodesShutdownRequest.exit(request.paramAsBoolean("exit", nodesShutdownRequest.exit()));
-        client.admin().cluster().nodesShutdown(nodesShutdownRequest, new ActionListener<NodesShutdownResponse>() {
+        client.admin().cluster().nodesShutdown(nodesShutdownRequest, new RestBuilderListener<NodesShutdownResponse>(channel) {
             @Override
-            public void onResponse(NodesShutdownResponse response) {
-                try {
-                    XContentBuilder builder = restContentBuilder(request);
-                    builder.startObject();
-                    builder.field("cluster_name", response.getClusterName().value());
-
-                    builder.startObject("nodes");
-                    for (DiscoveryNode node : response.getNodes()) {
-                        builder.startObject(node.id(), XContentBuilder.FieldCaseConversion.NONE);
-                        builder.field("name", node.name(), XContentBuilder.FieldCaseConversion.NONE);
-                        builder.endObject();
-                    }
-                    builder.endObject();
+            public RestResponse buildResponse(NodesShutdownResponse response, XContentBuilder builder) throws Exception {
+                builder.startObject();
+                builder.field("cluster_name", response.getClusterName().value());
 
+                builder.startObject("nodes");
+                for (DiscoveryNode node : response.getNodes()) {
+                    builder.startObject(node.id(), XContentBuilder.FieldCaseConversion.NONE);
+                    builder.field("name", node.name(), XContentBuilder.FieldCaseConversion.NONE);
                     builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(RestStatus.OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
                 }
-            }
+                builder.endObject();
 
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
+                builder.endObject();
+                return new BytesRestResponse(RestStatus.OK, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/cluster/node/stats/RestNodesStatsAction.java b/src/main/java/org/elasticsearch/rest/action/admin/cluster/node/stats/RestNodesStatsAction.java
index 8e8513823279a..55cb1606e0cd6 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/cluster/node/stats/RestNodesStatsAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/cluster/node/stats/RestNodesStatsAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.admin.cluster.node.stats;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.cluster.node.stats.NodesStatsRequest;
 import org.elasticsearch.action.admin.cluster.node.stats.NodesStatsResponse;
 import org.elasticsearch.action.admin.indices.stats.CommonStatsFlags;
@@ -28,11 +27,12 @@
 import org.elasticsearch.common.Strings;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.common.xcontent.XContentBuilder;
-import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
+import org.elasticsearch.rest.BaseRestHandler;
+import org.elasticsearch.rest.RestChannel;
+import org.elasticsearch.rest.RestController;
+import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.action.support.RestToXContentListener;
 
-import java.io.IOException;
 import java.util.Set;
 
 import static org.elasticsearch.rest.RestRequest.Method.GET;
@@ -111,28 +111,6 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
             nodesStatsRequest.indices().types(request.paramAsStringArray("types", null));
         }
 
-        client.admin().cluster().nodesStats(nodesStatsRequest, new ActionListener<NodesStatsResponse>() {
-            @Override
-            public void onResponse(NodesStatsResponse response) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
-                    response.toXContent(builder, request);
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(RestStatus.OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
-            }
-        });
+        client.admin().cluster().nodesStats(nodesStatsRequest, new RestToXContentListener<NodesStatsResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/cluster/repositories/delete/RestDeleteRepositoryAction.java b/src/main/java/org/elasticsearch/rest/action/admin/cluster/repositories/delete/RestDeleteRepositoryAction.java
index 4df5c1ee91583..c3cb9d6778a54 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/cluster/repositories/delete/RestDeleteRepositoryAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/cluster/repositories/delete/RestDeleteRepositoryAction.java
@@ -25,6 +25,7 @@
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.action.support.AcknowledgedRestListener;
 
 import static org.elasticsearch.client.Requests.deleteRepositoryRequest;
 import static org.elasticsearch.rest.RestRequest.Method.DELETE;
@@ -47,6 +48,6 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         deleteRepositoryRequest.listenerThreaded(false);
         deleteRepositoryRequest.timeout(request.paramAsTime("timeout", deleteRepositoryRequest.timeout()));
         deleteRepositoryRequest.masterNodeTimeout(request.paramAsTime("master_timeout", deleteRepositoryRequest.masterNodeTimeout()));
-        client.admin().cluster().deleteRepository(deleteRepositoryRequest, new AcknowledgedRestResponseActionListener<DeleteRepositoryResponse>(request, channel, logger));
+        client.admin().cluster().deleteRepository(deleteRepositoryRequest, new AcknowledgedRestListener<DeleteRepositoryResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/cluster/repositories/get/RestGetRepositoriesAction.java b/src/main/java/org/elasticsearch/rest/action/admin/cluster/repositories/get/RestGetRepositoriesAction.java
index ed8824e5dcaaf..2187769f77e61 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/cluster/repositories/get/RestGetRepositoriesAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/cluster/repositories/get/RestGetRepositoriesAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.admin.cluster.repositories.get;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.cluster.repositories.get.GetRepositoriesRequest;
 import org.elasticsearch.action.admin.cluster.repositories.get.GetRepositoriesResponse;
 import org.elasticsearch.client.Client;
@@ -30,7 +29,7 @@
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
 import java.io.IOException;
 
@@ -56,30 +55,16 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         GetRepositoriesRequest getRepositoriesRequest = getRepositoryRequest(repositories);
         getRepositoriesRequest.masterNodeTimeout(request.paramAsTime("master_timeout", getRepositoriesRequest.masterNodeTimeout()));
         getRepositoriesRequest.local(request.paramAsBoolean("local", getRepositoriesRequest.local()));
-        client.admin().cluster().getRepositories(getRepositoriesRequest, new ActionListener<GetRepositoriesResponse>() {
+        client.admin().cluster().getRepositories(getRepositoriesRequest, new RestBuilderListener<GetRepositoriesResponse>(channel) {
             @Override
-            public void onResponse(GetRepositoriesResponse response) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
-                    for (RepositoryMetaData repositoryMetaData : response.repositories()) {
-                        RepositoriesMetaData.FACTORY.toXContent(repositoryMetaData, builder, request);
-                    }
-                    builder.endObject();
-
-                    channel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
+            public RestResponse buildResponse(GetRepositoriesResponse response, XContentBuilder builder) throws Exception {
+                builder.startObject();
+                for (RepositoryMetaData repositoryMetaData : response.repositories()) {
+                    RepositoriesMetaData.FACTORY.toXContent(repositoryMetaData, builder, request);
                 }
-            }
+                builder.endObject();
 
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
+                return new BytesRestResponse(OK, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/cluster/repositories/put/RestPutRepositoryAction.java b/src/main/java/org/elasticsearch/rest/action/admin/cluster/repositories/put/RestPutRepositoryAction.java
index 743803a192821..f6aa31c83e640 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/cluster/repositories/put/RestPutRepositoryAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/cluster/repositories/put/RestPutRepositoryAction.java
@@ -25,6 +25,7 @@
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.action.support.AcknowledgedRestListener;
 
 import static org.elasticsearch.client.Requests.putRepositoryRequest;
 import static org.elasticsearch.rest.RestRequest.Method.POST;
@@ -50,6 +51,6 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         putRepositoryRequest.source(request.content().toUtf8());
         putRepositoryRequest.masterNodeTimeout(request.paramAsTime("master_timeout", putRepositoryRequest.masterNodeTimeout()));
         putRepositoryRequest.timeout(request.paramAsTime("timeout", putRepositoryRequest.timeout()));
-        client.admin().cluster().putRepository(putRepositoryRequest, new AcknowledgedRestResponseActionListener<PutRepositoryResponse>(request, channel, logger));
+        client.admin().cluster().putRepository(putRepositoryRequest, new AcknowledgedRestListener<PutRepositoryResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/cluster/reroute/RestClusterRerouteAction.java b/src/main/java/org/elasticsearch/rest/action/admin/cluster/reroute/RestClusterRerouteAction.java
index b6719f55bc1b2..48344c3ee7835 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/cluster/reroute/RestClusterRerouteAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/cluster/reroute/RestClusterRerouteAction.java
@@ -29,6 +29,7 @@
 import org.elasticsearch.common.xcontent.ToXContent;
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.action.support.AcknowledgedRestListener;
 
 import java.io.IOException;
 
@@ -47,7 +48,7 @@ public RestClusterRerouteAction(Settings settings, Client client, RestController
     }
 
     @Override
-    public void handleRequest(final RestRequest request, final RestChannel channel) {
+    public void handleRequest(final RestRequest request, final RestChannel channel) throws Exception {
         final ClusterRerouteRequest clusterRerouteRequest = Requests.clusterRerouteRequest();
         clusterRerouteRequest.listenerThreaded(false);
         clusterRerouteRequest.dryRun(request.paramAsBoolean("dry_run", clusterRerouteRequest.dryRun()));
@@ -55,19 +56,10 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         clusterRerouteRequest.timeout(request.paramAsTime("timeout", clusterRerouteRequest.timeout()));
         clusterRerouteRequest.masterNodeTimeout(request.paramAsTime("master_timeout", clusterRerouteRequest.masterNodeTimeout()));
         if (request.hasContent()) {
-            try {
-                clusterRerouteRequest.source(request.content());
-            } catch (Exception e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.warn("Failed to send response", e1);
-                }
-                return;
-            }
+            clusterRerouteRequest.source(request.content());
         }
 
-        client.admin().cluster().reroute(clusterRerouteRequest, new AcknowledgedRestResponseActionListener<ClusterRerouteResponse>(request, channel, logger) {
+        client.admin().cluster().reroute(clusterRerouteRequest, new AcknowledgedRestListener<ClusterRerouteResponse>(channel) {
             @Override
             protected void addCustomFields(XContentBuilder builder, ClusterRerouteResponse response) throws IOException {
                 builder.startObject("state");
@@ -82,14 +74,6 @@ protected void addCustomFields(XContentBuilder builder, ClusterRerouteResponse r
                     response.getExplanations().toXContent(builder, ToXContent.EMPTY_PARAMS);
                 }
             }
-
-            @Override
-            public void onFailure(Throwable e) {
-                if (logger.isDebugEnabled()) {
-                    logger.debug("failed to handle cluster reroute", e);
-                }
-                super.onFailure(e);
-            }
         });
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/cluster/settings/RestClusterGetSettingsAction.java b/src/main/java/org/elasticsearch/rest/action/admin/cluster/settings/RestClusterGetSettingsAction.java
index 37eb82cc08716..3f7341a02a2e2 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/cluster/settings/RestClusterGetSettingsAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/cluster/settings/RestClusterGetSettingsAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.admin.cluster.settings;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.cluster.state.ClusterStateRequest;
 import org.elasticsearch.action.admin.cluster.state.ClusterStateResponse;
 import org.elasticsearch.client.Client;
@@ -28,7 +27,7 @@
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
 import java.io.IOException;
 
@@ -49,36 +48,22 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
                 .routingTable(false)
                 .nodes(false);
         clusterStateRequest.local(request.paramAsBoolean("local", clusterStateRequest.local()));
-        client.admin().cluster().state(clusterStateRequest, new ActionListener<ClusterStateResponse>() {
+        client.admin().cluster().state(clusterStateRequest, new RestBuilderListener<ClusterStateResponse>(channel) {
             @Override
-            public void onResponse(ClusterStateResponse response) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
+            public RestResponse buildResponse(ClusterStateResponse response, XContentBuilder builder) throws Exception {
+                builder.startObject();
 
-                    builder.startObject("persistent");
-                    response.getState().metaData().persistentSettings().toXContent(builder, request);
-                    builder.endObject();
+                builder.startObject("persistent");
+                response.getState().metaData().persistentSettings().toXContent(builder, request);
+                builder.endObject();
 
-                    builder.startObject("transient");
-                    response.getState().metaData().transientSettings().toXContent(builder, request);
-                    builder.endObject();
+                builder.startObject("transient");
+                response.getState().metaData().transientSettings().toXContent(builder, request);
+                builder.endObject();
 
-                    builder.endObject();
+                builder.endObject();
 
-                    channel.sendResponse(new BytesRestResponse(RestStatus.OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
+                return new BytesRestResponse(RestStatus.OK, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/cluster/settings/RestClusterUpdateSettingsAction.java b/src/main/java/org/elasticsearch/rest/action/admin/cluster/settings/RestClusterUpdateSettingsAction.java
index 126e50a1fb085..3dc9a502ff125 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/cluster/settings/RestClusterUpdateSettingsAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/cluster/settings/RestClusterUpdateSettingsAction.java
@@ -28,6 +28,7 @@
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.common.xcontent.XContentFactory;
 import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.action.support.AcknowledgedRestListener;
 
 import java.io.IOException;
 import java.util.Map;
@@ -43,30 +44,20 @@ public RestClusterUpdateSettingsAction(Settings settings, Client client, RestCon
     }
 
     @Override
-    public void handleRequest(final RestRequest request, final RestChannel channel) {
+    public void handleRequest(final RestRequest request, final RestChannel channel) throws Exception {
         final ClusterUpdateSettingsRequest clusterUpdateSettingsRequest = Requests.clusterUpdateSettingsRequest();
         clusterUpdateSettingsRequest.listenerThreaded(false);
         clusterUpdateSettingsRequest.timeout(request.paramAsTime("timeout", clusterUpdateSettingsRequest.timeout()));
         clusterUpdateSettingsRequest.masterNodeTimeout(request.paramAsTime("master_timeout", clusterUpdateSettingsRequest.masterNodeTimeout()));
-        try {
-            Map<String, Object> source = XContentFactory.xContent(request.content()).createParser(request.content()).mapAndClose();
-            if (source.containsKey("transient")) {
-                clusterUpdateSettingsRequest.transientSettings((Map) source.get("transient"));
-            }
-            if (source.containsKey("persistent")) {
-                clusterUpdateSettingsRequest.persistentSettings((Map) source.get("persistent"));
-            }
-        } catch (Exception e) {
-            try {
-                channel.sendResponse(new BytesRestResponse(request, e));
-            } catch (IOException e1) {
-                logger.warn("Failed to send response", e1);
-            }
-            return;
+        Map<String, Object> source = XContentFactory.xContent(request.content()).createParser(request.content()).mapAndClose();
+        if (source.containsKey("transient")) {
+            clusterUpdateSettingsRequest.transientSettings((Map) source.get("transient"));
+        }
+        if (source.containsKey("persistent")) {
+            clusterUpdateSettingsRequest.persistentSettings((Map) source.get("persistent"));
         }
 
-        client.admin().cluster().updateSettings(clusterUpdateSettingsRequest, new AcknowledgedRestResponseActionListener<ClusterUpdateSettingsResponse>(request, channel, logger) {
-
+        client.admin().cluster().updateSettings(clusterUpdateSettingsRequest, new AcknowledgedRestListener<ClusterUpdateSettingsResponse>(channel) {
             @Override
             protected void addCustomFields(XContentBuilder builder, ClusterUpdateSettingsResponse response) throws IOException {
                 builder.startObject("persistent");
@@ -77,14 +68,6 @@ protected void addCustomFields(XContentBuilder builder, ClusterUpdateSettingsRes
                 response.getTransientSettings().toXContent(builder, request);
                 builder.endObject();
             }
-
-            @Override
-            public void onFailure(Throwable e) {
-                if (logger.isDebugEnabled()) {
-                    logger.debug("failed to handle cluster state", e);
-                }
-                super.onFailure(e);
-            }
         });
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/cluster/shards/RestClusterSearchShardsAction.java b/src/main/java/org/elasticsearch/rest/action/admin/cluster/shards/RestClusterSearchShardsAction.java
index e21b5b690dc7c..66d0f93c7f159 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/cluster/shards/RestClusterSearchShardsAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/cluster/shards/RestClusterSearchShardsAction.java
@@ -30,7 +30,7 @@
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
+import org.elasticsearch.rest.action.support.RestToXContentListener;
 
 import java.io.IOException;
 
@@ -64,28 +64,6 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         clusterSearchShardsRequest.preference(request.param("preference"));
         clusterSearchShardsRequest.indicesOptions(IndicesOptions.fromRequest(request, clusterSearchShardsRequest.indicesOptions()));
 
-        client.admin().cluster().searchShards(clusterSearchShardsRequest, new ActionListener<ClusterSearchShardsResponse>() {
-            @Override
-            public void onResponse(ClusterSearchShardsResponse response) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
-                    response.toXContent(builder, request);
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(RestStatus.OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
-            }
-        });
+        client.admin().cluster().searchShards(clusterSearchShardsRequest, new RestToXContentListener<ClusterSearchShardsResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/cluster/snapshots/create/RestCreateSnapshotAction.java b/src/main/java/org/elasticsearch/rest/action/admin/cluster/snapshots/create/RestCreateSnapshotAction.java
index c26a9307e820a..8cc3ee07b7246 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/cluster/snapshots/create/RestCreateSnapshotAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/cluster/snapshots/create/RestCreateSnapshotAction.java
@@ -19,17 +19,16 @@
 
 package org.elasticsearch.rest.action.admin.cluster.snapshots.create;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.cluster.snapshots.create.CreateSnapshotRequest;
 import org.elasticsearch.action.admin.cluster.snapshots.create.CreateSnapshotResponse;
 import org.elasticsearch.client.Client;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.common.xcontent.XContentBuilder;
-import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
-
-import java.io.IOException;
+import org.elasticsearch.rest.BaseRestHandler;
+import org.elasticsearch.rest.RestChannel;
+import org.elasticsearch.rest.RestController;
+import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.action.support.RestToXContentListener;
 
 import static org.elasticsearch.client.Requests.createSnapshotRequest;
 import static org.elasticsearch.rest.RestRequest.Method.POST;
@@ -54,29 +53,6 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         createSnapshotRequest.source(request.content().toUtf8());
         createSnapshotRequest.masterNodeTimeout(request.paramAsTime("master_timeout", createSnapshotRequest.masterNodeTimeout()));
         createSnapshotRequest.waitForCompletion(request.paramAsBoolean("wait_for_completion", false));
-        client.admin().cluster().createSnapshot(createSnapshotRequest, new ActionListener<CreateSnapshotResponse>() {
-            @Override
-            public void onResponse(CreateSnapshotResponse response) {
-                try {
-
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
-                    response.toXContent(builder, request);
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(response.status(), builder));
-                } catch (IOException e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
-            }
-        });
+        client.admin().cluster().createSnapshot(createSnapshotRequest, new RestToXContentListener<CreateSnapshotResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/cluster/snapshots/delete/RestDeleteSnapshotAction.java b/src/main/java/org/elasticsearch/rest/action/admin/cluster/snapshots/delete/RestDeleteSnapshotAction.java
index 9b69fd6f66ac0..36dc88c44559a 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/cluster/snapshots/delete/RestDeleteSnapshotAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/cluster/snapshots/delete/RestDeleteSnapshotAction.java
@@ -19,21 +19,16 @@
 
 package org.elasticsearch.rest.action.admin.cluster.snapshots.delete;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.cluster.snapshots.delete.DeleteSnapshotRequest;
 import org.elasticsearch.action.admin.cluster.snapshots.delete.DeleteSnapshotResponse;
 import org.elasticsearch.client.Client;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
-
-import java.io.IOException;
+import org.elasticsearch.rest.action.support.AcknowledgedRestListener;
 
 import static org.elasticsearch.client.Requests.deleteSnapshotRequest;
 import static org.elasticsearch.rest.RestRequest.Method.DELETE;
-import static org.elasticsearch.rest.RestStatus.OK;
 
 /**
  * Deletes a snapshot
@@ -50,6 +45,6 @@ public RestDeleteSnapshotAction(Settings settings, Client client, RestController
     public void handleRequest(final RestRequest request, final RestChannel channel) {
         DeleteSnapshotRequest deleteSnapshotRequest = deleteSnapshotRequest(request.param("repository"), request.param("snapshot"));
         deleteSnapshotRequest.masterNodeTimeout(request.paramAsTime("master_timeout", deleteSnapshotRequest.masterNodeTimeout()));
-        client.admin().cluster().deleteSnapshot(deleteSnapshotRequest, new AcknowledgedRestResponseActionListener<DeleteSnapshotResponse>(request, channel, logger));
+        client.admin().cluster().deleteSnapshot(deleteSnapshotRequest, new AcknowledgedRestListener<DeleteSnapshotResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/cluster/snapshots/get/RestGetSnapshotsAction.java b/src/main/java/org/elasticsearch/rest/action/admin/cluster/snapshots/get/RestGetSnapshotsAction.java
index af9970dad4f80..4c0f92e75c3c0 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/cluster/snapshots/get/RestGetSnapshotsAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/cluster/snapshots/get/RestGetSnapshotsAction.java
@@ -19,22 +19,20 @@
 
 package org.elasticsearch.rest.action.admin.cluster.snapshots.get;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.cluster.snapshots.get.GetSnapshotsRequest;
 import org.elasticsearch.action.admin.cluster.snapshots.get.GetSnapshotsResponse;
 import org.elasticsearch.client.Client;
 import org.elasticsearch.common.Strings;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.common.xcontent.XContentBuilder;
-import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
-
-import java.io.IOException;
+import org.elasticsearch.rest.BaseRestHandler;
+import org.elasticsearch.rest.RestChannel;
+import org.elasticsearch.rest.RestController;
+import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.action.support.RestToXContentListener;
 
 import static org.elasticsearch.client.Requests.getSnapshotsRequest;
 import static org.elasticsearch.rest.RestRequest.Method.GET;
-import static org.elasticsearch.rest.RestStatus.OK;
 
 /**
  * Returns information about snapshot
@@ -57,28 +55,6 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         }
         GetSnapshotsRequest getSnapshotsRequest = getSnapshotsRequest(repository).snapshots(snapshots);
         getSnapshotsRequest.masterNodeTimeout(request.paramAsTime("master_timeout", getSnapshotsRequest.masterNodeTimeout()));
-        client.admin().cluster().getSnapshots(getSnapshotsRequest, new ActionListener<GetSnapshotsResponse>() {
-            @Override
-            public void onResponse(GetSnapshotsResponse response) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
-                    response.toXContent(builder, request);
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (IOException e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
-            }
-        });
+        client.admin().cluster().getSnapshots(getSnapshotsRequest, new RestToXContentListener<GetSnapshotsResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/cluster/snapshots/restore/RestRestoreSnapshotAction.java b/src/main/java/org/elasticsearch/rest/action/admin/cluster/snapshots/restore/RestRestoreSnapshotAction.java
index 3593242247cba..059c6dd17cb75 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/cluster/snapshots/restore/RestRestoreSnapshotAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/cluster/snapshots/restore/RestRestoreSnapshotAction.java
@@ -19,21 +19,19 @@
 
 package org.elasticsearch.rest.action.admin.cluster.snapshots.restore;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.cluster.snapshots.restore.RestoreSnapshotRequest;
 import org.elasticsearch.action.admin.cluster.snapshots.restore.RestoreSnapshotResponse;
 import org.elasticsearch.client.Client;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.common.xcontent.XContentBuilder;
-import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
-
-import java.io.IOException;
+import org.elasticsearch.rest.BaseRestHandler;
+import org.elasticsearch.rest.RestChannel;
+import org.elasticsearch.rest.RestController;
+import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.action.support.RestToXContentListener;
 
 import static org.elasticsearch.client.Requests.restoreSnapshotRequest;
 import static org.elasticsearch.rest.RestRequest.Method.POST;
-import static org.elasticsearch.rest.RestStatus.OK;
 
 /**
  * Restores a snapshot
@@ -52,28 +50,6 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         restoreSnapshotRequest.masterNodeTimeout(request.paramAsTime("master_timeout", restoreSnapshotRequest.masterNodeTimeout()));
         restoreSnapshotRequest.waitForCompletion(request.paramAsBoolean("wait_for_completion", false));
         restoreSnapshotRequest.source(request.content().toUtf8());
-        client.admin().cluster().restoreSnapshot(restoreSnapshotRequest, new ActionListener<RestoreSnapshotResponse>() {
-            @Override
-            public void onResponse(RestoreSnapshotResponse response) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
-                    response.toXContent(builder, request);
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (IOException e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
-            }
-        });
+        client.admin().cluster().restoreSnapshot(restoreSnapshotRequest, new RestToXContentListener<RestoreSnapshotResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/cluster/snapshots/status/RestSnapshotsStatusAction.java b/src/main/java/org/elasticsearch/rest/action/admin/cluster/snapshots/status/RestSnapshotsStatusAction.java
index ac798f549cd7e..a4afab5622fe1 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/cluster/snapshots/status/RestSnapshotsStatusAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/cluster/snapshots/status/RestSnapshotsStatusAction.java
@@ -19,22 +19,20 @@
 
 package org.elasticsearch.rest.action.admin.cluster.snapshots.status;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.cluster.snapshots.status.SnapshotsStatusRequest;
 import org.elasticsearch.action.admin.cluster.snapshots.status.SnapshotsStatusResponse;
 import org.elasticsearch.client.Client;
 import org.elasticsearch.common.Strings;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.common.xcontent.XContentBuilder;
-import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
-
-import java.io.IOException;
+import org.elasticsearch.rest.BaseRestHandler;
+import org.elasticsearch.rest.RestChannel;
+import org.elasticsearch.rest.RestController;
+import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.action.support.RestToXContentListener;
 
 import static org.elasticsearch.client.Requests.snapshotsStatusRequest;
 import static org.elasticsearch.rest.RestRequest.Method.GET;
-import static org.elasticsearch.rest.RestStatus.OK;
 
 /**
  * Returns status of currently running snapshot
@@ -59,19 +57,6 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         SnapshotsStatusRequest snapshotsStatusResponse = snapshotsStatusRequest(repository).snapshots(snapshots);
 
         snapshotsStatusResponse.masterNodeTimeout(request.paramAsTime("master_timeout", snapshotsStatusResponse.masterNodeTimeout()));
-        client.admin().cluster().snapshotsStatus(snapshotsStatusResponse, new AbstractRestResponseActionListener<SnapshotsStatusResponse>(request, channel, logger) {
-            @Override
-            public void onResponse(SnapshotsStatusResponse response) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
-                    response.toXContent(builder, request);
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (IOException e) {
-                    onFailure(e);
-                }
-            }
-        });
+        client.admin().cluster().snapshotsStatus(snapshotsStatusResponse, new RestToXContentListener<SnapshotsStatusResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/cluster/state/RestClusterStateAction.java b/src/main/java/org/elasticsearch/rest/action/admin/cluster/state/RestClusterStateAction.java
index 2465fd4d3ad51..edcf327f3e9f0 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/cluster/state/RestClusterStateAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/cluster/state/RestClusterStateAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.admin.cluster.state;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.cluster.state.ClusterStateRequest;
 import org.elasticsearch.action.admin.cluster.state.ClusterStateResponse;
 import org.elasticsearch.client.Client;
@@ -31,7 +30,7 @@
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.common.xcontent.XContentBuilderString;
 import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
 import java.io.IOException;
 import java.util.Set;
@@ -78,31 +77,14 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
             clusterStateRequest.indexTemplates(request.paramAsStringArray("index_templates", Strings.EMPTY_ARRAY));
         }
 
-        client.admin().cluster().state(clusterStateRequest, new ActionListener<ClusterStateResponse>() {
+        client.admin().cluster().state(clusterStateRequest, new RestBuilderListener<ClusterStateResponse>(channel) {
             @Override
-            public void onResponse(ClusterStateResponse response) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
-                    builder.field(Fields.CLUSTER_NAME, response.getClusterName().value());
-                    response.getState().settingsFilter(settingsFilter).toXContent(builder, request);
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(RestStatus.OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                if (logger.isDebugEnabled()) {
-                    logger.debug("failed to handle cluster state", e);
-                }
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
+            public RestResponse buildResponse(ClusterStateResponse response, XContentBuilder builder) throws Exception {
+                builder.startObject();
+                builder.field(Fields.CLUSTER_NAME, response.getClusterName().value());
+                response.getState().settingsFilter(settingsFilter).toXContent(builder, request);
+                builder.endObject();
+                return new BytesRestResponse(RestStatus.OK, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/cluster/stats/RestClusterStatsAction.java b/src/main/java/org/elasticsearch/rest/action/admin/cluster/stats/RestClusterStatsAction.java
index b8058a275bf26..bd02edf9099ca 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/cluster/stats/RestClusterStatsAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/cluster/stats/RestClusterStatsAction.java
@@ -19,17 +19,16 @@
 
 package org.elasticsearch.rest.action.admin.cluster.stats;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.cluster.stats.ClusterStatsRequest;
 import org.elasticsearch.action.admin.cluster.stats.ClusterStatsResponse;
 import org.elasticsearch.client.Client;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.common.xcontent.XContentBuilder;
-import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
-
-import java.io.IOException;
+import org.elasticsearch.rest.BaseRestHandler;
+import org.elasticsearch.rest.RestChannel;
+import org.elasticsearch.rest.RestController;
+import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.action.support.RestToXContentListener;
 
 
 /**
@@ -48,28 +47,6 @@ public RestClusterStatsAction(Settings settings, Client client, RestController c
     public void handleRequest(final RestRequest request, final RestChannel channel) {
         ClusterStatsRequest clusterStatsRequest = new ClusterStatsRequest().nodesIds(request.paramAsStringArray("nodeId", null));
         clusterStatsRequest.listenerThreaded(false);
-        client.admin().cluster().clusterStats(clusterStatsRequest, new ActionListener<ClusterStatsResponse>() {
-            @Override
-            public void onResponse(ClusterStatsResponse response) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
-                    response.toXContent(builder, request);
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(RestStatus.OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
-            }
-        });
+        client.admin().cluster().clusterStats(clusterStatsRequest, new RestToXContentListener<ClusterStatsResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/cluster/tasks/RestPendingClusterTasksAction.java b/src/main/java/org/elasticsearch/rest/action/admin/cluster/tasks/RestPendingClusterTasksAction.java
index 9f72daf9cbc00..d2450e0282e71 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/cluster/tasks/RestPendingClusterTasksAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/cluster/tasks/RestPendingClusterTasksAction.java
@@ -19,17 +19,16 @@
 
 package org.elasticsearch.rest.action.admin.cluster.tasks;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.cluster.tasks.PendingClusterTasksRequest;
 import org.elasticsearch.action.admin.cluster.tasks.PendingClusterTasksResponse;
 import org.elasticsearch.client.Client;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.common.xcontent.XContentBuilder;
-import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
-
-import java.io.IOException;
+import org.elasticsearch.rest.BaseRestHandler;
+import org.elasticsearch.rest.RestChannel;
+import org.elasticsearch.rest.RestController;
+import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.action.support.RestToXContentListener;
 
 /**
  */
@@ -46,32 +45,6 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         PendingClusterTasksRequest pendingClusterTasksRequest = new PendingClusterTasksRequest();
         pendingClusterTasksRequest.masterNodeTimeout(request.paramAsTime("master_timeout", pendingClusterTasksRequest.masterNodeTimeout()));
         pendingClusterTasksRequest.local(request.paramAsBoolean("local", pendingClusterTasksRequest.local()));
-        client.admin().cluster().pendingClusterTasks(pendingClusterTasksRequest, new ActionListener<PendingClusterTasksResponse>() {
-
-            @Override
-            public void onResponse(PendingClusterTasksResponse response) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
-                    response.toXContent(builder, request);
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(RestStatus.OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                if (logger.isDebugEnabled()) {
-                    logger.debug("failed to get pending cluster tasks", e);
-                }
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
-            }
-        });
+        client.admin().cluster().pendingClusterTasks(pendingClusterTasksRequest, new RestToXContentListener<PendingClusterTasksResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/RestIndicesAliasesAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/RestIndicesAliasesAction.java
index 83a86d52738f5..a44f370c47e7b 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/RestIndicesAliasesAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/RestIndicesAliasesAction.java
@@ -29,8 +29,8 @@
 import org.elasticsearch.common.xcontent.XContentFactory;
 import org.elasticsearch.common.xcontent.XContentParser;
 import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.action.support.AcknowledgedRestListener;
 
-import java.io.IOException;
 import java.util.Map;
 
 import static org.elasticsearch.cluster.metadata.AliasAction.newAddAliasAction;
@@ -48,7 +48,7 @@ public RestIndicesAliasesAction(Settings settings, Client client, RestController
     }
 
     @Override
-    public void handleRequest(final RestRequest request, final RestChannel channel) {
+    public void handleRequest(final RestRequest request, final RestChannel channel) throws Exception {
         IndicesAliasesRequest indicesAliasesRequest = new IndicesAliasesRequest();
         indicesAliasesRequest.listenerThreaded(false);
         indicesAliasesRequest.masterNodeTimeout(request.paramAsTime("master_timeout", indicesAliasesRequest.masterNodeTimeout()));
@@ -131,14 +131,7 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
                     }
                 }
             }
-        } catch (Exception e) {
-            try {
-                channel.sendResponse(new BytesRestResponse(request, e));
-            } catch (IOException e1) {
-                logger.warn("Failed to send response", e1);
-            }
-            return;
         }
-        client.admin().indices().aliases(indicesAliasesRequest, new AcknowledgedRestResponseActionListener<IndicesAliasesResponse>(request, channel, logger));
+        client.admin().indices().aliases(indicesAliasesRequest, new AcknowledgedRestListener<IndicesAliasesResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/delete/RestIndexDeleteAliasesAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/delete/RestIndexDeleteAliasesAction.java
index 8cbd9d7709265..f5896d6b891ed 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/delete/RestIndexDeleteAliasesAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/delete/RestIndexDeleteAliasesAction.java
@@ -25,6 +25,7 @@
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.action.support.AcknowledgedRestListener;
 
 import static org.elasticsearch.rest.RestRequest.Method.DELETE;
 
@@ -48,6 +49,6 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         indicesAliasesRequest.removeAlias(indices, aliases);
         indicesAliasesRequest.masterNodeTimeout(request.paramAsTime("master_timeout", indicesAliasesRequest.masterNodeTimeout()));
 
-        client.admin().indices().aliases(indicesAliasesRequest, new AcknowledgedRestResponseActionListener<IndicesAliasesResponse>(request, channel, logger));
+        client.admin().indices().aliases(indicesAliasesRequest, new AcknowledgedRestListener<IndicesAliasesResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/get/RestGetAliasesAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/get/RestGetAliasesAction.java
index 57db88f150bdc..14d1676b6cc9c 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/get/RestGetAliasesAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/get/RestGetAliasesAction.java
@@ -20,7 +20,6 @@
 package org.elasticsearch.rest.action.admin.indices.alias.get;
 
 import com.carrotsearch.hppc.cursors.ObjectObjectCursor;
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.indices.alias.get.GetAliasesRequest;
 import org.elasticsearch.action.admin.indices.alias.get.GetAliasesResponse;
 import org.elasticsearch.action.support.IndicesOptions;
@@ -33,9 +32,8 @@
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.common.xcontent.XContentBuilderString;
 import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
-import java.io.IOException;
 import java.util.List;
 import java.util.Locale;
 
@@ -64,50 +62,33 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         getAliasesRequest.indicesOptions(IndicesOptions.fromRequest(request, getAliasesRequest.indicesOptions()));
         getAliasesRequest.local(request.paramAsBoolean("local", getAliasesRequest.local()));
 
-        client.admin().indices().getAliases(getAliasesRequest, new ActionListener<GetAliasesResponse>() {
-
+        client.admin().indices().getAliases(getAliasesRequest, new RestBuilderListener<GetAliasesResponse>(channel) {
             @Override
-            public void onResponse(GetAliasesResponse response) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    // empty body, if indices were specified but no aliases were
-                    if (indices.length > 0 && response.getAliases().isEmpty()) {
-                        channel.sendResponse(new BytesRestResponse(OK, RestXContentBuilder.emptyBuilder(request)));
-                        return;
-                    } else if (response.getAliases().isEmpty()) {
-                        String message = String.format(Locale.ROOT, "alias [%s] missing", toNamesString(getAliasesRequest.aliases()));
-                        builder.startObject()
-                                .field("error", message)
-                                .field("status", RestStatus.NOT_FOUND.getStatus())
-                                .endObject();
-                        channel.sendResponse(new BytesRestResponse(RestStatus.NOT_FOUND, builder));
-                        return;
-                    }
+            public RestResponse buildResponse(GetAliasesResponse response, XContentBuilder builder) throws Exception {
+                // empty body, if indices were specified but no aliases were
+                if (indices.length > 0 && response.getAliases().isEmpty()) {
+                    return new BytesRestResponse(OK, builder.startObject().endObject());
+                } else if (response.getAliases().isEmpty()) {
+                    String message = String.format(Locale.ROOT, "alias [%s] missing", toNamesString(getAliasesRequest.aliases()));
+                    builder.startObject()
+                            .field("error", message)
+                            .field("status", RestStatus.NOT_FOUND.getStatus())
+                            .endObject();
+                    return new BytesRestResponse(RestStatus.NOT_FOUND, builder);
+                }
 
-                    builder.startObject();
-                    for (ObjectObjectCursor<String, List<AliasMetaData>> entry : response.getAliases()) {
-                        builder.startObject(entry.key, XContentBuilder.FieldCaseConversion.NONE);
-                        builder.startObject(Fields.ALIASES);
-                        for (AliasMetaData alias : entry.value) {
-                            AliasMetaData.Builder.toXContent(alias, builder, ToXContent.EMPTY_PARAMS);
-                        }
-                        builder.endObject();
-                        builder.endObject();
+                builder.startObject();
+                for (ObjectObjectCursor<String, List<AliasMetaData>> entry : response.getAliases()) {
+                    builder.startObject(entry.key, XContentBuilder.FieldCaseConversion.NONE);
+                    builder.startObject(Fields.ALIASES);
+                    for (AliasMetaData alias : entry.value) {
+                        AliasMetaData.Builder.toXContent(alias, builder, ToXContent.EMPTY_PARAMS);
                     }
                     builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
+                    builder.endObject();
                 }
+                builder.endObject();
+                return new BytesRestResponse(OK, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/get/RestGetIndicesAliasesAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/get/RestGetIndicesAliasesAction.java
index d0bb8765e3e0b..0b6f8dbdce3af 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/get/RestGetIndicesAliasesAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/get/RestGetIndicesAliasesAction.java
@@ -19,7 +19,6 @@
 package org.elasticsearch.rest.action.admin.indices.alias.get;
 
 import com.carrotsearch.hppc.cursors.ObjectCursor;
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.cluster.state.ClusterStateRequest;
 import org.elasticsearch.action.admin.cluster.state.ClusterStateResponse;
 import org.elasticsearch.client.Client;
@@ -34,9 +33,7 @@
 import org.elasticsearch.common.xcontent.ToXContent;
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
-
-import java.io.IOException;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
 import static org.elasticsearch.common.Strings.isAllOrWildcard;
 import static org.elasticsearch.rest.RestRequest.Method.GET;
@@ -62,49 +59,35 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         final String[] aliases = Strings.splitStringByCommaToArray(request.param("name"));
 
         ClusterStateRequest clusterStateRequest = Requests.clusterStateRequest()
-                                .routingTable(false)
-                                .nodes(false)
-                                .indices(indices);
+                .routingTable(false)
+                .nodes(false)
+                .indices(indices);
         clusterStateRequest.local(request.paramAsBoolean("local", clusterStateRequest.local()));
         clusterStateRequest.listenerThreaded(false);
 
-        client.admin().cluster().state(clusterStateRequest, new ActionListener<ClusterStateResponse>() {
+        client.admin().cluster().state(clusterStateRequest, new RestBuilderListener<ClusterStateResponse>(channel) {
             @Override
-            public void onResponse(ClusterStateResponse response) {
-                try {
-                    MetaData metaData = response.getState().metaData();
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
+            public RestResponse buildResponse(ClusterStateResponse response, XContentBuilder builder) throws Exception {
+                MetaData metaData = response.getState().metaData();
+                builder.startObject();
 
-                    final boolean isAllAliasesRequested = isAllOrWildcard(aliases);
-                    for (IndexMetaData indexMetaData : metaData) {
-                        builder.startObject(indexMetaData.index(), XContentBuilder.FieldCaseConversion.NONE);
-                        builder.startObject("aliases");
+                final boolean isAllAliasesRequested = isAllOrWildcard(aliases);
+                for (IndexMetaData indexMetaData : metaData) {
+                    builder.startObject(indexMetaData.index(), XContentBuilder.FieldCaseConversion.NONE);
+                    builder.startObject("aliases");
 
-                        for (ObjectCursor<AliasMetaData> cursor : indexMetaData.aliases().values()) {
-                            if (isAllAliasesRequested || Regex.simpleMatch(aliases, cursor.value.alias())) {
-                                AliasMetaData.Builder.toXContent(cursor.value, builder, ToXContent.EMPTY_PARAMS);
-                            }
+                    for (ObjectCursor<AliasMetaData> cursor : indexMetaData.aliases().values()) {
+                        if (isAllAliasesRequested || Regex.simpleMatch(aliases, cursor.value.alias())) {
+                            AliasMetaData.Builder.toXContent(cursor.value, builder, ToXContent.EMPTY_PARAMS);
                         }
-
-                        builder.endObject();
-                        builder.endObject();
                     }
 
                     builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
+                    builder.endObject();
                 }
-            }
 
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
+                builder.endObject();
+                return new BytesRestResponse(OK, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/put/RestIndexPutAliasAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/put/RestIndexPutAliasAction.java
index 0d6a7b9101cf9..9d46a0c92d329 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/put/RestIndexPutAliasAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/put/RestIndexPutAliasAction.java
@@ -30,8 +30,8 @@
 import org.elasticsearch.common.xcontent.XContentFactory;
 import org.elasticsearch.common.xcontent.XContentParser;
 import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.action.support.AcknowledgedRestListener;
 
-import java.io.IOException;
 import java.util.Map;
 
 import static org.elasticsearch.rest.RestRequest.Method.POST;
@@ -50,7 +50,7 @@ public RestIndexPutAliasAction(Settings settings, Client client, RestController
         controller.registerHandler(PUT, "/_aliases/{name}", this);
         controller.registerHandler(PUT, "/{index}/_alias", this);
         controller.registerHandler(PUT, "/_alias", this);
-        
+
         controller.registerHandler(POST, "/{index}/_alias/{name}", this);
         controller.registerHandler(POST, "/_alias/{name}", this);
         controller.registerHandler(POST, "/{index}/_aliases/{name}", this);
@@ -60,7 +60,7 @@ public RestIndexPutAliasAction(Settings settings, Client client, RestController
     }
 
     @Override
-    public void handleRequest(final RestRequest request, final RestChannel channel) {
+    public void handleRequest(final RestRequest request, final RestChannel channel) throws Exception {
         String[] indices = Strings.splitStringByCommaToArray(request.param("index"));
         String alias = request.param("name");
         Map<String, Object> filter = null;
@@ -69,9 +69,7 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         String searchRouting = null;
 
         if (request.hasContent()) {
-            XContentParser parser = null;
-            try {
-                parser = XContentFactory.xContent(request.content()).createParser(request.content());
+            try (XContentParser parser = XContentFactory.xContent(request.content()).createParser(request.content())) {
                 XContentParser.Token token = parser.nextToken();
                 if (token == null) {
                     throw new ElasticsearchIllegalArgumentException("No index alias is specified");
@@ -98,27 +96,16 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
                         }
                     }
                 }
-            } catch (Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.warn("Failed to send response", e1);
-                }
-                return;
-            } finally {
-                if (parser != null) {
-                    parser.close();
-                }
             }
         }
 
         IndicesAliasesRequest indicesAliasesRequest = new IndicesAliasesRequest();
         indicesAliasesRequest.timeout(request.paramAsTime("timeout", indicesAliasesRequest.timeout()));
-        String[] aliases = new String[] {alias};
+        String[] aliases = new String[]{alias};
         IndicesAliasesRequest.AliasActions aliasAction = new AliasActions(AliasAction.Type.ADD, indices, aliases);
         indicesAliasesRequest.addAliasAction(aliasAction);
         indicesAliasesRequest.masterNodeTimeout(request.paramAsTime("master_timeout", indicesAliasesRequest.masterNodeTimeout()));
-        
+
 
         if (routing != null) {
             aliasAction.routing(routing);
@@ -132,6 +119,6 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         if (filter != null) {
             aliasAction.filter(filter);
         }
-        client.admin().indices().aliases(indicesAliasesRequest, new AcknowledgedRestResponseActionListener<IndicesAliasesResponse>(request, channel, logger));
+        client.admin().indices().aliases(indicesAliasesRequest, new AcknowledgedRestListener<IndicesAliasesResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/analyze/RestAnalyzeAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/analyze/RestAnalyzeAction.java
index 1678c811a29a9..54ea0291308d1 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/analyze/RestAnalyzeAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/analyze/RestAnalyzeAction.java
@@ -19,21 +19,19 @@
 package org.elasticsearch.rest.action.admin.indices.analyze;
 
 import org.elasticsearch.ElasticsearchIllegalArgumentException;
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.indices.analyze.AnalyzeRequest;
 import org.elasticsearch.action.admin.indices.analyze.AnalyzeResponse;
 import org.elasticsearch.client.Client;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.common.xcontent.XContentBuilder;
-import org.elasticsearch.rest.*;
-
-import java.io.IOException;
+import org.elasticsearch.rest.BaseRestHandler;
+import org.elasticsearch.rest.RestChannel;
+import org.elasticsearch.rest.RestController;
+import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.action.support.RestToXContentListener;
 
 import static org.elasticsearch.rest.RestRequest.Method.GET;
 import static org.elasticsearch.rest.RestRequest.Method.POST;
-import static org.elasticsearch.rest.RestStatus.OK;
-import static org.elasticsearch.rest.action.support.RestXContentBuilder.restContentBuilder;
 
 /**
  *
@@ -56,12 +54,7 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
             text = request.content().toUtf8();
         }
         if (text == null) {
-            try {
-                channel.sendResponse(new BytesRestResponse(request, new ElasticsearchIllegalArgumentException("text is missing")));
-            } catch (IOException e1) {
-                logger.warn("Failed to send response", e1);
-            }
-            return;
+            throw new ElasticsearchIllegalArgumentException("text is missing");
         }
 
         AnalyzeRequest analyzeRequest = new AnalyzeRequest(request.param("index"), text);
@@ -72,28 +65,6 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         analyzeRequest.tokenizer(request.param("tokenizer"));
         analyzeRequest.tokenFilters(request.paramAsStringArray("token_filters", request.paramAsStringArray("filters", analyzeRequest.tokenFilters())));
         analyzeRequest.charFilters(request.paramAsStringArray("char_filters", analyzeRequest.charFilters()));
-        client.admin().indices().analyze(analyzeRequest, new ActionListener<AnalyzeResponse>() {
-            @Override
-            public void onResponse(AnalyzeResponse response) {
-                try {
-                    XContentBuilder builder = restContentBuilder(request, null);
-                    builder.startObject();
-                    response.toXContent(builder, request);
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
-            }
-        });
+        client.admin().indices().analyze(analyzeRequest, new RestToXContentListener<AnalyzeResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/cache/clear/RestClearIndicesCacheAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/cache/clear/RestClearIndicesCacheAction.java
index 5b4d8444cc54d..79c5735977f00 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/cache/clear/RestClearIndicesCacheAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/cache/clear/RestClearIndicesCacheAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.admin.indices.cache.clear;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.indices.cache.clear.ClearIndicesCacheRequest;
 import org.elasticsearch.action.admin.indices.cache.clear.ClearIndicesCacheResponse;
 import org.elasticsearch.action.support.IndicesOptions;
@@ -30,13 +29,10 @@
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
-
-import java.io.IOException;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
 import static org.elasticsearch.rest.RestRequest.Method.GET;
 import static org.elasticsearch.rest.RestRequest.Method.POST;
-import static org.elasticsearch.rest.RestStatus.BAD_REQUEST;
 import static org.elasticsearch.rest.RestStatus.OK;
 import static org.elasticsearch.rest.action.support.RestActions.buildBroadcastShardsHeader;
 
@@ -60,69 +56,44 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         ClearIndicesCacheRequest clearIndicesCacheRequest = new ClearIndicesCacheRequest(Strings.splitStringByCommaToArray(request.param("index")));
         clearIndicesCacheRequest.listenerThreaded(false);
         clearIndicesCacheRequest.indicesOptions(IndicesOptions.fromRequest(request, clearIndicesCacheRequest.indicesOptions()));
-        try {
-            if (request.hasParam("filter")) {
-                clearIndicesCacheRequest.filterCache(request.paramAsBoolean("filter", clearIndicesCacheRequest.filterCache()));
-            }
-            if (request.hasParam("filter_cache")) {
-                clearIndicesCacheRequest.filterCache(request.paramAsBoolean("filter_cache", clearIndicesCacheRequest.filterCache()));
-            }
-            if (request.hasParam("field_data")) {
-                clearIndicesCacheRequest.fieldDataCache(request.paramAsBoolean("field_data", clearIndicesCacheRequest.fieldDataCache()));
-            }
-            if (request.hasParam("fielddata")) {
-                clearIndicesCacheRequest.fieldDataCache(request.paramAsBoolean("fielddata", clearIndicesCacheRequest.fieldDataCache()));
-            }
-            if (request.hasParam("id")) {
-                clearIndicesCacheRequest.idCache(request.paramAsBoolean("id", clearIndicesCacheRequest.idCache()));
-            }
-            if (request.hasParam("id_cache")) {
-                clearIndicesCacheRequest.idCache(request.paramAsBoolean("id_cache", clearIndicesCacheRequest.idCache()));
-            }
-            if (request.hasParam("recycler")) {
-                clearIndicesCacheRequest.recycler(request.paramAsBoolean("recycler", clearIndicesCacheRequest.recycler()));
-            }
-            clearIndicesCacheRequest.fields(request.paramAsStringArray("fields", clearIndicesCacheRequest.fields()));
-            clearIndicesCacheRequest.filterKeys(request.paramAsStringArray("filter_keys", clearIndicesCacheRequest.filterKeys()));
-
-            BroadcastOperationThreading operationThreading = BroadcastOperationThreading.fromString(request.param("operationThreading"), BroadcastOperationThreading.THREAD_PER_SHARD);
-            if (operationThreading == BroadcastOperationThreading.NO_THREADS) {
-                // since we don't spawn, don't allow no_threads, but change it to a single thread
-                operationThreading = BroadcastOperationThreading.THREAD_PER_SHARD;
-            }
-            clearIndicesCacheRequest.operationThreading(operationThreading);
-        } catch (Exception e) {
-            try {
-                XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                channel.sendResponse(new BytesRestResponse(BAD_REQUEST, builder.startObject().field("error", e.getMessage()).endObject()));
-            } catch (IOException e1) {
-                logger.error("Failed to send failure response", e1);
-            }
-            return;
+        if (request.hasParam("filter")) {
+            clearIndicesCacheRequest.filterCache(request.paramAsBoolean("filter", clearIndicesCacheRequest.filterCache()));
         }
-        client.admin().indices().clearCache(clearIndicesCacheRequest, new ActionListener<ClearIndicesCacheResponse>() {
-            @Override
-            public void onResponse(ClearIndicesCacheResponse response) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
-
-                    buildBroadcastShardsHeader(builder, response);
+        if (request.hasParam("filter_cache")) {
+            clearIndicesCacheRequest.filterCache(request.paramAsBoolean("filter_cache", clearIndicesCacheRequest.filterCache()));
+        }
+        if (request.hasParam("field_data")) {
+            clearIndicesCacheRequest.fieldDataCache(request.paramAsBoolean("field_data", clearIndicesCacheRequest.fieldDataCache()));
+        }
+        if (request.hasParam("fielddata")) {
+            clearIndicesCacheRequest.fieldDataCache(request.paramAsBoolean("fielddata", clearIndicesCacheRequest.fieldDataCache()));
+        }
+        if (request.hasParam("id")) {
+            clearIndicesCacheRequest.idCache(request.paramAsBoolean("id", clearIndicesCacheRequest.idCache()));
+        }
+        if (request.hasParam("id_cache")) {
+            clearIndicesCacheRequest.idCache(request.paramAsBoolean("id_cache", clearIndicesCacheRequest.idCache()));
+        }
+        if (request.hasParam("recycler")) {
+            clearIndicesCacheRequest.recycler(request.paramAsBoolean("recycler", clearIndicesCacheRequest.recycler()));
+        }
+        clearIndicesCacheRequest.fields(request.paramAsStringArray("fields", clearIndicesCacheRequest.fields()));
+        clearIndicesCacheRequest.filterKeys(request.paramAsStringArray("filter_keys", clearIndicesCacheRequest.filterKeys()));
 
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
+        BroadcastOperationThreading operationThreading = BroadcastOperationThreading.fromString(request.param("operationThreading"), BroadcastOperationThreading.THREAD_PER_SHARD);
+        if (operationThreading == BroadcastOperationThreading.NO_THREADS) {
+            // since we don't spawn, don't allow no_threads, but change it to a single thread
+            operationThreading = BroadcastOperationThreading.THREAD_PER_SHARD;
+        }
+        clearIndicesCacheRequest.operationThreading(operationThreading);
 
+        client.admin().indices().clearCache(clearIndicesCacheRequest, new RestBuilderListener<ClearIndicesCacheResponse>(channel) {
             @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
+            public RestResponse buildResponse(ClearIndicesCacheResponse response, XContentBuilder builder) throws Exception {
+                builder.startObject();
+                buildBroadcastShardsHeader(builder, response);
+                builder.endObject();
+                return new BytesRestResponse(OK, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/close/RestCloseIndexAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/close/RestCloseIndexAction.java
index 7d814f645b677..62b7a90051d16 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/close/RestCloseIndexAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/close/RestCloseIndexAction.java
@@ -27,6 +27,7 @@
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.action.support.AcknowledgedRestListener;
 
 /**
  *
@@ -47,6 +48,6 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         closeIndexRequest.masterNodeTimeout(request.paramAsTime("master_timeout", closeIndexRequest.masterNodeTimeout()));
         closeIndexRequest.timeout(request.paramAsTime("timeout", closeIndexRequest.timeout()));
         closeIndexRequest.indicesOptions(IndicesOptions.fromRequest(request, closeIndexRequest.indicesOptions()));
-        client.admin().indices().close(closeIndexRequest, new AcknowledgedRestResponseActionListener<CloseIndexResponse>(request, channel, logger));
+        client.admin().indices().close(closeIndexRequest, new AcknowledgedRestListener<CloseIndexResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/create/RestCreateIndexAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/create/RestCreateIndexAction.java
index 0329649328170..7aff3201a806e 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/create/RestCreateIndexAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/create/RestCreateIndexAction.java
@@ -24,9 +24,11 @@
 import org.elasticsearch.client.Client;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.rest.*;
-
-import java.io.IOException;
+import org.elasticsearch.rest.BaseRestHandler;
+import org.elasticsearch.rest.RestChannel;
+import org.elasticsearch.rest.RestController;
+import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.action.support.AcknowledgedRestListener;
 
 /**
  *
@@ -46,21 +48,10 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         CreateIndexRequest createIndexRequest = new CreateIndexRequest(request.param("index"));
         createIndexRequest.listenerThreaded(false);
         if (request.hasContent()) {
-            try {
-                createIndexRequest.source(request.content());
-            } catch (Exception e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.warn("Failed to send response", e1);
-                }
-                return;
-            }
+            createIndexRequest.source(request.content());
         }
-
         createIndexRequest.timeout(request.paramAsTime("timeout", createIndexRequest.timeout()));
         createIndexRequest.masterNodeTimeout(request.paramAsTime("master_timeout", createIndexRequest.masterNodeTimeout()));
-
-        client.admin().indices().create(createIndexRequest, new AcknowledgedRestResponseActionListener<CreateIndexResponse>(request, channel, logger));
+        client.admin().indices().create(createIndexRequest, new AcknowledgedRestListener<CreateIndexResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/delete/RestDeleteIndexAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/delete/RestDeleteIndexAction.java
index b00478b7b69f9..0a237185ef13b 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/delete/RestDeleteIndexAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/delete/RestDeleteIndexAction.java
@@ -27,6 +27,7 @@
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.action.support.AcknowledgedRestListener;
 
 /**
  *
@@ -47,6 +48,6 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         deleteIndexRequest.timeout(request.paramAsTime("timeout", deleteIndexRequest.timeout()));
         deleteIndexRequest.masterNodeTimeout(request.paramAsTime("master_timeout", deleteIndexRequest.masterNodeTimeout()));
         deleteIndexRequest.indicesOptions(IndicesOptions.fromRequest(request, deleteIndexRequest.indicesOptions()));
-        client.admin().indices().delete(deleteIndexRequest, new AcknowledgedRestResponseActionListener<DeleteIndexResponse>(request, channel, logger));
+        client.admin().indices().delete(deleteIndexRequest, new AcknowledgedRestListener<DeleteIndexResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/exists/indices/RestIndicesExistsAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/exists/indices/RestIndicesExistsAction.java
index fc8cb2e138698..6266115cb2a00 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/exists/indices/RestIndicesExistsAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/exists/indices/RestIndicesExistsAction.java
@@ -19,8 +19,6 @@
 
 package org.elasticsearch.rest.action.admin.indices.exists.indices;
 
-import org.elasticsearch.ExceptionsHelper;
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.indices.exists.indices.IndicesExistsRequest;
 import org.elasticsearch.action.admin.indices.exists.indices.IndicesExistsResponse;
 import org.elasticsearch.action.support.IndicesOptions;
@@ -30,6 +28,7 @@
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.settings.SettingsFilter;
 import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.action.support.RestResponseListener;
 
 import static org.elasticsearch.rest.RestRequest.Method.HEAD;
 import static org.elasticsearch.rest.RestStatus.NOT_FOUND;
@@ -57,28 +56,16 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         indicesExistsRequest.indicesOptions(IndicesOptions.fromRequest(request, indicesExistsRequest.indicesOptions()));
         indicesExistsRequest.local(request.paramAsBoolean("local", indicesExistsRequest.local()));
         indicesExistsRequest.listenerThreaded(false);
-        client.admin().indices().exists(indicesExistsRequest, new ActionListener<IndicesExistsResponse>() {
+        client.admin().indices().exists(indicesExistsRequest, new RestResponseListener<IndicesExistsResponse>(channel) {
             @Override
-            public void onResponse(IndicesExistsResponse response) {
-                try {
-                    if (response.isExists()) {
-                        channel.sendResponse(new BytesRestResponse(OK));
-                    } else {
-                        channel.sendResponse(new BytesRestResponse(NOT_FOUND));
-                    }
-                } catch (Throwable e) {
-                    onFailure(e);
+            public RestResponse buildResponse(IndicesExistsResponse response) {
+                if (response.isExists()) {
+                    return new BytesRestResponse(OK);
+                } else {
+                    return new BytesRestResponse(NOT_FOUND);
                 }
             }
 
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(ExceptionsHelper.status(e)));
-                } catch (Exception e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
-            }
         });
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/exists/types/RestTypesExistsAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/exists/types/RestTypesExistsAction.java
index e591e035be1a6..51bb842eeca46 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/exists/types/RestTypesExistsAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/exists/types/RestTypesExistsAction.java
@@ -18,8 +18,6 @@
  */
 package org.elasticsearch.rest.action.admin.indices.exists.types;
 
-import org.elasticsearch.ExceptionsHelper;
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.indices.exists.types.TypesExistsRequest;
 import org.elasticsearch.action.admin.indices.exists.types.TypesExistsResponse;
 import org.elasticsearch.action.support.IndicesOptions;
@@ -28,6 +26,7 @@
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.action.support.RestResponseListener;
 
 import static org.elasticsearch.rest.RestRequest.Method.HEAD;
 import static org.elasticsearch.rest.RestStatus.NOT_FOUND;
@@ -52,26 +51,13 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         typesExistsRequest.listenerThreaded(false);
         typesExistsRequest.local(request.paramAsBoolean("local", typesExistsRequest.local()));
         typesExistsRequest.indicesOptions(IndicesOptions.fromRequest(request, typesExistsRequest.indicesOptions()));
-        client.admin().indices().typesExists(typesExistsRequest, new ActionListener<TypesExistsResponse>() {
+        client.admin().indices().typesExists(typesExistsRequest, new RestResponseListener<TypesExistsResponse>(channel) {
             @Override
-            public void onResponse(TypesExistsResponse response) {
-                try {
-                    if (response.isExists()) {
-                        channel.sendResponse(new BytesRestResponse(OK));
-                    } else {
-                        channel.sendResponse(new BytesRestResponse(NOT_FOUND));
-                    }
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(ExceptionsHelper.status(e)));
-                } catch (Exception e1) {
-                    logger.error("Failed to send failure response", e1);
+            public RestResponse buildResponse(TypesExistsResponse response) throws Exception {
+                if (response.isExists()) {
+                    return new BytesRestResponse(OK);
+                } else {
+                    return new BytesRestResponse(NOT_FOUND);
                 }
             }
         });
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/flush/RestFlushAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/flush/RestFlushAction.java
index 7f73cce3936a4..66c73dc4ec51d 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/flush/RestFlushAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/flush/RestFlushAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.admin.indices.flush;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.indices.flush.FlushRequest;
 import org.elasticsearch.action.admin.indices.flush.FlushResponse;
 import org.elasticsearch.action.support.IndicesOptions;
@@ -30,9 +29,7 @@
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
-
-import java.io.IOException;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
 import static org.elasticsearch.rest.RestRequest.Method.GET;
 import static org.elasticsearch.rest.RestRequest.Method.POST;
@@ -67,29 +64,13 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         flushRequest.operationThreading(operationThreading);
         flushRequest.full(request.paramAsBoolean("full", flushRequest.full()));
         flushRequest.force(request.paramAsBoolean("force", flushRequest.force()));
-        client.admin().indices().flush(flushRequest, new ActionListener<FlushResponse>() {
-            @Override
-            public void onResponse(FlushResponse response) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
-
-                    buildBroadcastShardsHeader(builder, response);
-
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
+        client.admin().indices().flush(flushRequest, new RestBuilderListener<FlushResponse>(channel) {
             @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
+            public RestResponse buildResponse(FlushResponse response, XContentBuilder builder) throws Exception {
+                builder.startObject();
+                buildBroadcastShardsHeader(builder, response);
+                builder.endObject();
+                return new BytesRestResponse(OK, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/delete/RestDeleteMappingAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/delete/RestDeleteMappingAction.java
index d915609b243bf..489a921ef4563 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/delete/RestDeleteMappingAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/delete/RestDeleteMappingAction.java
@@ -27,6 +27,7 @@
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.action.support.AcknowledgedRestListener;
 
 import static org.elasticsearch.client.Requests.deleteMappingRequest;
 import static org.elasticsearch.rest.RestRequest.Method.DELETE;
@@ -56,6 +57,6 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         deleteMappingRequest.timeout(request.paramAsTime("timeout", deleteMappingRequest.timeout()));
         deleteMappingRequest.masterNodeTimeout(request.paramAsTime("master_timeout", deleteMappingRequest.masterNodeTimeout()));
         deleteMappingRequest.indicesOptions(IndicesOptions.fromRequest(request, deleteMappingRequest.indicesOptions()));
-        client.admin().indices().deleteMapping(deleteMappingRequest, new AcknowledgedRestResponseActionListener<DeleteMappingResponse>(request, channel, logger));
+        client.admin().indices().deleteMapping(deleteMappingRequest, new AcknowledgedRestListener<DeleteMappingResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/get/RestGetFieldMappingAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/get/RestGetFieldMappingAction.java
index 87f72747b70d1..96fc875b0985c 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/get/RestGetFieldMappingAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/get/RestGetFieldMappingAction.java
@@ -20,7 +20,6 @@
 package org.elasticsearch.rest.action.admin.indices.mapping.get;
 
 import com.google.common.collect.ImmutableMap;
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.indices.mapping.get.GetFieldMappingsRequest;
 import org.elasticsearch.action.admin.indices.mapping.get.GetFieldMappingsResponse;
 import org.elasticsearch.action.admin.indices.mapping.get.GetFieldMappingsResponse.FieldMappingMetaData;
@@ -32,7 +31,7 @@
 import org.elasticsearch.common.xcontent.ToXContent;
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
 import java.io.IOException;
 import java.util.Map;
@@ -40,7 +39,6 @@
 import static org.elasticsearch.rest.RestRequest.Method.GET;
 import static org.elasticsearch.rest.RestStatus.NOT_FOUND;
 import static org.elasticsearch.rest.RestStatus.OK;
-import static org.elasticsearch.rest.action.support.RestXContentBuilder.emptyBuilder;
 
 /**
  *
@@ -66,48 +64,31 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         getMappingsRequest.indices(indices).types(types).fields(fields).includeDefaults(request.paramAsBoolean("include_defaults", false));
         getMappingsRequest.indicesOptions(IndicesOptions.fromRequest(request, getMappingsRequest.indicesOptions()));
         getMappingsRequest.local(request.paramAsBoolean("local", getMappingsRequest.local()));
-        client.admin().indices().getFieldMappings(getMappingsRequest, new ActionListener<GetFieldMappingsResponse>() {
+        client.admin().indices().getFieldMappings(getMappingsRequest, new RestBuilderListener<GetFieldMappingsResponse>(channel) {
 
             @SuppressWarnings("unchecked")
             @Override
-            public void onResponse(GetFieldMappingsResponse response) {
-                try {
-                    ImmutableMap<String, ImmutableMap<String, ImmutableMap<String, FieldMappingMetaData>>> mappingsByIndex = response.mappings();
+            public RestResponse buildResponse(GetFieldMappingsResponse response, XContentBuilder builder) throws Exception {
+                ImmutableMap<String, ImmutableMap<String, ImmutableMap<String, FieldMappingMetaData>>> mappingsByIndex = response.mappings();
 
-                    boolean isPossibleSingleFieldRequest = indices.length == 1 && types.length == 1 && fields.length == 1;
-                    if (isPossibleSingleFieldRequest && isFieldMappingMissingField(mappingsByIndex)) {
-                        channel.sendResponse(new BytesRestResponse(OK, emptyBuilder(request)));
-                        return;
-                    }
-
-                    RestStatus status = OK;
-                    if (mappingsByIndex.isEmpty() && fields.length > 0) {
-                        status = NOT_FOUND;
-                    }
-
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
-                    response.toXContent(builder, ToXContent.EMPTY_PARAMS);
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(status, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
+                boolean isPossibleSingleFieldRequest = indices.length == 1 && types.length == 1 && fields.length == 1;
+                if (isPossibleSingleFieldRequest && isFieldMappingMissingField(mappingsByIndex)) {
+                    return new BytesRestResponse(OK, builder.startObject().endObject());
                 }
-            }
 
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
+                RestStatus status = OK;
+                if (mappingsByIndex.isEmpty() && fields.length > 0) {
+                    status = NOT_FOUND;
                 }
+                builder.startObject();
+                response.toXContent(builder, ToXContent.EMPTY_PARAMS);
+                builder.endObject();
+                return new BytesRestResponse(status, builder);
             }
         });
     }
 
     /**
-     *
      * Helper method to find out if the only included fieldmapping metadata is typed NULL, which means
      * that type and index exist, but the field did not
      */
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/get/RestGetMappingAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/get/RestGetMappingAction.java
index bb87b0b0061fa..0990cf95cd76b 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/get/RestGetMappingAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/get/RestGetMappingAction.java
@@ -20,7 +20,6 @@
 package org.elasticsearch.rest.action.admin.indices.mapping.get;
 
 import com.carrotsearch.hppc.cursors.ObjectObjectCursor;
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.indices.mapping.get.GetMappingsRequest;
 import org.elasticsearch.action.admin.indices.mapping.get.GetMappingsResponse;
 import org.elasticsearch.action.support.IndicesOptions;
@@ -36,9 +35,7 @@
 import org.elasticsearch.indices.IndexMissingException;
 import org.elasticsearch.indices.TypeMissingException;
 import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
-
-import java.io.IOException;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
 import static org.elasticsearch.rest.RestRequest.Method.GET;
 import static org.elasticsearch.rest.RestStatus.OK;
@@ -67,57 +64,39 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         getMappingsRequest.indices(indices).types(types);
         getMappingsRequest.indicesOptions(IndicesOptions.fromRequest(request, getMappingsRequest.indicesOptions()));
         getMappingsRequest.local(request.paramAsBoolean("local", getMappingsRequest.local()));
-        client.admin().indices().getMappings(getMappingsRequest, new ActionListener<GetMappingsResponse>() {
-
+        client.admin().indices().getMappings(getMappingsRequest, new RestBuilderListener<GetMappingsResponse>(channel) {
             @Override
-            public void onResponse(GetMappingsResponse response) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
-
-                    ImmutableOpenMap<String, ImmutableOpenMap<String, MappingMetaData>> mappingsByIndex = response.getMappings();
-                    if (mappingsByIndex.isEmpty()) {
-                        if (indices.length != 0 && types.length != 0) {
-                            channel.sendResponse(new BytesRestResponse(OK, RestXContentBuilder.emptyBuilder(request)));
-                        } else if (indices.length != 0) {
-                            channel.sendResponse(new BytesRestResponse(request, new IndexMissingException(new Index(indices[0]))));
-                        } else if (types.length != 0) {
-                            channel.sendResponse(new BytesRestResponse(request, new TypeMissingException(new Index("_all"), types[0])));
-                        } else {
-                            builder.endObject();
-                            channel.sendResponse(new BytesRestResponse(OK, builder));
-                        }
-                        return;
+            public RestResponse buildResponse(GetMappingsResponse response, XContentBuilder builder) throws Exception {
+                builder.startObject();
+                ImmutableOpenMap<String, ImmutableOpenMap<String, MappingMetaData>> mappingsByIndex = response.getMappings();
+                if (mappingsByIndex.isEmpty()) {
+                    if (indices.length != 0 && types.length != 0) {
+                        return new BytesRestResponse(OK, builder.endObject());
+                    } else if (indices.length != 0) {
+                        return new BytesRestResponse(channel, new IndexMissingException(new Index(indices[0])));
+                    } else if (types.length != 0) {
+                        return new BytesRestResponse(channel, new TypeMissingException(new Index("_all"), types[0]));
+                    } else {
+                        return new BytesRestResponse(OK, builder.endObject());
                     }
+                }
 
-                    for (ObjectObjectCursor<String, ImmutableOpenMap<String, MappingMetaData>> indexEntry : mappingsByIndex) {
-                        if (indexEntry.value.isEmpty()) {
-                            continue;
-                        }
-                        builder.startObject(indexEntry.key, XContentBuilder.FieldCaseConversion.NONE);
-                        builder.startObject(Fields.MAPPINGS);
-                        for (ObjectObjectCursor<String, MappingMetaData> typeEntry : indexEntry.value) {
-                            builder.field(typeEntry.key);
-                            builder.map(typeEntry.value.sourceAsMap());
-                        }
-                        builder.endObject();
-                        builder.endObject();
+                for (ObjectObjectCursor<String, ImmutableOpenMap<String, MappingMetaData>> indexEntry : mappingsByIndex) {
+                    if (indexEntry.value.isEmpty()) {
+                        continue;
                     }
-
+                    builder.startObject(indexEntry.key, XContentBuilder.FieldCaseConversion.NONE);
+                    builder.startObject(Fields.MAPPINGS);
+                    for (ObjectObjectCursor<String, MappingMetaData> typeEntry : indexEntry.value) {
+                        builder.field(typeEntry.key);
+                        builder.map(typeEntry.value.sourceAsMap());
+                    }
+                    builder.endObject();
                     builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
                 }
-            }
 
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
+                builder.endObject();
+                return new BytesRestResponse(OK, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/put/RestPutMappingAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/put/RestPutMappingAction.java
index b6e9a5ebabcd0..408729dd1539c 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/put/RestPutMappingAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/put/RestPutMappingAction.java
@@ -27,6 +27,7 @@
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.action.support.AcknowledgedRestListener;
 
 import static org.elasticsearch.client.Requests.putMappingRequest;
 import static org.elasticsearch.rest.RestRequest.Method.POST;
@@ -73,6 +74,6 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         putMappingRequest.ignoreConflicts(request.paramAsBoolean("ignore_conflicts", putMappingRequest.ignoreConflicts()));
         putMappingRequest.masterNodeTimeout(request.paramAsTime("master_timeout", putMappingRequest.masterNodeTimeout()));
         putMappingRequest.indicesOptions(IndicesOptions.fromRequest(request, putMappingRequest.indicesOptions()));
-        client.admin().indices().putMapping(putMappingRequest, new AcknowledgedRestResponseActionListener<PutMappingResponse>(request, channel, logger));
+        client.admin().indices().putMapping(putMappingRequest, new AcknowledgedRestListener<PutMappingResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/open/RestOpenIndexAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/open/RestOpenIndexAction.java
index a0a08fad34965..4c58f331374da 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/open/RestOpenIndexAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/open/RestOpenIndexAction.java
@@ -27,6 +27,7 @@
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.action.support.AcknowledgedRestListener;
 
 /**
  *
@@ -47,6 +48,6 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         openIndexRequest.timeout(request.paramAsTime("timeout", openIndexRequest.timeout()));
         openIndexRequest.masterNodeTimeout(request.paramAsTime("master_timeout", openIndexRequest.masterNodeTimeout()));
         openIndexRequest.indicesOptions(IndicesOptions.fromRequest(request, openIndexRequest.indicesOptions()));
-        client.admin().indices().open(openIndexRequest, new AcknowledgedRestResponseActionListener<OpenIndexResponse>(request, channel, logger));
+        client.admin().indices().open(openIndexRequest, new AcknowledgedRestListener<OpenIndexResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/optimize/RestOptimizeAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/optimize/RestOptimizeAction.java
index 4abc3c5d76ec4..ed95c4cc61f19 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/optimize/RestOptimizeAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/optimize/RestOptimizeAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.admin.indices.optimize;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.indices.optimize.OptimizeRequest;
 import org.elasticsearch.action.admin.indices.optimize.OptimizeResponse;
 import org.elasticsearch.action.support.IndicesOptions;
@@ -30,13 +29,10 @@
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
-
-import java.io.IOException;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
 import static org.elasticsearch.rest.RestRequest.Method.GET;
 import static org.elasticsearch.rest.RestRequest.Method.POST;
-import static org.elasticsearch.rest.RestStatus.BAD_REQUEST;
 import static org.elasticsearch.rest.RestStatus.OK;
 import static org.elasticsearch.rest.action.support.RestActions.buildBroadcastShardsHeader;
 
@@ -60,51 +56,26 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         OptimizeRequest optimizeRequest = new OptimizeRequest(Strings.splitStringByCommaToArray(request.param("index")));
         optimizeRequest.listenerThreaded(false);
         optimizeRequest.indicesOptions(IndicesOptions.fromRequest(request, optimizeRequest.indicesOptions()));
-        try {
-            optimizeRequest.waitForMerge(request.paramAsBoolean("wait_for_merge", optimizeRequest.waitForMerge()));
-            optimizeRequest.maxNumSegments(request.paramAsInt("max_num_segments", optimizeRequest.maxNumSegments()));
-            optimizeRequest.onlyExpungeDeletes(request.paramAsBoolean("only_expunge_deletes", optimizeRequest.onlyExpungeDeletes()));
-            optimizeRequest.flush(request.paramAsBoolean("flush", optimizeRequest.flush()));
-            optimizeRequest.force(request.paramAsBoolean("force", optimizeRequest.force()));
+        optimizeRequest.waitForMerge(request.paramAsBoolean("wait_for_merge", optimizeRequest.waitForMerge()));
+        optimizeRequest.maxNumSegments(request.paramAsInt("max_num_segments", optimizeRequest.maxNumSegments()));
+        optimizeRequest.onlyExpungeDeletes(request.paramAsBoolean("only_expunge_deletes", optimizeRequest.onlyExpungeDeletes()));
+        optimizeRequest.flush(request.paramAsBoolean("flush", optimizeRequest.flush()));
+        optimizeRequest.force(request.paramAsBoolean("force", optimizeRequest.force()));
 
-            BroadcastOperationThreading operationThreading = BroadcastOperationThreading.fromString(request.param("operation_threading"), BroadcastOperationThreading.THREAD_PER_SHARD);
-            if (operationThreading == BroadcastOperationThreading.NO_THREADS) {
-                // since we don't spawn, don't allow no_threads, but change it to a single thread
-                operationThreading = BroadcastOperationThreading.THREAD_PER_SHARD;
-            }
-            optimizeRequest.operationThreading(operationThreading);
-        } catch (Exception e) {
-            try {
-                XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                channel.sendResponse(new BytesRestResponse(BAD_REQUEST, builder.startObject().field("error", e.getMessage()).endObject()));
-            } catch (IOException e1) {
-                logger.error("Failed to send failure response", e1);
-            }
-            return;
+        BroadcastOperationThreading operationThreading = BroadcastOperationThreading.fromString(request.param("operation_threading"), BroadcastOperationThreading.THREAD_PER_SHARD);
+        if (operationThreading == BroadcastOperationThreading.NO_THREADS) {
+            // since we don't spawn, don't allow no_threads, but change it to a single thread
+            operationThreading = BroadcastOperationThreading.THREAD_PER_SHARD;
         }
-        client.admin().indices().optimize(optimizeRequest, new ActionListener<OptimizeResponse>() {
-            @Override
-            public void onResponse(OptimizeResponse response) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
-
-                    buildBroadcastShardsHeader(builder, response);
-
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
+        optimizeRequest.operationThreading(operationThreading);
 
+        client.admin().indices().optimize(optimizeRequest, new RestBuilderListener<OptimizeResponse>(channel) {
             @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
+            public RestResponse buildResponse(OptimizeResponse response, XContentBuilder builder) throws Exception {
+                builder.startObject();
+                buildBroadcastShardsHeader(builder, response);
+                builder.endObject();
+                return new BytesRestResponse(OK, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/recovery/RestRecoveryAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/recovery/RestRecoveryAction.java
index 4ba2f1fc01706..68f6290d64860 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/recovery/RestRecoveryAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/recovery/RestRecoveryAction.java
@@ -19,9 +19,6 @@
 
 package org.elasticsearch.rest.action.admin.indices.recovery;
 
-import java.io.IOException;
-
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.indices.recovery.RecoveryRequest;
 import org.elasticsearch.action.admin.indices.recovery.RecoveryResponse;
 import org.elasticsearch.action.support.IndicesOptions;
@@ -31,16 +28,15 @@
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
 import static org.elasticsearch.rest.RestRequest.Method.GET;
 import static org.elasticsearch.rest.RestStatus.OK;
-import static org.elasticsearch.rest.action.support.RestActions.buildBroadcastShardsHeader;
 
 /**
  * REST handler to report on index recoveries.
  */
-public class RestRecoveryAction extends BaseRestHandler  {
+public class RestRecoveryAction extends BaseRestHandler {
 
     @Inject
     public RestRecoveryAction(Settings settings, Client client, RestController controller) {
@@ -58,32 +54,16 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         recoveryRequest.listenerThreaded(false);
         recoveryRequest.indicesOptions(IndicesOptions.fromRequest(request, recoveryRequest.indicesOptions()));
 
-        client.admin().indices().recoveries(recoveryRequest, new ActionListener<RecoveryResponse>() {
-
-            @Override
-            public void onResponse(RecoveryResponse response) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-
-                    if (response.hasRecoveries()) {
-                        response.detailed(recoveryRequest.detailed());
-                        builder.startObject();
-                        response.toXContent(builder, request);
-                        builder.endObject();
-                    }
-                    channel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
+        client.admin().indices().recoveries(recoveryRequest, new RestBuilderListener<RecoveryResponse>(channel) {
             @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException ioe) {
-                    logger.error("Failed to send failure response", ioe);
+            public RestResponse buildResponse(RecoveryResponse response, XContentBuilder builder) throws Exception {
+                if (response.hasRecoveries()) {
+                    response.detailed(recoveryRequest.detailed());
+                    builder.startObject();
+                    response.toXContent(builder, request);
+                    builder.endObject();
                 }
+                return new BytesRestResponse(OK, builder);
             }
         });
 
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/refresh/RestRefreshAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/refresh/RestRefreshAction.java
index f9da4b33987cf..03a60d683467b 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/refresh/RestRefreshAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/refresh/RestRefreshAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.admin.indices.refresh;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.indices.refresh.RefreshRequest;
 import org.elasticsearch.action.admin.indices.refresh.RefreshResponse;
 import org.elasticsearch.action.support.IndicesOptions;
@@ -30,7 +29,7 @@
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
 import java.io.IOException;
 
@@ -66,29 +65,13 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
             operationThreading = BroadcastOperationThreading.THREAD_PER_SHARD;
         }
         refreshRequest.operationThreading(operationThreading);
-        client.admin().indices().refresh(refreshRequest, new ActionListener<RefreshResponse>() {
+        client.admin().indices().refresh(refreshRequest, new RestBuilderListener<RefreshResponse>(channel) {
             @Override
-            public void onResponse(RefreshResponse response) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
-
-                    buildBroadcastShardsHeader(builder, response);
-
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
+            public RestResponse buildResponse(RefreshResponse response, XContentBuilder builder) throws Exception {
+                builder.startObject();
+                buildBroadcastShardsHeader(builder, response);
+                builder.endObject();
+                return new BytesRestResponse(OK, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/segments/RestIndicesSegmentsAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/segments/RestIndicesSegmentsAction.java
index 87344325b84e2..120df719b5f25 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/segments/RestIndicesSegmentsAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/segments/RestIndicesSegmentsAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.admin.indices.segments;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.indices.segments.IndicesSegmentResponse;
 import org.elasticsearch.action.admin.indices.segments.IndicesSegmentsRequest;
 import org.elasticsearch.action.support.IndicesOptions;
@@ -30,9 +29,7 @@
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
-
-import java.io.IOException;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
 import static org.elasticsearch.rest.RestRequest.Method.GET;
 import static org.elasticsearch.rest.RestStatus.OK;
@@ -60,28 +57,14 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
             operationThreading = BroadcastOperationThreading.SINGLE_THREAD;
         }
         indicesSegmentsRequest.operationThreading(operationThreading);
-        client.admin().indices().segments(indicesSegmentsRequest, new ActionListener<IndicesSegmentResponse>() {
-            @Override
-            public void onResponse(IndicesSegmentResponse response) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
-                    buildBroadcastShardsHeader(builder, response);
-                    response.toXContent(builder, request);
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
+        client.admin().indices().segments(indicesSegmentsRequest, new RestBuilderListener<IndicesSegmentResponse>(channel) {
             @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
+            public RestResponse buildResponse(IndicesSegmentResponse response, XContentBuilder builder) throws Exception {
+                builder.startObject();
+                buildBroadcastShardsHeader(builder, response);
+                response.toXContent(builder, request);
+                builder.endObject();
+                return new BytesRestResponse(OK, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/settings/RestGetSettingsAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/settings/RestGetSettingsAction.java
index 3d0f4343f033f..fd83724934d94 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/settings/RestGetSettingsAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/settings/RestGetSettingsAction.java
@@ -20,7 +20,6 @@
 package org.elasticsearch.rest.action.admin.indices.settings;
 
 import com.carrotsearch.hppc.cursors.ObjectObjectCursor;
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.indices.settings.get.GetSettingsRequest;
 import org.elasticsearch.action.admin.indices.settings.get.GetSettingsResponse;
 import org.elasticsearch.action.support.IndicesOptions;
@@ -31,9 +30,7 @@
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.common.xcontent.XContentBuilderString;
 import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
-
-import java.io.IOException;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
 import static org.elasticsearch.rest.RestRequest.Method.GET;
 import static org.elasticsearch.rest.RestStatus.OK;
@@ -59,38 +56,24 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
                 .names(names);
         getSettingsRequest.local(request.paramAsBoolean("local", getSettingsRequest.local()));
 
-        client.admin().indices().getSettings(getSettingsRequest, new ActionListener<GetSettingsResponse>() {
+        client.admin().indices().getSettings(getSettingsRequest, new RestBuilderListener<GetSettingsResponse>(channel) {
 
             @Override
-            public void onResponse(GetSettingsResponse getSettingsResponse) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
-                    for (ObjectObjectCursor<String, Settings> cursor : getSettingsResponse.getIndexToSettings()) {
-                        // no settings, jump over it to shorten the response data
-                        if (cursor.value.getAsMap().isEmpty()) {
-                            continue;
-                        }
-                        builder.startObject(cursor.key, XContentBuilder.FieldCaseConversion.NONE);
-                        builder.startObject(Fields.SETTINGS);
-                        cursor.value.toXContent(builder, request);
-                        builder.endObject();
-                        builder.endObject();
+            public RestResponse buildResponse(GetSettingsResponse getSettingsResponse, XContentBuilder builder) throws Exception {
+                builder.startObject();
+                for (ObjectObjectCursor<String, Settings> cursor : getSettingsResponse.getIndexToSettings()) {
+                    // no settings, jump over it to shorten the response data
+                    if (cursor.value.getAsMap().isEmpty()) {
+                        continue;
                     }
+                    builder.startObject(cursor.key, XContentBuilder.FieldCaseConversion.NONE);
+                    builder.startObject(Fields.SETTINGS);
+                    cursor.value.toXContent(builder, request);
+                    builder.endObject();
                     builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (IOException e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
                 }
+                builder.endObject();
+                return new BytesRestResponse(OK, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/settings/RestUpdateSettingsAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/settings/RestUpdateSettingsAction.java
index 5b3252688984b..0697c0dc4d0bc 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/settings/RestUpdateSettingsAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/settings/RestUpdateSettingsAction.java
@@ -27,14 +27,12 @@
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.ImmutableSettings;
 import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.common.settings.SettingsException;
 import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.action.support.AcknowledgedRestListener;
 
-import java.io.IOException;
 import java.util.Map;
 
 import static org.elasticsearch.client.Requests.updateSettingsRequest;
-import static org.elasticsearch.rest.RestStatus.BAD_REQUEST;
 
 /**
  *
@@ -59,24 +57,15 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         ImmutableSettings.Builder updateSettings = ImmutableSettings.settingsBuilder();
         String bodySettingsStr = request.content().toUtf8();
         if (Strings.hasText(bodySettingsStr)) {
-            try {
-                Settings buildSettings = ImmutableSettings.settingsBuilder().loadFromSource(bodySettingsStr).build();
-                for (Map.Entry<String, String> entry : buildSettings.getAsMap().entrySet()) {
-                    String key = entry.getKey();
-                    String value = entry.getValue();
-                    // clean up in case the body is wrapped with "settings" : { ... }
-                    if (key.startsWith("settings.")) {
-                        key = key.substring("settings.".length());
-                    }
-                    updateSettings.put(key, value);
+            Settings buildSettings = ImmutableSettings.settingsBuilder().loadFromSource(bodySettingsStr).build();
+            for (Map.Entry<String, String> entry : buildSettings.getAsMap().entrySet()) {
+                String key = entry.getKey();
+                String value = entry.getValue();
+                // clean up in case the body is wrapped with "settings" : { ... }
+                if (key.startsWith("settings.")) {
+                    key = key.substring("settings.".length());
                 }
-            } catch (Exception e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, BAD_REQUEST, new SettingsException("Failed to parse index settings", e)));
-                } catch (IOException e1) {
-                    logger.warn("Failed to send response", e1);
-                }
-                return;
+                updateSettings.put(key, value);
             }
         }
         for (Map.Entry<String, String> entry : request.params().entrySet()) {
@@ -87,6 +76,6 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         }
         updateSettingsRequest.settings(updateSettings);
 
-        client.admin().indices().updateSettings(updateSettingsRequest, new AcknowledgedRestResponseActionListener<UpdateSettingsResponse>(request, channel, logger));
+        client.admin().indices().updateSettings(updateSettingsRequest, new AcknowledgedRestListener<UpdateSettingsResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/stats/RestIndicesStatsAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/stats/RestIndicesStatsAction.java
index 1d5bb756f5cf3..1c8885db407f0 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/stats/RestIndicesStatsAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/stats/RestIndicesStatsAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.admin.indices.stats;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest;
 import org.elasticsearch.action.admin.indices.stats.IndicesStatsResponse;
 import org.elasticsearch.action.support.IndicesOptions;
@@ -29,7 +28,7 @@
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
 import java.io.IOException;
 import java.util.Set;
@@ -100,28 +99,14 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
             indicesStatsRequest.fieldDataFields(request.paramAsStringArray("fielddata_fields", request.paramAsStringArray("fields", Strings.EMPTY_ARRAY)));
         }
 
-        client.admin().indices().stats(indicesStatsRequest, new ActionListener<IndicesStatsResponse>() {
+        client.admin().indices().stats(indicesStatsRequest, new RestBuilderListener<IndicesStatsResponse>(channel) {
             @Override
-            public void onResponse(IndicesStatsResponse response) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
-                    buildBroadcastShardsHeader(builder, response);
-                    response.toXContent(builder, request);
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
+            public RestResponse buildResponse(IndicesStatsResponse response, XContentBuilder builder) throws Exception {
+                builder.startObject();
+                buildBroadcastShardsHeader(builder, response);
+                response.toXContent(builder, request);
+                builder.endObject();
+                return new BytesRestResponse(OK, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/status/RestIndicesStatusAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/status/RestIndicesStatusAction.java
index 5b1a34d456d66..d94ab10f7b8d4 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/status/RestIndicesStatusAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/status/RestIndicesStatusAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.admin.indices.status;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.indices.status.IndicesStatusRequest;
 import org.elasticsearch.action.admin.indices.status.IndicesStatusResponse;
 import org.elasticsearch.action.support.IndicesOptions;
@@ -31,7 +30,7 @@
 import org.elasticsearch.common.settings.SettingsFilter;
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
 import java.io.IOException;
 
@@ -69,28 +68,14 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
             operationThreading = BroadcastOperationThreading.SINGLE_THREAD;
         }
         indicesStatusRequest.operationThreading(operationThreading);
-        client.admin().indices().status(indicesStatusRequest, new ActionListener<IndicesStatusResponse>() {
+        client.admin().indices().status(indicesStatusRequest, new RestBuilderListener<IndicesStatusResponse>(channel) {
             @Override
-            public void onResponse(IndicesStatusResponse response) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
-                    buildBroadcastShardsHeader(builder, response);
-                    response.toXContent(builder, request, settingsFilter);
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
+            public RestResponse buildResponse(IndicesStatusResponse response, XContentBuilder builder) throws Exception {
+                builder.startObject();
+                buildBroadcastShardsHeader(builder, response);
+                response.toXContent(builder, request, settingsFilter);
+                builder.endObject();
+                return new BytesRestResponse(OK, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/template/delete/RestDeleteIndexTemplateAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/template/delete/RestDeleteIndexTemplateAction.java
index eb2b525f02a3f..c270d758e4af7 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/template/delete/RestDeleteIndexTemplateAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/template/delete/RestDeleteIndexTemplateAction.java
@@ -24,6 +24,7 @@
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.action.support.AcknowledgedRestListener;
 
 /**
  *
@@ -41,6 +42,6 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         DeleteIndexTemplateRequest deleteIndexTemplateRequest = new DeleteIndexTemplateRequest(request.param("name"));
         deleteIndexTemplateRequest.listenerThreaded(false);
         deleteIndexTemplateRequest.masterNodeTimeout(request.paramAsTime("master_timeout", deleteIndexTemplateRequest.masterNodeTimeout()));
-        client.admin().indices().deleteTemplate(deleteIndexTemplateRequest, new AcknowledgedRestResponseActionListener<DeleteIndexTemplateResponse>(request, channel, logger));
+        client.admin().indices().deleteTemplate(deleteIndexTemplateRequest, new AcknowledgedRestListener<DeleteIndexTemplateResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/template/get/RestGetIndexTemplateAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/template/get/RestGetIndexTemplateAction.java
index 5e31b2bbce721..969a9e76fc6db 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/template/get/RestGetIndexTemplateAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/template/get/RestGetIndexTemplateAction.java
@@ -19,7 +19,6 @@
 package org.elasticsearch.rest.action.admin.indices.template.get;
 
 import com.google.common.collect.Maps;
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.indices.template.get.GetIndexTemplatesRequest;
 import org.elasticsearch.action.admin.indices.template.get.GetIndexTemplatesResponse;
 import org.elasticsearch.client.Client;
@@ -30,7 +29,7 @@
 import org.elasticsearch.common.xcontent.ToXContent;
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
 import java.util.Map;
 
@@ -61,38 +60,24 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
 
         final boolean implicitAll = getIndexTemplatesRequest.names().length == 0;
 
-        client.admin().indices().getTemplates(getIndexTemplatesRequest, new ActionListener<GetIndexTemplatesResponse>() {
+        client.admin().indices().getTemplates(getIndexTemplatesRequest, new RestBuilderListener<GetIndexTemplatesResponse>(channel) {
             @Override
-            public void onResponse(GetIndexTemplatesResponse getIndexTemplatesResponse) {
-                try {
-                    boolean templateExists = getIndexTemplatesResponse.getIndexTemplates().size() > 0;
+            public RestResponse buildResponse(GetIndexTemplatesResponse getIndexTemplatesResponse, XContentBuilder builder) throws Exception {
+                boolean templateExists = getIndexTemplatesResponse.getIndexTemplates().size() > 0;
 
-                    Map<String, String> paramsMap = Maps.newHashMap();
-                    paramsMap.put("reduce_mappings", "true");
-                    ToXContent.Params params = new ToXContent.DelegatingMapParams(paramsMap, request);
+                Map<String, String> paramsMap = Maps.newHashMap();
+                paramsMap.put("reduce_mappings", "true");
+                ToXContent.Params params = new ToXContent.DelegatingMapParams(paramsMap, request);
 
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
-                    for (IndexTemplateMetaData indexTemplateMetaData : getIndexTemplatesResponse.getIndexTemplates()) {
-                        IndexTemplateMetaData.Builder.toXContent(indexTemplateMetaData, builder, params);
-                    }
-                    builder.endObject();
-
-                    RestStatus restStatus = (templateExists || implicitAll) ? OK : NOT_FOUND;
-
-                    channel.sendResponse(new BytesRestResponse(restStatus, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
+                builder.startObject();
+                for (IndexTemplateMetaData indexTemplateMetaData : getIndexTemplatesResponse.getIndexTemplates()) {
+                    IndexTemplateMetaData.Builder.toXContent(indexTemplateMetaData, builder, params);
                 }
-            }
+                builder.endObject();
 
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (Exception e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
+                RestStatus restStatus = (templateExists || implicitAll) ? OK : NOT_FOUND;
+
+                return new BytesRestResponse(restStatus, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/template/head/RestHeadIndexTemplateAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/template/head/RestHeadIndexTemplateAction.java
index d7cba1fd5c1ee..7991da9acb2c4 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/template/head/RestHeadIndexTemplateAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/template/head/RestHeadIndexTemplateAction.java
@@ -18,14 +18,13 @@
  */
 package org.elasticsearch.rest.action.admin.indices.template.head;
 
-import org.elasticsearch.ExceptionsHelper;
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.indices.template.get.GetIndexTemplatesRequest;
 import org.elasticsearch.action.admin.indices.template.get.GetIndexTemplatesResponse;
 import org.elasticsearch.client.Client;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.action.support.RestResponseListener;
 
 import static org.elasticsearch.rest.RestRequest.Method.HEAD;
 import static org.elasticsearch.rest.RestStatus.NOT_FOUND;
@@ -47,28 +46,14 @@ public RestHeadIndexTemplateAction(Settings settings, Client client, RestControl
     public void handleRequest(final RestRequest request, final RestChannel channel) {
         GetIndexTemplatesRequest getIndexTemplatesRequest = new GetIndexTemplatesRequest(request.param("name"));
         getIndexTemplatesRequest.local(request.paramAsBoolean("local", getIndexTemplatesRequest.local()));
-        client.admin().indices().getTemplates(getIndexTemplatesRequest, new ActionListener<GetIndexTemplatesResponse>() {
+        client.admin().indices().getTemplates(getIndexTemplatesRequest, new RestResponseListener<GetIndexTemplatesResponse>(channel) {
             @Override
-            public void onResponse(GetIndexTemplatesResponse getIndexTemplatesResponse) {
-                try {
-                    boolean templateExists = getIndexTemplatesResponse.getIndexTemplates().size() > 0;
-
-                    if (templateExists) {
-                        channel.sendResponse(new BytesRestResponse(OK));
-                    } else {
-                        channel.sendResponse(new BytesRestResponse(NOT_FOUND));
-                    }
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(ExceptionsHelper.status(e)));
-                } catch (Exception e1) {
-                    logger.error("Failed to send failure response", e1);
+            public RestResponse buildResponse(GetIndexTemplatesResponse getIndexTemplatesResponse) {
+                boolean templateExists = getIndexTemplatesResponse.getIndexTemplates().size() > 0;
+                if (templateExists) {
+                    return new BytesRestResponse(OK);
+                } else {
+                    return new BytesRestResponse(NOT_FOUND);
                 }
             }
         });
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/template/put/RestPutIndexTemplateAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/template/put/RestPutIndexTemplateAction.java
index 29eeed0f0098c..3b271cd00fb68 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/template/put/RestPutIndexTemplateAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/template/put/RestPutIndexTemplateAction.java
@@ -24,6 +24,7 @@
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.action.support.AcknowledgedRestListener;
 
 import java.io.IOException;
 
@@ -55,20 +56,9 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         putRequest.template(request.param("template", putRequest.template()));
         putRequest.order(request.paramAsInt("order", putRequest.order()));
         putRequest.masterNodeTimeout(request.paramAsTime("master_timeout", putRequest.masterNodeTimeout()));
-
-        try {
-            putRequest.create(request.paramAsBoolean("create", false));
-            putRequest.cause(request.param("cause", ""));
-            putRequest.source(request.content());
-        } catch (Exception e) {
-            try {
-                channel.sendResponse(new BytesRestResponse(request, e));
-            } catch (IOException e1) {
-                logger.warn("Failed to send response", e1);
-            }
-            return;
-        }
-
-        client.admin().indices().putTemplate(putRequest, new AcknowledgedRestResponseActionListener<PutIndexTemplateResponse>(request, channel, logger));
+        putRequest.create(request.paramAsBoolean("create", false));
+        putRequest.cause(request.param("cause", ""));
+        putRequest.source(request.content());
+        client.admin().indices().putTemplate(putRequest, new AcknowledgedRestListener<PutIndexTemplateResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/validate/query/RestValidateQueryAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/validate/query/RestValidateQueryAction.java
index d3493d34fd76f..77b59649c76f5 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/validate/query/RestValidateQueryAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/validate/query/RestValidateQueryAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.admin.indices.validate.query;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.indices.validate.query.QueryExplanation;
 import org.elasticsearch.action.admin.indices.validate.query.ValidateQueryRequest;
 import org.elasticsearch.action.admin.indices.validate.query.ValidateQueryResponse;
@@ -33,13 +32,10 @@
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
 import org.elasticsearch.rest.action.support.RestActions;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
-
-import java.io.IOException;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
 import static org.elasticsearch.rest.RestRequest.Method.GET;
 import static org.elasticsearch.rest.RestRequest.Method.POST;
-import static org.elasticsearch.rest.RestStatus.BAD_REQUEST;
 import static org.elasticsearch.rest.RestStatus.OK;
 import static org.elasticsearch.rest.action.support.RestActions.buildBroadcastShardsHeader;
 
@@ -64,84 +60,60 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         ValidateQueryRequest validateQueryRequest = new ValidateQueryRequest(Strings.splitStringByCommaToArray(request.param("index")));
         validateQueryRequest.listenerThreaded(false);
         validateQueryRequest.indicesOptions(IndicesOptions.fromRequest(request, validateQueryRequest.indicesOptions()));
-        try {
-            BroadcastOperationThreading operationThreading = BroadcastOperationThreading.fromString(request.param("operation_threading"), BroadcastOperationThreading.SINGLE_THREAD);
-            if (operationThreading == BroadcastOperationThreading.NO_THREADS) {
-                // since we don't spawn, don't allow no_threads, but change it to a single thread
-                operationThreading = BroadcastOperationThreading.SINGLE_THREAD;
-            }
-            validateQueryRequest.operationThreading(operationThreading);
-            if (request.hasContent()) {
-                validateQueryRequest.source(request.content(), request.contentUnsafe());
+        BroadcastOperationThreading operationThreading = BroadcastOperationThreading.fromString(request.param("operation_threading"), BroadcastOperationThreading.SINGLE_THREAD);
+        if (operationThreading == BroadcastOperationThreading.NO_THREADS) {
+            // since we don't spawn, don't allow no_threads, but change it to a single thread
+            operationThreading = BroadcastOperationThreading.SINGLE_THREAD;
+        }
+        validateQueryRequest.operationThreading(operationThreading);
+        if (request.hasContent()) {
+            validateQueryRequest.source(request.content(), request.contentUnsafe());
+        } else {
+            String source = request.param("source");
+            if (source != null) {
+                validateQueryRequest.source(source);
             } else {
-                String source = request.param("source");
-                if (source != null) {
-                    validateQueryRequest.source(source);
-                } else {
-                    QuerySourceBuilder querySourceBuilder = RestActions.parseQuerySource(request);
-                    if (querySourceBuilder != null) {
-                        validateQueryRequest.source(querySourceBuilder);
-                    }
+                QuerySourceBuilder querySourceBuilder = RestActions.parseQuerySource(request);
+                if (querySourceBuilder != null) {
+                    validateQueryRequest.source(querySourceBuilder);
                 }
             }
-            validateQueryRequest.types(Strings.splitStringByCommaToArray(request.param("type")));
-            if (request.paramAsBoolean("explain", false)) {
-                validateQueryRequest.explain(true);
-            } else {
-                validateQueryRequest.explain(false);
-            }
-        } catch (Exception e) {
-            try {
-                XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                channel.sendResponse(new BytesRestResponse(BAD_REQUEST, builder.startObject().field("error", e.getMessage()).endObject()));
-            } catch (IOException e1) {
-                logger.error("Failed to send failure response", e1);
-            }
-            return;
+        }
+        validateQueryRequest.types(Strings.splitStringByCommaToArray(request.param("type")));
+        if (request.paramAsBoolean("explain", false)) {
+            validateQueryRequest.explain(true);
+        } else {
+            validateQueryRequest.explain(false);
         }
 
-        client.admin().indices().validateQuery(validateQueryRequest, new ActionListener<ValidateQueryResponse>() {
+        client.admin().indices().validateQuery(validateQueryRequest, new RestBuilderListener<ValidateQueryResponse>(channel) {
             @Override
-            public void onResponse(ValidateQueryResponse response) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
-                    builder.field("valid", response.isValid());
+            public RestResponse buildResponse(ValidateQueryResponse response, XContentBuilder builder) throws Exception {
+                builder.startObject();
+                builder.field("valid", response.isValid());
 
-                    buildBroadcastShardsHeader(builder, response);
+                buildBroadcastShardsHeader(builder, response);
 
-                    if (response.getQueryExplanation() != null && !response.getQueryExplanation().isEmpty()) {
-                        builder.startArray("explanations");
-                        for (QueryExplanation explanation : response.getQueryExplanation()) {
-                            builder.startObject();
-                            if (explanation.getIndex() != null) {
-                                builder.field("index", explanation.getIndex(), XContentBuilder.FieldCaseConversion.NONE);
-                            }
-                            builder.field("valid", explanation.isValid());
-                            if (explanation.getError() != null) {
-                                builder.field("error", explanation.getError());
-                            }
-                            if (explanation.getExplanation() != null) {
-                                builder.field("explanation", explanation.getExplanation());
-                            }
-                            builder.endObject();
+                if (response.getQueryExplanation() != null && !response.getQueryExplanation().isEmpty()) {
+                    builder.startArray("explanations");
+                    for (QueryExplanation explanation : response.getQueryExplanation()) {
+                        builder.startObject();
+                        if (explanation.getIndex() != null) {
+                            builder.field("index", explanation.getIndex(), XContentBuilder.FieldCaseConversion.NONE);
+                        }
+                        builder.field("valid", explanation.isValid());
+                        if (explanation.getError() != null) {
+                            builder.field("error", explanation.getError());
                         }
-                        builder.endArray();
+                        if (explanation.getExplanation() != null) {
+                            builder.field("explanation", explanation.getExplanation());
+                        }
+                        builder.endObject();
                     }
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
+                    builder.endArray();
                 }
+                builder.endObject();
+                return new BytesRestResponse(OK, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/delete/RestDeleteWarmerAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/delete/RestDeleteWarmerAction.java
index e54e4589921ad..eb02d775048d5 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/delete/RestDeleteWarmerAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/delete/RestDeleteWarmerAction.java
@@ -19,12 +19,17 @@
 package org.elasticsearch.rest.action.admin.indices.warmer.delete;
 
 import org.elasticsearch.action.admin.indices.warmer.delete.DeleteWarmerRequest;
+import org.elasticsearch.action.admin.indices.warmer.delete.DeleteWarmerResponse;
 import org.elasticsearch.action.support.IndicesOptions;
 import org.elasticsearch.client.Client;
 import org.elasticsearch.common.Strings;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.BaseRestHandler;
+import org.elasticsearch.rest.RestChannel;
+import org.elasticsearch.rest.RestController;
+import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.action.support.AcknowledgedRestListener;
 
 import static org.elasticsearch.rest.RestRequest.Method.DELETE;
 
@@ -49,6 +54,6 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         deleteWarmerRequest.timeout(request.paramAsTime("timeout", deleteWarmerRequest.timeout()));
         deleteWarmerRequest.masterNodeTimeout(request.paramAsTime("master_timeout", deleteWarmerRequest.masterNodeTimeout()));
         deleteWarmerRequest.indicesOptions(IndicesOptions.fromRequest(request, deleteWarmerRequest.indicesOptions()));
-        client.admin().indices().deleteWarmer(deleteWarmerRequest, new AcknowledgedRestResponseActionListener(request, channel, logger));
+        client.admin().indices().deleteWarmer(deleteWarmerRequest, new AcknowledgedRestListener<DeleteWarmerResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/get/RestGetWarmerAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/get/RestGetWarmerAction.java
index f16b679932e8a..9ab046ad37dbd 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/get/RestGetWarmerAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/get/RestGetWarmerAction.java
@@ -20,7 +20,6 @@
 
 import com.carrotsearch.hppc.cursors.ObjectObjectCursor;
 import com.google.common.collect.ImmutableList;
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.indices.warmer.get.GetWarmersRequest;
 import org.elasticsearch.action.admin.indices.warmer.get.GetWarmersResponse;
 import org.elasticsearch.action.support.IndicesOptions;
@@ -30,11 +29,9 @@
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 import org.elasticsearch.search.warmer.IndexWarmersMetaData;
 
-import java.io.IOException;
-
 import static org.elasticsearch.rest.RestRequest.Method.GET;
 import static org.elasticsearch.rest.RestStatus.OK;
 
@@ -64,42 +61,27 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         getWarmersRequest.indices(indices).types(types).warmers(names);
         getWarmersRequest.local(request.paramAsBoolean("local", getWarmersRequest.local()));
         getWarmersRequest.indicesOptions(IndicesOptions.fromRequest(request, getWarmersRequest.indicesOptions()));
-        client.admin().indices().getWarmers(getWarmersRequest, new ActionListener<GetWarmersResponse>() {
+        client.admin().indices().getWarmers(getWarmersRequest, new RestBuilderListener<GetWarmersResponse>(channel) {
 
             @Override
-            public void onResponse(GetWarmersResponse response) {
-                try {
-                    if (indices.length > 0 && response.warmers().isEmpty()) {
-                        channel.sendResponse(new BytesRestResponse(OK, RestXContentBuilder.emptyBuilder(request)));
-                        return;
-                    }
+            public RestResponse buildResponse(GetWarmersResponse response, XContentBuilder builder) throws Exception {
+                if (indices.length > 0 && response.warmers().isEmpty()) {
+                    return new BytesRestResponse(OK, builder.startObject().endObject());
+                }
 
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
-                    for (ObjectObjectCursor<String, ImmutableList<IndexWarmersMetaData.Entry>> entry : response.warmers()) {
-                        builder.startObject(entry.key, XContentBuilder.FieldCaseConversion.NONE);
-                        builder.startObject(IndexWarmersMetaData.TYPE, XContentBuilder.FieldCaseConversion.NONE);
-                        for (IndexWarmersMetaData.Entry warmerEntry : entry.value) {
-                            IndexWarmersMetaData.FACTORY.toXContent(warmerEntry, builder, request);
-                        }
-                        builder.endObject();
-                        builder.endObject();
+                builder.startObject();
+                for (ObjectObjectCursor<String, ImmutableList<IndexWarmersMetaData.Entry>> entry : response.warmers()) {
+                    builder.startObject(entry.key, XContentBuilder.FieldCaseConversion.NONE);
+                    builder.startObject(IndexWarmersMetaData.TYPE, XContentBuilder.FieldCaseConversion.NONE);
+                    for (IndexWarmersMetaData.Entry warmerEntry : entry.value) {
+                        IndexWarmersMetaData.FACTORY.toXContent(warmerEntry, builder, request);
                     }
                     builder.endObject();
-
-                    channel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
+                    builder.endObject();
                 }
-            }
+                builder.endObject();
 
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
+                return new BytesRestResponse(OK, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/put/RestPutWarmerAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/put/RestPutWarmerAction.java
index b0bc3b1f35eda..0d11a9d64257c 100644
--- a/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/put/RestPutWarmerAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/put/RestPutWarmerAction.java
@@ -19,13 +19,18 @@
 package org.elasticsearch.rest.action.admin.indices.warmer.put;
 
 import org.elasticsearch.action.admin.indices.warmer.put.PutWarmerRequest;
+import org.elasticsearch.action.admin.indices.warmer.put.PutWarmerResponse;
 import org.elasticsearch.action.search.SearchRequest;
 import org.elasticsearch.action.support.IndicesOptions;
 import org.elasticsearch.client.Client;
 import org.elasticsearch.common.Strings;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.BaseRestHandler;
+import org.elasticsearch.rest.RestChannel;
+import org.elasticsearch.rest.RestController;
+import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.action.support.AcknowledgedRestListener;
 
 import static org.elasticsearch.rest.RestRequest.Method.POST;
 import static org.elasticsearch.rest.RestRequest.Method.PUT;
@@ -40,15 +45,15 @@ public RestPutWarmerAction(Settings settings, Client client, RestController cont
         controller.registerHandler(PUT, "/_warmer/{name}", this);
         controller.registerHandler(PUT, "/{index}/_warmer/{name}", this);
         controller.registerHandler(PUT, "/{index}/{type}/_warmer/{name}", this);
-        
+
         controller.registerHandler(PUT, "/_warmers/{name}", this);
         controller.registerHandler(PUT, "/{index}/_warmers/{name}", this);
         controller.registerHandler(PUT, "/{index}/{type}/_warmers/{name}", this);
-        
+
         controller.registerHandler(POST, "/_warmer/{name}", this);
         controller.registerHandler(POST, "/{index}/_warmer/{name}", this);
         controller.registerHandler(POST, "/{index}/{type}/_warmer/{name}", this);
-        
+
         controller.registerHandler(POST, "/_warmers/{name}", this);
         controller.registerHandler(POST, "/{index}/_warmers/{name}", this);
         controller.registerHandler(POST, "/{index}/{type}/_warmers/{name}", this);
@@ -65,6 +70,6 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         putWarmerRequest.searchRequest(searchRequest);
         putWarmerRequest.timeout(request.paramAsTime("timeout", putWarmerRequest.timeout()));
         putWarmerRequest.masterNodeTimeout(request.paramAsTime("master_timeout", putWarmerRequest.masterNodeTimeout()));
-        client.admin().indices().putWarmer(putWarmerRequest,  new AcknowledgedRestResponseActionListener(request, channel, logger));
+        client.admin().indices().putWarmer(putWarmerRequest, new AcknowledgedRestListener<PutWarmerResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/bulk/RestBulkAction.java b/src/main/java/org/elasticsearch/rest/action/bulk/RestBulkAction.java
index 212adcfecbef8..01d62a407802d 100644
--- a/src/main/java/org/elasticsearch/rest/action/bulk/RestBulkAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/bulk/RestBulkAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.bulk;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.WriteConsistencyLevel;
 import org.elasticsearch.action.bulk.BulkItemResponse;
 import org.elasticsearch.action.bulk.BulkRequest;
@@ -36,14 +35,11 @@
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.common.xcontent.XContentBuilderString;
 import org.elasticsearch.rest.*;
-
-import java.io.IOException;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
 import static org.elasticsearch.rest.RestRequest.Method.POST;
 import static org.elasticsearch.rest.RestRequest.Method.PUT;
-import static org.elasticsearch.rest.RestStatus.BAD_REQUEST;
 import static org.elasticsearch.rest.RestStatus.OK;
-import static org.elasticsearch.rest.action.support.RestXContentBuilder.restContentBuilder;
 
 /**
  * <pre>
@@ -73,7 +69,7 @@ public RestBulkAction(Settings settings, Client client, RestController controlle
     }
 
     @Override
-    public void handleRequest(final RestRequest request, final RestChannel channel) {
+    public void handleRequest(final RestRequest request, final RestChannel channel) throws Exception {
         BulkRequest bulkRequest = Requests.bulkRequest();
         bulkRequest.listenerThreaded(false);
         String defaultIndex = request.param("index");
@@ -90,84 +86,60 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         }
         bulkRequest.timeout(request.paramAsTime("timeout", BulkShardRequest.DEFAULT_TIMEOUT));
         bulkRequest.refresh(request.paramAsBoolean("refresh", bulkRequest.refresh()));
-        try {
-            bulkRequest.add(request.content(), request.contentUnsafe(), defaultIndex, defaultType, defaultRouting, null, allowExplicitIndex);
-        } catch (Exception e) {
-            try {
-                XContentBuilder builder = restContentBuilder(request);
-                channel.sendResponse(new BytesRestResponse(BAD_REQUEST, builder.startObject().field("error", e.getMessage()).endObject()));
-            } catch (IOException e1) {
-                logger.error("Failed to send failure response", e1);
-            }
-            return;
-        }
+        bulkRequest.add(request.content(), request.contentUnsafe(), defaultIndex, defaultType, defaultRouting, null, allowExplicitIndex);
 
-        client.bulk(bulkRequest, new ActionListener<BulkResponse>() {
+        client.bulk(bulkRequest, new RestBuilderListener<BulkResponse>(channel) {
             @Override
-            public void onResponse(BulkResponse response) {
-                try {
-                    XContentBuilder builder = restContentBuilder(request);
+            public RestResponse buildResponse(BulkResponse response, XContentBuilder builder) throws Exception {
+                builder.startObject();
+                builder.field(Fields.TOOK, response.getTookInMillis());
+                builder.field(Fields.ERRORS, response.hasFailures());
+                builder.startArray(Fields.ITEMS);
+                for (BulkItemResponse itemResponse : response) {
                     builder.startObject();
-                    builder.field(Fields.TOOK, response.getTookInMillis());
-                    builder.field(Fields.ERRORS, response.hasFailures());
-                    builder.startArray(Fields.ITEMS);
-                    for (BulkItemResponse itemResponse : response) {
-                        builder.startObject();
-                        builder.startObject(itemResponse.getOpType());
-                        builder.field(Fields._INDEX, itemResponse.getIndex());
-                        builder.field(Fields._TYPE, itemResponse.getType());
-                        builder.field(Fields._ID, itemResponse.getId());
-                        long version = itemResponse.getVersion();
-                        if (version != -1) {
-                            builder.field(Fields._VERSION, itemResponse.getVersion());
-                        }
-                        if (itemResponse.isFailed()) {
-                            builder.field(Fields.STATUS, itemResponse.getFailure().getStatus().getStatus());
-                            builder.field(Fields.ERROR, itemResponse.getFailure().getMessage());
-                        } else {
-                            if (itemResponse.getResponse() instanceof DeleteResponse) {
-                                DeleteResponse deleteResponse = itemResponse.getResponse();
-                                if (deleteResponse.isFound()) {
-                                    builder.field(Fields.STATUS, RestStatus.OK.getStatus());
-                                } else {
-                                    builder.field(Fields.STATUS, RestStatus.NOT_FOUND.getStatus());
-                                }
-                                builder.field(Fields.FOUND, deleteResponse.isFound());
-                            } else if (itemResponse.getResponse() instanceof IndexResponse) {
-                                IndexResponse indexResponse = itemResponse.getResponse();
-                                if (indexResponse.isCreated()) {
-                                    builder.field(Fields.STATUS, RestStatus.CREATED.getStatus());
-                                } else {
-                                    builder.field(Fields.STATUS, RestStatus.OK.getStatus());
-                                }
-                            } else if (itemResponse.getResponse() instanceof UpdateResponse) {
-                                UpdateResponse updateResponse = itemResponse.getResponse();
-                                if (updateResponse.isCreated()) {
-                                    builder.field(Fields.STATUS, RestStatus.CREATED.getStatus());
-                                } else {
-                                    builder.field(Fields.STATUS, RestStatus.OK.getStatus());
-                                }
+                    builder.startObject(itemResponse.getOpType());
+                    builder.field(Fields._INDEX, itemResponse.getIndex());
+                    builder.field(Fields._TYPE, itemResponse.getType());
+                    builder.field(Fields._ID, itemResponse.getId());
+                    long version = itemResponse.getVersion();
+                    if (version != -1) {
+                        builder.field(Fields._VERSION, itemResponse.getVersion());
+                    }
+                    if (itemResponse.isFailed()) {
+                        builder.field(Fields.STATUS, itemResponse.getFailure().getStatus().getStatus());
+                        builder.field(Fields.ERROR, itemResponse.getFailure().getMessage());
+                    } else {
+                        if (itemResponse.getResponse() instanceof DeleteResponse) {
+                            DeleteResponse deleteResponse = itemResponse.getResponse();
+                            if (deleteResponse.isFound()) {
+                                builder.field(Fields.STATUS, RestStatus.OK.getStatus());
+                            } else {
+                                builder.field(Fields.STATUS, RestStatus.NOT_FOUND.getStatus());
+                            }
+                            builder.field(Fields.FOUND, deleteResponse.isFound());
+                        } else if (itemResponse.getResponse() instanceof IndexResponse) {
+                            IndexResponse indexResponse = itemResponse.getResponse();
+                            if (indexResponse.isCreated()) {
+                                builder.field(Fields.STATUS, RestStatus.CREATED.getStatus());
+                            } else {
+                                builder.field(Fields.STATUS, RestStatus.OK.getStatus());
+                            }
+                        } else if (itemResponse.getResponse() instanceof UpdateResponse) {
+                            UpdateResponse updateResponse = itemResponse.getResponse();
+                            if (updateResponse.isCreated()) {
+                                builder.field(Fields.STATUS, RestStatus.CREATED.getStatus());
+                            } else {
+                                builder.field(Fields.STATUS, RestStatus.OK.getStatus());
                             }
                         }
-                        builder.endObject();
-                        builder.endObject();
                     }
-                    builder.endArray();
-
                     builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
+                    builder.endObject();
                 }
-            }
+                builder.endArray();
 
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
+                builder.endObject();
+                return new BytesRestResponse(OK, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/cat/AbstractCatAction.java b/src/main/java/org/elasticsearch/rest/action/cat/AbstractCatAction.java
index 8c5f634fbf37f..43cc2e6aa7e0d 100644
--- a/src/main/java/org/elasticsearch/rest/action/cat/AbstractCatAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/cat/AbstractCatAction.java
@@ -20,6 +20,8 @@
 
 import org.elasticsearch.client.Client;
 import org.elasticsearch.common.Table;
+import org.elasticsearch.common.io.UTF8StreamWriter;
+import org.elasticsearch.common.io.stream.BytesStreamOutput;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.rest.*;
 
@@ -42,12 +44,13 @@ public AbstractCatAction(Settings settings, Client client) {
     abstract Table getTableWithHeader(final RestRequest request);
 
     @Override
-    public void handleRequest(final RestRequest request, final RestChannel channel) {
+    public void handleRequest(final RestRequest request, final RestChannel channel) throws Exception {
         boolean helpWanted = request.paramAsBoolean("help", false);
         if (helpWanted) {
             Table table = getTableWithHeader(request);
-            int[] width = buildHelpWidths(table, request, false);
-            StringBuilder out = new StringBuilder();
+            int[] width = buildHelpWidths(table, request);
+            BytesStreamOutput bytesOutput = channel.bytesOutput();
+            UTF8StreamWriter out = new UTF8StreamWriter().setOutput(bytesOutput);
             for (Table.Cell cell : table.getHeaders()) {
                 // need to do left-align always, so create new cells
                 pad(new Table.Cell(cell.value), width[0], request, out);
@@ -57,11 +60,10 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
                 pad(new Table.Cell(cell.attr.containsKey("desc") ? cell.attr.get("desc") : "not available"), width[2], request, out);
                 out.append("\n");
             }
-            channel.sendResponse(new BytesRestResponse(RestStatus.OK, out.toString()));
+            out.close();
+            channel.sendResponse(new BytesRestResponse(RestStatus.OK, BytesRestResponse.TEXT_CONTENT_TYPE, bytesOutput.bytes(), true));
         } else {
             doRequest(request, channel);
         }
     }
-
-
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/cat/RestAliasAction.java b/src/main/java/org/elasticsearch/rest/action/cat/RestAliasAction.java
index d8f180f2bd619..fcdeb7a3247dc 100644
--- a/src/main/java/org/elasticsearch/rest/action/cat/RestAliasAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/cat/RestAliasAction.java
@@ -19,7 +19,6 @@
 package org.elasticsearch.rest.action.cat;
 
 import com.carrotsearch.hppc.cursors.ObjectObjectCursor;
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.indices.alias.get.GetAliasesRequest;
 import org.elasticsearch.action.admin.indices.alias.get.GetAliasesResponse;
 import org.elasticsearch.client.Client;
@@ -28,13 +27,13 @@
 import org.elasticsearch.common.Table;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.rest.BytesRestResponse;
 import org.elasticsearch.rest.RestChannel;
 import org.elasticsearch.rest.RestController;
 import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.RestResponse;
+import org.elasticsearch.rest.action.support.RestResponseListener;
 import org.elasticsearch.rest.action.support.RestTable;
 
-import java.io.IOException;
 import java.util.List;
 
 import static org.elasticsearch.rest.RestRequest.Method.GET;
@@ -59,24 +58,11 @@ void doRequest(final RestRequest request, final RestChannel channel) {
                 new GetAliasesRequest();
         getAliasesRequest.local(request.paramAsBoolean("local", getAliasesRequest.local()));
 
-        client.admin().indices().getAliases(getAliasesRequest, new ActionListener<GetAliasesResponse>() {
+        client.admin().indices().getAliases(getAliasesRequest, new RestResponseListener<GetAliasesResponse>(channel) {
             @Override
-            public void onResponse(GetAliasesResponse response) {
-                try {
-                    Table tab = buildTable(request, response);
-                    channel.sendResponse(RestTable.buildResponse(tab, request, channel));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
+            public RestResponse buildResponse(GetAliasesResponse response) throws Exception {
+                Table tab = buildTable(request, response);
+                return RestTable.buildResponse(tab, channel);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/cat/RestAllocationAction.java b/src/main/java/org/elasticsearch/rest/action/cat/RestAllocationAction.java
index ec7bbc08cd4b3..ac72bce4e825c 100644
--- a/src/main/java/org/elasticsearch/rest/action/cat/RestAllocationAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/cat/RestAllocationAction.java
@@ -20,7 +20,6 @@
 package org.elasticsearch.rest.action.cat;
 
 import com.carrotsearch.hppc.ObjectIntOpenHashMap;
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.cluster.node.stats.NodeStats;
 import org.elasticsearch.action.admin.cluster.node.stats.NodesStatsRequest;
 import org.elasticsearch.action.admin.cluster.node.stats.NodesStatsResponse;
@@ -34,14 +33,14 @@
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.unit.ByteSizeValue;
-import org.elasticsearch.rest.BytesRestResponse;
 import org.elasticsearch.rest.RestChannel;
 import org.elasticsearch.rest.RestController;
 import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.RestResponse;
+import org.elasticsearch.rest.action.support.RestActionListener;
+import org.elasticsearch.rest.action.support.RestResponseListener;
 import org.elasticsearch.rest.action.support.RestTable;
 
-import java.io.IOException;
-
 import static org.elasticsearch.rest.RestRequest.Method.GET;
 
 
@@ -67,42 +66,20 @@ public void doRequest(final RestRequest request, final RestChannel channel) {
         clusterStateRequest.local(request.paramAsBoolean("local", clusterStateRequest.local()));
         clusterStateRequest.masterNodeTimeout(request.paramAsTime("master_timeout", clusterStateRequest.masterNodeTimeout()));
 
-        client.admin().cluster().state(clusterStateRequest, new ActionListener<ClusterStateResponse>() {
+        client.admin().cluster().state(clusterStateRequest, new RestActionListener<ClusterStateResponse>(channel) {
             @Override
-            public void onResponse(final ClusterStateResponse state) {
+            public void processResponse(final ClusterStateResponse state) {
                 NodesStatsRequest statsRequest = new NodesStatsRequest(nodes);
                 statsRequest.clear().fs(true);
 
-                client.admin().cluster().nodesStats(statsRequest, new ActionListener<NodesStatsResponse>() {
-                    @Override
-                    public void onResponse(NodesStatsResponse stats) {
-                        try {
-                            Table tab = buildTable(request, state, stats);
-                            channel.sendResponse(RestTable.buildResponse(tab, request, channel));
-                        } catch (Throwable e) {
-                            onFailure(e);
-                        }
-                    }
-
+                client.admin().cluster().nodesStats(statsRequest, new RestResponseListener<NodesStatsResponse>(channel) {
                     @Override
-                    public void onFailure(Throwable e) {
-                        try {
-                            channel.sendResponse(new BytesRestResponse(request, e));
-                        } catch (IOException e1) {
-                            logger.error("Failed to send failure response", e1);
-                        }
+                    public RestResponse buildResponse(NodesStatsResponse stats) throws Exception {
+                        Table tab = buildTable(request, state, stats);
+                        return RestTable.buildResponse(tab, channel);
                     }
                 });
             }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
-            }
         });
 
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/cat/RestCatAction.java b/src/main/java/org/elasticsearch/rest/action/cat/RestCatAction.java
index 2ff2fca1fa725..59da870af2202 100644
--- a/src/main/java/org/elasticsearch/rest/action/cat/RestCatAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/cat/RestCatAction.java
@@ -49,14 +49,6 @@ public RestCatAction(Settings settings, Client client, RestController controller
 
     @Override
     public void handleRequest(final RestRequest request, final RestChannel channel) {
-        try {
-            channel.sendResponse(new BytesRestResponse(RestStatus.OK, HELP));
-        } catch (Throwable t) {
-            try {
-                channel.sendResponse(new BytesRestResponse(request, t));
-            } catch (IOException e1) {
-                logger.error("Failed to send failure response", e1);
-            }
-        }
+        channel.sendResponse(new BytesRestResponse(RestStatus.OK, HELP));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/cat/RestCountAction.java b/src/main/java/org/elasticsearch/rest/action/cat/RestCountAction.java
index 10e80f1f31169..b108cd0e04107 100644
--- a/src/main/java/org/elasticsearch/rest/action/cat/RestCountAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/cat/RestCountAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.cat;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.count.CountRequest;
 import org.elasticsearch.action.count.CountResponse;
 import org.elasticsearch.action.support.QuerySourceBuilder;
@@ -29,16 +28,16 @@
 import org.elasticsearch.common.Table;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.rest.BytesRestResponse;
 import org.elasticsearch.rest.RestChannel;
 import org.elasticsearch.rest.RestController;
 import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.RestResponse;
 import org.elasticsearch.rest.action.support.RestActions;
+import org.elasticsearch.rest.action.support.RestResponseListener;
 import org.elasticsearch.rest.action.support.RestTable;
 import org.joda.time.format.DateTimeFormat;
 import org.joda.time.format.DateTimeFormatter;
 
-import java.io.IOException;
 import java.util.concurrent.TimeUnit;
 
 import static org.elasticsearch.rest.RestRequest.Method.GET;
@@ -74,23 +73,10 @@ public void doRequest(final RestRequest request, final RestChannel channel) {
             }
         }
 
-        client.count(countRequest, new ActionListener<CountResponse>() {
+        client.count(countRequest, new RestResponseListener<CountResponse>(channel) {
             @Override
-            public void onResponse(CountResponse countResponse) {
-                try {
-                    channel.sendResponse(RestTable.buildResponse(buildTable(request, countResponse), request, channel));
-                } catch (Throwable t) {
-                    onFailure(t);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable t) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, t));
-                } catch (IOException e) {
-                    logger.error("Failed to send failure response", e);
-                }
+            public RestResponse buildResponse(CountResponse countResponse) throws Exception {
+                return RestTable.buildResponse(buildTable(request, countResponse), channel);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/cat/RestHealthAction.java b/src/main/java/org/elasticsearch/rest/action/cat/RestHealthAction.java
index b9a06cb3064cc..364fb91df09fd 100644
--- a/src/main/java/org/elasticsearch/rest/action/cat/RestHealthAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/cat/RestHealthAction.java
@@ -19,22 +19,21 @@
 
 package org.elasticsearch.rest.action.cat;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.cluster.health.ClusterHealthRequest;
 import org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse;
 import org.elasticsearch.client.Client;
 import org.elasticsearch.common.Table;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.rest.BytesRestResponse;
 import org.elasticsearch.rest.RestChannel;
 import org.elasticsearch.rest.RestController;
 import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.RestResponse;
+import org.elasticsearch.rest.action.support.RestResponseListener;
 import org.elasticsearch.rest.action.support.RestTable;
 import org.joda.time.format.DateTimeFormat;
 import org.joda.time.format.DateTimeFormatter;
 
-import java.io.IOException;
 import java.util.Locale;
 import java.util.concurrent.TimeUnit;
 
@@ -57,23 +56,10 @@ void documentation(StringBuilder sb) {
     public void doRequest(final RestRequest request, final RestChannel channel) {
         ClusterHealthRequest clusterHealthRequest = new ClusterHealthRequest();
 
-        client.admin().cluster().health(clusterHealthRequest, new ActionListener<ClusterHealthResponse>() {
+        client.admin().cluster().health(clusterHealthRequest, new RestResponseListener<ClusterHealthResponse>(channel) {
             @Override
-            public void onResponse(final ClusterHealthResponse health) {
-                try {
-                    channel.sendResponse(RestTable.buildResponse(buildTable(health, request), request, channel));
-                } catch (Throwable t) {
-                    onFailure(t);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
+            public RestResponse buildResponse(final ClusterHealthResponse health) throws Exception {
+                return RestTable.buildResponse(buildTable(health, request), channel);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/cat/RestIndicesAction.java b/src/main/java/org/elasticsearch/rest/action/cat/RestIndicesAction.java
index c8567d1f6885f..2ef578f3f6304 100644
--- a/src/main/java/org/elasticsearch/rest/action/cat/RestIndicesAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/cat/RestIndicesAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.cat;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.cluster.health.ClusterHealthRequest;
 import org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse;
 import org.elasticsearch.action.admin.cluster.health.ClusterIndexHealth;
@@ -30,15 +29,18 @@
 import org.elasticsearch.action.admin.indices.stats.IndicesStatsResponse;
 import org.elasticsearch.client.Client;
 import org.elasticsearch.client.Requests;
-import org.elasticsearch.cluster.metadata.MetaData;
 import org.elasticsearch.common.Strings;
 import org.elasticsearch.common.Table;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.RestChannel;
+import org.elasticsearch.rest.RestController;
+import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.RestResponse;
+import org.elasticsearch.rest.action.support.RestActionListener;
+import org.elasticsearch.rest.action.support.RestResponseListener;
 import org.elasticsearch.rest.action.support.RestTable;
 
-import java.io.IOException;
 import java.util.Locale;
 
 import static org.elasticsearch.rest.RestRequest.Method.GET;
@@ -66,59 +68,28 @@ public void doRequest(final RestRequest request, final RestChannel channel) {
         clusterStateRequest.local(request.paramAsBoolean("local", clusterStateRequest.local()));
         clusterStateRequest.masterNodeTimeout(request.paramAsTime("master_timeout", clusterStateRequest.masterNodeTimeout()));
 
-        client.admin().cluster().state(clusterStateRequest, new ActionListener<ClusterStateResponse>() {
+        client.admin().cluster().state(clusterStateRequest, new RestActionListener<ClusterStateResponse>(channel) {
             @Override
-            public void onResponse(final ClusterStateResponse clusterStateResponse) {
+            public void processResponse(final ClusterStateResponse clusterStateResponse) {
                 final String[] concreteIndices = clusterStateResponse.getState().metaData().concreteIndicesIgnoreMissing(indices);
                 ClusterHealthRequest clusterHealthRequest = Requests.clusterHealthRequest(concreteIndices);
                 clusterHealthRequest.local(request.paramAsBoolean("local", clusterHealthRequest.local()));
-                client.admin().cluster().health(clusterHealthRequest, new ActionListener<ClusterHealthResponse>() {
+                client.admin().cluster().health(clusterHealthRequest, new RestActionListener<ClusterHealthResponse>(channel) {
                     @Override
-                    public void onResponse(final ClusterHealthResponse clusterHealthResponse) {
+                    public void processResponse(final ClusterHealthResponse clusterHealthResponse) {
                         IndicesStatsRequest indicesStatsRequest = new IndicesStatsRequest();
                         indicesStatsRequest.all();
-                        client.admin().indices().stats(indicesStatsRequest, new ActionListener<IndicesStatsResponse>() {
+                        client.admin().indices().stats(indicesStatsRequest, new RestResponseListener<IndicesStatsResponse>(channel) {
                             @Override
-                            public void onResponse(IndicesStatsResponse indicesStatsResponse) {
-                                try {
-                                    Table tab = buildTable(request, concreteIndices, clusterHealthResponse, indicesStatsResponse);
-                                    channel.sendResponse(RestTable.buildResponse(tab, request, channel));
-                                } catch (Throwable e) {
-                                    onFailure(e);
-                                }
-                            }
-
-                            @Override
-                            public void onFailure(Throwable e) {
-                                try {
-                                    channel.sendResponse(new BytesRestResponse(request, e));
-                                } catch (IOException e1) {
-                                    logger.error("Failed to send failure response", e1);
-                                }
+                            public RestResponse buildResponse(IndicesStatsResponse indicesStatsResponse) throws Exception {
+                                Table tab = buildTable(request, concreteIndices, clusterHealthResponse, indicesStatsResponse);
+                                return RestTable.buildResponse(tab, channel);
                             }
                         });
 
                     }
-
-                    @Override
-                    public void onFailure(Throwable e) {
-                        try {
-                            channel.sendResponse(new BytesRestResponse(request, e));
-                        } catch (IOException e1) {
-                            logger.error("Failed to send failure response", e1);
-                        }
-                    }
                 });
             }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
-            }
         });
     }
 
diff --git a/src/main/java/org/elasticsearch/rest/action/cat/RestMasterAction.java b/src/main/java/org/elasticsearch/rest/action/cat/RestMasterAction.java
index 98c09305b720f..653ff879f9de8 100644
--- a/src/main/java/org/elasticsearch/rest/action/cat/RestMasterAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/cat/RestMasterAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.cat;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.cluster.state.ClusterStateRequest;
 import org.elasticsearch.action.admin.cluster.state.ClusterStateResponse;
 import org.elasticsearch.client.Client;
@@ -28,14 +27,13 @@
 import org.elasticsearch.common.Table;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.rest.BytesRestResponse;
 import org.elasticsearch.rest.RestChannel;
 import org.elasticsearch.rest.RestController;
 import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.RestResponse;
+import org.elasticsearch.rest.action.support.RestResponseListener;
 import org.elasticsearch.rest.action.support.RestTable;
 
-import java.io.IOException;
-
 import static org.elasticsearch.rest.RestRequest.Method.GET;
 
 public class RestMasterAction extends AbstractCatAction {
@@ -58,23 +56,10 @@ public void doRequest(final RestRequest request, final RestChannel channel) {
         clusterStateRequest.local(request.paramAsBoolean("local", clusterStateRequest.local()));
         clusterStateRequest.masterNodeTimeout(request.paramAsTime("master_timeout", clusterStateRequest.masterNodeTimeout()));
 
-        client.admin().cluster().state(clusterStateRequest, new ActionListener<ClusterStateResponse>() {
-            @Override
-            public void onResponse(final ClusterStateResponse clusterStateResponse) {
-                try {
-                    channel.sendResponse(RestTable.buildResponse(buildTable(request, clusterStateResponse), request, channel));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
+        client.admin().cluster().state(clusterStateRequest, new RestResponseListener<ClusterStateResponse>(channel) {
             @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
+            public RestResponse buildResponse(final ClusterStateResponse clusterStateResponse) throws Exception {
+                return RestTable.buildResponse(buildTable(request, clusterStateResponse), channel);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/cat/RestNodesAction.java b/src/main/java/org/elasticsearch/rest/action/cat/RestNodesAction.java
index 7d825c74adb71..14e6b1b5ff8c0 100644
--- a/src/main/java/org/elasticsearch/rest/action/cat/RestNodesAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/cat/RestNodesAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.cat;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.cluster.node.info.NodeInfo;
 import org.elasticsearch.action.admin.cluster.node.info.NodesInfoRequest;
 import org.elasticsearch.action.admin.cluster.node.info.NodesInfoResponse;
@@ -36,13 +35,14 @@
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.transport.InetSocketTransportAddress;
-import org.elasticsearch.rest.BytesRestResponse;
 import org.elasticsearch.rest.RestChannel;
 import org.elasticsearch.rest.RestController;
 import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.RestResponse;
+import org.elasticsearch.rest.action.support.RestActionListener;
+import org.elasticsearch.rest.action.support.RestResponseListener;
 import org.elasticsearch.rest.action.support.RestTable;
 
-import java.io.IOException;
 import java.util.Locale;
 
 import static org.elasticsearch.rest.RestRequest.Method.GET;
@@ -67,56 +67,25 @@ public void doRequest(final RestRequest request, final RestChannel channel) {
         clusterStateRequest.local(request.paramAsBoolean("local", clusterStateRequest.local()));
         clusterStateRequest.masterNodeTimeout(request.paramAsTime("master_timeout", clusterStateRequest.masterNodeTimeout()));
 
-        client.admin().cluster().state(clusterStateRequest, new ActionListener<ClusterStateResponse>() {
+        client.admin().cluster().state(clusterStateRequest, new RestActionListener<ClusterStateResponse>(channel) {
             @Override
-            public void onResponse(final ClusterStateResponse clusterStateResponse) {
+            public void processResponse(final ClusterStateResponse clusterStateResponse) {
                 NodesInfoRequest nodesInfoRequest = new NodesInfoRequest();
                 nodesInfoRequest.clear().jvm(true).os(true).process(true);
-                client.admin().cluster().nodesInfo(nodesInfoRequest, new ActionListener<NodesInfoResponse>() {
+                client.admin().cluster().nodesInfo(nodesInfoRequest, new RestActionListener<NodesInfoResponse>(channel) {
                     @Override
-                    public void onResponse(final NodesInfoResponse nodesInfoResponse) {
+                    public void processResponse(final NodesInfoResponse nodesInfoResponse) {
                         NodesStatsRequest nodesStatsRequest = new NodesStatsRequest();
                         nodesStatsRequest.clear().jvm(true).os(true).fs(true).indices(true);
-                        client.admin().cluster().nodesStats(nodesStatsRequest, new ActionListener<NodesStatsResponse>() {
+                        client.admin().cluster().nodesStats(nodesStatsRequest, new RestResponseListener<NodesStatsResponse>(channel) {
                             @Override
-                            public void onResponse(NodesStatsResponse nodesStatsResponse) {
-                                try {
-                                    channel.sendResponse(RestTable.buildResponse(buildTable(request, clusterStateResponse, nodesInfoResponse, nodesStatsResponse), request, channel));
-                                } catch (Throwable e) {
-                                    onFailure(e);
-                                }
-                            }
-
-                            @Override
-                            public void onFailure(Throwable e) {
-                                try {
-                                    channel.sendResponse(new BytesRestResponse(request, e));
-                                } catch (IOException e1) {
-                                    logger.error("Failed to send failure response", e1);
-                                }
+                            public RestResponse buildResponse(NodesStatsResponse nodesStatsResponse) throws Exception {
+                                return RestTable.buildResponse(buildTable(request, clusterStateResponse, nodesInfoResponse, nodesStatsResponse), channel);
                             }
                         });
                     }
-
-                    @Override
-                    public void onFailure(Throwable e) {
-                        try {
-                            channel.sendResponse(new BytesRestResponse(request, e));
-                        } catch (IOException e1) {
-                            logger.error("Failed to send failure response", e1);
-                        }
-                    }
                 });
             }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
-            }
         });
     }
 
diff --git a/src/main/java/org/elasticsearch/rest/action/cat/RestPendingClusterTasksAction.java b/src/main/java/org/elasticsearch/rest/action/cat/RestPendingClusterTasksAction.java
index 8473aeccb3d09..f6dbe11fcb21a 100644
--- a/src/main/java/org/elasticsearch/rest/action/cat/RestPendingClusterTasksAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/cat/RestPendingClusterTasksAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.cat;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.cluster.tasks.PendingClusterTasksRequest;
 import org.elasticsearch.action.admin.cluster.tasks.PendingClusterTasksResponse;
 import org.elasticsearch.client.Client;
@@ -27,11 +26,13 @@
 import org.elasticsearch.common.Table;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.RestChannel;
+import org.elasticsearch.rest.RestController;
+import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.RestResponse;
+import org.elasticsearch.rest.action.support.RestResponseListener;
 import org.elasticsearch.rest.action.support.RestTable;
 
-import java.io.IOException;
-
 import static org.elasticsearch.rest.RestRequest.Method.GET;
 
 public class RestPendingClusterTasksAction extends AbstractCatAction {
@@ -51,24 +52,11 @@ public void doRequest(final RestRequest request, final RestChannel channel) {
         PendingClusterTasksRequest pendingClusterTasksRequest = new PendingClusterTasksRequest();
         pendingClusterTasksRequest.masterNodeTimeout(request.paramAsTime("master_timeout", pendingClusterTasksRequest.masterNodeTimeout()));
         pendingClusterTasksRequest.local(request.paramAsBoolean("local", pendingClusterTasksRequest.local()));
-        client.admin().cluster().pendingClusterTasks(pendingClusterTasksRequest, new ActionListener<PendingClusterTasksResponse>() {
-            @Override
-            public void onResponse(PendingClusterTasksResponse pendingClusterTasks) {
-                try {
-                    Table tab = buildTable(request, pendingClusterTasks);
-                    channel.sendResponse(RestTable.buildResponse(tab, request, channel));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
+        client.admin().cluster().pendingClusterTasks(pendingClusterTasksRequest, new RestResponseListener<PendingClusterTasksResponse>(channel) {
             @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
+            public RestResponse buildResponse(PendingClusterTasksResponse pendingClusterTasks) throws Exception {
+                Table tab = buildTable(request, pendingClusterTasks);
+                return RestTable.buildResponse(tab, channel);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/cat/RestPluginsAction.java b/src/main/java/org/elasticsearch/rest/action/cat/RestPluginsAction.java
index 747df7ee5a3cd..cdf8efffbe709 100644
--- a/src/main/java/org/elasticsearch/rest/action/cat/RestPluginsAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/cat/RestPluginsAction.java
@@ -31,10 +31,12 @@
 import org.elasticsearch.common.Table;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.rest.AbstractRestResponseActionListener;
 import org.elasticsearch.rest.RestChannel;
 import org.elasticsearch.rest.RestController;
 import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.RestResponse;
+import org.elasticsearch.rest.action.support.RestActionListener;
+import org.elasticsearch.rest.action.support.RestResponseListener;
 import org.elasticsearch.rest.action.support.RestTable;
 
 import static org.elasticsearch.rest.RestRequest.Method.GET;
@@ -59,19 +61,15 @@ public void doRequest(final RestRequest request, final RestChannel channel) {
         clusterStateRequest.local(request.paramAsBoolean("local", clusterStateRequest.local()));
         clusterStateRequest.masterNodeTimeout(request.paramAsTime("master_timeout", clusterStateRequest.masterNodeTimeout()));
 
-        client.admin().cluster().state(clusterStateRequest, new AbstractRestResponseActionListener<ClusterStateResponse>(request, channel, logger) {
+        client.admin().cluster().state(clusterStateRequest, new RestActionListener<ClusterStateResponse>(channel) {
             @Override
-            public void onResponse(final ClusterStateResponse clusterStateResponse) {
+            public void processResponse(final ClusterStateResponse clusterStateResponse) throws Exception {
                 NodesInfoRequest nodesInfoRequest = new NodesInfoRequest();
                 nodesInfoRequest.clear().plugins(true);
-                client.admin().cluster().nodesInfo(nodesInfoRequest, new AbstractRestResponseActionListener<NodesInfoResponse>(request, channel, logger) {
+                client.admin().cluster().nodesInfo(nodesInfoRequest, new RestResponseListener<NodesInfoResponse>(channel) {
                     @Override
-                    public void onResponse(final NodesInfoResponse nodesInfoResponse) {
-                        try {
-                            channel.sendResponse(RestTable.buildResponse(buildTable(request, clusterStateResponse, nodesInfoResponse), request, channel));
-                        } catch (Throwable e) {
-                            onFailure(e);
-                        }
+                    public RestResponse buildResponse(final NodesInfoResponse nodesInfoResponse) throws Exception {
+                        return RestTable.buildResponse(buildTable(request, clusterStateResponse, nodesInfoResponse), channel);
                     }
                 });
             }
diff --git a/src/main/java/org/elasticsearch/rest/action/cat/RestRecoveryAction.java b/src/main/java/org/elasticsearch/rest/action/cat/RestRecoveryAction.java
index 4490505779570..5c1fc478c600d 100644
--- a/src/main/java/org/elasticsearch/rest/action/cat/RestRecoveryAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/cat/RestRecoveryAction.java
@@ -19,8 +19,7 @@
 
 package org.elasticsearch.rest.action.cat;
 
-import org.elasticsearch.action.ActionListener;
-import org.elasticsearch.indices.recovery.RecoveryState;
+import org.apache.lucene.util.CollectionUtil;
 import org.elasticsearch.action.admin.indices.recovery.RecoveryRequest;
 import org.elasticsearch.action.admin.indices.recovery.RecoveryResponse;
 import org.elasticsearch.action.admin.indices.recovery.ShardRecoveryResponse;
@@ -30,16 +29,17 @@
 import org.elasticsearch.common.Table;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.rest.BytesRestResponse;
+import org.elasticsearch.indices.recovery.RecoveryState;
 import org.elasticsearch.rest.RestChannel;
 import org.elasticsearch.rest.RestController;
 import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.RestResponse;
+import org.elasticsearch.rest.action.support.RestResponseListener;
 import org.elasticsearch.rest.action.support.RestTable;
 
-import org.apache.lucene.util.CollectionUtil;
-
-import java.io.IOException;
-import java.util.*;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Locale;
 
 import static org.elasticsearch.rest.RestRequest.Method.GET;
 
@@ -71,28 +71,10 @@ public void doRequest(final RestRequest request, final RestChannel channel) {
         recoveryRequest.listenerThreaded(false);
         recoveryRequest.indicesOptions(IndicesOptions.fromRequest(request, recoveryRequest.indicesOptions()));
 
-        client.admin().indices().recoveries(recoveryRequest, new ActionListener<RecoveryResponse>() {
-
+        client.admin().indices().recoveries(recoveryRequest, new RestResponseListener<RecoveryResponse>(channel) {
             @Override
-            public void onResponse(final RecoveryResponse response) {
-                try {
-                    channel.sendResponse(RestTable.buildResponse(buildRecoveryTable(request, response), request, channel));
-                } catch (Throwable e) {
-                    try {
-                        channel.sendResponse(new BytesRestResponse(request, e));
-                    } catch (IOException e2) {
-                        logger.error("Unable to send recovery status response", e2);
-                    }
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
+            public RestResponse buildResponse(final RecoveryResponse response) throws Exception {
+                return RestTable.buildResponse(buildRecoveryTable(request, response), channel);
             }
         });
     }
@@ -122,8 +104,8 @@ Table getTableWithHeader(RestRequest request) {
      * buildRecoveryTable will build a table of recovery information suitable
      * for displaying at the command line.
      *
-     * @param request   A Rest request
-     * @param response  A recovery status response
+     * @param request  A Rest request
+     * @param response A recovery status response
      * @return A table containing index, shardId, node, target size, recovered size and percentage for each recovering replica
      */
     public Table buildRecoveryTable(RestRequest request, RecoveryResponse response) {
diff --git a/src/main/java/org/elasticsearch/rest/action/cat/RestSegmentsAction.java b/src/main/java/org/elasticsearch/rest/action/cat/RestSegmentsAction.java
index 0a95b3f516c11..587a69fcb3c14 100644
--- a/src/main/java/org/elasticsearch/rest/action/cat/RestSegmentsAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/cat/RestSegmentsAction.java
@@ -31,6 +31,8 @@
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.index.engine.Segment;
 import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.action.support.RestActionListener;
+import org.elasticsearch.rest.action.support.RestResponseListener;
 import org.elasticsearch.rest.action.support.RestTable;
 
 import java.io.IOException;
@@ -57,43 +59,19 @@ void doRequest(final RestRequest request, final RestChannel channel) {
         clusterStateRequest.masterNodeTimeout(request.paramAsTime("master_timeout", clusterStateRequest.masterNodeTimeout()));
         clusterStateRequest.clear().nodes(true).routingTable(true).indices(indices);
 
-        client.admin().cluster().state(clusterStateRequest, new AbstractRestResponseActionListener<ClusterStateResponse>(request, channel, logger) {
+        client.admin().cluster().state(clusterStateRequest, new RestActionListener<ClusterStateResponse>(channel) {
             @Override
-            public void onResponse(final ClusterStateResponse clusterStateResponse) {
+            public void processResponse(final ClusterStateResponse clusterStateResponse) {
                 final IndicesSegmentsRequest indicesSegmentsRequest = new IndicesSegmentsRequest();
                 indicesSegmentsRequest.indices(indices);
-
-                client.admin().indices().segments(indicesSegmentsRequest, new ActionListener<IndicesSegmentResponse>() {
+                client.admin().indices().segments(indicesSegmentsRequest, new RestResponseListener<IndicesSegmentResponse>(channel) {
                     @Override
-                    public void onResponse(final IndicesSegmentResponse indicesSegmentResponse) {
+                    public RestResponse buildResponse(final IndicesSegmentResponse indicesSegmentResponse) throws Exception {
                         final Map<String, IndexSegments> indicesSegments = indicesSegmentResponse.getIndices();
-                        try {
-                            Table tab = buildTable(request, clusterStateResponse, indicesSegments);
-                            channel.sendResponse(RestTable.buildResponse(tab, request, channel));
-                        } catch (Throwable e) {
-                            onFailure(e);
-                        }
-                    }
-
-                    @Override
-                    public void onFailure(Throwable e) {
-                        try {
-                            channel.sendResponse(new BytesRestResponse(request, e));
-                        } catch (IOException e1) {
-                            logger.error("Failed to send failure response", e1);
-                        }
+                        Table tab = buildTable(request, clusterStateResponse, indicesSegments);
+                        return RestTable.buildResponse(tab, channel);
                     }
                 });
-
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/cat/RestShardsAction.java b/src/main/java/org/elasticsearch/rest/action/cat/RestShardsAction.java
index cedabafcfb180..3e2fa39cfc97e 100644
--- a/src/main/java/org/elasticsearch/rest/action/cat/RestShardsAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/cat/RestShardsAction.java
@@ -30,7 +30,12 @@
 import org.elasticsearch.common.Table;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.RestChannel;
+import org.elasticsearch.rest.RestController;
+import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.RestResponse;
+import org.elasticsearch.rest.action.support.RestActionListener;
+import org.elasticsearch.rest.action.support.RestResponseListener;
 import org.elasticsearch.rest.action.support.RestTable;
 
 import static org.elasticsearch.rest.RestRequest.Method.GET;
@@ -57,19 +62,15 @@ public void doRequest(final RestRequest request, final RestChannel channel) {
         clusterStateRequest.local(request.paramAsBoolean("local", clusterStateRequest.local()));
         clusterStateRequest.masterNodeTimeout(request.paramAsTime("master_timeout", clusterStateRequest.masterNodeTimeout()));
         clusterStateRequest.clear().nodes(true).routingTable(true).indices(indices);
-        client.admin().cluster().state(clusterStateRequest, new AbstractRestResponseActionListener<ClusterStateResponse>(request, channel, logger) {
+        client.admin().cluster().state(clusterStateRequest, new RestActionListener<ClusterStateResponse>(channel) {
             @Override
-            public void onResponse(final ClusterStateResponse clusterStateResponse) {
+            public void processResponse(final ClusterStateResponse clusterStateResponse) {
                 IndicesStatsRequest indicesStatsRequest = new IndicesStatsRequest();
                 indicesStatsRequest.all();
-                client.admin().indices().stats(indicesStatsRequest, new AbstractRestResponseActionListener<IndicesStatsResponse>(request, channel, logger) {
+                client.admin().indices().stats(indicesStatsRequest, new RestResponseListener<IndicesStatsResponse>(channel) {
                     @Override
-                    public void onResponse(IndicesStatsResponse indicesStatsResponse) {
-                        try {
-                            channel.sendResponse(RestTable.buildResponse(buildTable(request, clusterStateResponse, indicesStatsResponse), request, channel));
-                        } catch (Throwable e) {
-                            onFailure(e);
-                        }
+                    public RestResponse buildResponse(IndicesStatsResponse indicesStatsResponse) throws Exception {
+                        return RestTable.buildResponse(buildTable(request, clusterStateResponse, indicesStatsResponse), channel);
                     }
                 });
             }
diff --git a/src/main/java/org/elasticsearch/rest/action/cat/RestThreadPoolAction.java b/src/main/java/org/elasticsearch/rest/action/cat/RestThreadPoolAction.java
index 1fb4163fa2f6c..b1989f5fe9a2f 100644
--- a/src/main/java/org/elasticsearch/rest/action/cat/RestThreadPoolAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/cat/RestThreadPoolAction.java
@@ -36,10 +36,12 @@
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.transport.InetSocketTransportAddress;
-import org.elasticsearch.rest.AbstractRestResponseActionListener;
 import org.elasticsearch.rest.RestChannel;
 import org.elasticsearch.rest.RestController;
 import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.RestResponse;
+import org.elasticsearch.rest.action.support.RestActionListener;
+import org.elasticsearch.rest.action.support.RestResponseListener;
 import org.elasticsearch.rest.action.support.RestTable;
 import org.elasticsearch.threadpool.ThreadPool;
 import org.elasticsearch.threadpool.ThreadPoolStats;
@@ -50,7 +52,7 @@
 
 public class RestThreadPoolAction extends AbstractCatAction {
 
-    private final static String[] SUPPORTED_NAMES = new String[] {
+    private final static String[] SUPPORTED_NAMES = new String[]{
             ThreadPool.Names.BULK,
             ThreadPool.Names.FLUSH,
             ThreadPool.Names.GENERIC,
@@ -67,7 +69,7 @@ public class RestThreadPoolAction extends AbstractCatAction {
             ThreadPool.Names.WARMER
     };
 
-    private final static String[] SUPPORTED_ALIASES = new String[] {
+    private final static String[] SUPPORTED_ALIASES = new String[]{
             "b",
             "f",
             "ge",
@@ -84,7 +86,7 @@ public class RestThreadPoolAction extends AbstractCatAction {
             "w"
     };
 
-    private final static String[] DEFAULT_THREAD_POOLS = new String[] {
+    private final static String[] DEFAULT_THREAD_POOLS = new String[]{
             ThreadPool.Names.BULK,
             ThreadPool.Names.INDEX,
             ThreadPool.Names.SEARCH,
@@ -123,24 +125,20 @@ public void doRequest(final RestRequest request, final RestChannel channel) {
         clusterStateRequest.masterNodeTimeout(request.paramAsTime("master_timeout", clusterStateRequest.masterNodeTimeout()));
         final String[] pools = fetchSortedPools(request, DEFAULT_THREAD_POOLS);
 
-        client.admin().cluster().state(clusterStateRequest, new AbstractRestResponseActionListener<ClusterStateResponse>(request, channel, logger) {
+        client.admin().cluster().state(clusterStateRequest, new RestActionListener<ClusterStateResponse>(channel) {
             @Override
-            public void onResponse(final ClusterStateResponse clusterStateResponse) {
+            public void processResponse(final ClusterStateResponse clusterStateResponse) {
                 NodesInfoRequest nodesInfoRequest = new NodesInfoRequest();
                 nodesInfoRequest.clear().process(true);
-                client.admin().cluster().nodesInfo(nodesInfoRequest, new AbstractRestResponseActionListener<NodesInfoResponse>(request, channel, logger) {
+                client.admin().cluster().nodesInfo(nodesInfoRequest, new RestActionListener<NodesInfoResponse>(channel) {
                     @Override
-                    public void onResponse(final NodesInfoResponse nodesInfoResponse) {
+                    public void processResponse(final NodesInfoResponse nodesInfoResponse) {
                         NodesStatsRequest nodesStatsRequest = new NodesStatsRequest();
                         nodesStatsRequest.clear().threadPool(true);
-                        client.admin().cluster().nodesStats(nodesStatsRequest, new AbstractRestResponseActionListener<NodesStatsResponse>(request, channel, logger) {
+                        client.admin().cluster().nodesStats(nodesStatsRequest, new RestResponseListener<NodesStatsResponse>(channel) {
                             @Override
-                            public void onResponse(NodesStatsResponse nodesStatsResponse) {
-                                try {
-                                    channel.sendResponse(RestTable.buildResponse(buildTable(request, clusterStateResponse, nodesInfoResponse, nodesStatsResponse, pools), request, channel));
-                                } catch (Throwable e) {
-                                    onFailure(e);
-                                }
+                            public RestResponse buildResponse(NodesStatsResponse nodesStatsResponse) throws Exception {
+                                return RestTable.buildResponse(buildTable(request, clusterStateResponse, nodesInfoResponse, nodesStatsResponse, pools), channel);
                             }
                         });
                     }
diff --git a/src/main/java/org/elasticsearch/rest/action/count/RestCountAction.java b/src/main/java/org/elasticsearch/rest/action/count/RestCountAction.java
index ca31643e19976..81a2b45f0c496 100644
--- a/src/main/java/org/elasticsearch/rest/action/count/RestCountAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/count/RestCountAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.count;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.count.CountRequest;
 import org.elasticsearch.action.count.CountResponse;
 import org.elasticsearch.action.support.IndicesOptions;
@@ -32,14 +31,11 @@
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
 import org.elasticsearch.rest.action.support.RestActions;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
-
-import java.io.IOException;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
 import static org.elasticsearch.action.count.CountRequest.DEFAULT_MIN_SCORE;
 import static org.elasticsearch.rest.RestRequest.Method.GET;
 import static org.elasticsearch.rest.RestRequest.Method.POST;
-import static org.elasticsearch.rest.RestStatus.BAD_REQUEST;
 import static org.elasticsearch.rest.action.support.RestActions.buildBroadcastShardsHeader;
 
 /**
@@ -63,64 +59,40 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         CountRequest countRequest = new CountRequest(Strings.splitStringByCommaToArray(request.param("index")));
         countRequest.indicesOptions(IndicesOptions.fromRequest(request, countRequest.indicesOptions()));
         countRequest.listenerThreaded(false);
-        try {
-            BroadcastOperationThreading operationThreading = BroadcastOperationThreading.fromString(request.param("operation_threading"), BroadcastOperationThreading.THREAD_PER_SHARD);
-            if (operationThreading == BroadcastOperationThreading.NO_THREADS) {
-                // since we don't spawn, don't allow no_threads, but change it to a single thread
-                operationThreading = BroadcastOperationThreading.SINGLE_THREAD;
-            }
-            countRequest.operationThreading(operationThreading);
-            if (request.hasContent()) {
-                countRequest.source(request.content(), request.contentUnsafe());
+        BroadcastOperationThreading operationThreading = BroadcastOperationThreading.fromString(request.param("operation_threading"), BroadcastOperationThreading.THREAD_PER_SHARD);
+        if (operationThreading == BroadcastOperationThreading.NO_THREADS) {
+            // since we don't spawn, don't allow no_threads, but change it to a single thread
+            operationThreading = BroadcastOperationThreading.SINGLE_THREAD;
+        }
+        countRequest.operationThreading(operationThreading);
+        if (request.hasContent()) {
+            countRequest.source(request.content(), request.contentUnsafe());
+        } else {
+            String source = request.param("source");
+            if (source != null) {
+                countRequest.source(source);
             } else {
-                String source = request.param("source");
-                if (source != null) {
-                    countRequest.source(source);
-                } else {
-                    QuerySourceBuilder querySourceBuilder = RestActions.parseQuerySource(request);
-                    if (querySourceBuilder != null) {
-                        countRequest.source(querySourceBuilder);
-                    }
+                QuerySourceBuilder querySourceBuilder = RestActions.parseQuerySource(request);
+                if (querySourceBuilder != null) {
+                    countRequest.source(querySourceBuilder);
                 }
             }
-            countRequest.routing(request.param("routing"));
-            countRequest.minScore(request.paramAsFloat("min_score", DEFAULT_MIN_SCORE));
-            countRequest.types(Strings.splitStringByCommaToArray(request.param("type")));
-            countRequest.preference(request.param("preference"));
-        } catch (Exception e) {
-            try {
-                XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                channel.sendResponse(new BytesRestResponse(BAD_REQUEST, builder.startObject().field("error", e.getMessage()).endObject()));
-            } catch (IOException e1) {
-                logger.error("Failed to send failure response", e1);
-            }
-            return;
         }
+        countRequest.routing(request.param("routing"));
+        countRequest.minScore(request.paramAsFloat("min_score", DEFAULT_MIN_SCORE));
+        countRequest.types(Strings.splitStringByCommaToArray(request.param("type")));
+        countRequest.preference(request.param("preference"));
 
-        client.count(countRequest, new ActionListener<CountResponse>() {
+        client.count(countRequest, new RestBuilderListener<CountResponse>(channel) {
             @Override
-            public void onResponse(CountResponse response) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
-                    builder.field("count", response.getCount());
+            public RestResponse buildResponse(CountResponse response, XContentBuilder builder) throws Exception {
+                builder.startObject();
+                builder.field("count", response.getCount());
 
-                    buildBroadcastShardsHeader(builder, response);
+                buildBroadcastShardsHeader(builder, response);
 
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(response.status(), builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
+                builder.endObject();
+                return new BytesRestResponse(response.status(), builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/delete/RestDeleteAction.java b/src/main/java/org/elasticsearch/rest/action/delete/RestDeleteAction.java
index 65f45731cdbd6..dd416dca6fa32 100644
--- a/src/main/java/org/elasticsearch/rest/action/delete/RestDeleteAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/delete/RestDeleteAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.delete;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.WriteConsistencyLevel;
 import org.elasticsearch.action.delete.DeleteRequest;
 import org.elasticsearch.action.delete.DeleteResponse;
@@ -32,9 +31,7 @@
 import org.elasticsearch.index.VersionType;
 import org.elasticsearch.rest.*;
 import org.elasticsearch.rest.action.support.RestActions;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
-
-import java.io.IOException;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
 import static org.elasticsearch.rest.RestRequest.Method.DELETE;
 import static org.elasticsearch.rest.RestStatus.NOT_FOUND;
@@ -74,35 +71,21 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
             deleteRequest.consistencyLevel(WriteConsistencyLevel.fromString(consistencyLevel));
         }
 
-        client.delete(deleteRequest, new ActionListener<DeleteResponse>() {
-            @Override
-            public void onResponse(DeleteResponse result) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject()
-                            .field(Fields.FOUND, result.isFound())
-                            .field(Fields._INDEX, result.getIndex())
-                            .field(Fields._TYPE, result.getType())
-                            .field(Fields._ID, result.getId())
-                            .field(Fields._VERSION, result.getVersion())
-                            .endObject();
-                    RestStatus status = OK;
-                    if (!result.isFound()) {
-                        status = NOT_FOUND;
-                    }
-                    channel.sendResponse(new BytesRestResponse(status, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
+        client.delete(deleteRequest, new RestBuilderListener<DeleteResponse>(channel) {
             @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
+            public RestResponse buildResponse(DeleteResponse result, XContentBuilder builder) throws Exception {
+                builder.startObject()
+                        .field(Fields.FOUND, result.isFound())
+                        .field(Fields._INDEX, result.getIndex())
+                        .field(Fields._TYPE, result.getType())
+                        .field(Fields._ID, result.getId())
+                        .field(Fields._VERSION, result.getVersion())
+                        .endObject();
+                RestStatus status = OK;
+                if (!result.isFound()) {
+                    status = NOT_FOUND;
                 }
+                return new BytesRestResponse(status, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/deletebyquery/RestDeleteByQueryAction.java b/src/main/java/org/elasticsearch/rest/action/deletebyquery/RestDeleteByQueryAction.java
index aafad28e5a30b..58dfaa135871e 100644
--- a/src/main/java/org/elasticsearch/rest/action/deletebyquery/RestDeleteByQueryAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/deletebyquery/RestDeleteByQueryAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.deletebyquery;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.ShardOperationFailedException;
 import org.elasticsearch.action.WriteConsistencyLevel;
 import org.elasticsearch.action.deletebyquery.DeleteByQueryRequest;
@@ -37,12 +36,9 @@
 import org.elasticsearch.common.xcontent.XContentBuilderString;
 import org.elasticsearch.rest.*;
 import org.elasticsearch.rest.action.support.RestActions;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
-
-import java.io.IOException;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
 import static org.elasticsearch.rest.RestRequest.Method.DELETE;
-import static org.elasticsearch.rest.RestStatus.PRECONDITION_FAILED;
 
 /**
  *
@@ -60,88 +56,64 @@ public RestDeleteByQueryAction(Settings settings, Client client, RestController
     public void handleRequest(final RestRequest request, final RestChannel channel) {
         DeleteByQueryRequest deleteByQueryRequest = new DeleteByQueryRequest(Strings.splitStringByCommaToArray(request.param("index")));
         deleteByQueryRequest.listenerThreaded(false);
-        try {
-            if (request.hasContent()) {
-                deleteByQueryRequest.source(request.content(), request.contentUnsafe());
+        if (request.hasContent()) {
+            deleteByQueryRequest.source(request.content(), request.contentUnsafe());
+        } else {
+            String source = request.param("source");
+            if (source != null) {
+                deleteByQueryRequest.source(source);
             } else {
-                String source = request.param("source");
-                if (source != null) {
-                    deleteByQueryRequest.source(source);
-                } else {
-                    QuerySourceBuilder querySourceBuilder = RestActions.parseQuerySource(request);
-                    if (querySourceBuilder != null) {
-                        deleteByQueryRequest.source(querySourceBuilder);
-                    }
+                QuerySourceBuilder querySourceBuilder = RestActions.parseQuerySource(request);
+                if (querySourceBuilder != null) {
+                    deleteByQueryRequest.source(querySourceBuilder);
                 }
             }
-            deleteByQueryRequest.types(Strings.splitStringByCommaToArray(request.param("type")));
-            deleteByQueryRequest.timeout(request.paramAsTime("timeout", ShardDeleteByQueryRequest.DEFAULT_TIMEOUT));
+        }
+        deleteByQueryRequest.types(Strings.splitStringByCommaToArray(request.param("type")));
+        deleteByQueryRequest.timeout(request.paramAsTime("timeout", ShardDeleteByQueryRequest.DEFAULT_TIMEOUT));
 
-            deleteByQueryRequest.routing(request.param("routing"));
-            String replicationType = request.param("replication");
-            if (replicationType != null) {
-                deleteByQueryRequest.replicationType(ReplicationType.fromString(replicationType));
-            }
-            String consistencyLevel = request.param("consistency");
-            if (consistencyLevel != null) {
-                deleteByQueryRequest.consistencyLevel(WriteConsistencyLevel.fromString(consistencyLevel));
-            }
-            deleteByQueryRequest.indicesOptions(IndicesOptions.fromRequest(request, deleteByQueryRequest.indicesOptions()));
-        } catch (Exception e) {
-            try {
-                XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                channel.sendResponse(new BytesRestResponse(PRECONDITION_FAILED, builder.startObject().field("error", e.getMessage()).endObject()));
-            } catch (IOException e1) {
-                logger.error("Failed to send failure response", e1);
-            }
-            return;
+        deleteByQueryRequest.routing(request.param("routing"));
+        String replicationType = request.param("replication");
+        if (replicationType != null) {
+            deleteByQueryRequest.replicationType(ReplicationType.fromString(replicationType));
+        }
+        String consistencyLevel = request.param("consistency");
+        if (consistencyLevel != null) {
+            deleteByQueryRequest.consistencyLevel(WriteConsistencyLevel.fromString(consistencyLevel));
         }
-        client.deleteByQuery(deleteByQueryRequest, new ActionListener<DeleteByQueryResponse>() {
+        deleteByQueryRequest.indicesOptions(IndicesOptions.fromRequest(request, deleteByQueryRequest.indicesOptions()));
+        client.deleteByQuery(deleteByQueryRequest, new RestBuilderListener<DeleteByQueryResponse>(channel) {
             @Override
-            public void onResponse(DeleteByQueryResponse result) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    RestStatus restStatus = result.status();
-                    builder.startObject();
-                    builder.startObject(Fields._INDICES);
-                    for (IndexDeleteByQueryResponse indexDeleteByQueryResponse : result.getIndices().values()) {
-                        builder.startObject(indexDeleteByQueryResponse.getIndex(), XContentBuilder.FieldCaseConversion.NONE);
+            public RestResponse buildResponse(DeleteByQueryResponse result, XContentBuilder builder) throws Exception {
+                RestStatus restStatus = result.status();
+                builder.startObject();
+                builder.startObject(Fields._INDICES);
+                for (IndexDeleteByQueryResponse indexDeleteByQueryResponse : result.getIndices().values()) {
+                    builder.startObject(indexDeleteByQueryResponse.getIndex(), XContentBuilder.FieldCaseConversion.NONE);
 
-                        builder.startObject(Fields._SHARDS);
-                        builder.field(Fields.TOTAL, indexDeleteByQueryResponse.getTotalShards());
-                        builder.field(Fields.SUCCESSFUL, indexDeleteByQueryResponse.getSuccessfulShards());
-                        builder.field(Fields.FAILED, indexDeleteByQueryResponse.getFailedShards());
-                        ShardOperationFailedException[] failures = indexDeleteByQueryResponse.getFailures();
-                        if (failures != null && failures.length > 0) {
-                            builder.startArray(Fields.FAILURES);
-                            for (ShardOperationFailedException shardFailure : failures) {
-                                builder.startObject();
-                                builder.field(Fields.INDEX, shardFailure.index());
-                                builder.field(Fields.SHARD, shardFailure.shardId());
-                                builder.field(Fields.REASON, shardFailure.reason());
-                                builder.endObject();
-                            }
-                            builder.endArray();
+                    builder.startObject(Fields._SHARDS);
+                    builder.field(Fields.TOTAL, indexDeleteByQueryResponse.getTotalShards());
+                    builder.field(Fields.SUCCESSFUL, indexDeleteByQueryResponse.getSuccessfulShards());
+                    builder.field(Fields.FAILED, indexDeleteByQueryResponse.getFailedShards());
+                    ShardOperationFailedException[] failures = indexDeleteByQueryResponse.getFailures();
+                    if (failures != null && failures.length > 0) {
+                        builder.startArray(Fields.FAILURES);
+                        for (ShardOperationFailedException shardFailure : failures) {
+                            builder.startObject();
+                            builder.field(Fields.INDEX, shardFailure.index());
+                            builder.field(Fields.SHARD, shardFailure.shardId());
+                            builder.field(Fields.REASON, shardFailure.reason());
+                            builder.endObject();
                         }
-                        builder.endObject();
-
-                        builder.endObject();
+                        builder.endArray();
                     }
                     builder.endObject();
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(restStatus, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
 
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
+                    builder.endObject();
                 }
+                builder.endObject();
+                builder.endObject();
+                return new BytesRestResponse(restStatus, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/explain/RestExplainAction.java b/src/main/java/org/elasticsearch/rest/action/explain/RestExplainAction.java
index e1486d426642a..9df3e59537de3 100644
--- a/src/main/java/org/elasticsearch/rest/action/explain/RestExplainAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/explain/RestExplainAction.java
@@ -21,7 +21,6 @@
 
 import org.apache.lucene.search.Explanation;
 import org.elasticsearch.ElasticsearchIllegalArgumentException;
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.explain.ExplainRequest;
 import org.elasticsearch.action.explain.ExplainResponse;
 import org.elasticsearch.action.support.QuerySourceBuilder;
@@ -36,6 +35,7 @@
 import org.elasticsearch.index.query.QueryBuilders;
 import org.elasticsearch.index.query.QueryStringQueryBuilder;
 import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 import org.elasticsearch.search.fetch.source.FetchSourceContext;
 
 import java.io.IOException;
@@ -44,7 +44,6 @@
 import static org.elasticsearch.rest.RestRequest.Method.POST;
 import static org.elasticsearch.rest.RestStatus.NOT_FOUND;
 import static org.elasticsearch.rest.RestStatus.OK;
-import static org.elasticsearch.rest.action.support.RestXContentBuilder.restContentBuilder;
 
 /**
  * Rest action for computing a score explanation for specific documents.
@@ -103,34 +102,28 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
 
         explainRequest.fetchSourceContext(FetchSourceContext.parseFromRestRequest(request));
 
-        client.explain(explainRequest, new ActionListener<ExplainResponse>() {
-
+        client.explain(explainRequest, new RestBuilderListener<ExplainResponse>(channel) {
             @Override
-            public void onResponse(ExplainResponse response) {
-                try {
-                    XContentBuilder builder = restContentBuilder(request);
-                    builder.startObject();
-                    builder.field(Fields._INDEX, explainRequest.index())
-                            .field(Fields._TYPE, explainRequest.type())
-                            .field(Fields._ID, explainRequest.id())
-                            .field(Fields.MATCHED, response.isMatch());
-
-                    if (response.hasExplanation()) {
-                        builder.startObject(Fields.EXPLANATION);
-                        buildExplanation(builder, response.getExplanation());
-                        builder.endObject();
-                    }
-                    GetResult getResult = response.getGetResult();
-                    if (getResult != null) {
-                        builder.startObject(Fields.GET);
-                        response.getGetResult().toXContentEmbedded(builder, request);
-                        builder.endObject();
-                    }
+            public RestResponse buildResponse(ExplainResponse response, XContentBuilder builder) throws Exception {
+                builder.startObject();
+                builder.field(Fields._INDEX, explainRequest.index())
+                        .field(Fields._TYPE, explainRequest.type())
+                        .field(Fields._ID, explainRequest.id())
+                        .field(Fields.MATCHED, response.isMatch());
+
+                if (response.hasExplanation()) {
+                    builder.startObject(Fields.EXPLANATION);
+                    buildExplanation(builder, response.getExplanation());
                     builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(response.isExists() ? OK : NOT_FOUND, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
                 }
+                GetResult getResult = response.getGetResult();
+                if (getResult != null) {
+                    builder.startObject(Fields.GET);
+                    response.getGetResult().toXContentEmbedded(builder, request);
+                    builder.endObject();
+                }
+                builder.endObject();
+                return new BytesRestResponse(response.isExists() ? OK : NOT_FOUND, builder);
             }
 
             private void buildExplanation(XContentBuilder builder, Explanation explanation) throws IOException {
@@ -147,15 +140,6 @@ private void buildExplanation(XContentBuilder builder, Explanation explanation)
                     builder.endArray();
                 }
             }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
-            }
         });
     }
 
diff --git a/src/main/java/org/elasticsearch/rest/action/get/RestGetAction.java b/src/main/java/org/elasticsearch/rest/action/get/RestGetAction.java
index 31ef77c1ce39c..981bde8459c51 100644
--- a/src/main/java/org/elasticsearch/rest/action/get/RestGetAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/get/RestGetAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.get;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.get.GetRequest;
 import org.elasticsearch.action.get.GetResponse;
 import org.elasticsearch.client.Client;
@@ -30,14 +29,12 @@
 import org.elasticsearch.index.VersionType;
 import org.elasticsearch.rest.*;
 import org.elasticsearch.rest.action.support.RestActions;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 import org.elasticsearch.search.fetch.source.FetchSourceContext;
 
-import java.io.IOException;
-
 import static org.elasticsearch.rest.RestRequest.Method.GET;
 import static org.elasticsearch.rest.RestStatus.NOT_FOUND;
 import static org.elasticsearch.rest.RestStatus.OK;
-import static org.elasticsearch.rest.action.support.RestXContentBuilder.restContentBuilder;
 
 /**
  *
@@ -74,29 +71,16 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
 
         getRequest.fetchSourceContext(FetchSourceContext.parseFromRestRequest(request));
 
-        client.get(getRequest, new ActionListener<GetResponse>() {
-            @Override
-            public void onResponse(GetResponse response) {
-
-                try {
-                    XContentBuilder builder = restContentBuilder(request);
-                    response.toXContent(builder, request);
-                    if (!response.isExists()) {
-                        channel.sendResponse(new BytesRestResponse(NOT_FOUND, builder));
-                    } else {
-                        channel.sendResponse(new BytesRestResponse(OK, builder));
-                    }
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
+        client.get(getRequest, new RestBuilderListener<GetResponse>(channel) {
             @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
+            public RestResponse buildResponse(GetResponse response, XContentBuilder builder) throws Exception {
+                builder.startObject();
+                response.toXContent(builder, request);
+                builder.endObject();
+                if (!response.isExists()) {
+                    return new BytesRestResponse(NOT_FOUND, builder);
+                } else {
+                    return new BytesRestResponse(OK, builder);
                 }
             }
         });
diff --git a/src/main/java/org/elasticsearch/rest/action/get/RestGetSourceAction.java b/src/main/java/org/elasticsearch/rest/action/get/RestGetSourceAction.java
index 497f261ce5a39..950028838569b 100644
--- a/src/main/java/org/elasticsearch/rest/action/get/RestGetSourceAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/get/RestGetSourceAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.get;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.ActionRequestValidationException;
 import org.elasticsearch.action.get.GetRequest;
 import org.elasticsearch.action.get.GetResponse;
@@ -27,8 +26,9 @@
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.xcontent.XContentBuilder;
+import org.elasticsearch.common.xcontent.XContentHelper;
 import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
+import org.elasticsearch.rest.action.support.RestResponseListener;
 import org.elasticsearch.search.fetch.source.FetchSourceContext;
 
 import java.io.IOException;
@@ -36,7 +36,6 @@
 import static org.elasticsearch.rest.RestRequest.Method.GET;
 import static org.elasticsearch.rest.RestStatus.NOT_FOUND;
 import static org.elasticsearch.rest.RestStatus.OK;
-import static org.elasticsearch.rest.action.support.RestXContentBuilder.restContentBuilder;
 
 /**
  *
@@ -66,35 +65,21 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
             try {
                 ActionRequestValidationException validationError = new ActionRequestValidationException();
                 validationError.addValidationError("fetching source can not be disabled");
-                channel.sendResponse(new BytesRestResponse(request, validationError));
+                channel.sendResponse(new BytesRestResponse(channel, validationError));
             } catch (IOException e) {
                 logger.error("Failed to send failure response", e);
             }
         }
 
-        client.get(getRequest, new ActionListener<GetResponse>() {
+        client.get(getRequest, new RestResponseListener<GetResponse>(channel) {
             @Override
-            public void onResponse(GetResponse response) {
-
-                try {
-                    XContentBuilder builder = restContentBuilder(request, response.getSourceInternal());
-                    if (!response.isExists()) {
-                        channel.sendResponse(new BytesRestResponse(NOT_FOUND, builder));
-                    } else {
-                        RestXContentBuilder.directSource(response.getSourceInternal(), builder, request);
-                        channel.sendResponse(new BytesRestResponse(OK, builder));
-                    }
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
+            public RestResponse buildResponse(GetResponse response) throws Exception {
+                XContentBuilder builder = channel.newBuilder(response.getSourceInternal());
+                if (!response.isExists()) {
+                    return new BytesRestResponse(NOT_FOUND, builder);
+                } else {
+                    XContentHelper.writeDirect(response.getSourceInternal(), builder, request);
+                    return new BytesRestResponse(OK, builder);
                 }
             }
         });
diff --git a/src/main/java/org/elasticsearch/rest/action/get/RestHeadAction.java b/src/main/java/org/elasticsearch/rest/action/get/RestHeadAction.java
index 54cb4e9f5150d..00cb76aacf751 100644
--- a/src/main/java/org/elasticsearch/rest/action/get/RestHeadAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/get/RestHeadAction.java
@@ -19,8 +19,6 @@
 
 package org.elasticsearch.rest.action.get;
 
-import org.elasticsearch.ExceptionsHelper;
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.get.GetRequest;
 import org.elasticsearch.action.get.GetResponse;
 import org.elasticsearch.client.Client;
@@ -28,6 +26,7 @@
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.action.support.RestResponseListener;
 
 import static org.elasticsearch.rest.RestRequest.Method.HEAD;
 import static org.elasticsearch.rest.RestStatus.NOT_FOUND;
@@ -59,26 +58,13 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         getRequest.fields(Strings.EMPTY_ARRAY);
         // TODO we can also just return the document size as Content-Length
 
-        client.get(getRequest, new ActionListener<GetResponse>() {
+        client.get(getRequest, new RestResponseListener<GetResponse>(channel) {
             @Override
-            public void onResponse(GetResponse response) {
-                try {
-                    if (!response.isExists()) {
-                        channel.sendResponse(new BytesRestResponse(NOT_FOUND));
-                    } else {
-                        channel.sendResponse(new BytesRestResponse(OK));
-                    }
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(ExceptionsHelper.status(e)));
-                } catch (Exception e1) {
-                    logger.error("Failed to send failure response", e1);
+            public RestResponse buildResponse(GetResponse response) {
+                if (!response.isExists()) {
+                    return new BytesRestResponse(NOT_FOUND);
+                } else {
+                    return new BytesRestResponse(OK);
                 }
             }
         });
diff --git a/src/main/java/org/elasticsearch/rest/action/get/RestMultiGetAction.java b/src/main/java/org/elasticsearch/rest/action/get/RestMultiGetAction.java
index ef928f37a9177..cf4eaa80a94e6 100644
--- a/src/main/java/org/elasticsearch/rest/action/get/RestMultiGetAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/get/RestMultiGetAction.java
@@ -29,6 +29,7 @@
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
 import org.elasticsearch.rest.action.support.RestActions;
+import org.elasticsearch.rest.action.support.RestToXContentListener;
 import org.elasticsearch.search.fetch.source.FetchSourceContext;
 
 import java.io.IOException;
@@ -37,7 +38,6 @@
 import static org.elasticsearch.rest.RestRequest.Method.POST;
 import static org.elasticsearch.rest.RestStatus.BAD_REQUEST;
 import static org.elasticsearch.rest.RestStatus.OK;
-import static org.elasticsearch.rest.action.support.RestXContentBuilder.restContentBuilder;
 
 public class RestMultiGetAction extends BaseRestHandler {
 
@@ -57,7 +57,7 @@ public RestMultiGetAction(Settings settings, Client client, RestController contr
     }
 
     @Override
-    public void handleRequest(final RestRequest request, final RestChannel channel) {
+    public void handleRequest(final RestRequest request, final RestChannel channel) throws Exception {
         MultiGetRequest multiGetRequest = new MultiGetRequest();
         multiGetRequest.listenerThreaded(false);
         multiGetRequest.refresh(request.paramAsBoolean("refresh", multiGetRequest.refresh()));
@@ -71,39 +71,8 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         }
 
         FetchSourceContext defaultFetchSource = FetchSourceContext.parseFromRestRequest(request);
+        multiGetRequest.add(request.param("index"), request.param("type"), sFields, defaultFetchSource, request.param("routing"), RestActions.getRestContent(request), allowExplicitIndex);
 
-        try {
-            multiGetRequest.add(request.param("index"), request.param("type"), sFields, defaultFetchSource, request.param("routing"), RestActions.getRestContent(request), allowExplicitIndex);
-        } catch (Exception e) {
-            try {
-                XContentBuilder builder = restContentBuilder(request);
-                channel.sendResponse(new BytesRestResponse(BAD_REQUEST, builder.startObject().field("error", e.getMessage()).endObject()));
-            } catch (IOException e1) {
-                logger.error("Failed to send failure response", e1);
-            }
-            return;
-        }
-
-        client.multiGet(multiGetRequest, new ActionListener<MultiGetResponse>() {
-            @Override
-            public void onResponse(MultiGetResponse response) {
-                try {
-                    XContentBuilder builder = restContentBuilder(request);
-                    response.toXContent(builder, request);
-                    channel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
-            }
-        });
+        client.multiGet(multiGetRequest, new RestToXContentListener<MultiGetResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/index/RestIndexAction.java b/src/main/java/org/elasticsearch/rest/action/index/RestIndexAction.java
index 731795775d9c1..4f267f5a2ccd7 100644
--- a/src/main/java/org/elasticsearch/rest/action/index/RestIndexAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/index/RestIndexAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.index;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.WriteConsistencyLevel;
 import org.elasticsearch.action.index.IndexRequest;
 import org.elasticsearch.action.index.IndexResponse;
@@ -32,7 +31,7 @@
 import org.elasticsearch.index.VersionType;
 import org.elasticsearch.rest.*;
 import org.elasticsearch.rest.action.support.RestActions;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
 import java.io.IOException;
 
@@ -87,7 +86,7 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
                 indexRequest.opType(IndexRequest.OpType.CREATE);
             } else {
                 try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
+                    XContentBuilder builder = channel.newBuilder();
                     channel.sendResponse(new BytesRestResponse(BAD_REQUEST, builder.startObject().field("error", "opType [" + sOpType + "] not allowed, either [index] or [create] are allowed").endObject()));
                 } catch (IOException e1) {
                     logger.warn("Failed to send response", e1);
@@ -103,35 +102,21 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         if (consistencyLevel != null) {
             indexRequest.consistencyLevel(WriteConsistencyLevel.fromString(consistencyLevel));
         }
-        client.index(indexRequest, new ActionListener<IndexResponse>() {
+        client.index(indexRequest, new RestBuilderListener<IndexResponse>(channel) {
             @Override
-            public void onResponse(IndexResponse response) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject()
-                            .field(Fields._INDEX, response.getIndex())
-                            .field(Fields._TYPE, response.getType())
-                            .field(Fields._ID, response.getId())
-                            .field(Fields._VERSION, response.getVersion())
-                            .field(Fields.CREATED, response.isCreated());
-                    builder.endObject();
-                    RestStatus status = OK;
-                    if (response.isCreated()) {
-                        status = CREATED;
-                    }
-                    channel.sendResponse(new BytesRestResponse(status, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
+            public RestResponse buildResponse(IndexResponse response, XContentBuilder builder) throws Exception {
+                builder.startObject()
+                        .field(Fields._INDEX, response.getIndex())
+                        .field(Fields._TYPE, response.getType())
+                        .field(Fields._ID, response.getId())
+                        .field(Fields._VERSION, response.getVersion())
+                        .field(Fields.CREATED, response.isCreated());
+                builder.endObject();
+                RestStatus status = OK;
+                if (response.isCreated()) {
+                    status = CREATED;
                 }
+                return new BytesRestResponse(status, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/main/RestMainAction.java b/src/main/java/org/elasticsearch/rest/action/main/RestMainAction.java
index 2775af0a6fb99..318217d5032d0 100644
--- a/src/main/java/org/elasticsearch/rest/action/main/RestMainAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/main/RestMainAction.java
@@ -23,7 +23,6 @@
 import org.elasticsearch.Build;
 import org.elasticsearch.ExceptionsHelper;
 import org.elasticsearch.Version;
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.admin.cluster.state.ClusterStateRequest;
 import org.elasticsearch.action.admin.cluster.state.ClusterStateResponse;
 import org.elasticsearch.client.Client;
@@ -32,7 +31,7 @@
 import org.elasticsearch.common.unit.TimeValue;
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
+import org.elasticsearch.rest.action.support.RestResponseListener;
 
 import static org.elasticsearch.rest.RestRequest.Method.GET;
 import static org.elasticsearch.rest.RestRequest.Method.HEAD;
@@ -59,61 +58,43 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         clusterStateRequest.masterNodeTimeout(TimeValue.timeValueMillis(0));
         clusterStateRequest.local(true);
         clusterStateRequest.clear().blocks(true);
-        client.admin().cluster().state(clusterStateRequest, new ActionListener<ClusterStateResponse>() {
+        client.admin().cluster().state(clusterStateRequest, new RestResponseListener<ClusterStateResponse>(channel) {
             @Override
-            public void onResponse(ClusterStateResponse response) {
+            public RestResponse buildResponse(ClusterStateResponse response) throws Exception {
                 RestStatus status = RestStatus.OK;
                 if (response.getState().blocks().hasGlobalBlock(RestStatus.SERVICE_UNAVAILABLE)) {
                     status = RestStatus.SERVICE_UNAVAILABLE;
                 }
                 if (request.method() == RestRequest.Method.HEAD) {
-                    channel.sendResponse(new BytesRestResponse(status));
-                    return;
+                    return new BytesRestResponse(status);
                 }
 
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
+                XContentBuilder builder = channel.newBuilder();
 
-                    // Default to pretty printing, but allow ?pretty=false to disable
-                    if (!request.hasParam("pretty")) {
-                        builder.prettyPrint().lfAtEnd();
-                    }
-
-                    builder.startObject();
-                    builder.field("status", status.getStatus());
-                    if (settings.get("name") != null) {
-                        builder.field("name", settings.get("name"));
-                    }
-                    builder.startObject("version")
-                            .field("number", version.number())
-                            .field("build_hash", Build.CURRENT.hash())
-                            .field("build_timestamp", Build.CURRENT.timestamp())
-                            .field("build_snapshot", version.snapshot)
-                                    // We use the lucene version from lucene constants since
-                                    // this includes bugfix release version as well and is already in
-                                    // the right format. We can also be sure that the format is maitained
-                                    // since this is also recorded in lucene segments and has BW compat
-                            .field("lucene_version", Constants.LUCENE_MAIN_VERSION)
-                            .endObject();
-                    builder.field("tagline", "You Know, for Search");
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(status, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
+                // Default to pretty printing, but allow ?pretty=false to disable
+                if (!request.hasParam("pretty")) {
+                    builder.prettyPrint().lfAtEnd();
                 }
-            }
 
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    if (request.method() == HEAD) {
-                        channel.sendResponse(new BytesRestResponse(ExceptionsHelper.status(e)));
-                    } else {
-                        channel.sendResponse(new BytesRestResponse(request, e));
-                    }
-                } catch (Exception e1) {
-                    logger.warn("Failed to send response", e);
+                builder.startObject();
+                builder.field("status", status.getStatus());
+                if (settings.get("name") != null) {
+                    builder.field("name", settings.get("name"));
                 }
+                builder.startObject("version")
+                        .field("number", version.number())
+                        .field("build_hash", Build.CURRENT.hash())
+                        .field("build_timestamp", Build.CURRENT.timestamp())
+                        .field("build_snapshot", version.snapshot)
+                                // We use the lucene version from lucene constants since
+                                // this includes bugfix release version as well and is already in
+                                // the right format. We can also be sure that the format is maitained
+                                // since this is also recorded in lucene segments and has BW compat
+                        .field("lucene_version", Constants.LUCENE_MAIN_VERSION)
+                        .endObject();
+                builder.field("tagline", "You Know, for Search");
+                builder.endObject();
+                return new BytesRestResponse(status, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/mlt/RestMoreLikeThisAction.java b/src/main/java/org/elasticsearch/rest/action/mlt/RestMoreLikeThisAction.java
index a87f8d58d7edd..be6cc09315b7a 100644
--- a/src/main/java/org/elasticsearch/rest/action/mlt/RestMoreLikeThisAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/mlt/RestMoreLikeThisAction.java
@@ -28,6 +28,7 @@
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.action.support.RestToXContentListener;
 import org.elasticsearch.search.Scroll;
 
 import java.io.IOException;
@@ -36,9 +37,7 @@
 import static org.elasticsearch.common.unit.TimeValue.parseTimeValue;
 import static org.elasticsearch.rest.RestRequest.Method.GET;
 import static org.elasticsearch.rest.RestRequest.Method.POST;
-import static org.elasticsearch.rest.RestStatus.BAD_REQUEST;
 import static org.elasticsearch.rest.RestStatus.OK;
-import static org.elasticsearch.rest.action.support.RestXContentBuilder.restContentBuilder;
 
 /**
  *
@@ -58,71 +57,39 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         mltRequest.routing(request.param("routing"));
 
         mltRequest.listenerThreaded(false);
-        try {
-            //TODO the ParseField class that encapsulates the supported names used for an attribute
-            //needs some work if it is to be used in a REST context like this too
-            // See the MoreLikeThisQueryParser constants that hold the valid syntax
-            mltRequest.fields(request.paramAsStringArray("mlt_fields", null));
-            mltRequest.percentTermsToMatch(request.paramAsFloat("percent_terms_to_match", -1));
-            mltRequest.minTermFreq(request.paramAsInt("min_term_freq", -1));
-            mltRequest.maxQueryTerms(request.paramAsInt("max_query_terms", -1));
-            mltRequest.stopWords(request.paramAsStringArray("stop_words", null));
-            mltRequest.minDocFreq(request.paramAsInt("min_doc_freq", -1));
-            mltRequest.maxDocFreq(request.paramAsInt("max_doc_freq", -1));
-            mltRequest.minWordLength(request.paramAsInt("min_word_len", request.paramAsInt("min_word_length",-1)));
-            mltRequest.maxWordLength(request.paramAsInt("max_word_len", request.paramAsInt("max_word_length",-1)));
-            mltRequest.boostTerms(request.paramAsFloat("boost_terms", -1));
+        //TODO the ParseField class that encapsulates the supported names used for an attribute
+        //needs some work if it is to be used in a REST context like this too
+        // See the MoreLikeThisQueryParser constants that hold the valid syntax
+        mltRequest.fields(request.paramAsStringArray("mlt_fields", null));
+        mltRequest.percentTermsToMatch(request.paramAsFloat("percent_terms_to_match", -1));
+        mltRequest.minTermFreq(request.paramAsInt("min_term_freq", -1));
+        mltRequest.maxQueryTerms(request.paramAsInt("max_query_terms", -1));
+        mltRequest.stopWords(request.paramAsStringArray("stop_words", null));
+        mltRequest.minDocFreq(request.paramAsInt("min_doc_freq", -1));
+        mltRequest.maxDocFreq(request.paramAsInt("max_doc_freq", -1));
+        mltRequest.minWordLength(request.paramAsInt("min_word_len", request.paramAsInt("min_word_length", -1)));
+        mltRequest.maxWordLength(request.paramAsInt("max_word_len", request.paramAsInt("max_word_length", -1)));
+        mltRequest.boostTerms(request.paramAsFloat("boost_terms", -1));
 
-            mltRequest.searchType(SearchType.fromString(request.param("search_type")));
-            mltRequest.searchIndices(request.paramAsStringArray("search_indices", null));
-            mltRequest.searchTypes(request.paramAsStringArray("search_types", null));
-            mltRequest.searchQueryHint(request.param("search_query_hint"));
-            mltRequest.searchSize(request.paramAsInt("search_size", mltRequest.searchSize()));
-            mltRequest.searchFrom(request.paramAsInt("search_from", mltRequest.searchFrom()));
-            String searchScroll = request.param("search_scroll");
-            if (searchScroll != null) {
-                mltRequest.searchScroll(new Scroll(parseTimeValue(searchScroll, null)));
-            }
-            if (request.hasContent()) {
-                mltRequest.searchSource(request.content(), request.contentUnsafe());
-            } else {
-                String searchSource = request.param("search_source");
-                if (searchSource != null) {
-                    mltRequest.searchSource(searchSource);
-                }
-            }
-        } catch (Exception e) {
-            try {
-                XContentBuilder builder = restContentBuilder(request);
-                channel.sendResponse(new BytesRestResponse(BAD_REQUEST, builder.startObject().field("error", e.getMessage()).endObject()));
-            } catch (IOException e1) {
-                logger.error("Failed to send failure response", e1);
-            }
-            return;
+        mltRequest.searchType(SearchType.fromString(request.param("search_type")));
+        mltRequest.searchIndices(request.paramAsStringArray("search_indices", null));
+        mltRequest.searchTypes(request.paramAsStringArray("search_types", null));
+        mltRequest.searchQueryHint(request.param("search_query_hint"));
+        mltRequest.searchSize(request.paramAsInt("search_size", mltRequest.searchSize()));
+        mltRequest.searchFrom(request.paramAsInt("search_from", mltRequest.searchFrom()));
+        String searchScroll = request.param("search_scroll");
+        if (searchScroll != null) {
+            mltRequest.searchScroll(new Scroll(parseTimeValue(searchScroll, null)));
         }
-
-        client.moreLikeThis(mltRequest, new ActionListener<SearchResponse>() {
-            @Override
-            public void onResponse(SearchResponse response) {
-                try {
-                    XContentBuilder builder = restContentBuilder(request);
-                    builder.startObject();
-                    response.toXContent(builder, request);
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
+        if (request.hasContent()) {
+            mltRequest.searchSource(request.content(), request.contentUnsafe());
+        } else {
+            String searchSource = request.param("search_source");
+            if (searchSource != null) {
+                mltRequest.searchSource(searchSource);
             }
+        }
 
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
-            }
-        });
+        client.moreLikeThis(mltRequest, new RestToXContentListener<SearchResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/percolate/RestMultiPercolateAction.java b/src/main/java/org/elasticsearch/rest/action/percolate/RestMultiPercolateAction.java
index fb3a9ae76b095..74328e2004e6d 100644
--- a/src/main/java/org/elasticsearch/rest/action/percolate/RestMultiPercolateAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/percolate/RestMultiPercolateAction.java
@@ -29,13 +29,13 @@
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
 import org.elasticsearch.rest.action.support.RestActions;
+import org.elasticsearch.rest.action.support.RestToXContentListener;
 
 import java.io.IOException;
 
 import static org.elasticsearch.rest.RestRequest.Method.GET;
 import static org.elasticsearch.rest.RestRequest.Method.POST;
 import static org.elasticsearch.rest.RestStatus.OK;
-import static org.elasticsearch.rest.action.support.RestXContentBuilder.restContentBuilder;
 
 /**
  *
@@ -59,45 +59,14 @@ public RestMultiPercolateAction(Settings settings, Client client, RestController
     }
 
     @Override
-    public void handleRequest(final RestRequest restRequest, final RestChannel restChannel) {
+    public void handleRequest(final RestRequest restRequest, final RestChannel restChannel) throws Exception {
         MultiPercolateRequest multiPercolateRequest = new MultiPercolateRequest();
         multiPercolateRequest.indicesOptions(IndicesOptions.fromRequest(restRequest, multiPercolateRequest.indicesOptions()));
         multiPercolateRequest.indices(Strings.splitStringByCommaToArray(restRequest.param("index")));
         multiPercolateRequest.documentType(restRequest.param("type"));
+        multiPercolateRequest.add(RestActions.getRestContent(restRequest), restRequest.contentUnsafe(), allowExplicitIndex);
 
-        try {
-            multiPercolateRequest.add(RestActions.getRestContent(restRequest), restRequest.contentUnsafe(), allowExplicitIndex);
-        } catch (Exception e) {
-            try {
-                restChannel.sendResponse(new BytesRestResponse(restRequest, e));
-            } catch (IOException e1) {
-                logger.error("Failed to send failure response", e1);
-            }
-            return;
-        }
-
-        client.multiPercolate(multiPercolateRequest, new ActionListener<MultiPercolateResponse>() {
-
-            @Override
-            public void onResponse(MultiPercolateResponse response) {
-                try {
-                    XContentBuilder builder = restContentBuilder(restRequest);
-                    response.toXContent(builder, restRequest);
-                    restChannel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (IOException e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    restChannel.sendResponse(new BytesRestResponse(restRequest, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
-            }
-        });
+        client.multiPercolate(multiPercolateRequest, new RestToXContentListener<MultiPercolateResponse>(restChannel));
     }
 
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/percolate/RestPercolateAction.java b/src/main/java/org/elasticsearch/rest/action/percolate/RestPercolateAction.java
index 54479af36b221..2c2d7c21b4a0b 100644
--- a/src/main/java/org/elasticsearch/rest/action/percolate/RestPercolateAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/percolate/RestPercolateAction.java
@@ -32,7 +32,7 @@
 import org.elasticsearch.index.VersionType;
 import org.elasticsearch.rest.*;
 import org.elasticsearch.rest.action.support.RestActions;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
+import org.elasticsearch.rest.action.support.RestToXContentListener;
 
 import java.io.IOException;
 
@@ -112,27 +112,7 @@ void executePercolate(final PercolateRequest percolateRequest, final RestRequest
             percolateRequest.operationThreading(operationThreading);
         }
 
-        client.percolate(percolateRequest, new ActionListener<PercolateResponse>() {
-            @Override
-            public void onResponse(PercolateResponse response) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(restRequest);
-                    response.toXContent(builder, restRequest);
-                    restChannel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    restChannel.sendResponse(new BytesRestResponse(restRequest, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
-            }
-        });
+        client.percolate(percolateRequest, new RestToXContentListener<PercolateResponse>(restChannel));
     }
 
     @Override
diff --git a/src/main/java/org/elasticsearch/rest/action/search/RestClearScrollAction.java b/src/main/java/org/elasticsearch/rest/action/search/RestClearScrollAction.java
index 0781db8047c21..b81c81cb5fc67 100644
--- a/src/main/java/org/elasticsearch/rest/action/search/RestClearScrollAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/search/RestClearScrollAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.search;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.search.ClearScrollRequest;
 import org.elasticsearch.action.search.ClearScrollResponse;
 import org.elasticsearch.client.Client;
@@ -27,15 +26,13 @@
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.xcontent.XContentBuilder;
-import org.elasticsearch.common.xcontent.XContentBuilderString;
 import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
-import java.io.IOException;
 import java.util.Arrays;
 
 import static org.elasticsearch.rest.RestRequest.Method.DELETE;
 import static org.elasticsearch.rest.RestStatus.OK;
-import static org.elasticsearch.rest.action.support.RestXContentBuilder.restContentBuilder;
 
 /**
  */
@@ -55,26 +52,12 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
 
         ClearScrollRequest clearRequest = new ClearScrollRequest();
         clearRequest.setScrollIds(Arrays.asList(splitScrollIds(scrollIds)));
-        client.clearScroll(clearRequest, new ActionListener<ClearScrollResponse>() {
+        client.clearScroll(clearRequest, new RestBuilderListener<ClearScrollResponse>(channel) {
             @Override
-            public void onResponse(ClearScrollResponse response) {
-                try {
-                    XContentBuilder builder = restContentBuilder(request);
-                    builder.startObject();
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
+            public RestResponse buildResponse(ClearScrollResponse response, XContentBuilder builder) throws Exception {
+                builder.startObject();
+                builder.endObject();
+                return new BytesRestResponse(OK, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/action/search/RestMultiSearchAction.java b/src/main/java/org/elasticsearch/rest/action/search/RestMultiSearchAction.java
index 9e1752f069119..147e235b7c8eb 100644
--- a/src/main/java/org/elasticsearch/rest/action/search/RestMultiSearchAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/search/RestMultiSearchAction.java
@@ -30,6 +30,7 @@
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
 import org.elasticsearch.rest.action.support.RestActions;
+import org.elasticsearch.rest.action.support.RestToXContentListener;
 
 import java.io.IOException;
 
@@ -37,7 +38,6 @@
 import static org.elasticsearch.rest.RestRequest.Method.POST;
 import static org.elasticsearch.rest.RestStatus.BAD_REQUEST;
 import static org.elasticsearch.rest.RestStatus.OK;
-import static org.elasticsearch.rest.action.support.RestXContentBuilder.restContentBuilder;
 
 /**
  */
@@ -60,48 +60,15 @@ public RestMultiSearchAction(Settings settings, Client client, RestController co
     }
 
     @Override
-    public void handleRequest(final RestRequest request, final RestChannel channel) {
+    public void handleRequest(final RestRequest request, final RestChannel channel) throws Exception {
         MultiSearchRequest multiSearchRequest = new MultiSearchRequest();
         multiSearchRequest.listenerThreaded(false);
 
         String[] indices = Strings.splitStringByCommaToArray(request.param("index"));
         String[] types = Strings.splitStringByCommaToArray(request.param("type"));
         IndicesOptions indicesOptions = IndicesOptions.fromRequest(request, multiSearchRequest.indicesOptions());
+        multiSearchRequest.add(RestActions.getRestContent(request), request.contentUnsafe(), indices, types, request.param("search_type"), request.param("routing"), indicesOptions, allowExplicitIndex);
 
-        try {
-            multiSearchRequest.add(RestActions.getRestContent(request), request.contentUnsafe(), indices, types, request.param("search_type"), request.param("routing"), indicesOptions, allowExplicitIndex);
-        } catch (Exception e) {
-            try {
-                XContentBuilder builder = restContentBuilder(request);
-                channel.sendResponse(new BytesRestResponse(BAD_REQUEST, builder.startObject().field("error", e.getMessage()).endObject()));
-            } catch (IOException e1) {
-                logger.error("Failed to send failure response", e1);
-            }
-            return;
-        }
-
-        client.multiSearch(multiSearchRequest, new ActionListener<MultiSearchResponse>() {
-            @Override
-            public void onResponse(MultiSearchResponse response) {
-                try {
-                    XContentBuilder builder = restContentBuilder(request);
-                    builder.startObject();
-                    response.toXContent(builder, request);
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
-            }
-        });
+        client.multiSearch(multiSearchRequest, new RestToXContentListener<MultiSearchResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/search/RestSearchAction.java b/src/main/java/org/elasticsearch/rest/action/search/RestSearchAction.java
index 2916dbddbb335..b18fc1bef80a5 100644
--- a/src/main/java/org/elasticsearch/rest/action/search/RestSearchAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/search/RestSearchAction.java
@@ -20,7 +20,6 @@
 package org.elasticsearch.rest.action.search;
 
 import org.elasticsearch.ElasticsearchIllegalArgumentException;
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.search.SearchOperationThreading;
 import org.elasticsearch.action.search.SearchRequest;
 import org.elasticsearch.action.search.SearchResponse;
@@ -29,22 +28,21 @@
 import org.elasticsearch.common.Strings;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.index.query.QueryBuilders;
 import org.elasticsearch.index.query.QueryStringQueryBuilder;
-import org.elasticsearch.rest.*;
+import org.elasticsearch.rest.BaseRestHandler;
+import org.elasticsearch.rest.RestChannel;
+import org.elasticsearch.rest.RestController;
+import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.action.support.RestToXContentListener;
 import org.elasticsearch.search.Scroll;
 import org.elasticsearch.search.builder.SearchSourceBuilder;
 import org.elasticsearch.search.fetch.source.FetchSourceContext;
 import org.elasticsearch.search.sort.SortOrder;
 
-import java.io.IOException;
-
 import static org.elasticsearch.common.unit.TimeValue.parseTimeValue;
 import static org.elasticsearch.rest.RestRequest.Method.GET;
 import static org.elasticsearch.rest.RestRequest.Method.POST;
-import static org.elasticsearch.rest.RestStatus.BAD_REQUEST;
-import static org.elasticsearch.rest.action.support.RestXContentBuilder.restContentBuilder;
 import static org.elasticsearch.search.suggest.SuggestBuilder.termSuggestion;
 
 /**
@@ -55,72 +53,34 @@ public class RestSearchAction extends BaseRestHandler {
     @Inject
     public RestSearchAction(Settings settings, Client client, RestController controller) {
         super(settings, client);
-        controller.registerHandler(GET,  "/_search", this);
+        controller.registerHandler(GET, "/_search", this);
         controller.registerHandler(POST, "/_search", this);
-        controller.registerHandler(GET,  "/{index}/_search", this);
+        controller.registerHandler(GET, "/{index}/_search", this);
         controller.registerHandler(POST, "/{index}/_search", this);
-        controller.registerHandler(GET,  "/{index}/{type}/_search", this);
+        controller.registerHandler(GET, "/{index}/{type}/_search", this);
         controller.registerHandler(POST, "/{index}/{type}/_search", this);
-        controller.registerHandler(GET,  "/_search/template", this);
+        controller.registerHandler(GET, "/_search/template", this);
         controller.registerHandler(POST, "/_search/template", this);
-        controller.registerHandler(GET,  "/{index}/_search/template", this);
+        controller.registerHandler(GET, "/{index}/_search/template", this);
         controller.registerHandler(POST, "/{index}/_search/template", this);
-        controller.registerHandler(GET,  "/{index}/{type}/_search/template", this);
+        controller.registerHandler(GET, "/{index}/{type}/_search/template", this);
         controller.registerHandler(POST, "/{index}/{type}/_search/template", this);
     }
 
     @Override
     public void handleRequest(final RestRequest request, final RestChannel channel) {
         SearchRequest searchRequest;
-        try {
-            searchRequest = RestSearchAction.parseSearchRequest(request);
-            searchRequest.listenerThreaded(false);
-            SearchOperationThreading operationThreading = SearchOperationThreading.fromString(request.param("operation_threading"), null);
-            if (operationThreading != null) {
-                if (operationThreading == SearchOperationThreading.NO_THREADS) {
-                    // since we don't spawn, don't allow no_threads, but change it to a single thread
-                    operationThreading = SearchOperationThreading.SINGLE_THREAD;
-                }
-                searchRequest.operationThreading(operationThreading);
-            }
-        } catch (Exception e) {
-            if (logger.isDebugEnabled()) {
-                logger.debug("failed to parse search request parameters", e);
+        searchRequest = RestSearchAction.parseSearchRequest(request);
+        searchRequest.listenerThreaded(false);
+        SearchOperationThreading operationThreading = SearchOperationThreading.fromString(request.param("operation_threading"), null);
+        if (operationThreading != null) {
+            if (operationThreading == SearchOperationThreading.NO_THREADS) {
+                // since we don't spawn, don't allow no_threads, but change it to a single thread
+                operationThreading = SearchOperationThreading.SINGLE_THREAD;
             }
-            try {
-                XContentBuilder builder = restContentBuilder(request);
-                channel.sendResponse(new BytesRestResponse(BAD_REQUEST, builder.startObject().field("error", e.getMessage()).endObject()));
-            } catch (IOException e1) {
-                logger.error("Failed to send failure response", e1);
-            }
-            return;
+            searchRequest.operationThreading(operationThreading);
         }
-        client.search(searchRequest, new ActionListener<SearchResponse>() {
-            @Override
-            public void onResponse(SearchResponse response) {
-                try {
-                    XContentBuilder builder = restContentBuilder(request);
-                    builder.startObject();
-                    response.toXContent(builder, request);
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(response.status(), builder));
-                } catch (Exception e) {
-                    if (logger.isDebugEnabled()) {
-                        logger.debug("failed to execute search (building response)", e);
-                    }
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
-            }
-        });
+        client.search(searchRequest, new RestToXContentListener<SearchResponse>(channel));
     }
 
     public static SearchRequest parseSearchRequest(RestRequest request) {
diff --git a/src/main/java/org/elasticsearch/rest/action/search/RestSearchScrollAction.java b/src/main/java/org/elasticsearch/rest/action/search/RestSearchScrollAction.java
index b64c1734103c6..0c41da18357b1 100644
--- a/src/main/java/org/elasticsearch/rest/action/search/RestSearchScrollAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/search/RestSearchScrollAction.java
@@ -29,6 +29,7 @@
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
 import org.elasticsearch.rest.action.support.RestActions;
+import org.elasticsearch.rest.action.support.RestToXContentListener;
 import org.elasticsearch.search.Scroll;
 
 import java.io.IOException;
@@ -36,8 +37,6 @@
 import static org.elasticsearch.common.unit.TimeValue.parseTimeValue;
 import static org.elasticsearch.rest.RestRequest.Method.GET;
 import static org.elasticsearch.rest.RestRequest.Method.POST;
-import static org.elasticsearch.rest.RestStatus.BAD_REQUEST;
-import static org.elasticsearch.rest.action.support.RestXContentBuilder.restContentBuilder;
 
 /**
  *
@@ -62,51 +61,18 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         }
         SearchScrollRequest searchScrollRequest = new SearchScrollRequest(scrollId);
         searchScrollRequest.listenerThreaded(false);
-        try {
-            String scroll = request.param("scroll");
-            if (scroll != null) {
-                searchScrollRequest.scroll(new Scroll(parseTimeValue(scroll, null)));
-            }
-            SearchOperationThreading operationThreading = SearchOperationThreading.fromString(request.param("operation_threading"), null);
-            if (operationThreading != null) {
-                if (operationThreading == SearchOperationThreading.NO_THREADS) {
-                    // since we don't spawn, don't allow no_threads, but change it to a single thread
-                    operationThreading = SearchOperationThreading.SINGLE_THREAD;
-                }
-                searchScrollRequest.operationThreading(operationThreading);
-            }
-        } catch (Exception e) {
-            try {
-                XContentBuilder builder = restContentBuilder(request);
-                channel.sendResponse(new BytesRestResponse(BAD_REQUEST, builder.startObject().field("error", e.getMessage()).endObject()));
-            } catch (IOException e1) {
-                logger.error("Failed to send failure response", e1);
-            }
-            return;
+        String scroll = request.param("scroll");
+        if (scroll != null) {
+            searchScrollRequest.scroll(new Scroll(parseTimeValue(scroll, null)));
         }
-
-        client.searchScroll(searchScrollRequest, new ActionListener<SearchResponse>() {
-            @Override
-            public void onResponse(SearchResponse response) {
-                try {
-                    XContentBuilder builder = restContentBuilder(request);
-                    builder.startObject();
-                    response.toXContent(builder, request);
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(response.status(), builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
+        SearchOperationThreading operationThreading = SearchOperationThreading.fromString(request.param("operation_threading"), null);
+        if (operationThreading != null) {
+            if (operationThreading == SearchOperationThreading.NO_THREADS) {
+                // since we don't spawn, don't allow no_threads, but change it to a single thread
+                operationThreading = SearchOperationThreading.SINGLE_THREAD;
             }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
-            }
-        });
+            searchScrollRequest.operationThreading(operationThreading);
+        }
+        client.searchScroll(searchScrollRequest, new RestToXContentListener<SearchResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/suggest/RestSuggestAction.java b/src/main/java/org/elasticsearch/rest/action/suggest/RestSuggestAction.java
index f443aa658c7fa..adf593fc8e2a1 100644
--- a/src/main/java/org/elasticsearch/rest/action/suggest/RestSuggestAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/suggest/RestSuggestAction.java
@@ -20,7 +20,6 @@
 package org.elasticsearch.rest.action.suggest;
 
 import org.elasticsearch.ElasticsearchIllegalArgumentException;
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.suggest.SuggestRequest;
 import org.elasticsearch.action.suggest.SuggestResponse;
 import org.elasticsearch.action.support.IndicesOptions;
@@ -31,14 +30,11 @@
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 import org.elasticsearch.search.suggest.Suggest;
 
-import java.io.IOException;
-
 import static org.elasticsearch.rest.RestRequest.Method.GET;
 import static org.elasticsearch.rest.RestRequest.Method.POST;
-import static org.elasticsearch.rest.RestStatus.BAD_REQUEST;
 import static org.elasticsearch.rest.RestStatus.OK;
 import static org.elasticsearch.rest.action.support.RestActions.buildBroadcastShardsHeader;
 
@@ -61,60 +57,36 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         SuggestRequest suggestRequest = new SuggestRequest(Strings.splitStringByCommaToArray(request.param("index")));
         suggestRequest.indicesOptions(IndicesOptions.fromRequest(request, suggestRequest.indicesOptions()));
         suggestRequest.listenerThreaded(false);
-        try {
-            BroadcastOperationThreading operationThreading = BroadcastOperationThreading.fromString(request.param("operation_threading"), BroadcastOperationThreading.THREAD_PER_SHARD);
-            if (operationThreading == BroadcastOperationThreading.NO_THREADS) {
-                // since we don't spawn, don't allow no_threads, but change it to a single thread
-                operationThreading = BroadcastOperationThreading.SINGLE_THREAD;
-            }
-            suggestRequest.operationThreading(operationThreading);
-            if (request.hasContent()) {
-                suggestRequest.suggest(request.content(), request.contentUnsafe());
+        BroadcastOperationThreading operationThreading = BroadcastOperationThreading.fromString(request.param("operation_threading"), BroadcastOperationThreading.THREAD_PER_SHARD);
+        if (operationThreading == BroadcastOperationThreading.NO_THREADS) {
+            // since we don't spawn, don't allow no_threads, but change it to a single thread
+            operationThreading = BroadcastOperationThreading.SINGLE_THREAD;
+        }
+        suggestRequest.operationThreading(operationThreading);
+        if (request.hasContent()) {
+            suggestRequest.suggest(request.content(), request.contentUnsafe());
+        } else {
+            String source = request.param("source");
+            if (source != null) {
+                suggestRequest.suggest(source);
             } else {
-                String source = request.param("source");
-                if (source != null) {
-                    suggestRequest.suggest(source);
-                } else {
-                    throw new ElasticsearchIllegalArgumentException("no content or source provided to execute suggestion");
-                }
-            }
-            suggestRequest.routing(request.param("routing"));
-            suggestRequest.preference(request.param("preference"));
-        } catch (Exception e) {
-            try {
-                XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                channel.sendResponse(new BytesRestResponse(BAD_REQUEST, builder.startObject().field("error", e.getMessage()).endObject()));
-            } catch (IOException e1) {
-                logger.error("Failed to send failure response", e1);
+                throw new ElasticsearchIllegalArgumentException("no content or source provided to execute suggestion");
             }
-            return;
         }
+        suggestRequest.routing(request.param("routing"));
+        suggestRequest.preference(request.param("preference"));
 
-        client.suggest(suggestRequest, new ActionListener<SuggestResponse>() {
-            @Override
-            public void onResponse(SuggestResponse response) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject();
-                    buildBroadcastShardsHeader(builder, response);
-                    Suggest suggest = response.getSuggest();
-                    if (suggest != null) {
-                        suggest.toXContent(builder, request);
-                    }
-                    builder.endObject();
-                    channel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
+        client.suggest(suggestRequest, new RestBuilderListener<SuggestResponse>(channel) {
             @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
+            public RestResponse buildResponse(SuggestResponse response, XContentBuilder builder) throws Exception {
+                builder.startObject();
+                buildBroadcastShardsHeader(builder, response);
+                Suggest suggest = response.getSuggest();
+                if (suggest != null) {
+                    suggest.toXContent(builder, request);
                 }
+                builder.endObject();
+                return new BytesRestResponse(OK, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/rest/AcknowledgedRestResponseActionListener.java b/src/main/java/org/elasticsearch/rest/action/support/AcknowledgedRestListener.java
similarity index 64%
rename from src/main/java/org/elasticsearch/rest/AcknowledgedRestResponseActionListener.java
rename to src/main/java/org/elasticsearch/rest/action/support/AcknowledgedRestListener.java
index ece66c3963b4b..f1c15c69ba93b 100644
--- a/src/main/java/org/elasticsearch/rest/AcknowledgedRestResponseActionListener.java
+++ b/src/main/java/org/elasticsearch/rest/action/support/AcknowledgedRestListener.java
@@ -16,13 +16,17 @@
  * specific language governing permissions and limitations
  * under the License.
  */
-package org.elasticsearch.rest;
+package org.elasticsearch.rest.action.support;
 
 import org.elasticsearch.action.support.master.AcknowledgedResponse;
 import org.elasticsearch.common.logging.ESLogger;
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.common.xcontent.XContentBuilderString;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
+import org.elasticsearch.rest.BytesRestResponse;
+import org.elasticsearch.rest.RestChannel;
+import org.elasticsearch.rest.RestResponse;
+import org.elasticsearch.rest.action.support.RestActionListener;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
 import java.io.IOException;
 
@@ -30,24 +34,19 @@
 
 /**
  */
-public class AcknowledgedRestResponseActionListener<T extends AcknowledgedResponse> extends AbstractRestResponseActionListener<T> {
+public class AcknowledgedRestListener<T extends AcknowledgedResponse> extends RestBuilderListener<T> {
 
-    public AcknowledgedRestResponseActionListener(RestRequest request, RestChannel channel, ESLogger logger) {
-        super(request, channel, logger);
+    public AcknowledgedRestListener(RestChannel channel) {
+        super(channel);
     }
 
     @Override
-    public void onResponse(T response) {
-        try {
-            XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-            builder.startObject()
-                    .field(Fields.ACKNOWLEDGED, response.isAcknowledged());
-            addCustomFields(builder, response);
-            builder.endObject();
-            channel.sendResponse(new BytesRestResponse(OK, builder));
-        } catch (IOException e) {
-            onFailure(e);
-        }
+    public RestResponse buildResponse(T response, XContentBuilder builder) throws Exception {
+        builder.startObject()
+                .field(Fields.ACKNOWLEDGED, response.isAcknowledged());
+        addCustomFields(builder, response);
+        builder.endObject();
+        return new BytesRestResponse(OK, builder);
     }
 
     /**
diff --git a/src/main/java/org/elasticsearch/rest/action/support/RestActionListener.java b/src/main/java/org/elasticsearch/rest/action/support/RestActionListener.java
new file mode 100644
index 0000000000000..211635ef98f6f
--- /dev/null
+++ b/src/main/java/org/elasticsearch/rest/action/support/RestActionListener.java
@@ -0,0 +1,65 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.rest.action.support;
+
+import org.elasticsearch.action.ActionListener;
+import org.elasticsearch.common.logging.ESLogger;
+import org.elasticsearch.common.logging.Loggers;
+import org.elasticsearch.rest.BytesRestResponse;
+import org.elasticsearch.rest.RestChannel;
+
+import java.io.IOException;
+
+/**
+ * An action listener that requires {@link #processResponse(Object)} to be implemented
+ * and will automatically handle failures.
+ */
+public abstract class RestActionListener<Response> implements ActionListener<Response> {
+
+    // we use static here so we won't have to pass the actual logger each time for a very rare case of logging
+    // where the settings don't matter that much
+    private static ESLogger logger = Loggers.getLogger(RestResponseListener.class);
+
+    protected final RestChannel channel;
+
+    protected RestActionListener(RestChannel channel) {
+        this.channel = channel;
+    }
+
+    @Override
+    public final void onResponse(Response response) {
+        try {
+            processResponse(response);
+        } catch (Throwable t) {
+            onFailure(t);
+        }
+    }
+
+    protected abstract void processResponse(Response response) throws Exception;
+
+    @Override
+    public final void onFailure(Throwable e) {
+        try {
+            channel.sendResponse(new BytesRestResponse(channel, e));
+        } catch (Throwable e1) {
+            logger.error("failed to send failure response", e1);
+        }
+    }
+}
diff --git a/src/main/java/org/elasticsearch/rest/action/support/RestBuilderListener.java b/src/main/java/org/elasticsearch/rest/action/support/RestBuilderListener.java
new file mode 100644
index 0000000000000..82c72e59f9b55
--- /dev/null
+++ b/src/main/java/org/elasticsearch/rest/action/support/RestBuilderListener.java
@@ -0,0 +1,44 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.rest.action.support;
+
+import org.elasticsearch.common.xcontent.XContentBuilder;
+import org.elasticsearch.rest.RestChannel;
+import org.elasticsearch.rest.RestResponse;
+
+/**
+ * A REST action listener that builds an {@link XContentBuilder} based response.
+ */
+public abstract class RestBuilderListener<Response> extends RestResponseListener<Response> {
+
+    public RestBuilderListener(RestChannel channel) {
+        super(channel);
+    }
+
+    @Override
+    public final RestResponse buildResponse(Response response) throws Exception {
+        return buildResponse(response, channel.newBuilder());
+    }
+
+    /**
+     * Builds a response to send back over the channel.
+     */
+    public abstract RestResponse buildResponse(Response response, XContentBuilder builder) throws Exception;
+}
diff --git a/src/main/java/org/elasticsearch/rest/AbstractRestResponseActionListener.java b/src/main/java/org/elasticsearch/rest/action/support/RestResponseListener.java
similarity index 52%
rename from src/main/java/org/elasticsearch/rest/AbstractRestResponseActionListener.java
rename to src/main/java/org/elasticsearch/rest/action/support/RestResponseListener.java
index e74934a761f1f..acdfaab737b7c 100644
--- a/src/main/java/org/elasticsearch/rest/AbstractRestResponseActionListener.java
+++ b/src/main/java/org/elasticsearch/rest/action/support/RestResponseListener.java
@@ -16,37 +16,35 @@
  * specific language governing permissions and limitations
  * under the License.
  */
-package org.elasticsearch.rest;
+
+package org.elasticsearch.rest.action.support;
 
 import org.elasticsearch.action.ActionListener;
-import org.elasticsearch.action.ActionResponse;
 import org.elasticsearch.common.logging.ESLogger;
+import org.elasticsearch.common.logging.Loggers;
+import org.elasticsearch.rest.BytesRestResponse;
+import org.elasticsearch.rest.RestChannel;
+import org.elasticsearch.rest.RestResponse;
 
 import java.io.IOException;
 
 /**
+ * A REST enabled action listener that has a basic onFailure implementation, and requires
+ * sub classes to only implement {@link #buildResponse(Object)}.
  */
-public abstract class AbstractRestResponseActionListener<T extends ActionResponse> implements ActionListener<T> {
-
-    protected final RestChannel channel;
-    protected final RestRequest request;
-    protected final ESLogger logger;
+public abstract class RestResponseListener<Response> extends RestActionListener<Response> {
 
-    public AbstractRestResponseActionListener(final RestRequest request, final RestChannel channel, final ESLogger logger) {
-        this.request = request;
-        this.channel = channel;
-        this.logger = logger;
+    protected RestResponseListener(RestChannel channel) {
+        super(channel);
     }
 
     @Override
-    public abstract void onResponse(T t);
-
-    @Override
-    public void onFailure(Throwable e) {
-        try {
-            channel.sendResponse(new BytesRestResponse(request, e));
-        } catch (IOException e1) {
-            logger.error("Failed to send failure response", e1);
-        }
+    protected final void processResponse(Response response) throws Exception {
+        channel.sendResponse(buildResponse(response));
     }
+
+    /**
+     * Builds the response to send back through the channel.
+     */
+    public abstract RestResponse buildResponse(Response response) throws Exception;
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/support/RestTable.java b/src/main/java/org/elasticsearch/rest/action/support/RestTable.java
index b6ba3ae4207bb..a2a9461ae772b 100644
--- a/src/main/java/org/elasticsearch/rest/action/support/RestTable.java
+++ b/src/main/java/org/elasticsearch/rest/action/support/RestTable.java
@@ -22,6 +22,8 @@
 import org.elasticsearch.common.Booleans;
 import org.elasticsearch.common.Strings;
 import org.elasticsearch.common.Table;
+import org.elasticsearch.common.io.UTF8StreamWriter;
+import org.elasticsearch.common.io.stream.BytesStreamOutput;
 import org.elasticsearch.common.unit.ByteSizeValue;
 import org.elasticsearch.common.unit.SizeValue;
 import org.elasticsearch.common.unit.TimeValue;
@@ -29,6 +31,7 @@
 import org.elasticsearch.common.xcontent.XContentType;
 import org.elasticsearch.rest.*;
 
+import java.io.IOException;
 import java.util.ArrayList;
 import java.util.List;
 
@@ -36,16 +39,18 @@
  */
 public class RestTable {
 
-    public static RestResponse buildResponse(Table table, RestRequest request, RestChannel channel) throws Exception {
+    public static RestResponse buildResponse(Table table, RestChannel channel) throws Exception {
+        RestRequest request = channel.request();
         XContentType xContentType = XContentType.fromRestContentType(request.param("format", request.header("Content-Type")));
         if (xContentType != null) {
-            return buildXContentBuilder(table, request, channel);
+            return buildXContentBuilder(table, channel);
         }
-        return buildTextPlainResponse(table, request, channel);
+        return buildTextPlainResponse(table, channel);
     }
 
-    public static RestResponse buildXContentBuilder(Table table, RestRequest request, RestChannel channel) throws Exception {
-        XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
+    public static RestResponse buildXContentBuilder(Table table, RestChannel channel) throws Exception {
+        RestRequest request = channel.request();
+        XContentBuilder builder = channel.newBuilder();
         List<DisplayHeader> displayHeaders = buildDisplayHeaders(table, request);
 
         builder.startArray();
@@ -61,13 +66,15 @@ public static RestResponse buildXContentBuilder(Table table, RestRequest request
         return new BytesRestResponse(RestStatus.OK, builder);
     }
 
-    public static RestResponse buildTextPlainResponse(Table table, RestRequest request, RestChannel channel) {
+    public static RestResponse buildTextPlainResponse(Table table, RestChannel channel) throws IOException {
+        RestRequest request = channel.request();
         boolean verbose = request.paramAsBoolean("v", false);
 
         List<DisplayHeader> headers = buildDisplayHeaders(table, request);
         int[] width = buildWidths(table, request, verbose, headers);
 
-        StringBuilder out = new StringBuilder();
+        BytesStreamOutput bytesOut = channel.bytesOutput();
+        UTF8StreamWriter out = new UTF8StreamWriter().setOutput(bytesOut);
         if (verbose) {
             for (int col = 0; col < headers.size(); col++) {
                 DisplayHeader header = headers.get(col);
@@ -85,8 +92,8 @@ public static RestResponse buildTextPlainResponse(Table table, RestRequest reque
             }
             out.append("\n");
         }
-
-        return new BytesRestResponse(RestStatus.OK, out.toString());
+        out.close();
+        return new BytesRestResponse(RestStatus.OK, BytesRestResponse.TEXT_CONTENT_TYPE, bytesOut.bytes(), true);
     }
 
     private static List<DisplayHeader> buildDisplayHeaders(Table table, RestRequest request) {
@@ -140,7 +147,7 @@ private static List<DisplayHeader> buildDisplayHeaders(Table table, RestRequest
         return display;
     }
 
-    public static int[] buildHelpWidths(Table table, RestRequest request, boolean verbose) {
+    public static int[] buildHelpWidths(Table table, RestRequest request) {
         int[] width = new int[3];
         for (Table.Cell cell : table.getHeaders()) {
             String v = renderValue(request, cell.value);
@@ -193,7 +200,7 @@ private static int[] buildWidths(Table table, RestRequest request, boolean verbo
         return width;
     }
 
-    public static void pad(Table.Cell cell, int width, RestRequest request, StringBuilder out) {
+    public static void pad(Table.Cell cell, int width, RestRequest request, UTF8StreamWriter out) throws IOException {
         String sValue = renderValue(request, cell.value);
         int length = sValue == null ? 0 : sValue.length();
         byte leftOver = (byte) (width - length);
diff --git a/src/main/java/org/elasticsearch/rest/action/support/RestToXContentListener.java b/src/main/java/org/elasticsearch/rest/action/support/RestToXContentListener.java
new file mode 100644
index 0000000000000..01f39662e3c9c
--- /dev/null
+++ b/src/main/java/org/elasticsearch/rest/action/support/RestToXContentListener.java
@@ -0,0 +1,50 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.rest.action.support;
+
+import org.elasticsearch.common.xcontent.ToXContent;
+import org.elasticsearch.common.xcontent.XContentBuilder;
+import org.elasticsearch.rest.BytesRestResponse;
+import org.elasticsearch.rest.RestChannel;
+import org.elasticsearch.rest.RestResponse;
+import org.elasticsearch.rest.RestStatus;
+
+/**
+ * A REST based action listener that assumes the response is of type {@link ToXContent} and automatically
+ * builds an XContent based response (wrapping the toXContent in startObject/endObject).
+ */
+public final class RestToXContentListener<Response extends ToXContent> extends RestResponseListener<Response> {
+
+    public RestToXContentListener(RestChannel channel) {
+        super(channel);
+    }
+
+    @Override
+    public final RestResponse buildResponse(Response response) throws Exception {
+        return buildResponse(response, channel.newBuilder());
+    }
+
+    public final RestResponse buildResponse(Response response, XContentBuilder builder) throws Exception {
+        builder.startObject();
+        response.toXContent(builder, channel.request());
+        builder.endObject();
+        return new BytesRestResponse(RestStatus.OK, builder);
+    }
+}
diff --git a/src/main/java/org/elasticsearch/rest/action/support/RestXContentBuilder.java b/src/main/java/org/elasticsearch/rest/action/support/RestXContentBuilder.java
deleted file mode 100644
index cff25557a669b..0000000000000
--- a/src/main/java/org/elasticsearch/rest/action/support/RestXContentBuilder.java
+++ /dev/null
@@ -1,84 +0,0 @@
-/*
- * Licensed to Elasticsearch under one or more contributor
- * license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright
- * ownership. Elasticsearch licenses this file to you under
- * the Apache License, Version 2.0 (the "License"); you may
- * not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *    http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.elasticsearch.rest.action.support;
-
-import org.elasticsearch.common.Nullable;
-import org.elasticsearch.common.bytes.BytesReference;
-import org.elasticsearch.common.io.stream.BytesStreamOutput;
-import org.elasticsearch.common.xcontent.*;
-import org.elasticsearch.rest.RestRequest;
-
-import java.io.IOException;
-
-/**
- *
- */
-public class RestXContentBuilder {
-
-    public static XContentBuilder restContentBuilder(RestRequest request) throws IOException {
-        // use the request body as the auto detect source (if it exists)
-        return restContentBuilder(request, request.hasContent() ? request.content() : null);
-    }
-
-    public static XContentBuilder restContentBuilder(RestRequest request, @Nullable BytesReference autoDetectSource) throws IOException {
-        XContentType contentType = XContentType.fromRestContentType(request.param("format", request.header("Content-Type")));
-        if (contentType == null) {
-            // try and guess it from the auto detect source
-            if (autoDetectSource != null) {
-                contentType = XContentFactory.xContentType(autoDetectSource);
-            }
-        }
-        if (contentType == null) {
-            // default to JSON
-            contentType = XContentType.JSON;
-        }
-        XContentBuilder builder = new XContentBuilder(XContentFactory.xContent(contentType), new BytesStreamOutput());
-        if (request.paramAsBoolean("pretty", false)) {
-            builder.prettyPrint().lfAtEnd();
-        }
-
-        builder.humanReadable(request.paramAsBoolean("human", builder.humanReadable()));
-
-        String casing = request.param("case");
-        if (casing != null && "camelCase".equals(casing)) {
-            builder.fieldCaseConversion(XContentBuilder.FieldCaseConversion.CAMELCASE);
-        } else {
-            // we expect all REST interfaces to write results in underscore casing, so
-            // no need for double casing
-            builder.fieldCaseConversion(XContentBuilder.FieldCaseConversion.NONE);
-        }
-        return builder;
-    }
-
-    public static XContentBuilder emptyBuilder(RestRequest request) throws IOException {
-        return restContentBuilder(request, request.hasContent() ? request.content() : null).startObject().endObject();
-    }
-
-    /**
-     * Directly writes the source to the output builder
-     */
-    public static void directSource(BytesReference source, XContentBuilder rawBuilder, ToXContent.Params params) throws IOException {
-        XContentHelper.writeDirect(source, rawBuilder, params);
-    }
-
-    public static void restDocumentSource(BytesReference source, XContentBuilder builder, ToXContent.Params params) throws IOException {
-        XContentHelper.writeRawField("_source", source, builder, params);
-    }
-}
diff --git a/src/main/java/org/elasticsearch/rest/action/termvector/RestMultiTermVectorsAction.java b/src/main/java/org/elasticsearch/rest/action/termvector/RestMultiTermVectorsAction.java
index 2938aa6f5a923..6badd0aedfb84 100644
--- a/src/main/java/org/elasticsearch/rest/action/termvector/RestMultiTermVectorsAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/termvector/RestMultiTermVectorsAction.java
@@ -30,11 +30,11 @@
 import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.rest.*;
 import org.elasticsearch.rest.action.support.RestActions;
+import org.elasticsearch.rest.action.support.RestToXContentListener;
 
 import static org.elasticsearch.rest.RestRequest.Method.GET;
 import static org.elasticsearch.rest.RestRequest.Method.POST;
 import static org.elasticsearch.rest.RestStatus.OK;
-import static org.elasticsearch.rest.action.support.RestXContentBuilder.restContentBuilder;
 
 public class RestMultiTermVectorsAction extends BaseRestHandler {
 
@@ -50,8 +50,7 @@ public RestMultiTermVectorsAction(Settings settings, Client client, RestControll
     }
 
     @Override
-    public void handleRequest(final RestRequest request, final RestChannel channel) {
-        
+    public void handleRequest(final RestRequest request, final RestChannel channel) throws Exception {
         MultiTermVectorsRequest multiTermVectorsRequest = new MultiTermVectorsRequest();
         multiTermVectorsRequest.listenerThreaded(false);
         TermVectorRequest template = new TermVectorRequest();
@@ -59,38 +58,8 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         template.type(request.param("type"));
         RestTermVectorAction.readURIParameters(template, request);
         multiTermVectorsRequest.ids(Strings.commaDelimitedListToStringArray(request.param("ids")));
+        multiTermVectorsRequest.add(template, RestActions.getRestContent(request));
 
-        try {
-            multiTermVectorsRequest.add(template, RestActions.getRestContent(request));
-        } catch (Throwable t) {
-            try {
-                channel.sendResponse(new BytesRestResponse(request, t));
-            } catch (Throwable tIO) {
-                logger.error("Failed to send failure response", tIO);
-            }
-            return;
-        }
-
-        client.multiTermVectors(multiTermVectorsRequest, new ActionListener<MultiTermVectorsResponse>() {
-            @Override
-            public void onResponse(MultiTermVectorsResponse response) {
-                try {
-                    XContentBuilder builder = restContentBuilder(request);
-                    response.toXContent(builder, request);
-                    channel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (Throwable t) {
-                    onFailure(t);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (Throwable t) {
-                    logger.error("Failed to send failure response", t);
-                }
-            }
-        });
+        client.multiTermVectors(multiTermVectorsRequest, new RestToXContentListener<MultiTermVectorsResponse>(channel));
     }
 }
diff --git a/src/main/java/org/elasticsearch/rest/action/termvector/RestTermVectorAction.java b/src/main/java/org/elasticsearch/rest/action/termvector/RestTermVectorAction.java
index 85ea2399c37da..700375e8ca69e 100644
--- a/src/main/java/org/elasticsearch/rest/action/termvector/RestTermVectorAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/termvector/RestTermVectorAction.java
@@ -19,28 +19,25 @@
 
 package org.elasticsearch.rest.action.termvector;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.termvector.TermVectorRequest;
 import org.elasticsearch.action.termvector.TermVectorResponse;
 import org.elasticsearch.client.Client;
 import org.elasticsearch.common.Strings;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.common.xcontent.XContentBuilder;
 import org.elasticsearch.common.xcontent.XContentFactory;
 import org.elasticsearch.common.xcontent.XContentParser;
-import org.elasticsearch.rest.*;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
+import org.elasticsearch.rest.BaseRestHandler;
+import org.elasticsearch.rest.RestChannel;
+import org.elasticsearch.rest.RestController;
+import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.action.support.RestToXContentListener;
 
-import java.io.IOException;
 import java.util.HashSet;
 import java.util.Set;
 
 import static org.elasticsearch.rest.RestRequest.Method.GET;
 import static org.elasticsearch.rest.RestRequest.Method.POST;
-import static org.elasticsearch.rest.RestStatus.BAD_REQUEST;
-import static org.elasticsearch.rest.RestStatus.OK;
-import static org.elasticsearch.rest.action.support.RestXContentBuilder.restContentBuilder;
 
 /**
  * This class parses the json request and translates it into a
@@ -56,23 +53,13 @@ public RestTermVectorAction(Settings settings, Client client, RestController con
     }
 
     @Override
-    public void handleRequest(final RestRequest request, final RestChannel channel) {
-
+    public void handleRequest(final RestRequest request, final RestChannel channel) throws Exception {
         TermVectorRequest termVectorRequest = new TermVectorRequest(request.param("index"), request.param("type"), request.param("id"));
         XContentParser parser = null;
         if (request.hasContent()) {
             try {
                 parser = XContentFactory.xContent(request.content()).createParser(request.content());
                 TermVectorRequest.parseRequest(termVectorRequest, parser);
-            } catch (IOException e) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    channel.sendResponse(new BytesRestResponse(BAD_REQUEST, builder.startObject().field("error", e.getMessage()).endObject()));
-
-                } catch (IOException e1) {
-                    logger.warn("Failed to send response", e1);
-                    return;
-                }
             } finally {
                 if (parser != null) {
                     parser.close();
@@ -81,28 +68,7 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
         }
         readURIParameters(termVectorRequest, request);
 
-        client.termVector(termVectorRequest, new ActionListener<TermVectorResponse>() {
-            @Override
-            public void onResponse(TermVectorResponse response) {
-                try {
-                    XContentBuilder builder = restContentBuilder(request);
-                    response.toXContent(builder, request);
-                    channel.sendResponse(new BytesRestResponse(OK, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
-                }
-            }
-
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
-                }
-            }
-        });
-
+        client.termVector(termVectorRequest, new RestToXContentListener<TermVectorResponse>(channel));
     }
 
     static public void readURIParameters(TermVectorRequest termVectorRequest, RestRequest request) {
diff --git a/src/main/java/org/elasticsearch/rest/action/update/RestUpdateAction.java b/src/main/java/org/elasticsearch/rest/action/update/RestUpdateAction.java
index 7a8cf3bb3075c..8c38979816990 100644
--- a/src/main/java/org/elasticsearch/rest/action/update/RestUpdateAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/update/RestUpdateAction.java
@@ -19,7 +19,6 @@
 
 package org.elasticsearch.rest.action.update;
 
-import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.WriteConsistencyLevel;
 import org.elasticsearch.action.index.IndexRequest;
 import org.elasticsearch.action.support.replication.ReplicationType;
@@ -34,9 +33,8 @@
 import org.elasticsearch.index.VersionType;
 import org.elasticsearch.rest.*;
 import org.elasticsearch.rest.action.support.RestActions;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
+import org.elasticsearch.rest.action.support.RestBuilderListener;
 
-import java.io.IOException;
 import java.util.Map;
 
 import static org.elasticsearch.rest.RestRequest.Method.POST;
@@ -54,7 +52,7 @@ public RestUpdateAction(Settings settings, Client client, RestController control
     }
 
     @Override
-    public void handleRequest(final RestRequest request, final RestChannel channel) {
+    public void handleRequest(final RestRequest request, final RestChannel channel) throws Exception {
         UpdateRequest updateRequest = new UpdateRequest(request.param("index"), request.param("type"), request.param("id"));
         updateRequest.listenerThreaded(false);
         updateRequest.routing(request.param("routing"));
@@ -91,75 +89,52 @@ public void handleRequest(final RestRequest request, final RestChannel channel)
 
         // see if we have it in the body
         if (request.hasContent()) {
-            try {
-                updateRequest.source(request.content());
-                IndexRequest upsertRequest = updateRequest.upsertRequest();
-                if (upsertRequest != null) {
-                    upsertRequest.routing(request.param("routing"));
-                    upsertRequest.parent(request.param("parent")); // order is important, set it after routing, so it will set the routing
-                    upsertRequest.timestamp(request.param("timestamp"));
-                    if (request.hasParam("ttl")) {
-                        upsertRequest.ttl(request.paramAsTime("ttl", null).millis());
-                    }
-                    upsertRequest.version(RestActions.parseVersion(request));
-                    upsertRequest.versionType(VersionType.fromString(request.param("version_type"), upsertRequest.versionType()));
+            updateRequest.source(request.content());
+            IndexRequest upsertRequest = updateRequest.upsertRequest();
+            if (upsertRequest != null) {
+                upsertRequest.routing(request.param("routing"));
+                upsertRequest.parent(request.param("parent")); // order is important, set it after routing, so it will set the routing
+                upsertRequest.timestamp(request.param("timestamp"));
+                if (request.hasParam("ttl")) {
+                    upsertRequest.ttl(request.paramAsTime("ttl", null).millis());
                 }
-                IndexRequest doc = updateRequest.doc();
-                if (doc != null) {
-                    doc.routing(request.param("routing"));
-                    doc.parent(request.param("parent")); // order is important, set it after routing, so it will set the routing
-                    doc.timestamp(request.param("timestamp"));
-                    if (request.hasParam("ttl")) {
-                        doc.ttl(request.paramAsTime("ttl", null).millis());
-                    }
-                    doc.version(RestActions.parseVersion(request));
-                    doc.versionType(VersionType.fromString(request.param("version_type"), doc.versionType()));
-                }                
-            } catch (Exception e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.warn("Failed to send response", e1);
+                upsertRequest.version(RestActions.parseVersion(request));
+                upsertRequest.versionType(VersionType.fromString(request.param("version_type"), upsertRequest.versionType()));
+            }
+            IndexRequest doc = updateRequest.doc();
+            if (doc != null) {
+                doc.routing(request.param("routing"));
+                doc.parent(request.param("parent")); // order is important, set it after routing, so it will set the routing
+                doc.timestamp(request.param("timestamp"));
+                if (request.hasParam("ttl")) {
+                    doc.ttl(request.paramAsTime("ttl", null).millis());
                 }
-                return;
+                doc.version(RestActions.parseVersion(request));
+                doc.versionType(VersionType.fromString(request.param("version_type"), doc.versionType()));
             }
         }
 
-        client.update(updateRequest, new ActionListener<UpdateResponse>() {
+        client.update(updateRequest, new RestBuilderListener<UpdateResponse>(channel) {
             @Override
-            public void onResponse(UpdateResponse response) {
-                try {
-                    XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
-                    builder.startObject()
-                            .field(Fields._INDEX, response.getIndex())
-                            .field(Fields._TYPE, response.getType())
-                            .field(Fields._ID, response.getId())
-                            .field(Fields._VERSION, response.getVersion());
-
-                    if (response.getGetResult() != null) {
-                        builder.startObject(Fields.GET);
-                        response.getGetResult().toXContentEmbedded(builder, request);
-                        builder.endObject();
-                    }
+            public RestResponse buildResponse(UpdateResponse response, XContentBuilder builder) throws Exception {
+                builder.startObject()
+                        .field(Fields._INDEX, response.getIndex())
+                        .field(Fields._TYPE, response.getType())
+                        .field(Fields._ID, response.getId())
+                        .field(Fields._VERSION, response.getVersion());
 
+                if (response.getGetResult() != null) {
+                    builder.startObject(Fields.GET);
+                    response.getGetResult().toXContentEmbedded(builder, request);
                     builder.endObject();
-                    RestStatus status = OK;
-                    if (response.isCreated()) {
-                        status = CREATED;
-                    }
-                    channel.sendResponse(new BytesRestResponse(status, builder));
-                } catch (Throwable e) {
-                    onFailure(e);
                 }
-            }
 
-            @Override
-            public void onFailure(Throwable e) {
-                try {
-                    channel.sendResponse(new BytesRestResponse(request, e));
-                } catch (IOException e1) {
-                    logger.error("Failed to send failure response", e1);
+                builder.endObject();
+                RestStatus status = OK;
+                if (response.isCreated()) {
+                    status = CREATED;
                 }
+                return new BytesRestResponse(status, builder);
             }
         });
     }
diff --git a/src/main/java/org/elasticsearch/search/internal/InternalSearchHit.java b/src/main/java/org/elasticsearch/search/internal/InternalSearchHit.java
index 9a92c49c568b6..883875700d907 100644
--- a/src/main/java/org/elasticsearch/search/internal/InternalSearchHit.java
+++ b/src/main/java/org/elasticsearch/search/internal/InternalSearchHit.java
@@ -36,7 +36,6 @@
 import org.elasticsearch.common.xcontent.XContentBuilderString;
 import org.elasticsearch.common.xcontent.XContentHelper;
 import org.elasticsearch.index.fielddata.fieldcomparator.BytesRefFieldComparatorSource;
-import org.elasticsearch.rest.action.support.RestXContentBuilder;
 import org.elasticsearch.search.SearchHit;
 import org.elasticsearch.search.SearchHitField;
 import org.elasticsearch.search.SearchShardTarget;
@@ -411,7 +410,7 @@ public XContentBuilder toXContent(XContentBuilder builder, Params params) throws
             builder.field(Fields._SCORE, score);
         }
         if (source != null) {
-            RestXContentBuilder.restDocumentSource(source, builder, params);
+            XContentHelper.writeRawField("_source", source, builder, params);
         }
         if (fields != null && !fields.isEmpty()) {
             builder.startObject(Fields.FIELDS);
diff --git a/src/test/java/org/elasticsearch/action/termvector/GetTermVectorCheckDocFreqTests.java b/src/test/java/org/elasticsearch/action/termvector/GetTermVectorCheckDocFreqTests.java
index f848388a536f6..bfabb1d64eae5 100644
--- a/src/test/java/org/elasticsearch/action/termvector/GetTermVectorCheckDocFreqTests.java
+++ b/src/test/java/org/elasticsearch/action/termvector/GetTermVectorCheckDocFreqTests.java
@@ -141,9 +141,10 @@ private void checkWithoutFieldStatistics(int numDocs, String[] values, int[] fre
         }
         assertThat(iterator.next(), Matchers.nullValue());
 
-        XContentBuilder xBuilder = new XContentFactory().jsonBuilder();
-
+        XContentBuilder xBuilder = XContentFactory.jsonBuilder();
+        xBuilder.startObject();
         response.toXContent(xBuilder, null);
+        xBuilder.endObject();
         BytesStream bytesStream = xBuilder.bytesStream();
         String utf8 = bytesStream.bytes().toUtf8();
         String expectedString = "{\"_index\":\"test\",\"_type\":\"type1\",\"_id\":\""
@@ -197,9 +198,10 @@ private void checkWithoutTermStatistics(int numDocs, String[] values, int[] freq
         }
         assertThat(iterator.next(), Matchers.nullValue());
 
-        XContentBuilder xBuilder = new XContentFactory().jsonBuilder();
-
+        XContentBuilder xBuilder = XContentFactory.jsonBuilder();
+        xBuilder.startObject();
         response.toXContent(xBuilder, null);
+        xBuilder.endObject();
         BytesStream bytesStream = xBuilder.bytesStream();
         String utf8 = bytesStream.bytes().toUtf8();
         String expectedString = "{\"_index\":\"test\",\"_type\":\"type1\",\"_id\":\""
@@ -257,7 +259,9 @@ private void checkAllInfo(int numDocs, String[] values, int[] freq, int[][] pos,
         assertThat(iterator.next(), Matchers.nullValue());
 
         XContentBuilder xBuilder = XContentFactory.jsonBuilder();
+        xBuilder.startObject();
         response.toXContent(xBuilder, ToXContent.EMPTY_PARAMS);
+        xBuilder.endObject();
         BytesStream bytesStream = xBuilder.bytesStream();
         String utf8 = bytesStream.bytes().toUtf8();
         String expectedString = "{\"_index\":\"test\",\"_type\":\"type1\",\"_id\":\""

From 1ec4f8f04b635fcd022be7f1fc3a0e641662e832 Mon Sep 17 00:00:00 2001
From: javanna <cavannaluca@gmail.com>
Date: Mon, 10 Mar 2014 13:48:17 +0100
Subject: [PATCH 23/54] [TEST] Replaced RestTestSuiteRunner with parametrized
 test that uses RandomizedRunner directly

ElasticsearchRestTests extends now ElasticsearchIntegrationTest and makes use of our ordinary test infrastructure, in particular all randomized aspects now come for free instead of having to maintain a separate (custom) tests runner

We previously parsed only the tests that needed to be run given the version of the cluster the tests are running against. This doesn't happen anymore as it didn't buy much and it would be harder to support as the tests get now parsed before the test cluster gets started. Thus all the tests are now parsed regardless of their skip sections, afterwards the ones that don't need to be run will be skipped through assume directives.

Fixed REST tests that rely on a specific number of shards as this change introduces also random number of shards and replicas (through randomIndexTemplate)

Closes #5654
---
 TESTING.asciidoc                              |  20 +-
 dev-tools/build_release.py                    |   2 +-
 pom.xml                                       |   9 +-
 .../test/cat.allocation/10_basic.yaml         |   2 +-
 rest-api-spec/test/cat.shards/10_basic.yaml   |  13 +-
 .../test/cluster.put_settings/10_basic.yaml   |   5 +
 rest-api-spec/test/create/40_routing.yaml     |   1 +
 .../test/create/55_parent_with_routing.yaml   |   1 +
 rest-api-spec/test/delete/30_routing.yaml     |   6 +
 rest-api-spec/test/delete/40_parent.yaml      |   2 +
 .../test/delete/45_parent_with_routing.yaml   |   1 +
 rest-api-spec/test/exists/40_routing.yaml     |   1 +
 .../test/exists/55_parent_with_routing.yaml   |   1 +
 rest-api-spec/test/get/40_routing.yaml        |   1 +
 .../test/get/55_parent_with_routing.yaml      |   1 +
 rest-api-spec/test/get_source/40_routing.yaml |   1 +
 .../get_source/55_parent_with_routing.yaml    |   1 +
 rest-api-spec/test/index/40_routing.yaml      |   1 +
 .../test/index/55_parent_with_routing.yaml    |   1 +
 .../test/indices.get_settings/10_basic.yaml   |  22 +-
 rest-api-spec/test/mget/30_parent.yaml        |   2 +
 rest-api-spec/test/mget/40_routing.yaml       |   1 +
 .../test/mget/55_parent_with_routing.yaml     |   1 +
 .../test/search/20_default_values.yaml        |  27 +-
 rest-api-spec/test/update/40_routing.yaml     |   1 +
 .../test/update/55_parent_with_routing.yaml   |   1 +
 .../StandaloneRandomizedContext.java          |  68 --
 .../test/ElasticsearchIntegrationTest.java    |   2 +-
 .../junit/listeners/ReproduceInfoPrinter.java |  34 +-
 .../test/rest/ElasticsearchRestTests.java     | 174 ++++-
 .../rest/{junit => }/RestTestCandidate.java   |  34 +-
 .../test/rest/RestTestExecutionContext.java   |  35 +-
 .../test/rest/client/RestClient.java          |  19 +-
 .../test/rest/junit/DescriptionHelper.java    |  79 ---
 .../rest/junit/RestReproduceInfoPrinter.java  | 107 ----
 .../test/rest/junit/RestTestSuiteRunner.java  | 598 ------------------
 .../test/rest/junit/RunAfter.java             |  56 --
 .../rest/parser/RestTestSectionParser.java    |  19 +-
 .../parser/RestTestSuiteParseContext.java     |   8 +-
 .../test/rest/parser/RestTestSuiteParser.java |  32 +-
 .../test/rest/parser/SetupSectionParser.java  |  26 +-
 .../test/rest/section/RestTestSuite.java      |   2 +-
 .../test/rest/test/AssertionParsersTests.java |  18 +-
 .../test/rest/test/DoSectionParserTests.java  |  28 +-
 .../test/rest/test/RestTestParserTests.java   | 181 +-----
 .../test/rest/test/SetSectionParserTests.java |   6 +-
 .../rest/test/SetupSectionParserTests.java    |  36 +-
 .../rest/test/SkipSectionParserTests.java     |  12 +-
 .../rest/test/TestSectionParserTests.java     |  38 +-
 49 files changed, 377 insertions(+), 1360 deletions(-)
 delete mode 100644 src/test/java/com/carrotsearch/randomizedtesting/StandaloneRandomizedContext.java
 rename src/test/java/org/elasticsearch/test/rest/{junit => }/RestTestCandidate.java (61%)
 delete mode 100644 src/test/java/org/elasticsearch/test/rest/junit/DescriptionHelper.java
 delete mode 100644 src/test/java/org/elasticsearch/test/rest/junit/RestReproduceInfoPrinter.java
 delete mode 100644 src/test/java/org/elasticsearch/test/rest/junit/RestTestSuiteRunner.java
 delete mode 100644 src/test/java/org/elasticsearch/test/rest/junit/RunAfter.java

diff --git a/TESTING.asciidoc b/TESTING.asciidoc
index 282c7e080363a..bf66dcfe0f22c 100644
--- a/TESTING.asciidoc
+++ b/TESTING.asciidoc
@@ -186,22 +186,18 @@ mvn test -Dtests.class=org.elasticsearch.test.rest.ElasticsearchRestTests
 `ElasticsearchRestTests` is the executable test class that runs all the
 yaml suites available within the `rest-api-spec` folder.
 
-The following are the options supported by the REST tests runner:
+The REST tests support all the options provided by the randomized runner, plus the following:
 
-* `tests.rest[true|false|host:port]`: determines whether the REST tests need
-to be run and if so whether to rely on an external cluster (providing host
-and port) or fire a test cluster (default). It's possible to provide a
-comma separated list of addresses to send requests in a round-robin fashion.
+* `tests.rest[true|false]`: determines whether the REST tests need to be run (default) or not.
 * `tests.rest.suite`: comma separated paths of the test suites to be run
 (by default loaded from /rest-api-spec/test). It is possible to run only a subset
 of the tests providing a sub-folder or even a single yaml file (the default
 /rest-api-spec/test prefix is optional when files are loaded from classpath)
 e.g. -Dtests.rest.suite=index,get,create/10_with_id
-* `tests.rest.section`: regex that allows to filter the test sections that
-are going to be run. If provided, only the section names that match (case
-insensitive) against it will be executed
 * `tests.rest.spec`: REST spec path (default /rest-api-spec/api)
-* `tests.iters`: runs multiple iterations
-* `tests.seed`: seed to base the random behaviours on
-* `tests.appendseed[true|false]`: enables adding the seed to each test
-section's description (default false)
+
+Note that the REST tests, like all the integration tests, can be run against an external
+cluster by specifying the `tests.cluster` property, which if present needs to contain a
+comma separated list of nodes to connect to (e.g. localhost:9300). A transport client will
+be created based on that and used for all the before|after test operations, and to extract
+the http addresses of the nodes so that REST requests can be sent to them.
diff --git a/dev-tools/build_release.py b/dev-tools/build_release.py
index 5eabf08c55652..97f895488d64b 100644
--- a/dev-tools/build_release.py
+++ b/dev-tools/build_release.py
@@ -388,7 +388,7 @@ def smoke_test_release(release, files, expected_hash, plugins):
           if version['build_hash'].strip() !=  expected_hash:
             raise RuntimeError('HEAD hash does not match expected [%s] but got [%s]' % (expected_hash, version['build_hash']))
           print('  Running REST Spec tests against package [%s]' % release_file)
-          run_mvn('test -Dtests.rest=%s -Dtests.class=*.*RestTests' % ("127.0.0.1:9200"))
+          run_mvn('test -Dtests.cluster=%s -Dtests.class=*.*RestTests' % ("127.0.0.1:9300"))
           print('  Verify if plugins are listed in _nodes')
           conn.request('GET', '/_nodes?plugin=true&pretty=true')
           res = conn.getresponse()
diff --git a/pom.xml b/pom.xml
index 883a7c69f2f3f..b5fbe68bd3ee0 100644
--- a/pom.xml
+++ b/pom.xml
@@ -52,7 +52,7 @@
         <dependency>
             <groupId>com.carrotsearch.randomizedtesting</groupId>
             <artifactId>randomizedtesting-runner</artifactId>
-            <version>2.1.1</version>
+            <version>2.1.2</version>
             <scope>test</scope>
         </dependency>
         <dependency>
@@ -387,7 +387,7 @@
             <plugin>
                 <groupId>com.carrotsearch.randomizedtesting</groupId>
                 <artifactId>junit4-maven-plugin</artifactId>
-                <version>2.1.1</version>
+                <version>2.1.2</version>
                 <executions>
                     <execution>
                         <id>tests</id>
@@ -445,6 +445,7 @@
                             <sysouts>${tests.verbose}</sysouts>
                             <seed>${tests.seed}</seed>
                             <haltOnFailure>${tests.failfast}</haltOnFailure>
+                            <uniqueSuiteNames>false</uniqueSuiteNames>
                             <systemProperties>
                                 <java.io.tmpdir>.</java.io.tmpdir> <!-- we use '.' since this is different per JVM-->
                                 <!-- RandomizedTesting library system properties -->
@@ -1138,12 +1139,8 @@
                                 <include>org/elasticsearch/cache/recycler/MockPageCacheRecycler.class</include>
                                 <include>org/apache/lucene/util/AbstractRandomizedTest.class</include>
                                 <include>org/apache/lucene/util/AbstractRandomizedTest$*.class</include>
-                                <!-- inner classes -->
-                                <include>com/carrotsearch/randomizedtesting/StandaloneRandomizedContext.class</include>
                             </includes>
                             <excludes>
-                                <!-- we only distribute the REST tests runner, not the executable test  -->
-                                <exclude>org/elasticsearch/test/rest/ElasticsearchRestTests.class</exclude>
                                 <!-- unit tests for yaml suite parser & rest spec parser need to be excluded -->
                                 <exclude>org/elasticsearch/test/rest/test/**/*</exclude>
                             </excludes>
diff --git a/rest-api-spec/test/cat.allocation/10_basic.yaml b/rest-api-spec/test/cat.allocation/10_basic.yaml
index 6250e42bd2f8b..ca9d5aa6e1ad9 100644
--- a/rest-api-spec/test/cat.allocation/10_basic.yaml
+++ b/rest-api-spec/test/cat.allocation/10_basic.yaml
@@ -55,7 +55,7 @@
   - match:
       $body: >
             /^
-              ( [1-5]               \s+
+              ( \d+                 \s+
                 \d+(\.\d+)?[kmgt]b  \s+
                 \d+(\.\d+)?[kmgt]b  \s+
                 \d+(\.\d+)?[kmgt]b  \s+
diff --git a/rest-api-spec/test/cat.shards/10_basic.yaml b/rest-api-spec/test/cat.shards/10_basic.yaml
index 223fdc2ae98c8..ef10f1b0b091d 100644
--- a/rest-api-spec/test/cat.shards/10_basic.yaml
+++ b/rest-api-spec/test/cat.shards/10_basic.yaml
@@ -9,12 +9,12 @@
                /^$/
 
   - do:
-      index:
-        index:  index1
-        type:   type1
-        id:     1
-        body:   { foo: bar }
-        refresh: true
+      indices.create:
+        index: index1
+        body:
+          settings:
+            number_of_shards: "5"
+            number_of_replicas: "1"
   - do:
        cluster.health:
          wait_for_status: yellow
@@ -30,6 +30,7 @@
         index: index2
         body:
           settings:
+            number_of_shards: "5"
             number_of_replicas: "0"
   - do:
        cluster.health:
diff --git a/rest-api-spec/test/cluster.put_settings/10_basic.yaml b/rest-api-spec/test/cluster.put_settings/10_basic.yaml
index e80ab747fa496..37396dac822c4 100644
--- a/rest-api-spec/test/cluster.put_settings/10_basic.yaml
+++ b/rest-api-spec/test/cluster.put_settings/10_basic.yaml
@@ -1,4 +1,9 @@
 ---
+setup:
+  - skip:
+      version: 0 - 999
+      reason:  leaves transient metadata behind, need to fix it
+---
 "Test put settings":
   - do:
       cluster.put_settings:
diff --git a/rest-api-spec/test/create/40_routing.yaml b/rest-api-spec/test/create/40_routing.yaml
index bc3fb84f5e994..e39cbd1287d06 100644
--- a/rest-api-spec/test/create/40_routing.yaml
+++ b/rest-api-spec/test/create/40_routing.yaml
@@ -7,6 +7,7 @@
        body:
          settings:
            index:
+             number_of_shards: 5
              number_of_replicas: 0
 
  - do:
diff --git a/rest-api-spec/test/create/55_parent_with_routing.yaml b/rest-api-spec/test/create/55_parent_with_routing.yaml
index ec9471842bc5f..8ce045a64b0ab 100644
--- a/rest-api-spec/test/create/55_parent_with_routing.yaml
+++ b/rest-api-spec/test/create/55_parent_with_routing.yaml
@@ -9,6 +9,7 @@
               test:
                 _parent: { type: "foo" }
             settings:
+              number_of_shards: 5
               number_of_replicas: 0
 
  - do:
diff --git a/rest-api-spec/test/delete/30_routing.yaml b/rest-api-spec/test/delete/30_routing.yaml
index b0d3ca3885f6d..857cf76a527b6 100644
--- a/rest-api-spec/test/delete/30_routing.yaml
+++ b/rest-api-spec/test/delete/30_routing.yaml
@@ -1,6 +1,12 @@
 ---
 "Routing":
 
+ - do:
+      indices.create:
+          index: test_1
+          body:
+            settings:
+              number_of_shards: 5
  - do:
       index:
           index:   test_1
diff --git a/rest-api-spec/test/delete/40_parent.yaml b/rest-api-spec/test/delete/40_parent.yaml
index 6e8beca79b440..cd250de331038 100644
--- a/rest-api-spec/test/delete/40_parent.yaml
+++ b/rest-api-spec/test/delete/40_parent.yaml
@@ -4,6 +4,8 @@
       indices.create:
           index: test_1
           body:
+            settings:
+              number_of_shards: 5
             mappings:
               test:
                 _parent: { type: "foo" }
diff --git a/rest-api-spec/test/delete/45_parent_with_routing.yaml b/rest-api-spec/test/delete/45_parent_with_routing.yaml
index 7a11db4e46749..8b11e82b75459 100644
--- a/rest-api-spec/test/delete/45_parent_with_routing.yaml
+++ b/rest-api-spec/test/delete/45_parent_with_routing.yaml
@@ -9,6 +9,7 @@
               test:
                 _parent: { type: "foo" }
             settings:
+              number_of_shards: 5
               number_of_replicas: 0
 
  - do:
diff --git a/rest-api-spec/test/exists/40_routing.yaml b/rest-api-spec/test/exists/40_routing.yaml
index 56ba443e3cda3..6b55a3bee37f7 100644
--- a/rest-api-spec/test/exists/40_routing.yaml
+++ b/rest-api-spec/test/exists/40_routing.yaml
@@ -7,6 +7,7 @@
        body:
          settings:
            index:
+             number_of_shards: 5
              number_of_replicas: 0
 
  - do:
diff --git a/rest-api-spec/test/exists/55_parent_with_routing.yaml b/rest-api-spec/test/exists/55_parent_with_routing.yaml
index 0e92aac6b2a12..bf617a23260f7 100644
--- a/rest-api-spec/test/exists/55_parent_with_routing.yaml
+++ b/rest-api-spec/test/exists/55_parent_with_routing.yaml
@@ -9,6 +9,7 @@
               test:
                 _parent: { type: "foo" }
             settings:
+              number_of_shards: 5
               number_of_replicas: 0
 
  - do:
diff --git a/rest-api-spec/test/get/40_routing.yaml b/rest-api-spec/test/get/40_routing.yaml
index f909cb0cbf78c..f464e662c3a3f 100644
--- a/rest-api-spec/test/get/40_routing.yaml
+++ b/rest-api-spec/test/get/40_routing.yaml
@@ -7,6 +7,7 @@
        body:
          settings:
            index:
+             number_of_shards: 5
              number_of_replicas: 0
 
  - do:
diff --git a/rest-api-spec/test/get/55_parent_with_routing.yaml b/rest-api-spec/test/get/55_parent_with_routing.yaml
index 43d60f5bd508e..c65a2b121982b 100644
--- a/rest-api-spec/test/get/55_parent_with_routing.yaml
+++ b/rest-api-spec/test/get/55_parent_with_routing.yaml
@@ -9,6 +9,7 @@
               test:
                 _parent: { type: "foo" }
             settings:
+              number_of_shards: 5
               number_of_replicas: 0
 
  - do:
diff --git a/rest-api-spec/test/get_source/40_routing.yaml b/rest-api-spec/test/get_source/40_routing.yaml
index f771dbb05937f..bd9a29184472a 100644
--- a/rest-api-spec/test/get_source/40_routing.yaml
+++ b/rest-api-spec/test/get_source/40_routing.yaml
@@ -7,6 +7,7 @@
        body:
          settings:
            index:
+             number_of_shards: 5
              number_of_replicas: 0
 
  - do:
diff --git a/rest-api-spec/test/get_source/55_parent_with_routing.yaml b/rest-api-spec/test/get_source/55_parent_with_routing.yaml
index 86fe2ba47bb5b..38ca9d5f22d15 100644
--- a/rest-api-spec/test/get_source/55_parent_with_routing.yaml
+++ b/rest-api-spec/test/get_source/55_parent_with_routing.yaml
@@ -9,6 +9,7 @@
               test:
                 _parent: { type: "foo" }
             settings:
+              number_of_shards: 5
               number_of_replicas: 0
 
  - do:
diff --git a/rest-api-spec/test/index/40_routing.yaml b/rest-api-spec/test/index/40_routing.yaml
index f909cb0cbf78c..f464e662c3a3f 100644
--- a/rest-api-spec/test/index/40_routing.yaml
+++ b/rest-api-spec/test/index/40_routing.yaml
@@ -7,6 +7,7 @@
        body:
          settings:
            index:
+             number_of_shards: 5
              number_of_replicas: 0
 
  - do:
diff --git a/rest-api-spec/test/index/55_parent_with_routing.yaml b/rest-api-spec/test/index/55_parent_with_routing.yaml
index 43d60f5bd508e..c65a2b121982b 100644
--- a/rest-api-spec/test/index/55_parent_with_routing.yaml
+++ b/rest-api-spec/test/index/55_parent_with_routing.yaml
@@ -9,6 +9,7 @@
               test:
                 _parent: { type: "foo" }
             settings:
+              number_of_shards: 5
               number_of_replicas: 0
 
  - do:
diff --git a/rest-api-spec/test/indices.get_settings/10_basic.yaml b/rest-api-spec/test/indices.get_settings/10_basic.yaml
index eaca930e694f5..a50be32e82c6a 100644
--- a/rest-api-spec/test/indices.get_settings/10_basic.yaml
+++ b/rest-api-spec/test/indices.get_settings/10_basic.yaml
@@ -3,9 +3,17 @@ setup:
   - do:
         indices.create:
           index: test_1
+          body:
+            settings:
+              number_of_shards: 5
+              number_of_replicas: 1
   - do:
         indices.create:
           index: test_2
+          body:
+            settings:
+              number_of_shards: 3
+              number_of_replicas: 0
 
 ---
 "Get /_settings":
@@ -15,8 +23,8 @@ setup:
 
  - match: { test_1.settings.index.number_of_shards: "5"}
  - match: { test_1.settings.index.number_of_replicas: "1"}
- - match: { test_2.settings.index.number_of_shards: "5"}
- - match: { test_2.settings.index.number_of_replicas: "1"}
+ - match: { test_2.settings.index.number_of_shards: "3"}
+ - match: { test_2.settings.index.number_of_replicas: "0"}
 
 ---
 "Get /{index}/_settings":
@@ -98,7 +106,7 @@ setup:
         name: index.number_of_shards
 
  - match: { test_1.settings.index.number_of_shards: "5"}
- - match: { test_2.settings.index.number_of_shards: "5"}
+ - match: { test_2.settings.index.number_of_shards: "3"}
  - is_false: test_1.settings.index.number_of_replicas
  - is_false: test_2.settings.index.number_of_replicas
 
@@ -111,7 +119,7 @@ setup:
         name: index.number_of_shards
 
  - match: { test_1.settings.index.number_of_shards: "5"}
- - match: { test_2.settings.index.number_of_shards: "5"}
+ - match: { test_2.settings.index.number_of_shards: "3"}
  - is_false: test_1.settings.index.number_of_replicas
  - is_false: test_2.settings.index.number_of_replicas
 
@@ -125,7 +133,7 @@ setup:
         name: index.number_of_shards
 
  - match: { test_1.settings.index.number_of_shards: "5"}
- - match: { test_2.settings.index.number_of_shards: "5"}
+ - match: { test_2.settings.index.number_of_shards: "3"}
  - is_false: test_1.settings.index.number_of_replicas
  - is_false: test_2.settings.index.number_of_replicas
 
@@ -138,7 +146,7 @@ setup:
         name: index.number_of_shards
 
  - match: { test_1.settings.index.number_of_shards: "5"}
- - match: { test_2.settings.index.number_of_shards: "5"}
+ - match: { test_2.settings.index.number_of_shards: "3"}
  - is_false: test_1.settings.index.number_of_replicas
  - is_false: test_2.settings.index.number_of_replicas
 
@@ -150,7 +158,7 @@ setup:
         index: '*2'
         name: index.number_of_shards
 
- - match: { test_2.settings.index.number_of_shards: "5"}
+ - match: { test_2.settings.index.number_of_shards: "3"}
  - is_false: test_1
  - is_false: test_2.settings.index.number_of_replicas
 
diff --git a/rest-api-spec/test/mget/30_parent.yaml b/rest-api-spec/test/mget/30_parent.yaml
index 491b8b8d5338a..9f29627445933 100644
--- a/rest-api-spec/test/mget/30_parent.yaml
+++ b/rest-api-spec/test/mget/30_parent.yaml
@@ -8,6 +8,8 @@
             mappings:
               test:
                 _parent: { type: "foo" }
+            settings:
+              number_of_shards: 5
  - do:
       cluster.health:
           wait_for_status: yellow
diff --git a/rest-api-spec/test/mget/40_routing.yaml b/rest-api-spec/test/mget/40_routing.yaml
index 96734da32932b..6c1884cf5d385 100644
--- a/rest-api-spec/test/mget/40_routing.yaml
+++ b/rest-api-spec/test/mget/40_routing.yaml
@@ -7,6 +7,7 @@
        body:
          settings:
            index:
+             number_of_shards: 5
              number_of_replicas: 0
 
  - do:
diff --git a/rest-api-spec/test/mget/55_parent_with_routing.yaml b/rest-api-spec/test/mget/55_parent_with_routing.yaml
index 039f5c58aacce..89e99d4c8813b 100644
--- a/rest-api-spec/test/mget/55_parent_with_routing.yaml
+++ b/rest-api-spec/test/mget/55_parent_with_routing.yaml
@@ -9,6 +9,7 @@
               test:
                 _parent: { type: "foo" }
             settings:
+              number_of_shards: 5
               number_of_replicas: 0
 
  - do:
diff --git a/rest-api-spec/test/search/20_default_values.yaml b/rest-api-spec/test/search/20_default_values.yaml
index e8f14340e339c..6921a58d8868c 100644
--- a/rest-api-spec/test/search/20_default_values.yaml
+++ b/rest-api-spec/test/search/20_default_values.yaml
@@ -34,10 +34,31 @@
               foo: bar
 
   - match: {hits.total: 2}
+
+  - do:
+      search:
+        index: test_1
+        type:  test
+        body:
+          query:
+            match:
+              foo: bar
+
+  - match: {hits.total: 1}
   - match: {hits.hits.0._index: test_1 }
   - match: {hits.hits.0._type:  test   }
   - match: {hits.hits.0._id:    "1"    }
 
-  - match: {hits.hits.1._index: test_2 }
-  - match: {hits.hits.1._type:  test   }
-  - match: {hits.hits.1._id:    "42"   }
+  - do:
+      search:
+        index: test_2
+        type:  test
+        body:
+          query:
+            match:
+              foo: bar
+
+  - match: {hits.total: 1}
+  - match: {hits.hits.0._index: test_2 }
+  - match: {hits.hits.0._type:  test   }
+  - match: {hits.hits.0._id:    "42"   }
diff --git a/rest-api-spec/test/update/40_routing.yaml b/rest-api-spec/test/update/40_routing.yaml
index 52cd938fb71f2..4b03a538b9245 100644
--- a/rest-api-spec/test/update/40_routing.yaml
+++ b/rest-api-spec/test/update/40_routing.yaml
@@ -7,6 +7,7 @@
        body:
          settings:
            index:
+             number_of_shards: 5
              number_of_replicas: 0
 
  - do:
diff --git a/rest-api-spec/test/update/55_parent_with_routing.yaml b/rest-api-spec/test/update/55_parent_with_routing.yaml
index cc828021500db..51dd91af3baa2 100644
--- a/rest-api-spec/test/update/55_parent_with_routing.yaml
+++ b/rest-api-spec/test/update/55_parent_with_routing.yaml
@@ -9,6 +9,7 @@
               test:
                 _parent: { type: "foo" }
             settings:
+              number_of_shards: 5
               number_of_replicas: 0
 
  - do:
diff --git a/src/test/java/com/carrotsearch/randomizedtesting/StandaloneRandomizedContext.java b/src/test/java/com/carrotsearch/randomizedtesting/StandaloneRandomizedContext.java
deleted file mode 100644
index bf890e8612622..0000000000000
--- a/src/test/java/com/carrotsearch/randomizedtesting/StandaloneRandomizedContext.java
+++ /dev/null
@@ -1,68 +0,0 @@
-/*
- * Licensed to Elasticsearch under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. Elasticsearch licenses this
- * file to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
- * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
- * License for the specific language governing permissions and limitations under
- * the License.
- */
-
-package com.carrotsearch.randomizedtesting;
-
-/**
- * Exposes methods that allow to use a {@link RandomizedContext} without using a {@link RandomizedRunner}
- * This was specifically needed by the REST tests since they run with a custom junit runner ({@link org.elasticsearch.test.rest.junit.RestTestSuiteRunner})
- */
-public final class StandaloneRandomizedContext {
-
-    private StandaloneRandomizedContext() {
-
-    }
-
-    /**
-     * Creates a new {@link RandomizedContext} associated to the current thread
-     */
-    public static void createRandomizedContext(Class<?> testClass, Randomness runnerRandomness) {
-        //the randomized runner is passed in as null, which is fine as long as we don't try to access it afterwards
-        RandomizedContext randomizedContext = RandomizedContext.create(Thread.currentThread().getThreadGroup(), testClass, null);
-        randomizedContext.push(runnerRandomness.clone(Thread.currentThread()));
-    }
-
-    /**
-     * Destroys the {@link RandomizedContext} associated to the current thread
-     */
-    public static void disposeRandomizedContext() {
-        RandomizedContext.current().dispose();
-    }
-
-    public static void pushRandomness(Randomness randomness) {
-        RandomizedContext.current().push(randomness);
-    }
-
-    public static void popAndDestroy() {
-        RandomizedContext.current().popAndDestroy();
-    }
-
-    /**
-     * Returns the string formatted seed associated to the current thread's randomized context
-     */
-    public static String getSeedAsString() {
-        return SeedUtils.formatSeed(RandomizedContext.current().getRandomness().getSeed());
-    }
-
-    /**
-     * Util method to extract the seed out of a {@link Randomness} instance
-     */
-    public static long getSeed(Randomness randomness) {
-        return randomness.getSeed();
-    }
-}
diff --git a/src/test/java/org/elasticsearch/test/ElasticsearchIntegrationTest.java b/src/test/java/org/elasticsearch/test/ElasticsearchIntegrationTest.java
index e3acdaf6b0bf1..4ce3f99297aef 100644
--- a/src/test/java/org/elasticsearch/test/ElasticsearchIntegrationTest.java
+++ b/src/test/java/org/elasticsearch/test/ElasticsearchIntegrationTest.java
@@ -1112,7 +1112,7 @@ public static void afterClass() throws IOException {
 
     }
 
-    private final static void initializeSuiteScope() throws Exception {
+    private static void initializeSuiteScope() throws Exception {
         Class<?> targetClass = getContext().getTargetClass();
         assert INSTANCE == null;
         if (isSuiteScope(targetClass)) {
diff --git a/src/test/java/org/elasticsearch/test/junit/listeners/ReproduceInfoPrinter.java b/src/test/java/org/elasticsearch/test/junit/listeners/ReproduceInfoPrinter.java
index 3d33ce1a2a8a0..f8b571edd7dd8 100644
--- a/src/test/java/org/elasticsearch/test/junit/listeners/ReproduceInfoPrinter.java
+++ b/src/test/java/org/elasticsearch/test/junit/listeners/ReproduceInfoPrinter.java
@@ -26,13 +26,18 @@
 import org.elasticsearch.common.logging.Loggers;
 import org.elasticsearch.test.ElasticsearchTestCase;
 import org.elasticsearch.test.TestCluster;
+import org.elasticsearch.test.rest.ElasticsearchRestTests;
 import org.junit.internal.AssumptionViolatedException;
 import org.junit.runner.Description;
 import org.junit.runner.notification.Failure;
 import org.junit.runner.notification.RunListener;
 
 import static com.carrotsearch.randomizedtesting.SysGlobals.SYSPROP_ITERATIONS;
+import static com.carrotsearch.randomizedtesting.SysGlobals.SYSPROP_TESTMETHOD;
 import static org.elasticsearch.test.ElasticsearchIntegrationTest.TESTS_CLUSTER;
+import static org.elasticsearch.test.rest.ElasticsearchRestTests.REST_TESTS_SPEC;
+import static org.elasticsearch.test.rest.ElasticsearchRestTests.REST_TESTS_SUITE;
+
 
 /**
  * A {@link RunListener} that emits to {@link System#err} a string with command
@@ -63,7 +68,13 @@ public void testFailure(Failure failure) throws Exception {
         final StringBuilder b = new StringBuilder();
         b.append("FAILURE  : ").append(d.getDisplayName()).append("\n");
         b.append("REPRODUCE WITH  : mvn test");
-        reproduceErrorMessageBuilder(b).appendAllOpts(failure.getDescription());
+        MavenMessageBuilder mavenMessageBuilder = new MavenMessageBuilder(b);
+        mavenMessageBuilder.appendAllOpts(failure.getDescription());
+
+        //ElasticsearchRestTests is a special case as it allows for additional parameters
+        if (ElasticsearchRestTests.class.isAssignableFrom(failure.getDescription().getTestClass())) {
+            mavenMessageBuilder.appendRestTestsProperties();
+        }
 
         b.append("\n");
         b.append("Throwable:\n");
@@ -74,10 +85,6 @@ public void testFailure(Failure failure) throws Exception {
         logger.error(b.toString());
     }
 
-    protected ReproduceErrorMessageBuilder reproduceErrorMessageBuilder(StringBuilder b) {
-        return new MavenMessageBuilder(b);
-    }
-
     protected TraceFormatting traces() {
         TraceFormatting traces = new TraceFormatting();
         try {
@@ -97,6 +104,12 @@ public MavenMessageBuilder(StringBuilder b) {
         @Override
         public ReproduceErrorMessageBuilder appendAllOpts(Description description) {
             super.appendAllOpts(description);
+
+            if (description.getMethodName() != null) {
+                //prints out the raw method description instead of methodName(description) which filters out the parameters
+                super.appendOpt(SYSPROP_TESTMETHOD(), "\"" + description.getMethodName() + "\"");
+            }
+
             return appendESProperties();
         }
 
@@ -108,9 +121,14 @@ public ReproduceErrorMessageBuilder appendOpt(String sysPropName, String value)
             if (sysPropName.equals(SYSPROP_ITERATIONS())) { // we don't want the iters to be in there!
                 return this;
             }
+            if (sysPropName.equals(SYSPROP_TESTMETHOD())) {
+                //don't print out the test method, we print it ourselves in appendAllOpts
+                //without filtering out the parameters (needed for REST tests)
+                return this;
+            }
             if (Strings.hasLength(value)) {
                 return super.appendOpt(sysPropName, value);
-            } 
+            }
             return this;
         }
 
@@ -123,6 +141,10 @@ public ReproduceErrorMessageBuilder appendESProperties() {
             return this;
         }
 
+        public ReproduceErrorMessageBuilder appendRestTestsProperties() {
+            return appendProperties(REST_TESTS_SUITE, REST_TESTS_SPEC);
+        }
+
         protected ReproduceErrorMessageBuilder appendProperties(String... properties) {
             for (String sysPropName : properties) {
                 if (Strings.hasLength(System.getProperty(sysPropName))) {
diff --git a/src/test/java/org/elasticsearch/test/rest/ElasticsearchRestTests.java b/src/test/java/org/elasticsearch/test/rest/ElasticsearchRestTests.java
index 26841aa6edfba..e68039821dd58 100644
--- a/src/test/java/org/elasticsearch/test/rest/ElasticsearchRestTests.java
+++ b/src/test/java/org/elasticsearch/test/rest/ElasticsearchRestTests.java
@@ -16,22 +16,174 @@
  * specific language governing permissions and limitations
  * under the License.
  */
+
 package org.elasticsearch.test.rest;
 
-import org.elasticsearch.test.rest.junit.RestTestSuiteRunner;
-import org.junit.runner.RunWith;
+import com.carrotsearch.randomizedtesting.SysGlobals;
+import com.carrotsearch.randomizedtesting.annotations.Name;
+import com.carrotsearch.randomizedtesting.annotations.ParametersFactory;
+import com.carrotsearch.randomizedtesting.annotations.ReplicateOnEachVm;
+import com.google.common.collect.Lists;
+import org.elasticsearch.cluster.routing.operation.hash.djb.DjbHashFunction;
+import org.elasticsearch.common.Strings;
+import org.elasticsearch.common.math.MathUtils;
+import org.elasticsearch.test.ElasticsearchIntegrationTest;
+import org.elasticsearch.test.rest.client.RestException;
+import org.elasticsearch.test.rest.parser.RestTestParseException;
+import org.elasticsearch.test.rest.parser.RestTestSuiteParser;
+import org.elasticsearch.test.rest.section.*;
+import org.elasticsearch.test.rest.spec.RestSpec;
+import org.elasticsearch.test.rest.support.FileUtils;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
 
-import static org.apache.lucene.util.LuceneTestCase.Slow;
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
 
 /**
- * Runs the clients test suite against an elasticsearch node, which can be an external node or an automatically created cluster.
- * Communicates with elasticsearch exclusively via REST layer.
- *
- * @see RestTestSuiteRunner for extensive documentation and all the supported options
+ * Runs the clients test suite against an elasticsearch cluster.
  */
-@Slow
-@RunWith(RestTestSuiteRunner.class)
-public class ElasticsearchRestTests {
+//tests distribution disabled for now since it causes reporting problems,
+// due to the non unique suite name
+//@ReplicateOnEachVm
+public class ElasticsearchRestTests extends ElasticsearchIntegrationTest {
+
+    public static final String REST_TESTS = "tests.rest";
+    public static final String REST_TESTS_SUITE = "tests.rest.suite";
+    public static final String REST_TESTS_SPEC = "tests.rest.spec";
+
+    private static final String DEFAULT_TESTS_PATH = "/rest-api-spec/test";
+    private static final String DEFAULT_SPEC_PATH = "/rest-api-spec/api";
+
+    private static final String PATHS_SEPARATOR = ",";
+
+    private static RestTestExecutionContext restTestExecutionContext;
+
+    //private static final int JVM_COUNT = systemPropertyAsInt(SysGlobals.CHILDVM_SYSPROP_JVM_COUNT, 1);
+    //private static final int CURRENT_JVM_ID = systemPropertyAsInt(SysGlobals.CHILDVM_SYSPROP_JVM_ID, 0);
+
+    private final RestTestCandidate testCandidate;
+
+    public ElasticsearchRestTests(@Name("yaml") RestTestCandidate testCandidate) {
+        this.testCandidate = testCandidate;
+    }
+
+    @ParametersFactory
+    public static Iterable<Object[]> parameters() throws IOException, RestTestParseException {
+        List<RestTestCandidate> restTestCandidates = collectTestCandidates();
+        List<Object[]> objects = Lists.newArrayList();
+        for (RestTestCandidate restTestCandidate : restTestCandidates) {
+            objects.add(new Object[]{restTestCandidate});
+        }
+        return objects;
+    }
+
+    private static List<RestTestCandidate> collectTestCandidates() throws RestTestParseException, IOException {
+        String[] paths = resolvePathsProperty(REST_TESTS_SUITE, DEFAULT_TESTS_PATH);
+        Map<String, Set<File>> yamlSuites = FileUtils.findYamlSuites(DEFAULT_TESTS_PATH, paths);
+
+        //yaml suites are grouped by directory (effectively by api)
+        List<String> apis = Lists.newArrayList(yamlSuites.keySet());
+
+        List<RestTestCandidate> testCandidates = Lists.newArrayList();
+        RestTestSuiteParser restTestSuiteParser = new RestTestSuiteParser();
+        for (String api : apis) {
+            List<File> yamlFiles = Lists.newArrayList(yamlSuites.get(api));
+            for (File yamlFile : yamlFiles) {
+                //tests distribution disabled for now since it causes reporting problems,
+                // due to the non unique suite name
+                //if (mustExecute(yamlFile.getAbsolutePath())) {
+                    RestTestSuite restTestSuite = restTestSuiteParser.parse(api, yamlFile);
+                    for (TestSection testSection : restTestSuite.getTestSections()) {
+                        testCandidates.add(new RestTestCandidate(restTestSuite, testSection));
+                    }
+                //}
+            }
+        }
+        return testCandidates;
+    }
+
+    /*private static boolean mustExecute(String test) {
+        //we distribute the tests across the forked jvms if > 1
+        if (JVM_COUNT > 1) {
+            int jvmId = MathUtils.mod(DjbHashFunction.DJB_HASH(test), JVM_COUNT);
+            if (jvmId != CURRENT_JVM_ID) {
+                return false;
+            }
+        }
+        return true;
+    }*/
+
+    private static String[] resolvePathsProperty(String propertyName, String defaultValue) {
+        String property = System.getProperty(propertyName);
+        if (!Strings.hasLength(property)) {
+            return new String[]{defaultValue};
+        } else {
+            return property.split(PATHS_SEPARATOR);
+        }
+    }
+
+    @BeforeClass
+    public static void initExecutionContext() throws IOException, RestException {
+        //skip REST tests if disabled through -Dtests.rest=false
+        assumeTrue(systemPropertyAsBoolean(REST_TESTS, true));
+
+        String[] specPaths = resolvePathsProperty(REST_TESTS_SPEC, DEFAULT_SPEC_PATH);
+        RestSpec restSpec = RestSpec.parseFrom(DEFAULT_SPEC_PATH, specPaths);
+        restTestExecutionContext = new RestTestExecutionContext(restSpec);
+    }
+
+    @AfterClass
+    public static void close() {
+        restTestExecutionContext.close();
+    }
+
+    @Before
+    public void reset() throws IOException, RestException {
+        restTestExecutionContext.resetClient(immutableCluster().httpAddresses());
+        restTestExecutionContext.clear();
+
+        assumeFalse(buildSkipMessage(testCandidate.getSuiteDescription(), testCandidate.getSetupSection().getSkipSection()),
+                testCandidate.getSetupSection().getSkipSection().skip(restTestExecutionContext.esVersion()));
+        assumeFalse(buildSkipMessage(testCandidate.getDescription(), testCandidate.getTestSection().getSkipSection()),
+                testCandidate.getTestSection().getSkipSection().skip(restTestExecutionContext.esVersion()));
+    }
+
+    private static String buildSkipMessage(String description, SkipSection skipSection) {
+        StringBuilder messageBuilder = new StringBuilder();
+        if (skipSection.isVersionCheck()) {
+            messageBuilder.append("[").append(description).append("] skipped, reason: [").append(skipSection.getReason()).append("] ");
+        } else {
+            messageBuilder.append("[").append(description).append("] skipped, reason: features ").append(skipSection.getFeatures()).append(" not supported");
+        }
+        return messageBuilder.toString();
+    }
+
+    @Test
+    public void test() throws IOException {
+        //let's check that there is something to run, otherwise there might be a problem with the test section
+        if (testCandidate.getTestSection().getExecutableSections().size() == 0) {
+            throw new IllegalArgumentException("No executable sections loaded for ["
+                    + testCandidate.getSuiteDescription() + "/" + testCandidate.getTestSection().getName() + "]");
+        }
+
+        if (!testCandidate.getSetupSection().isEmpty()) {
+            logger.info("start setup test [{}: {}]", testCandidate.getSuiteDescription(), testCandidate.getTestSection().getName());
+            for (DoSection doSection : testCandidate.getSetupSection().getDoSections()) {
+                doSection.execute(restTestExecutionContext);
+            }
+            logger.info("end setup test [{}: {}]", testCandidate.getSuiteDescription(), testCandidate.getTestSection().getName());
+        }
 
+        restTestExecutionContext.clear();
 
-}
\ No newline at end of file
+        for (ExecutableSection executableSection : testCandidate.getTestSection().getExecutableSections()) {
+            executableSection.execute(restTestExecutionContext);
+        }
+    }
+}
diff --git a/src/test/java/org/elasticsearch/test/rest/junit/RestTestCandidate.java b/src/test/java/org/elasticsearch/test/rest/RestTestCandidate.java
similarity index 61%
rename from src/test/java/org/elasticsearch/test/rest/junit/RestTestCandidate.java
rename to src/test/java/org/elasticsearch/test/rest/RestTestCandidate.java
index 2d7544dd81fbc..52d03887e2b4a 100644
--- a/src/test/java/org/elasticsearch/test/rest/junit/RestTestCandidate.java
+++ b/src/test/java/org/elasticsearch/test/rest/RestTestCandidate.java
@@ -16,37 +16,24 @@
  * specific language governing permissions and limitations
  * under the License.
  */
-package org.elasticsearch.test.rest.junit;
+package org.elasticsearch.test.rest;
 
 import org.elasticsearch.test.rest.section.RestTestSuite;
 import org.elasticsearch.test.rest.section.SetupSection;
 import org.elasticsearch.test.rest.section.TestSection;
-import org.junit.runner.Description;
 
 /**
- * Wraps {@link org.elasticsearch.test.rest.section.TestSection}s ready to be run,
- * properly enriched with the needed execution information.
- * The tests tree structure gets flattened to the leaves (test sections)
+ * Wraps {@link org.elasticsearch.test.rest.section.TestSection}s ready to be run.
+ * Each test section is associated to its {@link org.elasticsearch.test.rest.section.RestTestSuite}.
  */
 public class RestTestCandidate {
 
     private final RestTestSuite restTestSuite;
-    private final Description suiteDescription;
     private final TestSection testSection;
-    private final Description testDescription;
-    private final long seed;
 
-    static RestTestCandidate empty(RestTestSuite restTestSuite, Description suiteDescription) {
-        return new RestTestCandidate(restTestSuite, suiteDescription, null, null, -1);
-    }
-
-    RestTestCandidate(RestTestSuite restTestSuite, Description suiteDescription,
-                      TestSection testSection, Description testDescription, long seed) {
+    public RestTestCandidate(RestTestSuite restTestSuite, TestSection testSection) {
         this.restTestSuite = restTestSuite;
-        this.suiteDescription = suiteDescription;
         this.testSection = testSection;
-        this.testDescription = testDescription;
-        this.seed = seed;
     }
 
     public String getApi() {
@@ -61,12 +48,8 @@ public String getSuiteDescription() {
         return restTestSuite.getDescription();
     }
 
-    public Description describeSuite() {
-        return suiteDescription;
-    }
-
-    public Description describeTest() {
-        return testDescription;
+    public String getDescription() {
+        return getSuiteDescription() + "/" + testSection.getName();
     }
 
     public SetupSection getSetupSection() {
@@ -77,7 +60,8 @@ public TestSection getTestSection() {
         return testSection;
     }
 
-    public long getSeed() {
-        return seed;
+    @Override
+    public String toString() {
+        return getSuiteDescription() + "/" + testSection.getName();
     }
 }
diff --git a/src/test/java/org/elasticsearch/test/rest/RestTestExecutionContext.java b/src/test/java/org/elasticsearch/test/rest/RestTestExecutionContext.java
index 03b19c1a530e0..5322e660b5a43 100644
--- a/src/test/java/org/elasticsearch/test/rest/RestTestExecutionContext.java
+++ b/src/test/java/org/elasticsearch/test/rest/RestTestExecutionContext.java
@@ -44,17 +44,16 @@ public class RestTestExecutionContext implements Closeable {
 
     private static final ESLogger logger = Loggers.getLogger(RestTestExecutionContext.class);
 
-    private final RestClient restClient;
+    private final Stash stash = new Stash();
 
-    private final String esVersion;
+    private final RestSpec restSpec;
 
-    private final Stash stash = new Stash();
+    private RestClient restClient;
 
     private RestResponse response;
 
-    public RestTestExecutionContext(InetSocketAddress[] addresses, RestSpec restSpec) throws RestException, IOException {
-        this.restClient = new RestClient(addresses, restSpec);
-        this.esVersion = restClient.getEsVersion();
+    public RestTestExecutionContext(RestSpec restSpec) throws RestException, IOException {
+        this.restSpec = restSpec;
     }
 
     /**
@@ -104,15 +103,6 @@ private String bodyAsString(Map<String, Object> body) throws IOException {
         return XContentFactory.jsonBuilder().map(body).string();
     }
 
-    /**
-     * Calls an elasticsearch api internally without saving the obtained response in the context.
-     * Useful for internal calls (e.g. delete index during teardown)
-     * @throws RestException if the returned status code is non ok
-     */
-    public RestResponse callApiInternal(String apiName, String... params) throws IOException, RestException {
-        return restClient.callApi(apiName, params);
-    }
-
     private RestResponse callApiInternal(String apiName, Map<String, String> params, String body) throws IOException, RestException  {
         return restClient.callApi(apiName, params, body);
     }
@@ -124,11 +114,22 @@ public Object response(String path) throws IOException {
         return response.evaluate(path);
     }
 
+    /**
+     * Recreates the embedded REST client which will point to the given addresses
+     */
+    public void resetClient(InetSocketAddress[] addresses) throws IOException, RestException {
+        if (restClient == null) {
+            restClient = new RestClient(addresses, restSpec);
+        } else {
+            restClient.updateAddresses(addresses);
+        }
+    }
+
     /**
      * Clears the last obtained response and the stashed fields
      */
     public void clear() {
-        logger.debug("resetting response and stash");
+        logger.debug("resetting client, response and stash");
         response = null;
         stash.clear();
     }
@@ -141,7 +142,7 @@ public Stash stash() {
      * Returns the current es version as a string
      */
     public String esVersion() {
-        return esVersion;
+        return restClient.getEsVersion();
     }
 
     /**
diff --git a/src/test/java/org/elasticsearch/test/rest/client/RestClient.java b/src/test/java/org/elasticsearch/test/rest/client/RestClient.java
index e1e916b24fcd3..2d1ff1d20d8ad 100644
--- a/src/test/java/org/elasticsearch/test/rest/client/RestClient.java
+++ b/src/test/java/org/elasticsearch/test/rest/client/RestClient.java
@@ -48,7 +48,7 @@ public class RestClient implements Closeable {
     private final RestSpec restSpec;
     private final CloseableHttpClient httpClient;
 
-    private final InetSocketAddress[] addresses;
+    private InetSocketAddress[] addresses;
 
     private final String esVersion;
 
@@ -96,21 +96,10 @@ public String getEsVersion() {
     }
 
     /**
-     * Calls an api with the provided parameters
-     * @throws RestException if the obtained status code is non ok, unless the specific error code needs to be ignored
-     * according to the ignore parameter received as input (which won't get sent to elasticsearch)
+     * Allows to update the addresses the client needs to connect to
      */
-    public RestResponse callApi(String apiName, String... params) throws IOException, RestException {
-        if (params.length % 2 != 0) {
-            throw new IllegalArgumentException("The number of params passed must be even but was [" + params.length + "]");
-        }
-
-        Map<String, String> paramsMap = Maps.newHashMap();
-        for (int i = 0; i < params.length; i++) {
-            paramsMap.put(params[i++], params[i]);
-        }
-
-        return callApi(apiName, paramsMap, null);
+    public void updateAddresses(InetSocketAddress[] addresses) {
+        this.addresses = addresses;
     }
 
     /**
diff --git a/src/test/java/org/elasticsearch/test/rest/junit/DescriptionHelper.java b/src/test/java/org/elasticsearch/test/rest/junit/DescriptionHelper.java
deleted file mode 100644
index 44284a4d9e493..0000000000000
--- a/src/test/java/org/elasticsearch/test/rest/junit/DescriptionHelper.java
+++ /dev/null
@@ -1,79 +0,0 @@
-/*
- * Licensed to Elasticsearch under one or more contributor
- * license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright
- * ownership. Elasticsearch licenses this file to you under
- * the Apache License, Version 2.0 (the "License"); you may
- * not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *    http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.elasticsearch.test.rest.junit;
-
-import com.google.common.base.Joiner;
-import org.elasticsearch.test.rest.section.RestTestSuite;
-import org.elasticsearch.test.rest.section.TestSection;
-import org.junit.runner.Description;
-
-import java.util.Map;
-
-/**
- * Helper that knows how to assign proper junit {@link Description}s to each of the node in the tests tree
- */
-public final class DescriptionHelper {
-
-    private DescriptionHelper() {
-
-    }
-
-    /*
-    The following generated ids need to be unique throughout a tests run.
-    Ids are also shown by IDEs (with junit 4.11 unique ids can be different from what gets shown, not yet in 4.10).
-    Some tricks are applied to control what gets shown in IDEs in order to keep the ids unique and nice to see at the same time.
-     */
-
-    static Description createRootDescription(String name) {
-        return Description.createSuiteDescription(name);
-    }
-
-    static Description createApiDescription(String api) {
-        return Description.createSuiteDescription(api);
-    }
-
-    static Description createTestSuiteDescription(RestTestSuite restTestSuite) {
-        //e.g. "indices.open (10_basic)", which leads to 10_basic being shown by IDEs
-        String name = restTestSuite.getApi() + " (" + restTestSuite.getName() + ")";
-        return Description.createSuiteDescription(name);
-    }
-
-    static Description createTestSectionWithRepetitionsDescription(RestTestSuite restTestSuite, TestSection testSection) {
-        //e.g. "indices.open/10_basic (Basic test for index open/close)", which leads to
-        //"Basic test for index open/close" being shown by IDEs
-        String name = restTestSuite.getDescription() + " (" + testSection.getName() + ")";
-        return Description.createSuiteDescription(name);
-    }
-
-    static Description createTestSectionIterationDescription(RestTestSuite restTestSuite, TestSection testSection, Map<String, Object> args) {
-        //e.g. "Basic test for index open/close {#0} (indices.open/10_basic)" some IDEs might strip out the part between parentheses
-        String name = testSection.getName() + formatMethodArgs(args) + " ("  + restTestSuite.getDescription() + ")";
-        return Description.createSuiteDescription(name);
-    }
-
-    private static String formatMethodArgs(Map<String, Object> args) {
-        if (args == null || args.isEmpty()) return "";
-
-        StringBuilder b = new StringBuilder(" {");
-        Joiner.on(" ").withKeyValueSeparator("").appendTo(b, args);
-        b.append("}");
-
-        return b.toString();
-    }
-}
diff --git a/src/test/java/org/elasticsearch/test/rest/junit/RestReproduceInfoPrinter.java b/src/test/java/org/elasticsearch/test/rest/junit/RestReproduceInfoPrinter.java
deleted file mode 100644
index 735fbf66d05c4..0000000000000
--- a/src/test/java/org/elasticsearch/test/rest/junit/RestReproduceInfoPrinter.java
+++ /dev/null
@@ -1,107 +0,0 @@
-/*
- * Licensed to Elasticsearch under one or more contributor
- * license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright
- * ownership. Elasticsearch licenses this file to you under
- * the Apache License, Version 2.0 (the "License"); you may
- * not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *    http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.elasticsearch.test.rest.junit;
-
-import com.carrotsearch.randomizedtesting.ReproduceErrorMessageBuilder;
-import com.carrotsearch.randomizedtesting.StandaloneRandomizedContext;
-import com.carrotsearch.randomizedtesting.TraceFormatting;
-import org.elasticsearch.common.logging.ESLogger;
-import org.elasticsearch.common.logging.Loggers;
-import org.elasticsearch.test.junit.listeners.ReproduceInfoPrinter;
-import org.elasticsearch.test.rest.ElasticsearchRestTests;
-import org.elasticsearch.test.rest.junit.RestTestSuiteRunner.RunMode;
-import org.junit.runner.Description;
-
-import java.util.Arrays;
-
-import static com.carrotsearch.randomizedtesting.SysGlobals.SYSPROP_RANDOM_SEED;
-import static com.carrotsearch.randomizedtesting.SysGlobals.SYSPROP_TESTCLASS;
-import static org.elasticsearch.test.rest.junit.RestTestSuiteRunner.*;
-
-/**
- * A {@link org.junit.runner.notification.RunListener} that emits to {@link System#err} a string with command
- * line parameters allowing quick REST test re-run under MVN command line.
- */
-class RestReproduceInfoPrinter extends ReproduceInfoPrinter {
-
-    protected static final ESLogger logger = Loggers.getLogger(RestReproduceInfoPrinter.class);
-
-    private static boolean isTestCluster() {
-        return runMode() == RunMode.TEST_CLUSTER;
-    }
-
-    @Override
-    protected TraceFormatting traces() {
-        return new TraceFormatting(
-                Arrays.asList(
-                    "org.junit.",
-                    "junit.framework.",
-                    "sun.",
-                    "java.lang.reflect.",
-                    "com.carrotsearch.randomizedtesting.",
-                    "org.elasticsearch.test.rest.junit."
-                ));
-    }
-
-    @Override
-    protected ReproduceErrorMessageBuilder reproduceErrorMessageBuilder(StringBuilder b) {
-        return new MavenMessageBuilder(b);
-    }
-
-    private static class MavenMessageBuilder extends ReproduceInfoPrinter.MavenMessageBuilder {
-
-        public MavenMessageBuilder(StringBuilder b) {
-            super(b);
-        }
-
-        @Override
-        public ReproduceErrorMessageBuilder appendAllOpts(Description description) {
-
-            try {
-                appendOpt(SYSPROP_RANDOM_SEED(), StandaloneRandomizedContext.getSeedAsString());
-            } catch (IllegalStateException e) {
-                logger.warn("No context available when dumping reproduce options?");
-            }
-
-            //we know that ElasticsearchRestTests is the only one that runs with RestTestSuiteRunner
-            appendOpt(SYSPROP_TESTCLASS(), ElasticsearchRestTests.class.getName());
-
-            if (description.getClassName() != null) {
-                appendOpt(REST_TESTS_SUITE, description.getClassName());
-            }
-
-            appendRunnerProperties();
-            appendEnvironmentSettings();
-
-            appendProperties("es.logger.level");
-
-            if (isTestCluster()) {
-                appendProperties("es.node.mode", "es.node.local");
-            }
-
-            appendRestTestsProperties();
-
-            return this;
-        }
-
-        public ReproduceErrorMessageBuilder appendRestTestsProperties() {
-            return appendProperties(REST_TESTS_MODE, REST_TESTS_SPEC);
-        }
-    }
-}
diff --git a/src/test/java/org/elasticsearch/test/rest/junit/RestTestSuiteRunner.java b/src/test/java/org/elasticsearch/test/rest/junit/RestTestSuiteRunner.java
deleted file mode 100644
index b26a0229a7e67..0000000000000
--- a/src/test/java/org/elasticsearch/test/rest/junit/RestTestSuiteRunner.java
+++ /dev/null
@@ -1,598 +0,0 @@
-/*
- * Licensed to Elasticsearch under one or more contributor
- * license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright
- * ownership. Elasticsearch licenses this file to you under
- * the Apache License, Version 2.0 (the "License"); you may
- * not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *    http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.elasticsearch.test.rest.junit;
-
-import com.carrotsearch.hppc.hash.MurmurHash3;
-import com.carrotsearch.randomizedtesting.RandomizedTest;
-import com.carrotsearch.randomizedtesting.Randomness;
-import com.carrotsearch.randomizedtesting.SeedUtils;
-import com.google.common.collect.Lists;
-import org.elasticsearch.common.Strings;
-import org.elasticsearch.common.logging.ESLogger;
-import org.elasticsearch.common.logging.Loggers;
-import org.elasticsearch.common.transport.InetSocketTransportAddress;
-import org.elasticsearch.http.HttpServerTransport;
-import org.elasticsearch.test.ElasticsearchTestCase;
-import org.elasticsearch.test.TestCluster;
-import org.elasticsearch.test.rest.RestTestExecutionContext;
-import org.elasticsearch.test.rest.client.RestException;
-import org.elasticsearch.test.rest.client.RestResponse;
-import org.elasticsearch.test.rest.parser.RestTestParseException;
-import org.elasticsearch.test.rest.parser.RestTestSuiteParser;
-import org.elasticsearch.test.rest.section.DoSection;
-import org.elasticsearch.test.rest.section.ExecutableSection;
-import org.elasticsearch.test.rest.section.RestTestSuite;
-import org.elasticsearch.test.rest.section.TestSection;
-import org.elasticsearch.test.rest.spec.RestSpec;
-import org.elasticsearch.test.rest.support.Features;
-import org.elasticsearch.test.rest.support.FileUtils;
-import org.junit.runner.Description;
-import org.junit.runner.notification.Failure;
-import org.junit.runner.notification.RunNotifier;
-import org.junit.runners.ParentRunner;
-import org.junit.runners.model.InitializationError;
-import org.junit.runners.model.Statement;
-
-import java.io.File;
-import java.io.IOException;
-import java.net.InetSocketAddress;
-import java.util.*;
-import java.util.concurrent.atomic.AtomicInteger;
-import java.util.regex.Pattern;
-
-import static com.carrotsearch.randomizedtesting.SeedUtils.parseSeedChain;
-import static com.carrotsearch.randomizedtesting.StandaloneRandomizedContext.*;
-import static com.carrotsearch.randomizedtesting.SysGlobals.*;
-import static org.elasticsearch.test.TestCluster.clusterName;
-import static org.elasticsearch.test.rest.junit.DescriptionHelper.*;
-import static org.hamcrest.Matchers.equalTo;
-import static org.junit.Assert.assertThat;
-
-/**
- * JUnit runner for elasticsearch REST tests
- *
- * Supports the following options provided as system properties:
- * - tests.rest[true|false|host:port]: determines whether the REST tests need to be run and if so
- *                                     whether to rely on an external cluster (providing host and port) or fire a test cluster (default)
- * - tests.rest.suite: comma separated paths of the test suites to be run (by default loaded from /rest-api-spec/test)
- *                     it is possible to run only a subset of the tests providing a directory or a single yaml file
- *                     (the default /rest-api-spec/test prefix is optional when files are loaded from classpath)
- * - tests.rest.section: regex that allows to filter the test sections that are going to be run. If provided, only the
- *                       section names that match (case insensitive) against it will be executed
- * - tests.rest.spec: REST spec path (default /rest-api-spec/api)
- * - tests.iters: runs multiple iterations
- * - tests.seed: seed to base the random behaviours on
- * - tests.appendseed[true|false]: enables adding the seed to each test section's description (default false)
- *
- */
-public class RestTestSuiteRunner extends ParentRunner<RestTestCandidate> {
-
-    private static final ESLogger logger = Loggers.getLogger(RestTestSuiteRunner.class);
-
-    public static final String REST_TESTS_MODE = "tests.rest";
-    public static final String REST_TESTS_SUITE = "tests.rest.suite";
-    public static final String REST_TESTS_SECTION = "tests.rest.section";
-    public static final String REST_TESTS_SPEC = "tests.rest.spec";
-
-    private static final String DEFAULT_TESTS_PATH = "/rest-api-spec/test";
-    private static final String DEFAULT_SPEC_PATH = "/rest-api-spec/api";
-    private static final int DEFAULT_ITERATIONS = 1;
-
-    private static final String PATHS_SEPARATOR = ",";
-
-    private final RestTestExecutionContext restTestExecutionContext;
-    private final List<RestTestCandidate> restTestCandidates;
-    private final Description rootDescription;
-
-    private final RunMode runMode;
-
-    private final TestCluster testCluster;
-
-    private static final AtomicInteger sequencer = new AtomicInteger();
-
-    /** The runner's seed (master). */
-    private final Randomness runnerRandomness;
-
-    /**
-     * If {@link com.carrotsearch.randomizedtesting.SysGlobals#SYSPROP_RANDOM_SEED} property is used with two arguments
-     * (master:test_section) then this field contains test section level override.
-     */
-    private final Randomness testSectionRandomnessOverride;
-
-    enum RunMode {
-        NO, TEST_CLUSTER, EXTERNAL_CLUSTER
-    }
-
-    static RunMode runMode() {
-        String mode = System.getProperty(REST_TESTS_MODE);
-        if (!Strings.hasLength(mode)) {
-            //default true: we run the tests starting our own test cluster
-            mode = Boolean.TRUE.toString();
-        }
-
-        if (Boolean.FALSE.toString().equalsIgnoreCase(mode)) {
-            return RunMode.NO;
-        }
-        if (Boolean.TRUE.toString().equalsIgnoreCase(mode)) {
-            return RunMode.TEST_CLUSTER;
-        }
-        return RunMode.EXTERNAL_CLUSTER;
-    }
-
-    public RestTestSuiteRunner(Class<?> testClass) throws InitializationError {
-        super(testClass);
-
-        this.runMode = runMode();
-
-        if (runMode == RunMode.NO) {
-            //the tests won't be run. the run method will be called anyway but we'll just mark the whole suite as ignored
-            //no need to go ahead and parse the test suites then
-            this.runnerRandomness = null;
-            this.testSectionRandomnessOverride = null;
-            this.restTestExecutionContext = null;
-            this.restTestCandidates = null;
-            this.rootDescription = createRootDescription(getRootSuiteTitle());
-            this.rootDescription.addChild(createApiDescription("empty suite"));
-            this.testCluster = null;
-            return;
-        }
-
-        //the REST test suite is supposed to be run only once per jvm against either an external es node or a self started one
-        if (sequencer.getAndIncrement() > 0) {
-            throw new InitializationError("only one instance of RestTestSuiteRunner can be created per jvm");
-        }
-
-        //either read the seed from system properties (first one in the chain) or generate a new one
-        final String globalSeed = System.getProperty(SYSPROP_RANDOM_SEED());
-        final long initialSeed;
-        Randomness randomnessOverride = null;
-        if (Strings.hasLength(globalSeed)) {
-            final long[] seedChain = parseSeedChain(globalSeed);
-            if (seedChain.length == 0 || seedChain.length > 2) {
-                throw new IllegalArgumentException("Invalid system property "
-                        + SYSPROP_RANDOM_SEED() + " specification: " + globalSeed);
-            }
-            if (seedChain.length > 1) {
-                //read the test section level seed if present
-                randomnessOverride = new Randomness(seedChain[1]);
-            }
-            initialSeed = seedChain[0];
-        } else {
-            initialSeed = MurmurHash3.hash(System.nanoTime());
-        }
-        this.runnerRandomness = new Randomness(initialSeed);
-        this.testSectionRandomnessOverride = randomnessOverride;
-        logger.info("Master seed: {}", SeedUtils.formatSeed(initialSeed));
-
-        List<InetSocketAddress> addresses = Lists.newArrayList();
-        if (runMode == RunMode.TEST_CLUSTER) {
-            this.testCluster = new TestCluster(initialSeed, 1, 3,
-                    clusterName("REST-tests", ElasticsearchTestCase.CHILD_VM_ID, initialSeed));
-            this.testCluster.beforeTest(runnerRandomness.getRandom(), 0.0f);
-            for (HttpServerTransport httpServerTransport : testCluster.getInstances(HttpServerTransport.class)) {
-                addresses.add(((InetSocketTransportAddress) httpServerTransport.boundAddress().publishAddress()).address());
-            }
-        } else {
-            this.testCluster = null;
-            String testsMode = System.getProperty(REST_TESTS_MODE);
-            String[] stringAddresses = testsMode.split(",");
-            for (String stringAddress : stringAddresses) {
-                String[] split = stringAddress.split(":");
-                if (split.length < 2) {
-                    throw new InitializationError("address [" + testsMode + "] not valid");
-                }
-                try {
-                    addresses.add(new InetSocketAddress(split[0], Integer.valueOf(split[1])));
-                } catch(NumberFormatException e) {
-                    throw new InitializationError("port is not valid, expected number but was [" + split[1] + "]");
-                }
-            }
-        }
-
-        try {
-            String[] specPaths = resolvePathsProperty(REST_TESTS_SPEC, DEFAULT_SPEC_PATH);
-            RestSpec restSpec = RestSpec.parseFrom(DEFAULT_SPEC_PATH, specPaths);
-            this.restTestExecutionContext = new RestTestExecutionContext(addresses.toArray(new InetSocketAddress[addresses.size()]), restSpec);
-            this.rootDescription = createRootDescription(getRootSuiteTitle());
-            this.restTestCandidates = collectTestCandidates(rootDescription);
-        } catch (InitializationError e) {
-          stopTestCluster();
-            throw e;
-        } catch (Throwable e) {
-            stopTestCluster();
-            throw new InitializationError(e);
-        }
-    }
-
-    /**
-     * Parse the test suites and creates the test candidates to be run, together with their junit descriptions.
-     * The descriptions will be part of a tree containing api/yaml file/test section/eventual multiple iterations.
-     * The test candidates will be instead flattened out to the leaves level (iterations), the part that needs to be run.
-     */
-    protected List<RestTestCandidate> collectTestCandidates(Description rootDescription) throws InitializationError, IOException {
-
-        String[] paths = resolvePathsProperty(REST_TESTS_SUITE, DEFAULT_TESTS_PATH);
-        Map<String, Set<File>> yamlSuites = FileUtils.findYamlSuites(DEFAULT_TESTS_PATH, paths);
-
-        String sectionFilter = System.getProperty(REST_TESTS_SECTION);
-        Pattern sectionFilterPattern = null;
-        if (Strings.hasLength(sectionFilter)) {
-            sectionFilterPattern = Pattern.compile(sectionFilter, Pattern.CASE_INSENSITIVE);
-        }
-
-        int iterations = determineTestSectionIterationCount();
-        boolean appendSeedParameter = RandomizedTest.systemPropertyAsBoolean(SYSPROP_APPEND_SEED(), false);
-
-        //we iterate over the files and we shuffle them (grouped by api, and by yaml file)
-        //meanwhile we create the junit descriptions and test candidates (one per iteration)
-
-        //yaml suites are grouped by directory (effectively by api)
-        List<String> apis = Lists.newArrayList(yamlSuites.keySet());
-        Collections.shuffle(apis, runnerRandomness.getRandom());
-
-        final boolean fixedSeed = testSectionRandomnessOverride != null;
-        final boolean hasRepetitions = iterations > 1;
-
-        List<Throwable> parseExceptions = Lists.newArrayList();
-        List<RestTestCandidate> testCandidates = Lists.newArrayList();
-        RestTestSuiteParser restTestSuiteParser = new RestTestSuiteParser();
-        for (String api : apis) {
-
-            Description apiDescription = createApiDescription(api);
-
-            List<File> yamlFiles = Lists.newArrayList(yamlSuites.get(api));
-            Collections.shuffle(yamlFiles, runnerRandomness.getRandom());
-
-            for (File yamlFile : yamlFiles) {
-                RestTestSuite restTestSuite;
-                try {
-                    restTestSuite = restTestSuiteParser.parse(restTestExecutionContext.esVersion(), api, yamlFile);
-                } catch (RestTestParseException e) {
-                    parseExceptions.add(e);
-                    //we continue so that we collect all parse errors and show them all at once
-                    continue;
-                }
-
-                Description testSuiteDescription = createTestSuiteDescription(restTestSuite);
-
-                if (restTestSuite.getTestSections().size() == 0) {
-                    assert restTestSuite.getSetupSection().getSkipSection().skip(restTestExecutionContext.esVersion());
-                    testCandidates.add(RestTestCandidate.empty(restTestSuite, testSuiteDescription));
-                    continue;
-                }
-
-                Collections.shuffle(restTestSuite.getTestSections(), runnerRandomness.getRandom());
-
-                for (TestSection testSection : restTestSuite.getTestSections()) {
-
-                    if (sectionFilterPattern != null) {
-                        if (!sectionFilterPattern.matcher(testSection.getName()).find()) {
-                            continue;
-                        }
-                    }
-
-                    //no need to generate seed if we are going to skip the test section
-                    if (testSection.getSkipSection().skip(restTestExecutionContext.esVersion())) {
-                        Description testSectionDescription = createTestSectionIterationDescription(restTestSuite, testSection, null);
-                        testSuiteDescription.addChild(testSectionDescription);
-                        testCandidates.add(new RestTestCandidate(restTestSuite, testSuiteDescription, testSection, testSectionDescription, -1));
-                        continue;
-                    }
-
-                    Description parentDescription;
-                    if (hasRepetitions) {
-                        //additional level to group multiple iterations under the same test section's node
-                        parentDescription = createTestSectionWithRepetitionsDescription(restTestSuite, testSection);
-                        testSuiteDescription.addChild(parentDescription);
-                    } else {
-                        parentDescription = testSuiteDescription;
-                    }
-
-                    final long testSectionSeed = determineTestSectionSeed(restTestSuite.getDescription() + "/" + testSection.getName());
-                    for (int i = 0; i < iterations; i++) {
-                        //test section name argument needs to be unique here
-                        long thisSeed = (fixedSeed ? testSectionSeed : testSectionSeed ^ MurmurHash3.hash((long) i));
-
-                        final LinkedHashMap<String, Object> args = new LinkedHashMap<>();
-                        if (hasRepetitions) {
-                            args.put("#", i);
-                        }
-                        if (hasRepetitions || appendSeedParameter) {
-                            args.put("seed=", SeedUtils.formatSeedChain(runnerRandomness, new Randomness(thisSeed)));
-                        }
-
-                        Description testSectionDescription = createTestSectionIterationDescription(restTestSuite, testSection, args);
-                        parentDescription.addChild(testSectionDescription);
-                        testCandidates.add(new RestTestCandidate(restTestSuite, testSuiteDescription, testSection, testSectionDescription, thisSeed));
-                    }
-                }
-
-                //we add the suite only if it has at least a section left
-                if (testSuiteDescription.getChildren().size() > 0) {
-                    apiDescription.addChild(testSuiteDescription);
-                }
-            }
-
-            //we add the api only if it has at least a suite left
-            if (apiDescription.getChildren().size() > 0) {
-                rootDescription.addChild(apiDescription);
-            }
-        }
-
-        if (!parseExceptions.isEmpty()) {
-            throw new InitializationError(parseExceptions);
-        }
-
-        if (rootDescription.getChildren().size() == 0) {
-            throw new InitializationError("No tests to run");
-        }
-
-        return testCandidates;
-    }
-
-    protected String getRootSuiteTitle() {
-        if (runMode == RunMode.NO) {
-            return "elasticsearch REST Tests - not run";
-        }
-        if (runMode == RunMode.TEST_CLUSTER) {
-            return String.format(Locale.ROOT, "elasticsearch REST Tests - test cluster");
-        }
-        if (runMode == RunMode.EXTERNAL_CLUSTER) {
-            return String.format(Locale.ROOT, "elasticsearch REST Tests - external cluster %s", System.getProperty(REST_TESTS_MODE));
-        }
-        throw new UnsupportedOperationException("runMode [" + runMode + "] not supported");
-    }
-
-    private int determineTestSectionIterationCount() {
-        int iterations = RandomizedTest.systemPropertyAsInt(SYSPROP_ITERATIONS(), DEFAULT_ITERATIONS);
-        if (iterations < 1) {
-            throw new IllegalArgumentException("System property " + SYSPROP_ITERATIONS() + " must be >= 1 but was [" + iterations + "]");
-        }
-        return iterations;
-    }
-
-    protected static String[] resolvePathsProperty(String propertyName, String defaultValue) {
-        String property = System.getProperty(propertyName);
-        if (!Strings.hasLength(property)) {
-            return new String[]{defaultValue};
-        } else {
-            return property.split(PATHS_SEPARATOR);
-        }
-    }
-
-    /**
-     * Determine a given test section's initial random seed
-     */
-    private long determineTestSectionSeed(String testSectionName) {
-        if (testSectionRandomnessOverride != null) {
-            return getSeed(testSectionRandomnessOverride);
-        }
-
-        // We assign each test section a different starting hash based on the global seed
-        // and a hash of their name (so that the order of sections does not matter, only their names)
-        return getSeed(runnerRandomness) ^ MurmurHash3.hash((long) testSectionName.hashCode());
-    }
-
-    @Override
-    protected List<RestTestCandidate> getChildren() {
-        return restTestCandidates;
-    }
-
-    @Override
-    public Description getDescription() {
-        return rootDescription;
-    }
-
-    @Override
-    protected Description describeChild(RestTestCandidate child) {
-        return child.describeTest();
-    }
-
-    @Override
-    protected Statement classBlock(RunNotifier notifier) {
-        //we remove support for @BeforeClass & @AfterClass and JUnit Rules (as we don't call super)
-        Statement statement = childrenInvoker(notifier);
-        statement = withExecutionContextClose(statement);
-        if (testCluster != null) {
-            return withTestClusterClose(statement);
-        }
-        return statement;
-    }
-
-    protected Statement withExecutionContextClose(Statement statement) {
-        return new RunAfter(statement, new Statement() {
-            @Override
-            public void evaluate() throws Throwable {
-                restTestExecutionContext.close();
-            }
-        });
-    }
-
-    protected Statement withTestClusterClose(Statement statement) {
-        return new RunAfter(statement, new Statement() {
-            @Override
-            public void evaluate() throws Throwable {
-                stopTestCluster();
-            }
-        });
-    }
-
-    @Override
-    public void run(final RunNotifier notifier) {
-
-        if (runMode == RunMode.NO) {
-            notifier.fireTestIgnored(rootDescription.getChildren().get(0));
-            return;
-        }
-        final RestReproduceInfoPrinter restReproduceInfoPrinter = new RestReproduceInfoPrinter();
-        notifier.addListener(restReproduceInfoPrinter);
-        try {
-            //the test suite gets run on a separate thread as the randomized context is per thread
-            //once the randomized context is disposed it's not possible to create it again on the same thread
-            final Thread thread = new Thread() {
-                @Override
-                public void run() {
-                    try {
-                        createRandomizedContext(getTestClass().getJavaClass(), runnerRandomness);
-                        RestTestSuiteRunner.super.run(notifier);
-                    } finally {
-                        disposeRandomizedContext();
-                    }
-                }
-            };
-
-            thread.start();
-            try {
-                thread.join();
-            } catch (InterruptedException e) {
-                notifier.fireTestFailure(new Failure(getDescription(),
-                        new RuntimeException("Interrupted while waiting for the suite runner? Weird.", e)));
-            }
-        } finally {
-            // remove the listener once the suite is done otherwise it will print
-            // a bogus line if a subsequent test fails that is not a
-            // REST test. The RunNotifier is used across suites!
-            notifier.removeListener(restReproduceInfoPrinter);
-        }
-    }
-
-    @Override
-    protected void runChild(RestTestCandidate testCandidate, RunNotifier notifier) {
-
-        //if the while suite needs to be skipped, no test sections were loaded, only an empty one that we need to mark as ignored
-        if (testCandidate.getSetupSection().getSkipSection().skip(restTestExecutionContext.esVersion())) {
-            if (logger.isInfoEnabled()) {
-                if (testCandidate.getSetupSection().getSkipSection().isVersionCheck()) {
-                    logger.info("skipped test suite [{}]\nreason: {}\nskip versions: {} (current version: {})",
-                            testCandidate.getSuiteDescription(), testCandidate.getSetupSection().getSkipSection().getReason(),
-                            testCandidate.getSetupSection().getSkipSection().getVersion(), restTestExecutionContext.esVersion());
-                } else {
-                    logger.info("skipped test suite [{}]\nreason: feature not supported\nrequired features: {} (supported features: {})",
-                            testCandidate.getSuiteDescription(), testCandidate.getSetupSection().getSkipSection().getFeatures(), Features.getSupported());
-                }
-            }
-            notifier.fireTestIgnored(testCandidate.describeSuite());
-            return;
-        }
-
-        //from now on no more empty test candidates are expected
-        assert testCandidate.getTestSection() != null;
-
-        if (testCandidate.getTestSection().getSkipSection().skip(restTestExecutionContext.esVersion())) {
-            if (logger.isInfoEnabled()) {
-                if (testCandidate.getTestSection().getSkipSection().isVersionCheck()) {
-                    logger.info("skipped test [{}/{}]\nreason: {}\nskip versions: {} (current version: {})",
-                            testCandidate.getSuiteDescription(), testCandidate.getTestSection().getName(),
-                            testCandidate.getTestSection().getSkipSection().getReason(),
-                            testCandidate.getTestSection().getSkipSection().getVersion(), restTestExecutionContext.esVersion());
-                } else {
-                    logger.info("skipped test [{}/{}]\nreason: feature not supported\nrequired features: {} (supported features: {})",
-                            testCandidate.getSuiteDescription(), testCandidate.getTestSection().getName(),
-                            testCandidate.getTestSection().getSkipSection().getFeatures(), Features.getSupported());
-                }
-            }
-
-            notifier.fireTestIgnored(testCandidate.describeTest());
-            return;
-        }
-
-        runLeaf(methodBlock(testCandidate), testCandidate.describeTest(), notifier);
-    }
-
-    protected Statement methodBlock(final RestTestCandidate testCandidate) {
-        return new Statement() {
-            @Override
-            public void evaluate() throws Throwable {
-                final String testThreadName = "TEST-" + testCandidate.getSuiteDescription() +
-                        "." + testCandidate.getTestSection().getName() + "-seed#" + SeedUtils.formatSeedChain(runnerRandomness);
-                // This has a side effect of setting up a nested context for the test thread.
-                final String restoreName = Thread.currentThread().getName();
-                try {
-                    Thread.currentThread().setName(testThreadName);
-                    pushRandomness(new Randomness(testCandidate.getSeed()));
-                    runTestSection(testCandidate);
-                } finally {
-                    Thread.currentThread().setName(restoreName);
-                    popAndDestroy();
-                }
-            }
-        };
-    }
-
-    protected void runTestSection(RestTestCandidate testCandidate)
-            throws IOException, RestException {
-
-        //let's check that there is something to run, otherwise there might be a problem with the test section
-        if (testCandidate.getTestSection().getExecutableSections().size() == 0) {
-            throw new IllegalArgumentException("No executable sections loaded for ["
-                    + testCandidate.getSuiteDescription() + "/" + testCandidate.getTestSection().getName() + "]");
-        }
-
-        logger.info("cleaning up before test [{}: {}]", testCandidate.getSuiteDescription(), testCandidate.getTestSection().getName());
-        tearDown();
-
-        logger.info("start test [{}: {}]", testCandidate.getSuiteDescription(), testCandidate.getTestSection().getName());
-
-        if (!testCandidate.getSetupSection().isEmpty()) {
-            logger.info("start setup test [{}: {}]", testCandidate.getSuiteDescription(), testCandidate.getTestSection().getName());
-            for (DoSection doSection : testCandidate.getSetupSection().getDoSections()) {
-                doSection.execute(restTestExecutionContext);
-            }
-            logger.info("end setup test [{}: {}]", testCandidate.getSuiteDescription(), testCandidate.getTestSection().getName());
-        }
-
-        restTestExecutionContext.clear();
-
-        for (ExecutableSection executableSection : testCandidate.getTestSection().getExecutableSections()) {
-            executableSection.execute(restTestExecutionContext);
-        }
-
-        logger.info("end test [{}: {}]", testCandidate.getSuiteDescription(), testCandidate.getTestSection().getName());
-
-        logger.info("cleaning up after test [{}: {}]", testCandidate.getSuiteDescription(), testCandidate.getTestSection().getName());
-        tearDown();
-    }
-
-    private void tearDown() throws IOException, RestException {
-        wipeIndices();
-        wipeTemplates();
-        restTestExecutionContext.clear();
-    }
-
-    private void wipeIndices() throws IOException, RestException {
-        logger.debug("deleting all indices");
-        RestResponse restResponse = restTestExecutionContext.callApiInternal("indices.delete", "index", "_all");
-        assertThat(restResponse.getStatusCode(), equalTo(200));
-    }
-
-    @SuppressWarnings("unchecked")
-    public void wipeTemplates() throws IOException, RestException {
-        logger.debug("deleting all templates");
-        RestResponse restResponse = restTestExecutionContext.callApiInternal("indices.delete_template", "name", "*");
-        assertThat(restResponse.getStatusCode(), equalTo(200));
-    }
-
-    private void stopTestCluster() {
-        if (runMode == RunMode.TEST_CLUSTER) {
-            assert testCluster != null;
-            testCluster.afterTest();
-            testCluster.close();
-        }
-    }
-}
diff --git a/src/test/java/org/elasticsearch/test/rest/junit/RunAfter.java b/src/test/java/org/elasticsearch/test/rest/junit/RunAfter.java
deleted file mode 100644
index c7869359e8cb8..0000000000000
--- a/src/test/java/org/elasticsearch/test/rest/junit/RunAfter.java
+++ /dev/null
@@ -1,56 +0,0 @@
-/*
- * Licensed to Elasticsearch under one or more contributor
- * license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright
- * ownership. Elasticsearch licenses this file to you under
- * the Apache License, Version 2.0 (the "License"); you may
- * not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *    http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.elasticsearch.test.rest.junit;
-
-import com.google.common.collect.Lists;
-import org.junit.runners.model.MultipleFailureException;
-import org.junit.runners.model.Statement;
-
-import java.util.List;
-
-/**
- * {@link Statement} that allows to run a specific statement after another one
- */
-public class RunAfter extends Statement {
-
-    private final Statement next;
-    private final Statement after;
-
-    public RunAfter(Statement next, Statement after) {
-        this.next = next;
-        this.after = after;
-    }
-
-    @Override
-    public void evaluate() throws Throwable {
-        List<Throwable> errors = Lists.newArrayList();
-        try {
-            next.evaluate();
-        } catch (Throwable e) {
-            errors.add(e);
-        } finally {
-            try {
-                after.evaluate();
-            } catch (Throwable e) {
-                errors.add(e);
-            }
-        }
-        MultipleFailureException.assertEmpty(errors);
-    }
-}
diff --git a/src/test/java/org/elasticsearch/test/rest/parser/RestTestSectionParser.java b/src/test/java/org/elasticsearch/test/rest/parser/RestTestSectionParser.java
index 7ff06f477a59b..076361511193c 100644
--- a/src/test/java/org/elasticsearch/test/rest/parser/RestTestSectionParser.java
+++ b/src/test/java/org/elasticsearch/test/rest/parser/RestTestSectionParser.java
@@ -25,9 +25,6 @@
 
 /**
  * Parser for a complete test section
- *
- * Depending on the elasticsearch version the tests are going to run against, test sections might need to get skipped
- * In that case the relevant test sections parsing is entirely skipped
  */
 public class RestTestSectionParser implements RestTestFragmentParser<TestSection> {
 
@@ -39,21 +36,9 @@ public TestSection parse(RestTestSuiteParseContext parseContext) throws IOExcept
         parser.nextToken();
         testSection.setSkipSection(parseContext.parseSkipSection());
 
-        boolean skip = testSection.getSkipSection().skip(parseContext.getCurrentVersion());
-
         while ( parser.currentToken() != XContentParser.Token.END_ARRAY) {
-            if (skip) {
-                //if there was a skip section, there was a setup section as well, which means that we are sure
-                // the current token is at the beginning of a new object
-                assert parser.currentToken() == XContentParser.Token.START_OBJECT;
-                //we need to be at the beginning of an object to be able to skip children
-                parser.skipChildren();
-                //after skipChildren we are at the end of the skipped object, need to move on
-                parser.nextToken();
-            } else {
-                parseContext.advanceToFieldName();
-                testSection.addExecutableSection(parseContext.parseExecutableSection());
-            }
+            parseContext.advanceToFieldName();
+            testSection.addExecutableSection(parseContext.parseExecutableSection());
         }
 
         parser.nextToken();
diff --git a/src/test/java/org/elasticsearch/test/rest/parser/RestTestSuiteParseContext.java b/src/test/java/org/elasticsearch/test/rest/parser/RestTestSuiteParseContext.java
index 187ab5a1877c9..10110adf4b038 100644
--- a/src/test/java/org/elasticsearch/test/rest/parser/RestTestSuiteParseContext.java
+++ b/src/test/java/org/elasticsearch/test/rest/parser/RestTestSuiteParseContext.java
@@ -53,13 +53,11 @@ public class RestTestSuiteParseContext {
     private final String api;
     private final String suiteName;
     private final XContentParser parser;
-    private final String currentVersion;
 
-    public RestTestSuiteParseContext(String api, String suiteName, XContentParser parser, String currentVersion) {
+    public RestTestSuiteParseContext(String api, String suiteName, XContentParser parser) {
         this.api = api;
         this.suiteName = suiteName;
         this.parser = parser;
-        this.currentVersion = currentVersion;
     }
 
     public String getApi() {
@@ -74,10 +72,6 @@ public XContentParser parser() {
         return parser;
     }
 
-    public String getCurrentVersion() {
-        return currentVersion;
-    }
-
     public SetupSection parseSetupSection() throws IOException, RestTestParseException {
 
         advanceToFieldName();
diff --git a/src/test/java/org/elasticsearch/test/rest/parser/RestTestSuiteParser.java b/src/test/java/org/elasticsearch/test/rest/parser/RestTestSuiteParser.java
index efceed7f857c4..56a95871ac1dd 100644
--- a/src/test/java/org/elasticsearch/test/rest/parser/RestTestSuiteParser.java
+++ b/src/test/java/org/elasticsearch/test/rest/parser/RestTestSuiteParser.java
@@ -31,13 +31,10 @@
 
 /**
  * Parser for a complete test suite (yaml file)
- *
- * Depending on the elasticsearch version the tests are going to run against, a whole test suite might need to get skipped
- * In that case the relevant test sections parsing is entirely skipped
  */
 public class RestTestSuiteParser implements RestTestFragmentParser<RestTestSuite> {
 
-    public RestTestSuite parse(String currentVersion, String api, File file) throws IOException, RestTestParseException {
+    public RestTestSuite parse(String api, File file) throws IOException, RestTestParseException {
 
         if (!file.isFile()) {
             throw new IllegalArgumentException(file.getAbsolutePath() + " is not a file");
@@ -62,9 +59,14 @@ public RestTestSuite parse(String currentVersion, String api, File file) throws
             IOUtils.close(randomAccessFile);
         }
 
-        try (XContentParser parser = YamlXContent.yamlXContent.createParser(new FileInputStream(file))) {
-            RestTestSuiteParseContext testParseContext = new RestTestSuiteParseContext(api, filename, parser, currentVersion);
+        XContentParser parser = YamlXContent.yamlXContent.createParser(new FileInputStream(file));
+        try {
+            RestTestSuiteParseContext testParseContext = new RestTestSuiteParseContext(api, filename, parser);
             return parse(testParseContext);
+        } catch(Exception e) {
+            throw new RestTestParseException("Error parsing " + api + "/" + filename, e);
+        } finally {
+            parser.close();
         }
     }
 
@@ -79,8 +81,6 @@ public RestTestSuite parse(RestTestSuiteParseContext parseContext) throws IOExce
 
         restTestSuite.setSetupSection(parseContext.parseSetupSection());
 
-        boolean skip = restTestSuite.getSetupSection().getSkipSection().skip(parseContext.getCurrentVersion());
-
         while(true) {
             //the "---" section separator is not understood by the yaml parser. null is returned, same as when the parser is closed
             //we need to somehow distinguish between a null in the middle of a test ("---")
@@ -91,19 +91,9 @@ public RestTestSuite parse(RestTestSuiteParseContext parseContext) throws IOExce
                 }
             }
 
-            if (skip) {
-                //if there was a skip section, there was a setup section as well, which means that we are sure
-                // the current token is at the beginning of a new object
-                assert parser.currentToken() == XContentParser.Token.START_OBJECT;
-                //we need to be at the beginning of an object to be able to skip children
-                parser.skipChildren();
-                //after skipChildren we are at the end of the skipped object, need to move on
-                parser.nextToken();
-            } else {
-                TestSection testSection = parseContext.parseTestSection();
-                if (!restTestSuite.addTestSection(testSection)) {
-                    throw new RestTestParseException("duplicate test section [" + testSection.getName() + "] found in [" + restTestSuite.getDescription() + "]");
-                }
+            TestSection testSection = parseContext.parseTestSection();
+            if (!restTestSuite.addTestSection(testSection)) {
+                throw new RestTestParseException("duplicate test section [" + testSection.getName() + "] found in [" + restTestSuite.getDescription() + "]");
             }
         }
 
diff --git a/src/test/java/org/elasticsearch/test/rest/parser/SetupSectionParser.java b/src/test/java/org/elasticsearch/test/rest/parser/SetupSectionParser.java
index e03675552f961..2a2e39ea74490 100644
--- a/src/test/java/org/elasticsearch/test/rest/parser/SetupSectionParser.java
+++ b/src/test/java/org/elasticsearch/test/rest/parser/SetupSectionParser.java
@@ -36,27 +36,15 @@ public SetupSection parse(RestTestSuiteParseContext parseContext) throws IOExcep
         SetupSection setupSection = new SetupSection();
         setupSection.setSkipSection(parseContext.parseSkipSection());
 
-        boolean skip = setupSection.getSkipSection().skip(parseContext.getCurrentVersion());
-
         while (parser.currentToken() != XContentParser.Token.END_ARRAY) {
-            if (skip) {
-                //if there was a skip section, there was a setup section as well, which means that we are sure
-                // the current token is at the beginning of a new object
-                assert parser.currentToken() == XContentParser.Token.START_OBJECT;
-                //we need to be at the beginning of an object to be able to skip children
-                parser.skipChildren();
-                //after skipChildren we are at the end of the skipped object, need to move on
-                parser.nextToken();
-            } else {
-                parseContext.advanceToFieldName();
-                if (!"do".equals(parser.currentName())) {
-                    throw new RestTestParseException("section [" + parser.currentName() + "] not supported within setup section");
-                }
-
-                parser.nextToken();
-                setupSection.addDoSection(parseContext.parseDoSection());
-                parser.nextToken();
+            parseContext.advanceToFieldName();
+            if (!"do".equals(parser.currentName())) {
+                throw new RestTestParseException("section [" + parser.currentName() + "] not supported within setup section");
             }
+
+            parser.nextToken();
+            setupSection.addDoSection(parseContext.parseDoSection());
+            parser.nextToken();
         }
 
         parser.nextToken();
diff --git a/src/test/java/org/elasticsearch/test/rest/section/RestTestSuite.java b/src/test/java/org/elasticsearch/test/rest/section/RestTestSuite.java
index 5737e1eb221ff..e1abf099498f1 100644
--- a/src/test/java/org/elasticsearch/test/rest/section/RestTestSuite.java
+++ b/src/test/java/org/elasticsearch/test/rest/section/RestTestSuite.java
@@ -51,7 +51,7 @@ public String getName() {
     }
 
     //describes the rest test suite (e.g. index/10_with_id)
-    //useful also to reproduce failures (RestReproduceInfoPrinter)
+    //useful also to reproduce failures
     public String getDescription() {
         return api + "/" + name;
     }
diff --git a/src/test/java/org/elasticsearch/test/rest/test/AssertionParsersTests.java b/src/test/java/org/elasticsearch/test/rest/test/AssertionParsersTests.java
index 9d9ba78000cc5..ac1090e01d422 100644
--- a/src/test/java/org/elasticsearch/test/rest/test/AssertionParsersTests.java
+++ b/src/test/java/org/elasticsearch/test/rest/test/AssertionParsersTests.java
@@ -37,7 +37,7 @@ public void testParseIsTrue() throws Exception {
         );
 
         IsTrueParser isTrueParser = new IsTrueParser();
-        IsTrueAssertion trueAssertion = isTrueParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        IsTrueAssertion trueAssertion = isTrueParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(trueAssertion, notNullValue());
         assertThat(trueAssertion.getField(), equalTo("get.fields._timestamp"));
@@ -50,7 +50,7 @@ public void testParseIsFalse() throws Exception {
         );
 
         IsFalseParser isFalseParser = new IsFalseParser();
-        IsFalseAssertion falseAssertion = isFalseParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        IsFalseAssertion falseAssertion = isFalseParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(falseAssertion, notNullValue());
         assertThat(falseAssertion.getField(), equalTo("docs.1._source"));
@@ -63,7 +63,7 @@ public void testParseGreaterThan() throws Exception {
         );
 
         GreaterThanParser greaterThanParser = new GreaterThanParser();
-        GreaterThanAssertion greaterThanAssertion = greaterThanParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        GreaterThanAssertion greaterThanAssertion = greaterThanParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         assertThat(greaterThanAssertion, notNullValue());
         assertThat(greaterThanAssertion.getField(), equalTo("field"));
         assertThat(greaterThanAssertion.getExpectedValue(), instanceOf(Integer.class));
@@ -77,7 +77,7 @@ public void testParseLessThan() throws Exception {
         );
 
         LessThanParser lessThanParser = new LessThanParser();
-        LessThanAssertion lessThanAssertion = lessThanParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        LessThanAssertion lessThanAssertion = lessThanParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         assertThat(lessThanAssertion, notNullValue());
         assertThat(lessThanAssertion.getField(), equalTo("field"));
         assertThat(lessThanAssertion.getExpectedValue(), instanceOf(Integer.class));
@@ -91,7 +91,7 @@ public void testParseLength() throws Exception {
         );
 
         LengthParser lengthParser = new LengthParser();
-        LengthAssertion lengthAssertion = lengthParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        LengthAssertion lengthAssertion = lengthParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         assertThat(lengthAssertion, notNullValue());
         assertThat(lengthAssertion.getField(), equalTo("_id"));
         assertThat(lengthAssertion.getExpectedValue(), instanceOf(Integer.class));
@@ -106,7 +106,7 @@ public void testParseMatchSimpleIntegerValue() throws Exception {
         );
 
         MatchParser matchParser = new MatchParser();
-        MatchAssertion matchAssertion = matchParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        MatchAssertion matchAssertion = matchParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(matchAssertion, notNullValue());
         assertThat(matchAssertion.getField(), equalTo("field"));
@@ -122,7 +122,7 @@ public void testParseMatchSimpleStringValue() throws Exception {
         );
 
         MatchParser matchParser = new MatchParser();
-        MatchAssertion matchAssertion = matchParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        MatchAssertion matchAssertion = matchParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(matchAssertion, notNullValue());
         assertThat(matchAssertion.getField(), equalTo("foo"));
@@ -138,7 +138,7 @@ public void testParseMatchArray() throws Exception {
         );
 
         MatchParser matchParser = new MatchParser();
-        MatchAssertion matchAssertion = matchParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        MatchAssertion matchAssertion = matchParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(matchAssertion, notNullValue());
         assertThat(matchAssertion.getField(), equalTo("matches"));
@@ -157,7 +157,7 @@ public void testParseMatchSourceValues() throws Exception {
         );
 
         MatchParser matchParser = new MatchParser();
-        MatchAssertion matchAssertion = matchParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        MatchAssertion matchAssertion = matchParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(matchAssertion, notNullValue());
         assertThat(matchAssertion.getField(), equalTo("_source"));
diff --git a/src/test/java/org/elasticsearch/test/rest/test/DoSectionParserTests.java b/src/test/java/org/elasticsearch/test/rest/test/DoSectionParserTests.java
index e580f0f4e3dc6..911bb8999baff 100644
--- a/src/test/java/org/elasticsearch/test/rest/test/DoSectionParserTests.java
+++ b/src/test/java/org/elasticsearch/test/rest/test/DoSectionParserTests.java
@@ -47,7 +47,7 @@ public void testParseDoSectionNoBody() throws Exception {
         );
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         ApiCallSection apiCallSection = doSection.getApiCallSection();
 
         assertThat(apiCallSection, notNullValue());
@@ -66,7 +66,7 @@ public void testParseDoSectionNoParamsNoBody() throws Exception {
         );
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         ApiCallSection apiCallSection = doSection.getApiCallSection();
 
         assertThat(apiCallSection, notNullValue());
@@ -87,7 +87,7 @@ public void testParseDoSectionWithJsonBody() throws Exception {
         );
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         ApiCallSection apiCallSection = doSection.getApiCallSection();
 
         assertThat(apiCallSection, notNullValue());
@@ -120,7 +120,7 @@ public void testParseDoSectionWithJsonMultipleBodiesAsLongString() throws Except
         );
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         ApiCallSection apiCallSection = doSection.getApiCallSection();
 
         assertThat(apiCallSection, notNullValue());
@@ -147,7 +147,7 @@ public void testParseDoSectionWithJsonMultipleBodiesRepeatedProperty() throws Ex
         );
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         ApiCallSection apiCallSection = doSection.getApiCallSection();
 
         assertThat(apiCallSection, notNullValue());
@@ -172,7 +172,7 @@ public void testParseDoSectionWithYamlBody() throws Exception {
         String body = "{ \"_source\": [ \"include.field1\", \"include.field2\" ], \"query\": { \"match_all\": {} }}";
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         ApiCallSection apiCallSection = doSection.getApiCallSection();
 
         assertThat(apiCallSection, notNullValue());
@@ -209,7 +209,7 @@ public void testParseDoSectionWithYamlMultipleBodies() throws Exception {
         bodies[3] = "{ \"f1\":\"v2\", \"f2\": 47 }";
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         ApiCallSection apiCallSection = doSection.getApiCallSection();
 
         assertThat(apiCallSection, notNullValue());
@@ -243,7 +243,7 @@ public void testParseDoSectionWithYamlMultipleBodiesRepeatedProperty() throws Ex
         bodies[1] = "{ \"f1\":\"v1\", \"f2\": 42 }";
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         ApiCallSection apiCallSection = doSection.getApiCallSection();
 
         assertThat(apiCallSection, notNullValue());
@@ -273,7 +273,7 @@ public void testParseDoSectionWithYamlBodyMultiGet() throws Exception {
                 "]}";
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         ApiCallSection apiCallSection = doSection.getApiCallSection();
 
         assertThat(apiCallSection, notNullValue());
@@ -295,7 +295,7 @@ public void testParseDoSectionWithBodyStringified() throws Exception {
         );
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         ApiCallSection apiCallSection = doSection.getApiCallSection();
 
         assertThat(apiCallSection, notNullValue());
@@ -322,7 +322,7 @@ public void testParseDoSectionWithBodiesStringifiedAndNot() throws Exception {
         String body = "{ \"size\": 100, \"query\": { \"match_all\": {} } }";
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         ApiCallSection apiCallSection = doSection.getApiCallSection();
 
         assertThat(apiCallSection.getApi(), equalTo("index"));
@@ -344,7 +344,7 @@ public void testParseDoSectionWithCatch() throws Exception {
         );
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(doSection.getCatch(), equalTo("missing"));
         assertThat(doSection.getApiCallSection(), notNullValue());
@@ -360,7 +360,7 @@ public void testParseDoSectionWithoutClientCallSection() throws Exception {
         );
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
     }
 
     @Test
@@ -373,7 +373,7 @@ public void testParseDoSectionMultivaluedField() throws Exception {
         );
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(doSection.getCatch(), nullValue());
         assertThat(doSection.getApiCallSection(), notNullValue());
diff --git a/src/test/java/org/elasticsearch/test/rest/test/RestTestParserTests.java b/src/test/java/org/elasticsearch/test/rest/test/RestTestParserTests.java
index 9a06328097eed..88af4c6de2b68 100644
--- a/src/test/java/org/elasticsearch/test/rest/test/RestTestParserTests.java
+++ b/src/test/java/org/elasticsearch/test/rest/test/RestTestParserTests.java
@@ -84,7 +84,7 @@ public void testParseTestSetupAndSections() throws Exception {
         );
 
         RestTestSuiteParser testParser = new RestTestSuiteParser();
-        RestTestSuite restTestSuite = testParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.5"));
+        RestTestSuite restTestSuite = testParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(restTestSuite, notNullValue());
         assertThat(restTestSuite.getName(), equalTo("suite"));
@@ -136,179 +136,6 @@ public void testParseTestSetupAndSections() throws Exception {
         assertThat(matchAssertion.getExpectedValue().toString(), equalTo("whitespace"));
     }
 
-    @Test
-    public void testParseTestSetupAndSectionsSkipLastSection() throws Exception {
-        parser = YamlXContent.yamlXContent.createParser(
-                "setup:\n" +
-                        "  - do:\n" +
-                        "        indices.create:\n" +
-                        "          index: test_index\n" +
-                        "\n" +
-                        "---\n" +
-                        "\"Get index mapping\":\n" +
-                        "  - do:\n" +
-                        "      indices.get_mapping:\n" +
-                        "        index: test_index\n" +
-                        "\n" +
-                        "  - match: {test_index.test_type.properties.text.type:     string}\n" +
-                        "  - match: {test_index.test_type.properties.text.analyzer: whitespace}\n" +
-                        "\n" +
-                        "---\n" +
-                        "\"Get type mapping - pre 1.0\":\n" +
-                        "\n" +
-                        "  - skip:\n" +
-                        "      version:     \"0.90.9 - 999\"\n" +
-                        "      reason:      \"for newer versions the index name is always returned\"\n" +
-                        "\n" +
-                        "  - do:\n" +
-                        "      indices.get_mapping:\n" +
-                        "        index: test_index\n" +
-                        "        type: test_type\n" +
-                        "\n" +
-                        "  - match: {test_type.properties.text.type:     string}\n" +
-                        "  - match: {test_type.properties.text.analyzer: whitespace}\n"
-        );
-
-        RestTestSuiteParser testParser = new RestTestSuiteParser();
-        RestTestSuite restTestSuite = testParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "1.0.0"));
-
-        assertThat(restTestSuite, notNullValue());
-        assertThat(restTestSuite.getName(), equalTo("suite"));
-        assertThat(restTestSuite.getSetupSection(), notNullValue());
-        assertThat(restTestSuite.getSetupSection().getSkipSection().isEmpty(), equalTo(true));
-
-        assertThat(restTestSuite.getSetupSection().getDoSections().size(), equalTo(1));
-        assertThat(restTestSuite.getSetupSection().getDoSections().get(0).getApiCallSection().getApi(), equalTo("indices.create"));
-        assertThat(restTestSuite.getSetupSection().getDoSections().get(0).getApiCallSection().getParams().size(), equalTo(1));
-        assertThat(restTestSuite.getSetupSection().getDoSections().get(0).getApiCallSection().getParams().get("index"), equalTo("test_index"));
-
-        assertThat(restTestSuite.getTestSections().size(), equalTo(2));
-
-        assertThat(restTestSuite.getTestSections().get(0).getName(), equalTo("Get index mapping"));
-        assertThat(restTestSuite.getTestSections().get(0).getSkipSection().isEmpty(), equalTo(true));
-        assertThat(restTestSuite.getTestSections().get(0).getExecutableSections().size(), equalTo(3));
-        assertThat(restTestSuite.getTestSections().get(0).getExecutableSections().get(0), instanceOf(DoSection.class));
-        DoSection doSection = (DoSection) restTestSuite.getTestSections().get(0).getExecutableSections().get(0);
-        assertThat(doSection.getApiCallSection().getApi(), equalTo("indices.get_mapping"));
-        assertThat(doSection.getApiCallSection().getParams().size(), equalTo(1));
-        assertThat(doSection.getApiCallSection().getParams().get("index"), equalTo("test_index"));
-        assertThat(restTestSuite.getTestSections().get(0).getExecutableSections().get(1), instanceOf(MatchAssertion.class));
-        MatchAssertion matchAssertion = (MatchAssertion) restTestSuite.getTestSections().get(0).getExecutableSections().get(1);
-        assertThat(matchAssertion.getField(), equalTo("test_index.test_type.properties.text.type"));
-        assertThat(matchAssertion.getExpectedValue().toString(), equalTo("string"));
-        assertThat(restTestSuite.getTestSections().get(0).getExecutableSections().get(2), instanceOf(MatchAssertion.class));
-        matchAssertion = (MatchAssertion) restTestSuite.getTestSections().get(0).getExecutableSections().get(2);
-        assertThat(matchAssertion.getField(), equalTo("test_index.test_type.properties.text.analyzer"));
-        assertThat(matchAssertion.getExpectedValue().toString(), equalTo("whitespace"));
-
-        assertThat(restTestSuite.getTestSections().get(1).getName(), equalTo("Get type mapping - pre 1.0"));
-        assertThat(restTestSuite.getTestSections().get(1).getSkipSection().isEmpty(), equalTo(false));
-        assertThat(restTestSuite.getTestSections().get(1).getSkipSection().getReason(), equalTo("for newer versions the index name is always returned"));
-        assertThat(restTestSuite.getTestSections().get(1).getSkipSection().getVersion(), equalTo("0.90.9 - 999"));
-        assertThat(restTestSuite.getTestSections().get(1).getExecutableSections().size(), equalTo(0));
-    }
-
-    @Test
-    public void testParseTestSetupAndSectionsSkipEntireFile() throws Exception {
-        parser = YamlXContent.yamlXContent.createParser(
-                "setup:\n" +
-                        "  - skip:\n" +
-                        "      version:     \"0.90.3 - 0.90.6\"\n" +
-                        "      reason:      \"test skip entire file\"\n" +
-                        "  - do:\n" +
-                        "        indices.create:\n" +
-                        "          index: test_index\n" +
-                        "\n" +
-                        "---\n" +
-                        "\"Get index mapping\":\n" +
-                        "  - do:\n" +
-                        "      indices.get_mapping:\n" +
-                        "        index: test_index\n" +
-                        "\n" +
-                        "  - match: {test_index.test_type.properties.text.type:     string}\n" +
-                        "  - match: {test_index.test_type.properties.text.analyzer: whitespace}\n" +
-                        "\n" +
-                        "---\n" +
-                        "\"Get type mapping - pre 1.0\":\n" +
-                        "\n" +
-                        "  - skip:\n" +
-                        "      version:     \"0.90.9 - 999\"\n" +
-                        "      reason:      \"for newer versions the index name is always returned\"\n" +
-                        "\n" +
-                        "  - do:\n" +
-                        "      indices.get_mapping:\n" +
-                        "        index: test_index\n" +
-                        "        type: test_type\n" +
-                        "\n" +
-                        "  - match: {test_type.properties.text.type:     string}\n" +
-                        "  - match: {test_type.properties.text.analyzer: whitespace}\n"
-        );
-
-        RestTestSuiteParser testParser = new RestTestSuiteParser();
-        RestTestSuite restTestSuite = testParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.5"));
-
-        assertThat(restTestSuite, notNullValue());
-        assertThat(restTestSuite.getName(), equalTo("suite"));
-        assertThat(restTestSuite.getSetupSection(), notNullValue());
-
-        assertThat(restTestSuite.getSetupSection().getSkipSection().isEmpty(), equalTo(false));
-        assertThat(restTestSuite.getSetupSection().getSkipSection().getVersion(), equalTo("0.90.3 - 0.90.6"));
-        assertThat(restTestSuite.getSetupSection().getSkipSection().getReason(), equalTo("test skip entire file"));
-
-        assertThat(restTestSuite.getSetupSection().getDoSections().size(), equalTo(0));
-
-        assertThat(restTestSuite.getTestSections().size(), equalTo(0));
-    }
-
-    @Test
-    public void testParseTestSetupAndSectionsSkipEntireFileNoDo() throws Exception {
-        parser = YamlXContent.yamlXContent.createParser(
-                "setup:\n" +
-                        "  - skip:\n" +
-                        "      version:     \"0.90.3 - 0.90.6\"\n" +
-                        "      reason:      \"test skip entire file\"\n" +
-                        "\n" +
-                        "---\n" +
-                        "\"Get index mapping\":\n" +
-                        "  - do:\n" +
-                        "      indices.get_mapping:\n" +
-                        "        index: test_index\n" +
-                        "\n" +
-                        "  - match: {test_index.test_type.properties.text.type:     string}\n" +
-                        "  - match: {test_index.test_type.properties.text.analyzer: whitespace}\n" +
-                        "\n" +
-                        "---\n" +
-                        "\"Get type mapping - pre 1.0\":\n" +
-                        "\n" +
-                        "  - skip:\n" +
-                        "      version:     \"0.90.9 - 999\"\n" +
-                        "      reason:      \"for newer versions the index name is always returned\"\n" +
-                        "\n" +
-                        "  - do:\n" +
-                        "      indices.get_mapping:\n" +
-                        "        index: test_index\n" +
-                        "        type: test_type\n" +
-                        "\n" +
-                        "  - match: {test_type.properties.text.type:     string}\n" +
-                        "  - match: {test_type.properties.text.analyzer: whitespace}\n"
-        );
-
-        RestTestSuiteParser testParser = new RestTestSuiteParser();
-        RestTestSuite restTestSuite = testParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.5"));
-
-        assertThat(restTestSuite, notNullValue());
-        assertThat(restTestSuite.getName(), equalTo("suite"));
-        assertThat(restTestSuite.getSetupSection(), notNullValue());
-
-        assertThat(restTestSuite.getSetupSection().getSkipSection().isEmpty(), equalTo(false));
-        assertThat(restTestSuite.getSetupSection().getSkipSection().getVersion(), equalTo("0.90.3 - 0.90.6"));
-        assertThat(restTestSuite.getSetupSection().getSkipSection().getReason(), equalTo("test skip entire file"));
-
-        assertThat(restTestSuite.getSetupSection().getDoSections().size(), equalTo(0));
-
-        assertThat(restTestSuite.getTestSections().size(), equalTo(0));
-    }
-
     @Test
     public void testParseTestSingleTestSection() throws Exception {
         parser = YamlXContent.yamlXContent.createParser(
@@ -342,7 +169,7 @@ public void testParseTestSingleTestSection() throws Exception {
         );
 
         RestTestSuiteParser testParser = new RestTestSuiteParser();
-        RestTestSuite restTestSuite = testParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.5"));
+        RestTestSuite restTestSuite = testParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(restTestSuite, notNullValue());
         assertThat(restTestSuite.getName(), equalTo("suite"));
@@ -456,7 +283,7 @@ public void testParseTestMultipleTestSections() throws Exception {
         );
 
         RestTestSuiteParser testParser = new RestTestSuiteParser();
-        RestTestSuite restTestSuite = testParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.5"));
+        RestTestSuite restTestSuite = testParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(restTestSuite, notNullValue());
         assertThat(restTestSuite.getName(), equalTo("suite"));
@@ -531,6 +358,6 @@ public void testParseTestDuplicateTestSections() throws Exception {
         );
 
         RestTestSuiteParser testParser = new RestTestSuiteParser();
-        testParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.5"));
+        testParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
     }
 }
diff --git a/src/test/java/org/elasticsearch/test/rest/test/SetSectionParserTests.java b/src/test/java/org/elasticsearch/test/rest/test/SetSectionParserTests.java
index fb9d7b930c248..7e2724a167127 100644
--- a/src/test/java/org/elasticsearch/test/rest/test/SetSectionParserTests.java
+++ b/src/test/java/org/elasticsearch/test/rest/test/SetSectionParserTests.java
@@ -38,7 +38,7 @@ public void testParseSetSectionSingleValue() throws Exception {
 
         SetSectionParser setSectionParser = new SetSectionParser();
 
-        SetSection setSection = setSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        SetSection setSection = setSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(setSection, notNullValue());
         assertThat(setSection.getStash(), notNullValue());
@@ -54,7 +54,7 @@ public void testParseSetSectionMultipleValues() throws Exception {
 
         SetSectionParser setSectionParser = new SetSectionParser();
 
-        SetSection setSection = setSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        SetSection setSection = setSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(setSection, notNullValue());
         assertThat(setSection.getStash(), notNullValue());
@@ -72,6 +72,6 @@ public void testParseSetSectionNoValues() throws Exception {
 
         SetSectionParser setSectionParser = new SetSectionParser();
 
-        setSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        setSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
     }
 }
\ No newline at end of file
diff --git a/src/test/java/org/elasticsearch/test/rest/test/SetupSectionParserTests.java b/src/test/java/org/elasticsearch/test/rest/test/SetupSectionParserTests.java
index 191bc3d5fd3bc..2c0d351caca46 100644
--- a/src/test/java/org/elasticsearch/test/rest/test/SetupSectionParserTests.java
+++ b/src/test/java/org/elasticsearch/test/rest/test/SetupSectionParserTests.java
@@ -48,7 +48,7 @@ public void testParseSetupSection() throws Exception {
         );
 
         SetupSectionParser setupSectionParser = new SetupSectionParser();
-        SetupSection setupSection = setupSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        SetupSection setupSection = setupSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(setupSection, notNullValue());
         assertThat(setupSection.getSkipSection().isEmpty(), equalTo(true));
@@ -57,38 +57,6 @@ public void testParseSetupSection() throws Exception {
         assertThat(setupSection.getDoSections().get(1).getApiCallSection().getApi(), equalTo("index2"));
     }
 
-    @Test
-    public void testParseSetupAndSkipSectionSkip() throws Exception {
-
-        parser = YamlXContent.yamlXContent.createParser(
-                "  - skip:\n" +
-                "      version:  \"0.90.0 - 0.90.7\"\n" +
-                "      reason:   \"Update doesn't return metadata fields, waiting for #3259\"\n" +
-                "  - do:\n" +
-                "      index1:\n" +
-                "        index:  test_1\n" +
-                "        type:   test\n" +
-                "        id:     1\n" +
-                "        body:   { \"include\": { \"field1\": \"v1\", \"field2\": \"v2\" }, \"count\": 1 }\n" +
-                "  - do:\n" +
-                "      index2:\n" +
-                "        index:  test_1\n" +
-                "        type:   test\n" +
-                "        id:     2\n" +
-                "        body:   { \"include\": { \"field1\": \"v1\", \"field2\": \"v2\" }, \"count\": 1 }\n"
-        );
-
-        SetupSectionParser setupSectionParser = new SetupSectionParser();
-        SetupSection setupSection = setupSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.5"));
-
-        assertThat(setupSection, notNullValue());
-        assertThat(setupSection.getSkipSection().isEmpty(), equalTo(false));
-        assertThat(setupSection.getSkipSection(), notNullValue());
-        assertThat(setupSection.getSkipSection().getVersion(), equalTo("0.90.0 - 0.90.7"));
-        assertThat(setupSection.getSkipSection().getReason(), equalTo("Update doesn't return metadata fields, waiting for #3259"));
-        assertThat(setupSection.getDoSections().size(), equalTo(0));
-    }
-
     @Test
     public void testParseSetupAndSkipSectionNoSkip() throws Exception {
 
@@ -111,7 +79,7 @@ public void testParseSetupAndSkipSectionNoSkip() throws Exception {
         );
 
         SetupSectionParser setupSectionParser = new SetupSectionParser();
-        SetupSection setupSection = setupSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.8"));
+        SetupSection setupSection = setupSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(setupSection, notNullValue());
         assertThat(setupSection.getSkipSection().isEmpty(), equalTo(false));
diff --git a/src/test/java/org/elasticsearch/test/rest/test/SkipSectionParserTests.java b/src/test/java/org/elasticsearch/test/rest/test/SkipSectionParserTests.java
index 4209e90329e93..899ea1da04ced 100644
--- a/src/test/java/org/elasticsearch/test/rest/test/SkipSectionParserTests.java
+++ b/src/test/java/org/elasticsearch/test/rest/test/SkipSectionParserTests.java
@@ -38,7 +38,7 @@ public void testParseSkipSectionVersionNoFeature() throws Exception {
 
         SkipSectionParser skipSectionParser = new SkipSectionParser();
 
-        SkipSection skipSection = skipSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        SkipSection skipSection = skipSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(skipSection, notNullValue());
         assertThat(skipSection.getVersion(), equalTo("0 - 0.90.2"));
@@ -54,7 +54,7 @@ public void testParseSkipSectionFeatureNoVersion() throws Exception {
 
         SkipSectionParser skipSectionParser = new SkipSectionParser();
 
-        SkipSection skipSection = skipSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        SkipSection skipSection = skipSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(skipSection, notNullValue());
         assertThat(skipSection.getVersion(), nullValue());
@@ -71,7 +71,7 @@ public void testParseSkipSectionFeaturesNoVersion() throws Exception {
 
         SkipSectionParser skipSectionParser = new SkipSectionParser();
 
-        SkipSection skipSection = skipSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        SkipSection skipSection = skipSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(skipSection, notNullValue());
         assertThat(skipSection.getVersion(), nullValue());
@@ -92,7 +92,7 @@ public void testParseSkipSectionBothFeatureAndVersion() throws Exception {
 
         SkipSectionParser skipSectionParser = new SkipSectionParser();
 
-        skipSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        skipSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
     }
 
     @Test(expected = RestTestParseException.class)
@@ -102,7 +102,7 @@ public void testParseSkipSectionNoReason() throws Exception {
         );
 
         SkipSectionParser skipSectionParser = new SkipSectionParser();
-        skipSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        skipSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
     }
 
     @Test(expected = RestTestParseException.class)
@@ -112,6 +112,6 @@ public void testParseSkipSectionNoVersionNorFeature() throws Exception {
         );
 
         SkipSectionParser skipSectionParser = new SkipSectionParser();
-        skipSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        skipSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
     }
 }
\ No newline at end of file
diff --git a/src/test/java/org/elasticsearch/test/rest/test/TestSectionParserTests.java b/src/test/java/org/elasticsearch/test/rest/test/TestSectionParserTests.java
index f98f67f9d4d65..3ef2f23450282 100644
--- a/src/test/java/org/elasticsearch/test/rest/test/TestSectionParserTests.java
+++ b/src/test/java/org/elasticsearch/test/rest/test/TestSectionParserTests.java
@@ -42,7 +42,7 @@ public void testParseTestSectionWithDoSection() throws Exception {
         );
 
         RestTestSectionParser testSectionParser = new RestTestSectionParser();
-        TestSection testSection = testSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        TestSection testSection = testSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(testSection, notNullValue());
         assertThat(testSection.getName(), equalTo("First test section"));
@@ -56,34 +56,6 @@ public void testParseTestSectionWithDoSection() throws Exception {
         assertThat(doSection.getApiCallSection().hasBody(), equalTo(false));
     }
 
-    @Test
-    public void testParseTestSectionWithDoSetAndSkipSectionsSkip() throws Exception {
-        String yaml =
-                "\"First test section\": \n" +
-                "  - skip:\n" +
-                "      version:  \"0.90.0 - 0.90.7\"\n" +
-                "      reason:   \"Update doesn't return metadata fields, waiting for #3259\"\n" +
-                "  - do :\n" +
-                "      catch: missing\n" +
-                "      indices.get_warmer:\n" +
-                "          index: test_index\n" +
-                "          name: test_warmer\n" +
-                "  - set: {_scroll_id: scroll_id}";
-
-
-        RestTestSectionParser testSectionParser = new RestTestSectionParser();
-        parser = YamlXContent.yamlXContent.createParser(yaml);
-        TestSection testSection = testSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
-
-        assertThat(testSection, notNullValue());
-        assertThat(testSection.getName(), equalTo("First test section"));
-        assertThat(testSection.getSkipSection(), notNullValue());
-        assertThat(testSection.getSkipSection().getVersion(), equalTo("0.90.0 - 0.90.7"));
-        assertThat(testSection.getSkipSection().getReason(), equalTo("Update doesn't return metadata fields, waiting for #3259"));
-        //skip parsing when needed
-        assertThat(testSection.getExecutableSections().size(), equalTo(0));
-    }
-
     @Test
     public void testParseTestSectionWithDoSetAndSkipSectionsNoSkip() throws Exception {
         String yaml =
@@ -101,7 +73,7 @@ public void testParseTestSectionWithDoSetAndSkipSectionsNoSkip() throws Exceptio
 
         RestTestSectionParser testSectionParser = new RestTestSectionParser();
         parser = YamlXContent.yamlXContent.createParser(yaml);
-        TestSection testSection = testSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.8"));
+        TestSection testSection = testSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(testSection, notNullValue());
         assertThat(testSection.getName(), equalTo("First test section"));
@@ -139,7 +111,7 @@ public void testParseTestSectionWithMultipleDoSections() throws Exception {
         );
 
         RestTestSectionParser testSectionParser = new RestTestSectionParser();
-        TestSection testSection = testSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.5"));
+        TestSection testSection = testSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(testSection, notNullValue());
         assertThat(testSection.getName(), equalTo("Basic"));
@@ -193,7 +165,7 @@ public void testParseTestSectionWithDoSectionsAndAssertions() throws Exception {
         );
 
         RestTestSectionParser testSectionParser = new RestTestSectionParser();
-        TestSection testSection = testSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.5"));
+        TestSection testSection = testSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(testSection, notNullValue());
         assertThat(testSection.getName(), equalTo("Basic"));
@@ -265,7 +237,7 @@ public void testSmallSection() throws Exception {
                 "  - is_true: nodes\n" +
                 "  - is_true: cluster_name\n");
         RestTestSectionParser testSectionParser = new RestTestSectionParser();
-        TestSection testSection = testSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.5"));
+        TestSection testSection = testSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         assertThat(testSection, notNullValue());
         assertThat(testSection.getName(), equalTo("node_info test"));
         assertThat(testSection.getExecutableSections().size(), equalTo(3));

From f1a8aadb635061d26f687be9fe642166e6cd7647 Mon Sep 17 00:00:00 2001
From: Simon Willnauer <simonw@apache.org>
Date: Mon, 7 Apr 2014 18:17:25 +0200
Subject: [PATCH 24/54] [TEST] null out static resources in base test classes

---
 .../org/elasticsearch/test/rest/ElasticsearchRestTests.java | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/src/test/java/org/elasticsearch/test/rest/ElasticsearchRestTests.java b/src/test/java/org/elasticsearch/test/rest/ElasticsearchRestTests.java
index e68039821dd58..8d3dee1e4fda0 100644
--- a/src/test/java/org/elasticsearch/test/rest/ElasticsearchRestTests.java
+++ b/src/test/java/org/elasticsearch/test/rest/ElasticsearchRestTests.java
@@ -19,14 +19,10 @@
 
 package org.elasticsearch.test.rest;
 
-import com.carrotsearch.randomizedtesting.SysGlobals;
 import com.carrotsearch.randomizedtesting.annotations.Name;
 import com.carrotsearch.randomizedtesting.annotations.ParametersFactory;
-import com.carrotsearch.randomizedtesting.annotations.ReplicateOnEachVm;
 import com.google.common.collect.Lists;
-import org.elasticsearch.cluster.routing.operation.hash.djb.DjbHashFunction;
 import org.elasticsearch.common.Strings;
-import org.elasticsearch.common.math.MathUtils;
 import org.elasticsearch.test.ElasticsearchIntegrationTest;
 import org.elasticsearch.test.rest.client.RestException;
 import org.elasticsearch.test.rest.parser.RestTestParseException;
@@ -135,12 +131,14 @@ public static void initExecutionContext() throws IOException, RestException {
 
         String[] specPaths = resolvePathsProperty(REST_TESTS_SPEC, DEFAULT_SPEC_PATH);
         RestSpec restSpec = RestSpec.parseFrom(DEFAULT_SPEC_PATH, specPaths);
+        assert restTestExecutionContext == null;
         restTestExecutionContext = new RestTestExecutionContext(restSpec);
     }
 
     @AfterClass
     public static void close() {
         restTestExecutionContext.close();
+        restTestExecutionContext = null;
     }
 
     @Before

From 9a2dba2b8ecf226d22a9d2cd68b162ac56261630 Mon Sep 17 00:00:00 2001
From: Sean Gallagher <sean.gallagher@elasticsearch.com>
Date: Mon, 7 Apr 2014 12:43:10 -0400
Subject: [PATCH 25/54] Fixed upgrade.asciidoc typo and incorrect usage.
 Author: Sean Gallagher Date: 4 Apr 2014 Closes 5651

---
 docs/reference/setup/upgrade.asciidoc | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/docs/reference/setup/upgrade.asciidoc b/docs/reference/setup/upgrade.asciidoc
index b52d8eeab215d..f3bf3df193d82 100644
--- a/docs/reference/setup/upgrade.asciidoc
+++ b/docs/reference/setup/upgrade.asciidoc
@@ -119,11 +119,12 @@ This syntax is from versions prior to 1.0:
 
 * When the cluster is up and running, shard reallocation can be enabled.
 
-This synax is from release 1.0 and later:
+This syntax is from release 1.0 and later:
 [source,sh]
 ------------------------------------------------------
 	curl -XPUT localhost:9200/_cluster/settings -d '{
      		"persistent" : {
+		"cluster.routing.allocation.disable_allocation": false,
          	"cluster.routing.allocation.enable" : "all"
      		}
  	}'

From f283a9f3a5d026fd7930f0e63e049d8c479ef2f7 Mon Sep 17 00:00:00 2001
From: javanna <cavannaluca@gmail.com>
Date: Mon, 7 Apr 2014 18:59:25 +0200
Subject: [PATCH 26/54] [TEST] specified number_of_shards 5 to make sure the
 two docs end up in different shards that can be refreshed separately

---
 rest-api-spec/test/delete/50_refresh.yaml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/rest-api-spec/test/delete/50_refresh.yaml b/rest-api-spec/test/delete/50_refresh.yaml
index b550789e2cfaf..ef7a3045514e8 100644
--- a/rest-api-spec/test/delete/50_refresh.yaml
+++ b/rest-api-spec/test/delete/50_refresh.yaml
@@ -7,6 +7,7 @@
           body:
               settings:
                   refresh_interval: -1
+                  number_of_shards: 5
                   number_of_replicas: 0
  - do:
       cluster.health:

From fb338ef753fa2afb6e0de9bcd47dc704a53091e3 Mon Sep 17 00:00:00 2001
From: Andrew Selden <andrew.selden@elasticsearch.com>
Date: Mon, 7 Apr 2014 10:38:57 -0700
Subject: [PATCH 27/54] Make writePrimitive*() and readPrimitive*() methods
 public.

These utility methods are useful for client code to read/write arrays of
primitive types.
---
 .../org/elasticsearch/common/io/stream/StreamInput.java   | 8 ++++----
 .../org/elasticsearch/common/io/stream/StreamOutput.java  | 8 ++++----
 2 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/src/main/java/org/elasticsearch/common/io/stream/StreamInput.java b/src/main/java/org/elasticsearch/common/io/stream/StreamInput.java
index aa01ae96fb126..776e85f2f3c8e 100644
--- a/src/main/java/org/elasticsearch/common/io/stream/StreamInput.java
+++ b/src/main/java/org/elasticsearch/common/io/stream/StreamInput.java
@@ -446,7 +446,7 @@ public Object readGenericValue() throws IOException {
         }
     }
 
-    private Object readPrimitiveIntArray() throws IOException {
+    public int[] readPrimitiveIntArray() throws IOException {
         int length = readVInt();
         int[] values = new int[length];
         for(int i=0; i<length; i++) {
@@ -455,7 +455,7 @@ private Object readPrimitiveIntArray() throws IOException {
         return values;
     }
     
-    private Object readPrimitiveLongArray() throws IOException {
+    public long[] readPrimitiveLongArray() throws IOException {
         int length = readVInt();
         long[] values = new long[length];
         for(int i=0; i<length; i++) {
@@ -464,7 +464,7 @@ private Object readPrimitiveLongArray() throws IOException {
         return values;
     }
     
-    private Object readPrimitiveFloatArray() throws IOException {
+    public float[] readPrimitiveFloatArray() throws IOException {
         int length = readVInt();
         float[] values = new float[length];
         for(int i=0; i<length; i++) {
@@ -473,7 +473,7 @@ private Object readPrimitiveFloatArray() throws IOException {
         return values;
     }
     
-    private Object readPrimitiveDoubleArray() throws IOException {
+    public double[] readPrimitiveDoubleArray() throws IOException {
         int length = readVInt();
         double[] values = new double[length];
         for(int i=0; i<length; i++) {
diff --git a/src/main/java/org/elasticsearch/common/io/stream/StreamOutput.java b/src/main/java/org/elasticsearch/common/io/stream/StreamOutput.java
index e119b3e0ba8af..c5e3f2b09be53 100644
--- a/src/main/java/org/elasticsearch/common/io/stream/StreamOutput.java
+++ b/src/main/java/org/elasticsearch/common/io/stream/StreamOutput.java
@@ -425,28 +425,28 @@ public void writeGenericValue(@Nullable Object value) throws IOException {
         }
     }
 
-    private void writePrimitiveIntArray(int[] value) throws IOException {
+    public void writePrimitiveIntArray(int[] value) throws IOException {
         writeVInt(value.length);
         for (int i=0; i<value.length; i++) {
             writeInt(value[i]);
         }
     }
     
-    private void writePrimitiveLongArray(long[] value) throws IOException {
+    public void writePrimitiveLongArray(long[] value) throws IOException {
         writeVInt(value.length);
         for (int i=0; i<value.length; i++) {
             writeLong(value[i]);
         }
     }
     
-    private void writePrimitiveFloatArray(float[] value) throws IOException {
+    public void writePrimitiveFloatArray(float[] value) throws IOException {
         writeVInt(value.length);
         for (int i=0; i<value.length; i++) {
             writeFloat(value[i]);
         }
     }
     
-    private void writePrimitiveDoubleArray(double[] value) throws IOException {
+    public void writePrimitiveDoubleArray(double[] value) throws IOException {
         writeVInt(value.length);
         for (int i=0; i<value.length; i++) {
             writeDouble(value[i]);

From e3187d5b9a58b47f9613d12f089fcf090448ae8c Mon Sep 17 00:00:00 2001
From: Matt Weber <matt@mattweber.org>
Date: Fri, 4 Apr 2014 21:06:19 -0700
Subject: [PATCH 28/54] Update LongHash to work like BytesRefHash.

- rename "key" to "get"
- update and add more tests

Closes #5693
---
 .../elasticsearch/common/util/LongHash.java   |  37 ++--
 .../bucket/geogrid/GeoHashGridAggregator.java |  21 +--
 .../bucket/histogram/HistogramAggregator.java |   8 +-
 .../SignificantLongTermsAggregator.java       |  14 +-
 .../bucket/terms/DoubleTermsAggregator.java   |  14 +-
 .../bucket/terms/LongTermsAggregator.java     |  14 +-
 .../common/util/LongHashTests.java            | 168 ++++++++++++++++--
 7 files changed, 193 insertions(+), 83 deletions(-)

diff --git a/src/main/java/org/elasticsearch/common/util/LongHash.java b/src/main/java/org/elasticsearch/common/util/LongHash.java
index dbaf2af3632f1..3e1c317949d2b 100644
--- a/src/main/java/org/elasticsearch/common/util/LongHash.java
+++ b/src/main/java/org/elasticsearch/common/util/LongHash.java
@@ -23,10 +23,10 @@
 
 /**
  * Specialized hash table implementation similar to BytesRefHash that maps
- *  long values to ids. Collisions are resolved with open addressing and linear
- *  probing, growth is smooth thanks to {@link BigArrays} and capacity is always
- *  a multiple of 2 for faster identification of buckets.
- *  This class is not thread-safe.
+ * long values to ids. Collisions are resolved with open addressing and linear
+ * probing, growth is smooth thanks to {@link BigArrays} and capacity is always
+ * a multiple of 2 for faster identification of buckets.
+ * This class is not thread-safe.
  */
 // IDs are internally stored as id + 1 so that 0 encodes for an empty slot
 public final class LongHash extends AbstractHash {
@@ -41,14 +41,14 @@ public LongHash(long capacity, BigArrays bigArrays) {
     //Constructor with configurable capacity and load factor.
     public LongHash(long capacity, float maxLoadFactor, BigArrays bigArrays) {
         super(capacity, maxLoadFactor, bigArrays);
-        keys = bigArrays.newLongArray(capacity(), false);
+        keys = bigArrays.newLongArray(capacity, false);
     }
 
     /**
      * Return the key at <code>0 &lte; index &lte; capacity()</code>. The result is undefined if the slot is unused.
      */
-    public long key(long index) {
-        return keys.get(index);
+    public long get(long id) {
+        return keys.get(id);
     }
 
     /**
@@ -58,7 +58,7 @@ public long find(long key) {
         final long slot = slot(hash(key), mask);
         for (long index = slot; ; index = nextSlot(index, mask)) {
             final long id = id(index);
-            if (id == -1 || keys.get(index) == key) {
+            if (id == -1 || keys.get(id) == key) {
                 return id;
             }
         }
@@ -71,25 +71,28 @@ private long set(long key, long id) {
             final long curId = id(index);
             if (curId == -1) { // means unset
                 id(index, id);
-                keys.set(index, key);
+                append(id, key);
                 ++size;
                 return id;
-            } else if (keys.get(index) == key) {
+            } else if (keys.get(curId) == key) {
                 return -1 - curId;
             }
         }
     }
 
+    private void append(long id, long key) {
+        keys = bigArrays.grow(keys, id + 1);
+        keys.set(id, key);
+    }
+
     private void reset(long key, long id) {
         final long slot = slot(hash(key), mask);
         for (long index = slot; ; index = nextSlot(index, mask)) {
             final long curId = id(index);
             if (curId == -1) { // means unset
                 id(index, id);
-                keys.set(index, key);
+                append(id, key);
                 break;
-            } else {
-                assert keys.get(index) != key;
             }
         }
     }
@@ -107,17 +110,11 @@ public long add(long key) {
         return set(key, size);
     }
 
-    @Override
-    protected void resize(long capacity) {
-        super.resize(capacity);
-        keys = bigArrays.resize(keys, capacity);
-    }
-
     @Override
     protected void removeAndAdd(long index) {
         final long id = id(index, -1);
         assert id >= 0;
-        final long key = keys.set(index, 0);
+        final long key = keys.set(id, 0);
         reset(key, id);
     }
 
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/geogrid/GeoHashGridAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/geogrid/GeoHashGridAggregator.java
index e43c1360f894d..1961ab69bc62a 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/geogrid/GeoHashGridAggregator.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/geogrid/GeoHashGridAggregator.java
@@ -35,7 +35,7 @@
 
 /**
  * Aggregates data expressed as GeoHash longs (for efficiency's sake) but formats results as Geohash strings.
- * 
+ *
  */
 
 public class GeoHashGridAggregator extends BucketsAggregator {
@@ -100,19 +100,14 @@ public InternalGeoHashGrid buildAggregation(long owningBucketOrdinal) {
 
         InternalGeoHashGrid.BucketPriorityQueue ordered = new InternalGeoHashGrid.BucketPriorityQueue(size);
         OrdinalBucket spare = null;
-        for (long i = 0; i < bucketOrds.capacity(); ++i) {
-            final long ord = bucketOrds.id(i);
-            if (ord < 0) {
-                // slot is not allocated
-                continue;
-            }
-
+        for (long i = 0; i < bucketOrds.size(); i++) {
             if (spare == null) {
                 spare = new OrdinalBucket();
             }
-            spare.geohashAsLong = bucketOrds.key(i);
-            spare.docCount = bucketDocCount(ord);
-            spare.bucketOrd = ord;
+
+            spare.geohashAsLong = bucketOrds.get(i);
+            spare.docCount = bucketDocCount(i);
+            spare.bucketOrd = i;
             spare = (OrdinalBucket) ordered.insertWithOverflow(spare);
         }
 
@@ -129,8 +124,8 @@ public InternalGeoHashGrid buildAggregation(long owningBucketOrdinal) {
     public InternalGeoHashGrid buildEmptyAggregation() {
         return new InternalGeoHashGrid(name, requiredSize, Collections.<InternalGeoHashGrid.Bucket>emptyList());
     }
-    
-    
+
+
     @Override
     public void doRelease() {
         Releasables.release(bucketOrds);
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/HistogramAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/HistogramAggregator.java
index fab705e25f09d..0fc081aeba8d4 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/HistogramAggregator.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/HistogramAggregator.java
@@ -111,12 +111,8 @@ public void collect(int doc, long owningBucketOrdinal) throws IOException {
     public InternalAggregation buildAggregation(long owningBucketOrdinal) {
         assert owningBucketOrdinal == 0;
         List<InternalHistogram.Bucket> buckets = new ArrayList<>((int) bucketOrds.size());
-        for (long i = 0; i < bucketOrds.capacity(); ++i) {
-            final long ord = bucketOrds.id(i);
-            if (ord < 0) {
-                continue; // slot is not allocated
-            }
-            buckets.add(histogramFactory.createBucket(rounding.valueForKey(bucketOrds.key(i)), bucketDocCount(ord), bucketAggregations(ord), formatter));
+        for (long i = 0; i < bucketOrds.size(); i++) {
+            buckets.add(histogramFactory.createBucket(rounding.valueForKey(bucketOrds.get(i)), bucketDocCount(i), bucketAggregations(i), formatter));
         }
 
         CollectionUtil.introSort(buckets, order.comparator());
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/SignificantLongTermsAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/SignificantLongTermsAggregator.java
index 8fed8059a00ef..feca0bb857f8d 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/SignificantLongTermsAggregator.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/SignificantLongTermsAggregator.java
@@ -66,18 +66,12 @@ public SignificantLongTerms buildAggregation(long owningBucketOrdinal) {
 
         BucketSignificancePriorityQueue ordered = new BucketSignificancePriorityQueue(size);
         SignificantLongTerms.Bucket spare = null;
-        for (long i = 0; i < bucketOrds.capacity(); i++) {
-            final long ord = bucketOrds.id(i);
-            if (ord < 0) {
-                // slot is not allocated
-                continue;
-            }
-
+        for (long i = 0; i < bucketOrds.size(); i++) {
             if (spare == null) {
                 spare = new SignificantLongTerms.Bucket(0, 0, 0, 0, 0, null);
             }
-            spare.term = bucketOrds.key(i);
-            spare.subsetDf = bucketDocCount(ord);
+            spare.term = bucketOrds.get(i);
+            spare.subsetDf = bucketDocCount(i);
             spare.subsetSize = subsetSize;
             spare.supersetDf = termsAggFactory.getBackgroundFrequency(spare.term);
             spare.supersetSize = supersetSize;
@@ -86,7 +80,7 @@ public SignificantLongTerms buildAggregation(long owningBucketOrdinal) {
             // Back at the central reducer these properties will be updated with global stats
             spare.updateScore();
 
-            spare.bucketOrd = ord;
+            spare.bucketOrd = i;
             spare = (SignificantLongTerms.Bucket) ordered.insertWithOverflow(spare);
         }
 
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/DoubleTermsAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/DoubleTermsAggregator.java
index c9a44ea24e22c..8e2909837ddab 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/DoubleTermsAggregator.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/DoubleTermsAggregator.java
@@ -110,19 +110,13 @@ public DoubleTerms buildAggregation(long owningBucketOrdinal) {
 
         BucketPriorityQueue ordered = new BucketPriorityQueue(size, order.comparator(this));
         DoubleTerms.Bucket spare = null;
-        for (long i = 0; i < bucketOrds.capacity(); ++i) {
-            final long ord = bucketOrds.id(i);
-            if (ord < 0) {
-                // slot is not allocated
-                continue;
-            }
-
+        for (long i = 0; i < bucketOrds.size(); i++) {
             if (spare == null) {
                 spare = new DoubleTerms.Bucket(0, 0, null);
             }
-            spare.term = Double.longBitsToDouble(bucketOrds.key(i));
-            spare.docCount = bucketDocCount(ord);
-            spare.bucketOrd = ord;
+            spare.term = Double.longBitsToDouble(bucketOrds.get(i));
+            spare.docCount = bucketDocCount(i);
+            spare.bucketOrd = i;
             spare = (DoubleTerms.Bucket) ordered.insertWithOverflow(spare);
         }
 
diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/LongTermsAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/LongTermsAggregator.java
index cb1a222429314..c710f363ead76 100644
--- a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/LongTermsAggregator.java
+++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/LongTermsAggregator.java
@@ -110,19 +110,13 @@ public InternalAggregation buildAggregation(long owningBucketOrdinal) {
 
         BucketPriorityQueue ordered = new BucketPriorityQueue(size, order.comparator(this));
         LongTerms.Bucket spare = null;
-        for (long i = 0; i < bucketOrds.capacity(); ++i) {
-            final long ord = bucketOrds.id(i);
-            if (ord < 0) {
-                // slot is not allocated
-                continue;
-            }
-
+        for (long i = 0; i < bucketOrds.size(); i++) {
             if (spare == null) {
                 spare = new LongTerms.Bucket(0, 0, null);
             }
-            spare.term = bucketOrds.key(i);
-            spare.docCount = bucketDocCount(ord);
-            spare.bucketOrd = ord;
+            spare.term = bucketOrds.get(i);
+            spare.docCount = bucketDocCount(i);
+            spare.bucketOrd = i;
             spare = (LongTerms.Bucket) ordered.insertWithOverflow(spare);
         }
 
diff --git a/src/test/java/org/elasticsearch/common/util/LongHashTests.java b/src/test/java/org/elasticsearch/common/util/LongHashTests.java
index 042b1cbb61343..e4acf0a7e9e2a 100644
--- a/src/test/java/org/elasticsearch/common/util/LongHashTests.java
+++ b/src/test/java/org/elasticsearch/common/util/LongHashTests.java
@@ -22,14 +22,31 @@
 import com.carrotsearch.hppc.LongLongMap;
 import com.carrotsearch.hppc.LongLongOpenHashMap;
 import com.carrotsearch.hppc.cursors.LongLongCursor;
-import org.elasticsearch.common.util.BigArraysTests;
-import org.elasticsearch.common.util.LongHash;
 import org.elasticsearch.test.ElasticsearchTestCase;
+import org.junit.Test;
 
-import java.util.Iterator;
+import java.util.*;
 
 public class LongHashTests extends ElasticsearchTestCase {
 
+    LongHash hash;
+
+    private void newHash() {
+        if (hash != null) {
+            hash.release();
+        }
+
+        // Test high load factors to make sure that collision resolution works fine
+        final float maxLoadFactor = 0.6f + randomFloat() * 0.39f;
+        hash = new LongHash(randomIntBetween(0, 100), maxLoadFactor, BigArraysTests.randombigArrays());
+    }
+
+    @Override
+    public void setUp() throws Exception {
+        super.setUp();
+        newHash();
+    }
+
     public void testDuell() {
         final Long[] values = new Long[randomIntBetween(1, 100000)];
         for (int i = 0; i < values.length; ++i) {
@@ -37,34 +54,157 @@ public void testDuell() {
         }
         final LongLongMap valueToId = new LongLongOpenHashMap();
         final long[] idToValue = new long[values.length];
-        // Test high load factors to make sure that collision resolution works fine
-        final float maxLoadFactor = 0.6f + randomFloat() * 0.39f;
-        final LongHash longHash = new LongHash(randomIntBetween(0, 100), maxLoadFactor, BigArraysTests.randombigArrays());
         final int iters = randomInt(1000000);
         for (int i = 0; i < iters; ++i) {
             final Long value = randomFrom(values);
             if (valueToId.containsKey(value)) {
-                assertEquals(- 1 - valueToId.get(value), longHash.add(value));
+                assertEquals(-1 - valueToId.get(value), hash.add(value));
             } else {
-                assertEquals(valueToId.size(), longHash.add(value));
+                assertEquals(valueToId.size(), hash.add(value));
                 idToValue[valueToId.size()] = value;
                 valueToId.put(value, valueToId.size());
             }
         }
 
-        assertEquals(valueToId.size(), longHash.size());
+        assertEquals(valueToId.size(), hash.size());
         for (Iterator<LongLongCursor> iterator = valueToId.iterator(); iterator.hasNext(); ) {
             final LongLongCursor next = iterator.next();
-            assertEquals(next.value, longHash.find(next.key));
+            assertEquals(next.value, hash.find(next.key));
         }
 
-        for (long i = 0; i < longHash.capacity(); ++i) {
-            final long id = longHash.id(i);
+        for (long i = 0; i < hash.capacity(); ++i) {
+            final long id = hash.id(i);
             if (id >= 0) {
-                assertEquals(idToValue[(int) id], longHash.key(i));
+                assertEquals(idToValue[(int) id], hash.get(id));
+            }
+        }
+
+        for (long i = 0; i < hash.size(); i++) {
+            assertEquals(idToValue[(int) i], hash.get(i));
+        }
+
+        hash.release();
+    }
+
+    @Test
+    public void testSize() {
+        int num = scaledRandomIntBetween(2, 20);
+        for (int j = 0; j < num; j++) {
+            final int mod = 1 + randomInt(40);
+            for (int i = 0; i < 797; i++) {
+                long count = hash.size();
+                long key = hash.add(randomLong());
+                if (key < 0)
+                    assertEquals(hash.size(), count);
+                else
+                    assertEquals(hash.size(), count + 1);
+                if (i % mod == 0) {
+                    newHash();
+                }
+            }
+        }
+        hash.release();
+    }
+
+    @Test
+    public void testKey() {
+        int num = scaledRandomIntBetween(2, 20);
+        for (int j = 0; j < num; j++) {
+            Map<Long, Long> longs = new HashMap<>();
+            int uniqueCount = 0;
+            for (int i = 0; i < 797; i++) {
+                long ref = randomLong();
+                long count = hash.size();
+                long key = hash.add(ref);
+                if (key >= 0) {
+                    assertNull(longs.put(ref, key));
+                    assertEquals(uniqueCount, key);
+                    uniqueCount++;
+                    assertEquals(hash.size(), count + 1);
+                } else {
+                    assertTrue((-key) - 1L < count);
+                    assertEquals(hash.size(), count);
+                }
+            }
+
+            for (Map.Entry<Long, Long> entry : longs.entrySet()) {
+                long expected = entry.getKey();
+                long keyIdx = entry.getValue();
+                assertEquals(expected, hash.get(keyIdx));
+            }
+
+            newHash();
+        }
+        hash.release();
+    }
+
+    @Test
+    public void testAdd() {
+        int num = scaledRandomIntBetween(2, 20);
+        for (int j = 0; j < num; j++) {
+            Set<Long> longs = new HashSet<>();
+            int uniqueCount = 0;
+            for (int i = 0; i < 797; i++) {
+                long ref = randomLong();
+                long count = hash.size();
+                long key = hash.add(ref);
+                if (key >= 0) {
+                    assertTrue(longs.add(ref));
+                    assertEquals(uniqueCount, key);
+                    assertEquals(hash.size(), count + 1);
+                    uniqueCount++;
+                } else {
+                    assertFalse(longs.add(ref));
+                    assertTrue((-key) - 1 < count);
+                    assertEquals(ref, hash.get((-key) - 1));
+                    assertEquals(count, hash.size());
+                }
             }
+
+            assertAllIn(longs, hash);
+            newHash();
+        }
+        hash.release();
+    }
+
+    @Test
+    public void testFind() throws Exception {
+        int num = scaledRandomIntBetween(2, 20);
+        for (int j = 0; j < num; j++) {
+            Set<Long> longs = new HashSet<>();
+            int uniqueCount = 0;
+            for (int i = 0; i < 797; i++) {
+                long ref = randomLong();
+                long count = hash.size();
+                long key = hash.find(ref);
+                if (key >= 0) { // found in hash
+                    assertFalse(longs.add(ref));
+                    assertTrue(key < count);
+                    assertEquals(ref, hash.get(key));
+                    assertEquals(count, hash.size());
+                } else {
+                    key = hash.add(ref);
+                    assertTrue(longs.add(ref));
+                    assertEquals(uniqueCount, key);
+                    assertEquals(hash.size(), count + 1);
+                    uniqueCount++;
+                }
+            }
+
+            assertAllIn(longs, hash);
+            newHash();
+        }
+        hash.release();
+    }
+
+    private static void assertAllIn(Set<Long> longs, LongHash hash) {
+        long count = hash.size();
+        for (Long l : longs) {
+            long key = hash.add(l); // add again to check duplicates
+            assertEquals(l.longValue(), hash.get((-key) - 1));
+            assertEquals(count, hash.size());
+            assertTrue("key: " + key + " count: " + count + " long: " + l, key < count);
         }
-        longHash.release();
     }
 
 }

From 033e46f8afee2483e278c2f38ff2de19bd3cc835 Mon Sep 17 00:00:00 2001
From: Andrew Selden <andrew.selden@elasticsearch.com>
Date: Mon, 7 Apr 2014 11:10:29 -0700
Subject: [PATCH 29/54] Rename readPrimitive*Array()/writePrimitive*Array()
 methods

Make method names shorter and easier to read.

Closes #5711
---
 .../common/io/stream/StreamInput.java            | 16 ++++++++--------
 .../common/io/stream/StreamOutput.java           | 16 ++++++++--------
 2 files changed, 16 insertions(+), 16 deletions(-)

diff --git a/src/main/java/org/elasticsearch/common/io/stream/StreamInput.java b/src/main/java/org/elasticsearch/common/io/stream/StreamInput.java
index 776e85f2f3c8e..9f88e0d462564 100644
--- a/src/main/java/org/elasticsearch/common/io/stream/StreamInput.java
+++ b/src/main/java/org/elasticsearch/common/io/stream/StreamInput.java
@@ -434,19 +434,19 @@ public Object readGenericValue() throws IOException {
             case 16:
                 return readShort();
             case 17:
-                return readPrimitiveIntArray();
+                return readIntArray();
             case 18:
-                return readPrimitiveLongArray();
+                return readLongArray();
             case 19:
-                return readPrimitiveFloatArray();
+                return readFloatArray();
             case 20:
-                return readPrimitiveDoubleArray();
+                return readDoubleArray();
             default:
                 throw new IOException("Can't read unknown type [" + type + "]");
         }
     }
 
-    public int[] readPrimitiveIntArray() throws IOException {
+    public int[] readIntArray() throws IOException {
         int length = readVInt();
         int[] values = new int[length];
         for(int i=0; i<length; i++) {
@@ -455,7 +455,7 @@ public int[] readPrimitiveIntArray() throws IOException {
         return values;
     }
     
-    public long[] readPrimitiveLongArray() throws IOException {
+    public long[] readLongArray() throws IOException {
         int length = readVInt();
         long[] values = new long[length];
         for(int i=0; i<length; i++) {
@@ -464,7 +464,7 @@ public long[] readPrimitiveLongArray() throws IOException {
         return values;
     }
     
-    public float[] readPrimitiveFloatArray() throws IOException {
+    public float[] readFloatArray() throws IOException {
         int length = readVInt();
         float[] values = new float[length];
         for(int i=0; i<length; i++) {
@@ -473,7 +473,7 @@ public float[] readPrimitiveFloatArray() throws IOException {
         return values;
     }
     
-    public double[] readPrimitiveDoubleArray() throws IOException {
+    public double[] readDoubleArray() throws IOException {
         int length = readVInt();
         double[] values = new double[length];
         for(int i=0; i<length; i++) {
diff --git a/src/main/java/org/elasticsearch/common/io/stream/StreamOutput.java b/src/main/java/org/elasticsearch/common/io/stream/StreamOutput.java
index c5e3f2b09be53..ce6ea15911f38 100644
--- a/src/main/java/org/elasticsearch/common/io/stream/StreamOutput.java
+++ b/src/main/java/org/elasticsearch/common/io/stream/StreamOutput.java
@@ -410,43 +410,43 @@ public void writeGenericValue(@Nullable Object value) throws IOException {
             writeShort((Short) value);
         } else if (type == int[].class) {
             writeByte((byte) 17);
-            writePrimitiveIntArray((int[]) value);
+            writeIntArray((int[]) value);
         } else if (type == long[].class) {
             writeByte((byte) 18);
-            writePrimitiveLongArray((long[]) value);
+            writeLongArray((long[]) value);
         } else if (type == float[].class) {
             writeByte((byte) 19);
-            writePrimitiveFloatArray((float[]) value);
+            writeFloatArray((float[]) value);
         } else if (type == double[].class) {
             writeByte((byte) 20);
-            writePrimitiveDoubleArray((double[]) value);
+            writeDoubleArray((double[]) value);
         } else {
             throw new IOException("Can't write type [" + type + "]");
         }
     }
 
-    public void writePrimitiveIntArray(int[] value) throws IOException {
+    public void writeIntArray(int[] value) throws IOException {
         writeVInt(value.length);
         for (int i=0; i<value.length; i++) {
             writeInt(value[i]);
         }
     }
     
-    public void writePrimitiveLongArray(long[] value) throws IOException {
+    public void writeLongArray(long[] value) throws IOException {
         writeVInt(value.length);
         for (int i=0; i<value.length; i++) {
             writeLong(value[i]);
         }
     }
     
-    public void writePrimitiveFloatArray(float[] value) throws IOException {
+    public void writeFloatArray(float[] value) throws IOException {
         writeVInt(value.length);
         for (int i=0; i<value.length; i++) {
             writeFloat(value[i]);
         }
     }
     
-    public void writePrimitiveDoubleArray(double[] value) throws IOException {
+    public void writeDoubleArray(double[] value) throws IOException {
         writeVInt(value.length);
         for (int i=0; i<value.length; i++) {
             writeDouble(value[i]);

From 5138083e13b96b47a3175517ffd9a9b7c3a83fcf Mon Sep 17 00:00:00 2001
From: Sean Gallagher <sean.gallagher@elasticsearch.com>
Date: Tue, 1 Apr 2014 12:28:08 -0400
Subject: [PATCH 30/54] Author: Sean Gallagher Date: Tue Apr 1 12:28:00 2014

Added upgrade.asciidoc and links to it from setup.asciidoc

Author: Sean Gallagher
Date: Apr 1 2014

Added upgrade.asciidoc

Add upgrade instructions
Author: Sean Gallagher
Date: 4/4/14
Closes issue #5651

Fixed upgrade.asciidoc typo and incorrect usage.
Author: Sean Gallagher
Date: 4 Apr 2014
Closes 5651
---
 README.textile                        |   4 +
 docs/reference/setup.asciidoc         |   2 +
 docs/reference/setup/upgrade.asciidoc | 133 ++++++++++++++++++++++++++
 3 files changed, 139 insertions(+)
 create mode 100644 docs/reference/setup/upgrade.asciidoc

diff --git a/README.textile b/README.textile
index 9d8fba6b45d8b..fcf53bdba8c4d 100644
--- a/README.textile
+++ b/README.textile
@@ -206,6 +206,10 @@ The distribution will be created under @target/releases@.
 See the "TESTING":TESTING.asciidoc file for more information about
 running the Elasticsearch test suite.
 
+h3. Upgrading to Elasticsearch 1.x?
+
+In order to ensure a smooth upgrade process from earlier versions of Elasticsearch (< 0.90.x), it is recommended to perform a full cluster restart. Please see the "Upgrading" section of the "setup reference":http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html.
+
 h1. License
 
 <pre>
diff --git a/docs/reference/setup.asciidoc b/docs/reference/setup.asciidoc
index cf413cd918a67..12324483f89c2 100644
--- a/docs/reference/setup.asciidoc
+++ b/docs/reference/setup.asciidoc
@@ -61,3 +61,5 @@ include::setup/as-a-service-win.asciidoc[]
 include::setup/dir-layout.asciidoc[]
 
 include::setup/repositories.asciidoc[]
+
+include::setup/upgrade.asciidoc[]
diff --git a/docs/reference/setup/upgrade.asciidoc b/docs/reference/setup/upgrade.asciidoc
new file mode 100644
index 0000000000000..f3bf3df193d82
--- /dev/null
+++ b/docs/reference/setup/upgrade.asciidoc
@@ -0,0 +1,133 @@
+[[setup-upgrade]]
+== Upgrading
+
+Elasticsearch can usually be upgraded using a rolling upgrade process, resulting in no interruption of service.  This section details how to perform both rolling and restart upgrades.
+
+[float]
+[[backup]]
+=== Back Up Your Data!
+
+Before performing an upgrade, it's a good idea to back up the data on your system.  This will allow you to roll back in the event of a problem with the upgrade.  The upgrades sometimes include upgrades to the Lucene libraries used by Elasticsearch to access the index files, and after an index file has been updated to work with a new version of Lucene, it may not be accessible to the versions of Lucene present in earlier Elasticsearch releases.
+
+[float]
+==== 0.90.x and earlier
+
+To back up a running 0.90.x system, first disable index flushing.  This will prevent indices from being flushed to disk while the backup is in process:
+
+[source,sh]
+-----------------------------------
+$ curl -XPUT 'http://localhost:9200/_all/_settings' -d '{
+    "index": {
+        "translog.disable_flush": "true"
+    }
+}'
+-----------------------------------
+
+Then disable reallocation.  This will prevent the cluster from moving data files from one node to another while the backup is in process:
+
+[source,sh]
+-----------------------------------
+$ curl -XPUT 'http://localhost:9200/_cluster/settings' -d '{
+    "transient" : {
+        "cluster.routing.allocation.disable_allocation": "true"
+    }
+}'
+-----------------------------------
+
+After reallocation and index flushing are disabled, initiate a backup of Elasticsearch's data path using your favorite backup method (tar, storage array snapshots, backup software).  When the backup is complete and data no longer needs to be read from the Elasticsearch data path, reallocation and index flushing must be re-enabled:
+
+[source,sh]
+-----------------------------------
+$ curl -XPUT 'http://localhost:9200/_all/_settings' -d '{
+    "index": {
+        "translog.disable_flush": "false"
+    }
+}'
+
+$ curl -XPUT 'http://localhost:9200/_cluster/settings' -d '{
+    "transient" : {
+        "cluster.routing.allocation.disable_allocation": "false"
+    }
+}'
+-----------------------------------
+
+[float]
+==== 1.0 and later
+
+To back up a running 1.0 or later system, it is simplest to use the snapshot feature.  Complete instructions for backup and restore with snapshots are available http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html[here].
+
+[float]
+[[rolling-upgrades]]
+=== Rolling Upgrades
+
+Within major releases after release 1.0, rolling upgrades are supported.  To perform a rolling upgrade:
+
+* Shut down a single node within the cluster.
+
+[source,sh]
+--------------------------------------------
+curl -XPOST 'http://localhost:9200/_cluster/nodes/_local/_shutdown'
+--------------------------------------------
+
+* Confirm that all shards are correctly reallocated to the remaining running nodes.
+
+* Upgrade the stopped node.  To upgrade using a zip or compressed tarball from elasticsearch.org:
+** Extract the zip or tarball to a new directory, usually in the same volume as the current Elasticsearch installation.  Do not overwrite the existing installation, as the downloaded archive will contain a default elasticsearch.yml file and will overwrite your existing configuration.
+** Copy the configuration files from the old Elasticsearch installation's config directory to the new Elasticsearch installation's config directory.  Copy data files if necessary.  If data files are not located within the tarball's extraction directory, they will not have to be moved.
+** The simplest solution for moving from one version to another is to have a symbolic link for 'elasticsearch' that points to the currently running version.  This link can be easily updated and will provide a stable access point to the most recent version.  Update this symbolic link if it is being used.
+
+*To upgrade using a .deb or .rpm package:
+
+** Use rpm or deb to install the new package.  All files should be placed in their proper locations, and config files should not be overwritten.
+
+** Start the now upgraded node.  Observe that all shards are properly allocated back to the node and that it joins the cluster smoothly.
+
+Repeat this process for all remaining nodes.
+
+It may be possible to perform the upgrade by installing the new software while the service is running.  This would reduce downtime by ensuring the service was ready to run on the new version as soon as it is stopped on the node being upgraded.  This can be done by installing the new version in its own directory and using the symbolic link method outlined above.  It is important to test this procedure first to be sure that site-specific configuration data and production indices will not be overwritten during the upgrade process.
+
+[float]
+[[restart-upgrade]]
+=== Upgrading from 0.90.x or earlier to 1.0 or later
+
+Elasticsearch releases prior to 1.0 and releases after 1.0 are not compatible with each other, so a rolling upgrade is not possible.  In order to upgrade a pre-1.0 system to 1.0 or later, a full cluster stop and start is required.  In order to perform this upgrade:
+
+* Disable shard reallocation (optional).  This is done to allow for a faster startup after cluster shutdown.  If this step is not performed, the nodes will immediately start trying to replicate shards to each other on startup and will spend a lot of time on wasted I/O.  With shard reallocation disabled, the nodes will join the cluster with their indices intact, without attempting to rebalance.  After startup is complete, reallocation will be turned back on.
+
+This syntax is from versions prior to 1.0:
+
+[source,sh]
+--------------------------------------------------
+	curl -XPUT localhost:9200/_cluster/settings -d '{
+		"persistent" : {
+		"cluster.routing.allocation.disable_allocation" : true
+		}
+	}'
+--------------------------------------------------
+
+* Stop all Elasticsearch services on all nodes in the cluster.
+[source,sh]
+------------------------------------------------------
+	curl -XPOST 'http://localhost:9200/_shutdown'
+------------------------------------------------------
+
+* On the first node to be upgraded, extract the archive or install the new package as described above in the Rolling Upgrades section.  Repeat for all nodes.
+
+* After upgrading Elasticsearch on all nodes is complete, the cluster can be started by starting each node individually.
+** Start master-eligible nodes first, one at a time.  Verify that a quorum has been reached and a master has been elected before proceeding.
+** Start data and client nodes one at a time, verifying that they successfully join the cluster.
+
+* When the cluster is up and running, shard reallocation can be enabled.
+
+This syntax is from release 1.0 and later:
+[source,sh]
+------------------------------------------------------
+	curl -XPUT localhost:9200/_cluster/settings -d '{
+     		"persistent" : {
+		"cluster.routing.allocation.disable_allocation": false,
+         	"cluster.routing.allocation.enable" : "all"
+     		}
+ 	}'
+------------------------------------------------------
+
+The cluster upgrade can be streamlined by installing the software before stopping cluster services.  If this is done, testing must be performed to ensure that no production data or configuration files are overwritten prior to restart.

From f2181d5cbfb6f8763b707e4647f719f81f6f5d0d Mon Sep 17 00:00:00 2001
From: Simon Willnauer <simonw@apache.org>
Date: Sun, 6 Apr 2014 09:28:11 +0200
Subject: [PATCH 31/54] [TEST] Be more verbose if ClusterStatsTests fails

---
 .../cluster/stats/ClusterStatsTests.java      | 29 ++++++++++---------
 1 file changed, 15 insertions(+), 14 deletions(-)

diff --git a/src/test/java/org/elasticsearch/action/admin/cluster/stats/ClusterStatsTests.java b/src/test/java/org/elasticsearch/action/admin/cluster/stats/ClusterStatsTests.java
index bb6c46bcaaec7..c82eef9db6169 100644
--- a/src/test/java/org/elasticsearch/action/admin/cluster/stats/ClusterStatsTests.java
+++ b/src/test/java/org/elasticsearch/action/admin/cluster/stats/ClusterStatsTests.java
@@ -131,27 +131,28 @@ public void testValuesSmokeScreen() {
         index("test1", "type", "1", "f", "f");
 
         ClusterStatsResponse response = client().admin().cluster().prepareClusterStats().get();
-        assertThat(response.getTimestamp(), Matchers.greaterThan(946681200000l)); // 1 Jan 2000
-        assertThat(response.indicesStats.getStore().getSizeInBytes(), Matchers.greaterThan(0l));
+        String msg = response.toString();
+        assertThat(msg, response.getTimestamp(), Matchers.greaterThan(946681200000l)); // 1 Jan 2000
+        assertThat(msg, response.indicesStats.getStore().getSizeInBytes(), Matchers.greaterThan(0l));
 
-        assertThat(response.nodesStats.getFs().getTotal().bytes(), Matchers.greaterThan(0l));
-        assertThat(response.nodesStats.getJvm().getVersions().size(), Matchers.greaterThan(0));
+        assertThat(msg, response.nodesStats.getFs().getTotal().bytes(), Matchers.greaterThan(0l));
+        assertThat(msg, response.nodesStats.getJvm().getVersions().size(), Matchers.greaterThan(0));
         if (sigarService.sigarAvailable()) {
             // We only get those if we have sigar
-            assertThat(response.nodesStats.getOs().getAvailableProcessors(), Matchers.greaterThan(0));
-            assertThat(response.nodesStats.getOs().getAvailableMemory().bytes(), Matchers.greaterThan(0l));
-            assertThat(response.nodesStats.getOs().getCpus().size(), Matchers.greaterThan(0));
+            assertThat(msg, response.nodesStats.getOs().getAvailableProcessors(), Matchers.greaterThan(0));
+            assertThat(msg, response.nodesStats.getOs().getAvailableMemory().bytes(), Matchers.greaterThan(0l));
+            assertThat(msg, response.nodesStats.getOs().getCpus().size(), Matchers.greaterThan(0));
         }
-        assertThat(response.nodesStats.getVersions().size(), Matchers.greaterThan(0));
-        assertThat(response.nodesStats.getVersions().contains(Version.CURRENT), Matchers.equalTo(true));
-        assertThat(response.nodesStats.getPlugins().size(), Matchers.greaterThanOrEqualTo(0));
+        assertThat(msg, response.nodesStats.getVersions().size(), Matchers.greaterThan(0));
+        assertThat(msg, response.nodesStats.getVersions().contains(Version.CURRENT), Matchers.equalTo(true));
+        assertThat(msg, response.nodesStats.getPlugins().size(), Matchers.greaterThanOrEqualTo(0));
 
-        assertThat(response.nodesStats.getProcess().count, Matchers.greaterThan(0));
+        assertThat(msg, response.nodesStats.getProcess().count, Matchers.greaterThan(0));
         // 0 happens when not supported on platform
-        assertThat(response.nodesStats.getProcess().getAvgOpenFileDescriptors(), Matchers.greaterThanOrEqualTo(0L));
+        assertThat(msg, response.nodesStats.getProcess().getAvgOpenFileDescriptors(), Matchers.greaterThanOrEqualTo(0L));
         // these can be -1 if not supported on platform
-        assertThat(response.nodesStats.getProcess().getMinOpenFileDescriptors(), Matchers.greaterThanOrEqualTo(-1L));
-        assertThat(response.nodesStats.getProcess().getMaxOpenFileDescriptors(), Matchers.greaterThanOrEqualTo(-1L));
+        assertThat(msg, response.nodesStats.getProcess().getMinOpenFileDescriptors(), Matchers.greaterThanOrEqualTo(-1L));
+        assertThat(msg, response.nodesStats.getProcess().getMaxOpenFileDescriptors(), Matchers.greaterThanOrEqualTo(-1L));
 
     }
 }

From befa83338513bd8296551c57261ac8d505df9ecc Mon Sep 17 00:00:00 2001
From: Simon Willnauer <simonw@apache.org>
Date: Mon, 7 Apr 2014 22:05:47 +0200
Subject: [PATCH 32/54] Make sure successful operations are correct if second
 phase is fast

In TransportSearchTypeAction we  need to increment successful ops
first before we increment and compare the exit condition otherwise if we
are fast we could concurrently update totalOps but then preempt one
of the threads which can cause the successor to read a wrong value from
successfulOps if second phase is very fast ie. searchType == count etc.
This can cause wrong success stats in the search response.
---
 .../action/search/type/TransportSearchTypeAction.java     | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/src/main/java/org/elasticsearch/action/search/type/TransportSearchTypeAction.java b/src/main/java/org/elasticsearch/action/search/type/TransportSearchTypeAction.java
index 6b5897602b968..54e98ebe8cf39 100644
--- a/src/main/java/org/elasticsearch/action/search/type/TransportSearchTypeAction.java
+++ b/src/main/java/org/elasticsearch/action/search/type/TransportSearchTypeAction.java
@@ -246,11 +246,13 @@ public void onFailure(Throwable t) {
         void onFirstPhaseResult(int shardIndex, ShardRouting shard, FirstResult result, ShardIterator shardIt) {
             result.shardTarget(new SearchShardTarget(shard.currentNodeId(), shard.index(), shard.id()));
             processFirstPhaseResult(shardIndex, shard, result);
-
+            // we need to increment successful ops first before we compare the exit condition otherwise if we
+            // are fast we could concurrently update totalOps but then preempt one of the threads which can
+            // cause the successor to read a wrong value from successfulOps if second phase is very fast ie. count etc.
+            successulOps.incrementAndGet();
             // increment all the "future" shards to update the total ops since we some may work and some may not...
             // and when that happens, we break on total ops, so we must maintain them
-            int xTotalOps = totalOps.addAndGet(shardIt.remaining() + 1);
-            successulOps.incrementAndGet();
+            final int xTotalOps = totalOps.addAndGet(shardIt.remaining() + 1);
             if (xTotalOps == expectedTotalOps) {
                 try {
                     innerMoveToSecondPhase();

From 49c74e08859736d09add31436554dfa76e395dd8 Mon Sep 17 00:00:00 2001
From: Simon Willnauer <simonw@apache.org>
Date: Mon, 7 Apr 2014 22:09:09 +0200
Subject: [PATCH 33/54] Rename successulOps to successfulOps in
 TransportSearchTypeAction

---
 .../action/search/type/TransportSearchCountAction.java      | 2 +-
 .../search/type/TransportSearchDfsQueryAndFetchAction.java  | 4 ++--
 .../search/type/TransportSearchDfsQueryThenFetchAction.java | 6 +++---
 .../search/type/TransportSearchQueryAndFetchAction.java     | 2 +-
 .../search/type/TransportSearchQueryThenFetchAction.java    | 4 ++--
 .../action/search/type/TransportSearchScanAction.java       | 2 +-
 .../action/search/type/TransportSearchTypeAction.java       | 6 +++---
 7 files changed, 13 insertions(+), 13 deletions(-)

diff --git a/src/main/java/org/elasticsearch/action/search/type/TransportSearchCountAction.java b/src/main/java/org/elasticsearch/action/search/type/TransportSearchCountAction.java
index 96c2d24adda51..a19db2c7b7264 100644
--- a/src/main/java/org/elasticsearch/action/search/type/TransportSearchCountAction.java
+++ b/src/main/java/org/elasticsearch/action/search/type/TransportSearchCountAction.java
@@ -78,7 +78,7 @@ protected void moveToSecondPhase() throws Exception {
             if (request.scroll() != null) {
                 scrollId = buildScrollId(request.searchType(), firstResults, null);
             }
-            listener.onResponse(new SearchResponse(internalResponse, scrollId, expectedSuccessfulOps, successulOps.get(), buildTookInMillis(), buildShardFailures()));
+            listener.onResponse(new SearchResponse(internalResponse, scrollId, expectedSuccessfulOps, successfulOps.get(), buildTookInMillis(), buildShardFailures()));
         }
     }
 }
diff --git a/src/main/java/org/elasticsearch/action/search/type/TransportSearchDfsQueryAndFetchAction.java b/src/main/java/org/elasticsearch/action/search/type/TransportSearchDfsQueryAndFetchAction.java
index 215672bc21da4..d29355d851fef 100644
--- a/src/main/java/org/elasticsearch/action/search/type/TransportSearchDfsQueryAndFetchAction.java
+++ b/src/main/java/org/elasticsearch/action/search/type/TransportSearchDfsQueryAndFetchAction.java
@@ -158,7 +158,7 @@ void onSecondPhaseFailure(Throwable t, QuerySearchRequest querySearchRequest, in
                 logger.debug("[{}] Failed to execute query phase", t, querySearchRequest.id());
             }
             this.addShardFailure(shardIndex, dfsResult.shardTarget(), t);
-            successulOps.decrementAndGet();
+            successfulOps.decrementAndGet();
             if (counter.decrementAndGet() == 0) {
                 finishHim();
             }
@@ -185,7 +185,7 @@ void innerFinishHim() throws Exception {
             if (request.scroll() != null) {
                 scrollId = TransportSearchHelper.buildScrollId(request.searchType(), firstResults, null);
             }
-            listener.onResponse(new SearchResponse(internalResponse, scrollId, expectedSuccessfulOps, successulOps.get(), buildTookInMillis(), buildShardFailures()));
+            listener.onResponse(new SearchResponse(internalResponse, scrollId, expectedSuccessfulOps, successfulOps.get(), buildTookInMillis(), buildShardFailures()));
         }
     }
 }
diff --git a/src/main/java/org/elasticsearch/action/search/type/TransportSearchDfsQueryThenFetchAction.java b/src/main/java/org/elasticsearch/action/search/type/TransportSearchDfsQueryThenFetchAction.java
index f52a84c680b16..b168a40ab4670 100644
--- a/src/main/java/org/elasticsearch/action/search/type/TransportSearchDfsQueryThenFetchAction.java
+++ b/src/main/java/org/elasticsearch/action/search/type/TransportSearchDfsQueryThenFetchAction.java
@@ -168,7 +168,7 @@ void onQueryFailure(Throwable t, QuerySearchRequest querySearchRequest, int shar
                 logger.debug("[{}] Failed to execute query phase", t, querySearchRequest.id());
             }
             this.addShardFailure(shardIndex, dfsResult.shardTarget(), t);
-            successulOps.decrementAndGet();
+            successfulOps.decrementAndGet();
             if (counter.decrementAndGet() == 0) {
                 executeFetchPhase();
             }
@@ -272,7 +272,7 @@ void onFetchFailure(Throwable t, FetchSearchRequest fetchSearchRequest, int shar
                 logger.debug("[{}] Failed to execute fetch phase", t, fetchSearchRequest.id());
             }
             this.addShardFailure(shardIndex, shardTarget, t);
-            successulOps.decrementAndGet();
+            successfulOps.decrementAndGet();
             if (counter.decrementAndGet() == 0) {
                 finishHim();
             }
@@ -298,7 +298,7 @@ void innerFinishHim() throws Exception {
             if (request.scroll() != null) {
                 scrollId = TransportSearchHelper.buildScrollId(request.searchType(), firstResults, null);
             }
-            listener.onResponse(new SearchResponse(internalResponse, scrollId, expectedSuccessfulOps, successulOps.get(), buildTookInMillis(), buildShardFailures()));
+            listener.onResponse(new SearchResponse(internalResponse, scrollId, expectedSuccessfulOps, successfulOps.get(), buildTookInMillis(), buildShardFailures()));
         }
     }
 }
diff --git a/src/main/java/org/elasticsearch/action/search/type/TransportSearchQueryAndFetchAction.java b/src/main/java/org/elasticsearch/action/search/type/TransportSearchQueryAndFetchAction.java
index 334bb140d2e60..946512f22403b 100644
--- a/src/main/java/org/elasticsearch/action/search/type/TransportSearchQueryAndFetchAction.java
+++ b/src/main/java/org/elasticsearch/action/search/type/TransportSearchQueryAndFetchAction.java
@@ -91,7 +91,7 @@ private void innerFinishHim() throws IOException {
             if (request.scroll() != null) {
                 scrollId = buildScrollId(request.searchType(), firstResults, null);
             }
-            listener.onResponse(new SearchResponse(internalResponse, scrollId, expectedSuccessfulOps, successulOps.get(), buildTookInMillis(), buildShardFailures()));
+            listener.onResponse(new SearchResponse(internalResponse, scrollId, expectedSuccessfulOps, successfulOps.get(), buildTookInMillis(), buildShardFailures()));
         }
     }
 }
diff --git a/src/main/java/org/elasticsearch/action/search/type/TransportSearchQueryThenFetchAction.java b/src/main/java/org/elasticsearch/action/search/type/TransportSearchQueryThenFetchAction.java
index d713069ef39b5..ee92c694209f9 100644
--- a/src/main/java/org/elasticsearch/action/search/type/TransportSearchQueryThenFetchAction.java
+++ b/src/main/java/org/elasticsearch/action/search/type/TransportSearchQueryThenFetchAction.java
@@ -172,7 +172,7 @@ void onFetchFailure(Throwable t, FetchSearchRequest fetchSearchRequest, int shar
                 logger.debug("[{}] Failed to execute fetch phase", t, fetchSearchRequest.id());
             }
             this.addShardFailure(shardIndex, shardTarget, t);
-            successulOps.decrementAndGet();
+            successfulOps.decrementAndGet();
             if (counter.decrementAndGet() == 0) {
                 finishHim();
             }
@@ -198,7 +198,7 @@ void innerFinishHim() throws Exception {
             if (request.scroll() != null) {
                 scrollId = TransportSearchHelper.buildScrollId(request.searchType(), firstResults, null);
             }
-            listener.onResponse(new SearchResponse(internalResponse, scrollId, expectedSuccessfulOps, successulOps.get(), buildTookInMillis(), buildShardFailures()));
+            listener.onResponse(new SearchResponse(internalResponse, scrollId, expectedSuccessfulOps, successfulOps.get(), buildTookInMillis(), buildShardFailures()));
         }
     }
 }
diff --git a/src/main/java/org/elasticsearch/action/search/type/TransportSearchScanAction.java b/src/main/java/org/elasticsearch/action/search/type/TransportSearchScanAction.java
index b7e5c10a3f385..f9a12585329ff 100644
--- a/src/main/java/org/elasticsearch/action/search/type/TransportSearchScanAction.java
+++ b/src/main/java/org/elasticsearch/action/search/type/TransportSearchScanAction.java
@@ -75,7 +75,7 @@ protected void moveToSecondPhase() throws Exception {
             if (request.scroll() != null) {
                 scrollId = buildScrollId(request.searchType(), firstResults, ImmutableMap.of("total_hits", Long.toString(internalResponse.hits().totalHits())));
             }
-            listener.onResponse(new SearchResponse(internalResponse, scrollId, expectedSuccessfulOps, successulOps.get(), buildTookInMillis(), buildShardFailures()));
+            listener.onResponse(new SearchResponse(internalResponse, scrollId, expectedSuccessfulOps, successfulOps.get(), buildTookInMillis(), buildShardFailures()));
         }
     }
 }
diff --git a/src/main/java/org/elasticsearch/action/search/type/TransportSearchTypeAction.java b/src/main/java/org/elasticsearch/action/search/type/TransportSearchTypeAction.java
index 54e98ebe8cf39..325ac54717f61 100644
--- a/src/main/java/org/elasticsearch/action/search/type/TransportSearchTypeAction.java
+++ b/src/main/java/org/elasticsearch/action/search/type/TransportSearchTypeAction.java
@@ -89,7 +89,7 @@ protected abstract class BaseAsyncAction<FirstResult extends SearchPhaseResult>
         protected final int expectedSuccessfulOps;
         private final int expectedTotalOps;
 
-        protected final AtomicInteger successulOps = new AtomicInteger();
+        protected final AtomicInteger successfulOps = new AtomicInteger();
         private final AtomicInteger totalOps = new AtomicInteger();
 
         protected final AtomicArray<FirstResult> firstResults;
@@ -249,7 +249,7 @@ void onFirstPhaseResult(int shardIndex, ShardRouting shard, FirstResult result,
             // we need to increment successful ops first before we compare the exit condition otherwise if we
             // are fast we could concurrently update totalOps but then preempt one of the threads which can
             // cause the successor to read a wrong value from successfulOps if second phase is very fast ie. count etc.
-            successulOps.incrementAndGet();
+            successfulOps.incrementAndGet();
             // increment all the "future" shards to update the total ops since we some may work and some may not...
             // and when that happens, we break on total ops, so we must maintain them
             final int xTotalOps = totalOps.addAndGet(shardIt.remaining() + 1);
@@ -283,7 +283,7 @@ void onFirstPhaseResult(final int shardIndex, @Nullable ShardRouting shard, @Nul
                         logger.trace("{}: Failed to execute [{}]", t, shard, request);
                     }
                 }
-                if (successulOps.get() == 0) {
+                if (successfulOps.get() == 0) {
                     if (logger.isDebugEnabled()) {
                         logger.debug("All shards failed for phase: [{}]", firstPhaseName(), t);
                     }

From a1d0eee6bf1755cce5de2104eb2c72b77e31abbc Mon Sep 17 00:00:00 2001
From: Shay Banon <kimchy@gmail.com>
Date: Mon, 7 Apr 2014 22:34:47 +0200
Subject: [PATCH 34/54] [TEST] return the correct transport instance in mock
 transport

---
 .../elasticsearch/test/transport/MockTransportService.java  | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/src/test/java/org/elasticsearch/test/transport/MockTransportService.java b/src/test/java/org/elasticsearch/test/transport/MockTransportService.java
index 9e920a7dedba3..be54ab7dd936a 100644
--- a/src/test/java/org/elasticsearch/test/transport/MockTransportService.java
+++ b/src/test/java/org/elasticsearch/test/transport/MockTransportService.java
@@ -234,12 +234,14 @@ public void removeLifecycleListener(LifecycleListener listener) {
 
         @Override
         public Transport start() throws ElasticsearchException {
-            return transport.start();
+            transport.start();
+            return this;
         }
 
         @Override
         public Transport stop() throws ElasticsearchException {
-            return transport.stop();
+            transport.stop();
+            return this;
         }
 
         @Override

From 48031b62369fad5d46994d087bea97d21d6f765e Mon Sep 17 00:00:00 2001
From: Andrew O'Brien <obrien.andrew@gmail.com>
Date: Mon, 7 Apr 2014 17:47:39 -0400
Subject: [PATCH 35/54] Fixes typo in "Scan" search type documention

---
 docs/reference/search/request/search-type.asciidoc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/reference/search/request/search-type.asciidoc b/docs/reference/search/request/search-type.asciidoc
index 6f4c7bae79e43..50afc2b593bec 100644
--- a/docs/reference/search/request/search-type.asciidoc
+++ b/docs/reference/search/request/search-type.asciidoc
@@ -109,7 +109,7 @@ curl -XGET 'localhost:9200/_search?search_type=scan&scroll=10m&size=50' -d '
 '
 --------------------------------------------------
 
-The `scroll` parameter control the keep alive time of the scrolling
+The `scroll` parameter controls the keep alive time of the scrolling
 request and initiates the scrolling process. The timeout applies per
 round trip (i.e. between the previous scan scroll request, to the next).
 

From dcc6a6e13831a362bc620fe3e02e9b2d761de293 Mon Sep 17 00:00:00 2001
From: Bill Hwang <bill.hwang@elasticsearch.com>
Date: Mon, 7 Apr 2014 15:13:11 -0700
Subject: [PATCH 36/54] [BUILD] Remove site dependencies generation

Removed dependencies library generation.  It stalls Jenkins static analysis job
---
 pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/pom.xml b/pom.xml
index b5fbe68bd3ee0..76359dd482322 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1414,7 +1414,7 @@
               <reportSets>
                 <reportSet>
                   <reports>
-                    <report>dependencies</report>
+                    <report>index</report>
                   </reports>
                 </reportSet>
               </reportSets>

From b4c506b952d0808367830f6015bb3e13a7d56767 Mon Sep 17 00:00:00 2001
From: Simon Willnauer <simonw@apache.org>
Date: Tue, 8 Apr 2014 10:07:53 +0200
Subject: [PATCH 37/54] [TEST] use no replicas in MLT tests - doc mappers need
 to be present on all nodes

---
 rest-api-spec/test/mlt/10_basic.yaml | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/rest-api-spec/test/mlt/10_basic.yaml b/rest-api-spec/test/mlt/10_basic.yaml
index e914607b997b8..c29468133af19 100644
--- a/rest-api-spec/test/mlt/10_basic.yaml
+++ b/rest-api-spec/test/mlt/10_basic.yaml
@@ -1,5 +1,13 @@
 ---
 "Basic mlt":
+  - do:
+     indices.create:
+       index:    test_1
+       body:
+         settings:
+           index:
+             number_of_replicas: 0
+
   - do:
       index:
           index:  test_1
@@ -13,7 +21,7 @@
   - do:
       cluster.health:
            wait_for_status: green
-           timeout: 1s   
+
   - do:
       mlt:
           index:  test_1

From 0cebbf162449325addd713acb74952e43f66b438 Mon Sep 17 00:00:00 2001
From: David Pilato <david@pilato.fr>
Date: Tue, 8 Apr 2014 10:55:26 +0200
Subject: [PATCH 38/54] Text Query has been replaced by Match Query

Closes #5727.
---
 README.textile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/README.textile b/README.textile
index fcf53bdba8c4d..8330e3b7b4ee7 100644
--- a/README.textile
+++ b/README.textile
@@ -86,7 +86,7 @@ We can also use the JSON query language Elasticsearch provides instead of a quer
 curl -XGET 'http://localhost:9200/twitter/tweet/_search?pretty=true' -d '
 { 
     "query" : { 
-        "text" : { "user": "kimchy" }
+        "match" : { "user": "kimchy" }
     } 
 }'
 </pre>

From fd8a6ac382c4b2527ed166f934775712e7f1d349 Mon Sep 17 00:00:00 2001
From: Simon Willnauer <simonw@apache.org>
Date: Tue, 8 Apr 2014 11:11:56 +0200
Subject: [PATCH 39/54] [TEST] make BulkTest more robust if test infra is slow

---
 .../org/elasticsearch/document/BulkTests.java | 21 +++++++++++++------
 1 file changed, 15 insertions(+), 6 deletions(-)

diff --git a/src/test/java/org/elasticsearch/document/BulkTests.java b/src/test/java/org/elasticsearch/document/BulkTests.java
index b017f7a761408..bc78a61474419 100644
--- a/src/test/java/org/elasticsearch/document/BulkTests.java
+++ b/src/test/java/org/elasticsearch/document/BulkTests.java
@@ -569,21 +569,27 @@ public void preParsingSourceDueToIdShouldNotBreakCompleteBulkRequest() throws Ex
 
     @Test
     public void testThatBulkProcessorCountIsCorrect() throws InterruptedException {
-        final BlockingQueue<BulkResponse> responseQueue = new SynchronousQueue();
-
+        final AtomicReference<BulkResponse> responseRef = new AtomicReference<>();
+        final AtomicReference<Throwable> failureRef = new AtomicReference<>();
+        final CountDownLatch latch = new CountDownLatch(1);
         BulkProcessor.Listener listener = new BulkProcessor.Listener() {
             @Override
             public void beforeBulk(long executionId, BulkRequest request) {}
 
             @Override
             public void afterBulk(long executionId, BulkRequest request, BulkResponse response) {
-                responseQueue.add(response);
+                responseRef.set(response);
+                latch.countDown();
             }
 
             @Override
-            public void afterBulk(long executionId, BulkRequest request, Throwable failure) {}
+            public void afterBulk(long executionId, BulkRequest request, Throwable failure) {
+                failureRef.set(failure);
+                latch.countDown();
+            }
         };
 
+
         try (BulkProcessor processor = BulkProcessor.builder(client(), listener).setBulkActions(5)
                                         .setConcurrentRequests(1).setName("foo").build()) {
             Map<String, Object> data = Maps.newHashMap();
@@ -595,8 +601,11 @@ public void afterBulk(long executionId, BulkRequest request, Throwable failure)
             processor.add(new IndexRequest("test", "test", "4").source(data));
             processor.add(new IndexRequest("test", "test", "5").source(data));
 
-            BulkResponse response = responseQueue.poll(5, TimeUnit.SECONDS);
-            assertThat("Could not get a bulk response in 5 seconds", response, is(notNullValue()));
+            latch.await();
+            BulkResponse response = responseRef.get();
+            Throwable error = failureRef.get();
+            assertThat(error, nullValue());
+            assertThat("Could not get a bulk response even after an explicit flush.", response, notNullValue());
             assertThat(response.getItems().length, is(5));
         }
     }

From c58b823e9f076b52e796919346c3cd95eed43ef0 Mon Sep 17 00:00:00 2001
From: Simon Willnauer <simonw@apache.org>
Date: Tue, 8 Apr 2014 11:38:08 +0200
Subject: [PATCH 40/54] [TEST] Add more randomization to bulk tests

---
 .../java/org/elasticsearch/document/BulkTests.java | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/src/test/java/org/elasticsearch/document/BulkTests.java b/src/test/java/org/elasticsearch/document/BulkTests.java
index bc78a61474419..713807835e109 100644
--- a/src/test/java/org/elasticsearch/document/BulkTests.java
+++ b/src/test/java/org/elasticsearch/document/BulkTests.java
@@ -216,7 +216,11 @@ public void testBulkUpdate_largerVolume() throws Exception {
         createIndex("test");
         ensureGreen();
 
-        int numDocs = 2000;
+        int numDocs = scaledRandomIntBetween(100, 2000);
+        if (numDocs % 2 == 1) {
+            numDocs++; // this test needs an even num of docs
+        }
+        logger.info("Bulk-Indexing {} docs", numDocs);
         BulkRequestBuilder builder = client().prepareBulk();
         for (int i = 0; i < numDocs; i++) {
             builder.add(
@@ -356,11 +360,11 @@ public void testBulkIndexingWhileInitializing() throws Exception {
                         .put(indexSettings())
                         .put("index.number_of_replicas", replica)));
 
-        int numDocs = 5000;
-        int bulk = 50;
+        int numDocs = scaledRandomIntBetween(100, 5000);
+        int bulk = scaledRandomIntBetween(1, 99);
         for (int i = 0; i < numDocs; ) {
-            BulkRequestBuilder builder = client().prepareBulk();
-            for (int j = 0; j < bulk; j++, i++) {
+            final BulkRequestBuilder builder = client().prepareBulk();
+            for (int j = 0; j < bulk && i < numDocs; j++, i++) {
                 builder.add(client().prepareIndex("test", "type1", Integer.toString(i)).setSource("val", i));
             }
             logger.info("bulk indexing {}-{}", i - bulk, i - 1);

From e8467f0978154f3419e479d3030348e6bb1c7e1b Mon Sep 17 00:00:00 2001
From: Boaz Leskes <b.leskes@gmail.com>
Date: Tue, 8 Apr 2014 10:28:31 +0200
Subject: [PATCH 41/54] Failed shards could be re-assigned to the same nodes if
 multiple replicas failed at once

After a shard fails on a node we assign a new replica on another node. This is important in order to avoid failing again due to node specific problems. In the rare case where two different replicas of the same shard failed in a short time span, we may fail to do so and assign one of them back to the node it's currently on. This happens if both shard failed events are processed within the same batch on the master.

Closes #5725
---
 .../routing/allocation/RoutingAllocation.java | 17 ++++-
 .../cluster/IndicesClusterStateService.java   |  6 +-
 .../allocation/FailedShardsRoutingTests.java  | 74 ++++++++++++++++---
 3 files changed, 80 insertions(+), 17 deletions(-)

diff --git a/src/main/java/org/elasticsearch/cluster/routing/allocation/RoutingAllocation.java b/src/main/java/org/elasticsearch/cluster/routing/allocation/RoutingAllocation.java
index 07c46fe76415c..71f2adb430a69 100644
--- a/src/main/java/org/elasticsearch/cluster/routing/allocation/RoutingAllocation.java
+++ b/src/main/java/org/elasticsearch/cluster/routing/allocation/RoutingAllocation.java
@@ -29,7 +29,9 @@
 import org.elasticsearch.index.shard.ShardId;
 
 import java.util.HashMap;
+import java.util.HashSet;
 import java.util.Map;
+import java.util.Set;
 
 /**
  * The {@link RoutingAllocation} keep the state of the current allocation
@@ -107,7 +109,7 @@ public RoutingExplanations explanations() {
 
     private final ClusterInfo clusterInfo;
 
-    private Map<ShardId, String> ignoredShardToNodes = null;
+    private Map<ShardId, Set<String>> ignoredShardToNodes = null;
 
     private boolean ignoreDisable = false;
 
@@ -199,11 +201,20 @@ public void addIgnoreShardForNode(ShardId shardId, String nodeId) {
         if (ignoredShardToNodes == null) {
             ignoredShardToNodes = new HashMap<>();
         }
-        ignoredShardToNodes.put(shardId, nodeId);
+        Set<String> nodes = ignoredShardToNodes.get(shardId);
+        if (nodes == null) {
+            nodes = new HashSet<>();
+            ignoredShardToNodes.put(shardId, nodes);
+        }
+        nodes.add(nodeId);
     }
 
     public boolean shouldIgnoreShardForNode(ShardId shardId, String nodeId) {
-        return ignoredShardToNodes != null && nodeId.equals(ignoredShardToNodes.get(shardId));
+        if (ignoredShardToNodes == null) {
+            return false;
+        }
+        Set<String> nodes = ignoredShardToNodes.get(shardId);
+        return nodes != null && nodes.contains(nodeId);
     }
 
     /**
diff --git a/src/main/java/org/elasticsearch/indices/cluster/IndicesClusterStateService.java b/src/main/java/org/elasticsearch/indices/cluster/IndicesClusterStateService.java
index f02edb5432178..c7c140f77f4b0 100644
--- a/src/main/java/org/elasticsearch/indices/cluster/IndicesClusterStateService.java
+++ b/src/main/java/org/elasticsearch/indices/cluster/IndicesClusterStateService.java
@@ -60,11 +60,7 @@
 import org.elasticsearch.index.shard.service.IndexShard;
 import org.elasticsearch.index.shard.service.InternalIndexShard;
 import org.elasticsearch.indices.IndicesService;
-import org.elasticsearch.indices.recovery.RecoveryFailedException;
-import org.elasticsearch.indices.recovery.RecoveryState;
-import org.elasticsearch.indices.recovery.RecoveryStatus;
-import org.elasticsearch.indices.recovery.RecoveryTarget;
-import org.elasticsearch.indices.recovery.StartRecoveryRequest;
+import org.elasticsearch.indices.recovery.*;
 import org.elasticsearch.threadpool.ThreadPool;
 
 import java.util.HashMap;
diff --git a/src/test/java/org/elasticsearch/cluster/routing/allocation/FailedShardsRoutingTests.java b/src/test/java/org/elasticsearch/cluster/routing/allocation/FailedShardsRoutingTests.java
index bf21cef731e75..26abdb606402b 100644
--- a/src/test/java/org/elasticsearch/cluster/routing/allocation/FailedShardsRoutingTests.java
+++ b/src/test/java/org/elasticsearch/cluster/routing/allocation/FailedShardsRoutingTests.java
@@ -23,10 +23,7 @@
 import org.elasticsearch.cluster.metadata.IndexMetaData;
 import org.elasticsearch.cluster.metadata.MetaData;
 import org.elasticsearch.cluster.node.DiscoveryNodes;
-import org.elasticsearch.cluster.routing.ImmutableShardRouting;
-import org.elasticsearch.cluster.routing.RoutingNodes;
-import org.elasticsearch.cluster.routing.RoutingTable;
-import org.elasticsearch.cluster.routing.ShardRouting;
+import org.elasticsearch.cluster.routing.*;
 import org.elasticsearch.cluster.routing.allocation.command.AllocationCommands;
 import org.elasticsearch.cluster.routing.allocation.command.MoveAllocationCommand;
 import org.elasticsearch.common.logging.ESLogger;
@@ -34,6 +31,8 @@
 import org.elasticsearch.test.ElasticsearchAllocationTestCase;
 import org.junit.Test;
 
+import java.util.ArrayList;
+
 import static org.elasticsearch.cluster.routing.ShardRoutingState.*;
 import static org.elasticsearch.common.settings.ImmutableSettings.settingsBuilder;
 import static org.hamcrest.Matchers.*;
@@ -63,8 +62,8 @@ public void testFailedShardPrimaryRelocatingToAndFrom() {
 
         logger.info("--> adding 2 nodes on same rack and do rerouting");
         clusterState = ClusterState.builder(clusterState).nodes(DiscoveryNodes.builder()
-                .put(newNode("node1"))
-                .put(newNode("node2"))
+                        .put(newNode("node1"))
+                        .put(newNode("node2"))
         ).build();
 
         RoutingAllocation.Result rerouteResult = allocation.reroute(clusterState);
@@ -85,7 +84,7 @@ public void testFailedShardPrimaryRelocatingToAndFrom() {
 
         logger.info("--> adding additional node");
         clusterState = ClusterState.builder(clusterState).nodes(DiscoveryNodes.builder(clusterState.nodes())
-                .put(newNode("node3"))
+                        .put(newNode("node3"))
         ).build();
         rerouteResult = allocation.reroute(clusterState);
         clusterState = ClusterState.builder(clusterState).routingTable(rerouteResult.routingTable()).build();
@@ -101,7 +100,7 @@ public void testFailedShardPrimaryRelocatingToAndFrom() {
 
         logger.info("--> moving primary shard to node3");
         rerouteResult = allocation.reroute(clusterState, new AllocationCommands(
-                new MoveAllocationCommand(clusterState.routingTable().index("test").shard(0).primaryShard().shardId(), clusterState.routingTable().index("test").shard(0).primaryShard().currentNodeId(), "node3"))
+                        new MoveAllocationCommand(clusterState.routingTable().index("test").shard(0).primaryShard().shardId(), clusterState.routingTable().index("test").shard(0).primaryShard().currentNodeId(), "node3"))
         );
         assertThat(rerouteResult.changed(), equalTo(true));
         clusterState = ClusterState.builder(clusterState).routingTable(rerouteResult.routingTable()).build();
@@ -117,7 +116,7 @@ public void testFailedShardPrimaryRelocatingToAndFrom() {
 
         logger.info("--> moving primary shard to node3");
         rerouteResult = allocation.reroute(clusterState, new AllocationCommands(
-                new MoveAllocationCommand(clusterState.routingTable().index("test").shard(0).primaryShard().shardId(), clusterState.routingTable().index("test").shard(0).primaryShard().currentNodeId(), "node3"))
+                        new MoveAllocationCommand(clusterState.routingTable().index("test").shard(0).primaryShard().shardId(), clusterState.routingTable().index("test").shard(0).primaryShard().currentNodeId(), "node3"))
         );
         assertThat(rerouteResult.changed(), equalTo(true));
         clusterState = ClusterState.builder(clusterState).routingTable(rerouteResult.routingTable()).build();
@@ -273,6 +272,63 @@ public void firstAllocationFailureSingleNode() {
         assertThat(strategy.applyFailedShard(clusterState, new ImmutableShardRouting("test", 0, "node1", true, INITIALIZING, 0)).changed(), equalTo(false));
     }
 
+    @Test
+    public void singleShardMultipleAllocationFailures() {
+        AllocationService strategy = createAllocationService(settingsBuilder()
+                .put("cluster.routing.allocation.concurrent_recoveries", 10)
+                .put("cluster.routing.allocation.allow_rebalance", "always")
+                .build());
+
+        logger.info("Building initial routing table");
+        int numberOfReplicas = scaledRandomIntBetween(2, 10);
+        MetaData metaData = MetaData.builder()
+                .put(IndexMetaData.builder("test").numberOfShards(1).numberOfReplicas(numberOfReplicas))
+                .build();
+
+        RoutingTable routingTable = RoutingTable.builder()
+                .addAsNew(metaData.index("test"))
+                .build();
+
+        ClusterState clusterState = ClusterState.builder(org.elasticsearch.cluster.ClusterName.DEFAULT).metaData(metaData).routingTable(routingTable).build();
+
+        logger.info("Adding {} nodes and performing rerouting", numberOfReplicas + 1);
+        DiscoveryNodes.Builder nodeBuilder = DiscoveryNodes.builder();
+        for (int i = 0; i < numberOfReplicas + 1; i++) {
+            nodeBuilder.put(newNode("node" + Integer.toString(i)));
+        }
+        clusterState = ClusterState.builder(clusterState).nodes(nodeBuilder).build();
+        while (!clusterState.routingTable().shardsWithState(UNASSIGNED).isEmpty()) {
+            // start all initializing
+            clusterState = ClusterState.builder(clusterState)
+                    .routingTable(strategy
+                                    .applyStartedShards(clusterState, clusterState.routingTable().shardsWithState(INITIALIZING)).routingTable()
+                    )
+                    .build();
+            // and assign more unassigned
+            clusterState = ClusterState.builder(clusterState).routingTable(strategy.reroute(clusterState).routingTable()).build();
+        }
+
+        int shardsToFail = randomIntBetween(1, numberOfReplicas);
+        ArrayList<ShardRouting> failedShards = new ArrayList<>();
+        RoutingNodes routingNodes = clusterState.routingNodes();
+        for (int i = 0; i < shardsToFail; i++) {
+            String n = "node" + Integer.toString(randomInt(numberOfReplicas));
+            logger.info("failing shard on node [{}]", n);
+            ShardRouting shardToFail = routingNodes.node(n).get(0);
+            failedShards.add(new MutableShardRouting(shardToFail));
+        }
+
+        routingTable = strategy.applyFailedShards(clusterState, failedShards).routingTable();
+
+        clusterState = ClusterState.builder(clusterState).routingTable(routingTable).build();
+        routingNodes = clusterState.routingNodes();
+        for (ShardRouting failedShard : failedShards) {
+            if (!routingNodes.node(failedShard.currentNodeId()).isEmpty()) {
+                fail("shard " + failedShard + " was re-assigned to it's node");
+            }
+        }
+    }
+
     @Test
     public void firstAllocationFailureTwoNodes() {
         AllocationService strategy = createAllocationService(settingsBuilder()

From de13d707ed6e091504946ca1007cf52c695ec72a Mon Sep 17 00:00:00 2001
From: Simon Willnauer <simonw@apache.org>
Date: Tue, 8 Apr 2014 12:40:37 +0200
Subject: [PATCH 42/54] [TEST] Wait for LANGUID events to be processed before
 pulling stats

---
 .../action/admin/cluster/stats/ClusterStatsTests.java          | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/src/test/java/org/elasticsearch/action/admin/cluster/stats/ClusterStatsTests.java b/src/test/java/org/elasticsearch/action/admin/cluster/stats/ClusterStatsTests.java
index c82eef9db6169..c2eb77e37da7c 100644
--- a/src/test/java/org/elasticsearch/action/admin/cluster/stats/ClusterStatsTests.java
+++ b/src/test/java/org/elasticsearch/action/admin/cluster/stats/ClusterStatsTests.java
@@ -23,6 +23,7 @@
 import org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse;
 import org.elasticsearch.action.admin.cluster.health.ClusterHealthStatus;
 import org.elasticsearch.client.Requests;
+import org.elasticsearch.common.Priority;
 import org.elasticsearch.common.settings.ImmutableSettings;
 import org.elasticsearch.monitor.sigar.SigarService;
 import org.elasticsearch.test.ElasticsearchIntegrationTest;
@@ -45,7 +46,7 @@ private void assertCounts(ClusterStatsNodes.Counts counts, int total, int master
 
     private void waitForNodes(int numNodes) {
         ClusterHealthResponse actionGet = client().admin().cluster()
-                .health(Requests.clusterHealthRequest().waitForNodes(Integer.toString(numNodes))).actionGet();
+                .health(Requests.clusterHealthRequest().waitForEvents(Priority.LANGUID).waitForNodes(Integer.toString(numNodes))).actionGet();
         assertThat(actionGet.isTimedOut(), is(false));
     }
 

From 7eb8b0d0c0b2ef3adcfd7b1e10ea0ca9a50c1adc Mon Sep 17 00:00:00 2001
From: Simon Willnauer <simonw@apache.org>
Date: Tue, 8 Apr 2014 15:14:08 +0200
Subject: [PATCH 43/54] [TEST] Log where locks are created from if they are
 still open on close()

---
 .../test/store/MockDirectoryHelper.java       | 25 ++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/src/test/java/org/elasticsearch/test/store/MockDirectoryHelper.java b/src/test/java/org/elasticsearch/test/store/MockDirectoryHelper.java
index 4e9d07d286ec4..9050ae9204fca 100644
--- a/src/test/java/org/elasticsearch/test/store/MockDirectoryHelper.java
+++ b/src/test/java/org/elasticsearch/test/store/MockDirectoryHelper.java
@@ -21,6 +21,7 @@
 
 import com.carrotsearch.randomizedtesting.SeedUtils;
 import org.apache.lucene.store.Directory;
+import org.apache.lucene.store.Lock;
 import org.apache.lucene.store.MMapDirectory;
 import org.apache.lucene.store.MockDirectoryWrapper;
 import org.apache.lucene.store.MockDirectoryWrapper.Throttling;
@@ -36,11 +37,12 @@
 import org.elasticsearch.index.store.fs.NioFsDirectoryService;
 import org.elasticsearch.index.store.fs.SimpleFsDirectoryService;
 import org.elasticsearch.index.store.ram.RamDirectoryService;
-import org.elasticsearch.test.TestCluster;
 
 import java.io.IOException;
+import java.util.Map;
 import java.util.Random;
 import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
 
 public class MockDirectoryHelper {
     public static final String RANDOM_IO_EXCEPTION_RATE = "index.store.mock.random.io_exception_rate";
@@ -124,6 +126,7 @@ public DirectoryService randomRamDirectoryService() {
 
     public static final class ElasticsearchMockDirectoryWrapper extends MockDirectoryWrapper {
 
+        private final Map<String, Exception> locks = new ConcurrentHashMap<>();
         private final ESLogger logger;
         private final boolean crash;
         private RuntimeException closeException;
@@ -136,6 +139,9 @@ public ElasticsearchMockDirectoryWrapper(Random random, Directory delegate, ESLo
 
         @Override
         public synchronized void close() throws IOException {
+            for (Exception ex : locks.values()) {
+                logger.info("Lock still open - opened from: ", ex);
+            }
             try {
                 super.close();
             } catch (RuntimeException ex) {
@@ -159,5 +165,22 @@ public synchronized void crash() throws IOException {
                 super.crash();
             }
         }
+
+        @Override
+        public Lock makeLock(String lockName) {
+            Lock l = super.makeLock(lockName);
+            if (l != null) {
+                locks.put(lockName, new Exception());
+            }
+            return l;
+        }
+
+        public void clearLock(String lockName) throws IOException {
+            try {
+                super.clearLock(lockName);
+            } finally {
+                locks.remove(lockName);
+            }
+        }
     }
 }

From a98b3fa1ffecbb72dd406338500bc99e21cd90b5 Mon Sep 17 00:00:00 2001
From: Simon Willnauer <simonw@apache.org>
Date: Tue, 8 Apr 2014 16:18:15 +0200
Subject: [PATCH 44/54] Revert "[TEST] Log where locks are created from if they
 are still open on close()"

This reverts commit 7eb8b0d0c0b2ef3adcfd7b1e10ea0ca9a50c1adc.
---
 .../test/store/MockDirectoryHelper.java       | 25 +------------------
 1 file changed, 1 insertion(+), 24 deletions(-)

diff --git a/src/test/java/org/elasticsearch/test/store/MockDirectoryHelper.java b/src/test/java/org/elasticsearch/test/store/MockDirectoryHelper.java
index 9050ae9204fca..4e9d07d286ec4 100644
--- a/src/test/java/org/elasticsearch/test/store/MockDirectoryHelper.java
+++ b/src/test/java/org/elasticsearch/test/store/MockDirectoryHelper.java
@@ -21,7 +21,6 @@
 
 import com.carrotsearch.randomizedtesting.SeedUtils;
 import org.apache.lucene.store.Directory;
-import org.apache.lucene.store.Lock;
 import org.apache.lucene.store.MMapDirectory;
 import org.apache.lucene.store.MockDirectoryWrapper;
 import org.apache.lucene.store.MockDirectoryWrapper.Throttling;
@@ -37,12 +36,11 @@
 import org.elasticsearch.index.store.fs.NioFsDirectoryService;
 import org.elasticsearch.index.store.fs.SimpleFsDirectoryService;
 import org.elasticsearch.index.store.ram.RamDirectoryService;
+import org.elasticsearch.test.TestCluster;
 
 import java.io.IOException;
-import java.util.Map;
 import java.util.Random;
 import java.util.Set;
-import java.util.concurrent.ConcurrentHashMap;
 
 public class MockDirectoryHelper {
     public static final String RANDOM_IO_EXCEPTION_RATE = "index.store.mock.random.io_exception_rate";
@@ -126,7 +124,6 @@ public DirectoryService randomRamDirectoryService() {
 
     public static final class ElasticsearchMockDirectoryWrapper extends MockDirectoryWrapper {
 
-        private final Map<String, Exception> locks = new ConcurrentHashMap<>();
         private final ESLogger logger;
         private final boolean crash;
         private RuntimeException closeException;
@@ -139,9 +136,6 @@ public ElasticsearchMockDirectoryWrapper(Random random, Directory delegate, ESLo
 
         @Override
         public synchronized void close() throws IOException {
-            for (Exception ex : locks.values()) {
-                logger.info("Lock still open - opened from: ", ex);
-            }
             try {
                 super.close();
             } catch (RuntimeException ex) {
@@ -165,22 +159,5 @@ public synchronized void crash() throws IOException {
                 super.crash();
             }
         }
-
-        @Override
-        public Lock makeLock(String lockName) {
-            Lock l = super.makeLock(lockName);
-            if (l != null) {
-                locks.put(lockName, new Exception());
-            }
-            return l;
-        }
-
-        public void clearLock(String lockName) throws IOException {
-            try {
-                super.clearLock(lockName);
-            } finally {
-                locks.remove(lockName);
-            }
-        }
     }
 }

From 5b6fd6de428a1ae438b0a269cf6842cc55a1c681 Mon Sep 17 00:00:00 2001
From: Simon Willnauer <simonw@apache.org>
Date: Tue, 8 Apr 2014 17:46:29 +0200
Subject: [PATCH 45/54] [TEST] Fix testRandomDirectoryIOExceptions to wait for
 green on reopen & add more assertions

---
 .../SearchWithRandomExceptionsTests.java      | 25 +++++++++++++------
 1 file changed, 18 insertions(+), 7 deletions(-)

diff --git a/src/test/java/org/elasticsearch/search/basic/SearchWithRandomExceptionsTests.java b/src/test/java/org/elasticsearch/search/basic/SearchWithRandomExceptionsTests.java
index fc883d9c037b1..9f3475f3b6c13 100644
--- a/src/test/java/org/elasticsearch/search/basic/SearchWithRandomExceptionsTests.java
+++ b/src/test/java/org/elasticsearch/search/basic/SearchWithRandomExceptionsTests.java
@@ -50,6 +50,7 @@
 
 import static org.elasticsearch.common.settings.ImmutableSettings.settingsBuilder;
 import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked;
+import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertHitCount;
 
 public class SearchWithRandomExceptionsTests extends ElasticsearchIntegrationTest {
     
@@ -101,7 +102,7 @@ public void testRandomDirectoryIOExceptions() throws IOException, InterruptedExc
             numInitialDocs = between(10, 100);
             ensureYellow();
             for (int i = 0; i < numInitialDocs ; i++) {
-                client().prepareIndex("test", "type", "" + i).setTimeout(TimeValue.timeValueSeconds(1)).setSource("test", "init").get();
+                client().prepareIndex("test", "initial", "" + i).setTimeout(TimeValue.timeValueSeconds(1)).setSource("test", "init").get();
             }
             client().admin().indices().prepareRefresh("test").execute().get();
             client().admin().indices().prepareFlush("test").execute().get();
@@ -141,14 +142,16 @@ public void testRandomDirectoryIOExceptions() throws IOException, InterruptedExc
         long numCreated = 0;
         boolean[] added = new boolean[numDocs];
         for (int i = 0; i < numDocs ; i++) {
+            added[i] = false;
             try {
-                IndexResponse indexResponse = client().prepareIndex("test", "type", "" + (i + numInitialDocs)).setTimeout(TimeValue.timeValueSeconds(1)).setSource("test", English.intToEnglish(i)).get();
+                IndexResponse indexResponse = client().prepareIndex("test", "type", Integer.toString(i)).setTimeout(TimeValue.timeValueSeconds(1)).setSource("test", English.intToEnglish(i)).get();
                 if (indexResponse.isCreated()) {
                     numCreated++;
                     added[i] = true;
                 }
             } catch (ElasticsearchException ex) {
             }
+
         }
         NumShards test = getNumShards("test");
         logger.info("Start Refresh");
@@ -160,11 +163,19 @@ public void testRandomDirectoryIOExceptions() throws IOException, InterruptedExc
         for (int i = 0; i < numSearches; i++) {
             try {
                 int docToQuery = between(0, numDocs-1);
+                long expectedResults = added[docToQuery] ? 1 : 0;
                 logger.info("Searching for [test:{}]", English.intToEnglish(docToQuery));
-                SearchResponse searchResponse = client().prepareSearch().setQuery(QueryBuilders.matchQuery("test", English.intToEnglish(docToQuery))).get();
+                SearchResponse searchResponse = client().prepareSearch().setTypes("type").setQuery(QueryBuilders.matchQuery("test", English.intToEnglish(docToQuery))).get();
                 logger.info("Successful shards: [{}]  numShards: [{}]", searchResponse.getSuccessfulShards(), test.numPrimaries);
+                if (searchResponse.getSuccessfulShards() == test.numPrimaries && !refreshFailed) {
+                    assertThat(searchResponse.getHits().getTotalHits(), Matchers.equalTo(expectedResults));
+                }
                 // check match all
-                searchResponse = client().prepareSearch().setQuery(QueryBuilders.matchAllQuery()).get();
+                searchResponse = client().prepareSearch().setTypes("type").setQuery(QueryBuilders.matchAllQuery()).get();
+                logger.info("Match all Successful shards: [{}]  numShards: [{}]", searchResponse.getSuccessfulShards(), test.numPrimaries);
+                if (searchResponse.getSuccessfulShards() == test.numPrimaries && !refreshFailed) {
+                    assertThat(searchResponse.getHits().getTotalHits(), Matchers.equalTo(numCreated));
+                }
             } catch (SearchPhaseExecutionException ex) {
                 if (expectAllShardsFailed || refreshResponse.getSuccessfulShards() == 0) {
                     logger.info("expected SearchPhaseException: [{}]", ex.getMessage());
@@ -181,9 +192,9 @@ public void testRandomDirectoryIOExceptions() throws IOException, InterruptedExc
                     .put(MockDirectoryHelper.RANDOM_IO_EXCEPTION_RATE, 0)
                     .put(MockDirectoryHelper.RANDOM_IO_EXCEPTION_RATE_ON_OPEN, 0));
             client().admin().indices().prepareOpen("test").execute().get();
-            ensureYellow();
-            SearchResponse searchResponse = client().prepareSearch().setQuery(QueryBuilders.matchQuery("test", "init")).get();
-            assertThat(searchResponse.getHits().totalHits(), Matchers.equalTo(numInitialDocs));
+            ensureGreen();
+            SearchResponse searchResponse = client().prepareSearch().setTypes("initial").setQuery(QueryBuilders.matchQuery("test", "init")).get();
+            assertHitCount(searchResponse, numInitialDocs);
         }
     }
 

From adc9a250ab942b523ebd87464713ef082f1c8115 Mon Sep 17 00:00:00 2001
From: Simon Willnauer <simonw@apache.org>
Date: Tue, 8 Apr 2014 14:20:34 +0200
Subject: [PATCH 46/54] Fix P/C assertions for rewrite reader

Inner queries must be rewritten as soon as a weight is pulled ie. must
be non-null.

Closes #5731
---
 .../child/ChildrenConstantScoreQuery.java     | 56 ++++++++++---------
 .../index/search/child/ChildrenQuery.java     | 11 ++--
 .../child/ParentConstantScoreQuery.java       | 42 +++++++-------
 .../index/search/child/ParentQuery.java       | 12 ++--
 4 files changed, 60 insertions(+), 61 deletions(-)

diff --git a/src/main/java/org/elasticsearch/index/search/child/ChildrenConstantScoreQuery.java b/src/main/java/org/elasticsearch/index/search/child/ChildrenConstantScoreQuery.java
index ce6e46ef2e946..2475531573e9c 100644
--- a/src/main/java/org/elasticsearch/index/search/child/ChildrenConstantScoreQuery.java
+++ b/src/main/java/org/elasticsearch/index/search/child/ChildrenConstantScoreQuery.java
@@ -98,37 +98,41 @@ public Query clone() {
 
     @Override
     public Weight createWeight(IndexSearcher searcher) throws IOException {
-        SearchContext searchContext = SearchContext.current();
-        BytesRefHash parentIds = new BytesRefHash(512, searchContext.bigArrays());
-        ParentIdCollector collector = new ParentIdCollector(parentType, parentChildIndexFieldData, parentIds);
-        final Query childQuery;
-        if (rewrittenChildQuery == null) {
-            childQuery = rewrittenChildQuery = searcher.rewrite(originalChildQuery);
-        } else {
+        final SearchContext searchContext = SearchContext.current();
+        final BytesRefHash parentIds = new BytesRefHash(512, searchContext.bigArrays());
+        boolean releaseParentIds = true;
+        try {
+            final ParentIdCollector collector = new ParentIdCollector(parentType, parentChildIndexFieldData, parentIds);
+            assert rewrittenChildQuery != null;
             assert rewriteIndexReader == searcher.getIndexReader()  : "not equal, rewriteIndexReader=" + rewriteIndexReader + " searcher.getIndexReader()=" + searcher.getIndexReader();
-            childQuery = rewrittenChildQuery;
-        }
-        IndexSearcher indexSearcher = new IndexSearcher(searcher.getIndexReader());
-        indexSearcher.setSimilarity(searcher.getSimilarity());
-        indexSearcher.search(childQuery, collector);
+            final Query childQuery = rewrittenChildQuery;
+            IndexSearcher indexSearcher = new IndexSearcher(searcher.getIndexReader());
+            indexSearcher.setSimilarity(searcher.getSimilarity());
+            indexSearcher.search(childQuery, collector);
 
-        long remaining = parentIds.size();
-        if (remaining == 0) {
-            Releasables.release(parentIds);
-            return Queries.newMatchNoDocsQuery().createWeight(searcher);
-        }
+            long remaining = parentIds.size();
+            if (remaining == 0) {
+                return Queries.newMatchNoDocsQuery().createWeight(searcher);
+            }
 
-        Filter shortCircuitFilter = null;
-        if (remaining == 1) {
-            BytesRef id = parentIds.get(0, new BytesRef());
-            shortCircuitFilter = new TermFilter(new Term(UidFieldMapper.NAME, Uid.createUidAsBytes(parentType, id)));
-        } else if (remaining <= shortCircuitParentDocSet) {
-            shortCircuitFilter = new ParentIdsFilter(parentType, nonNestedDocsFilter, parentIds);
+            Filter shortCircuitFilter = null;
+            if (remaining == 1) {
+                BytesRef id = parentIds.get(0, new BytesRef());
+                shortCircuitFilter = new TermFilter(new Term(UidFieldMapper.NAME, Uid.createUidAsBytes(parentType, id)));
+            } else if (remaining <= shortCircuitParentDocSet) {
+                shortCircuitFilter = new ParentIdsFilter(parentType, nonNestedDocsFilter, parentIds);
+            }
+            final ParentWeight parentWeight = new ParentWeight(parentFilter, shortCircuitFilter, parentIds);
+            searchContext.addReleasable(parentWeight);
+            releaseParentIds = false;
+            return parentWeight;
+        } finally {
+           if (releaseParentIds) {
+               Releasables.release(parentIds);
+           }
         }
 
-        ParentWeight parentWeight = new ParentWeight(parentFilter, shortCircuitFilter, parentIds);
-        searchContext.addReleasable(parentWeight);
-        return parentWeight;
+
     }
 
     private final class ParentWeight extends Weight implements Releasable  {
diff --git a/src/main/java/org/elasticsearch/index/search/child/ChildrenQuery.java b/src/main/java/org/elasticsearch/index/search/child/ChildrenQuery.java
index f1c28a8c5a80f..ee01eea69c604 100644
--- a/src/main/java/org/elasticsearch/index/search/child/ChildrenQuery.java
+++ b/src/main/java/org/elasticsearch/index/search/child/ChildrenQuery.java
@@ -147,13 +147,10 @@ public void extractTerms(Set<Term> terms) {
     @Override
     public Weight createWeight(IndexSearcher searcher) throws IOException {
         SearchContext searchContext = SearchContext.current();
-        final Query childQuery;
-        if (rewrittenChildQuery == null) {
-            childQuery = rewrittenChildQuery = searcher.rewrite(originalChildQuery);
-        } else {
-            assert rewriteIndexReader == searcher.getIndexReader() : "not equal, rewriteIndexReader=" + rewriteIndexReader + " searcher.getIndexReader()=" + searcher.getIndexReader();
-            childQuery = rewrittenChildQuery;
-        }
+        assert rewrittenChildQuery != null;
+        assert rewriteIndexReader == searcher.getIndexReader() : "not equal, rewriteIndexReader=" + rewriteIndexReader + " searcher.getIndexReader()=" + searcher.getIndexReader();
+        final Query childQuery = rewrittenChildQuery;
+
         IndexSearcher indexSearcher = new IndexSearcher(searcher.getIndexReader());
         indexSearcher.setSimilarity(searcher.getSimilarity());
 
diff --git a/src/main/java/org/elasticsearch/index/search/child/ParentConstantScoreQuery.java b/src/main/java/org/elasticsearch/index/search/child/ParentConstantScoreQuery.java
index 11095773f2c5c..2c6893094949e 100644
--- a/src/main/java/org/elasticsearch/index/search/child/ParentConstantScoreQuery.java
+++ b/src/main/java/org/elasticsearch/index/search/child/ParentConstantScoreQuery.java
@@ -87,29 +87,31 @@ public Query clone() {
 
     @Override
     public Weight createWeight(IndexSearcher searcher) throws IOException {
-        SearchContext searchContext = SearchContext.current();
-        BytesRefHash parentIds = new BytesRefHash(512, searchContext.bigArrays());
-        ParentIdsCollector collector = new ParentIdsCollector(parentType, parentChildIndexFieldData, parentIds);
-
-        final Query parentQuery;
-        if (rewrittenParentQuery != null) {
-            parentQuery = rewrittenParentQuery;
-        } else {
+        final SearchContext searchContext = SearchContext.current();
+        final BytesRefHash parentIds = new BytesRefHash(512, searchContext.bigArrays());
+        boolean releaseParentIds = true;
+        try {
+            ParentIdsCollector collector = new ParentIdsCollector(parentType, parentChildIndexFieldData, parentIds);
+            assert rewrittenParentQuery != null;
             assert rewriteIndexReader == searcher.getIndexReader() : "not equal, rewriteIndexReader=" + rewriteIndexReader + " searcher.getIndexReader()=" + searcher.getIndexReader();
-            parentQuery = rewrittenParentQuery = originalParentQuery.rewrite(searcher.getIndexReader());
-        }
-        IndexSearcher indexSearcher = new IndexSearcher(searcher.getIndexReader());
-        indexSearcher.setSimilarity(searcher.getSimilarity());
-        indexSearcher.search(parentQuery, collector);
+            final Query parentQuery = rewrittenParentQuery;
+            IndexSearcher indexSearcher = new IndexSearcher(searcher.getIndexReader());
+            indexSearcher.setSimilarity(searcher.getSimilarity());
+            indexSearcher.search(parentQuery, collector);
 
-        if (parentIds.size() == 0) {
-            Releasables.release(parentIds);
-            return Queries.newMatchNoDocsQuery().createWeight(searcher);
-        }
+            if (parentIds.size() == 0) {
+                return Queries.newMatchNoDocsQuery().createWeight(searcher);
+            }
 
-        ChildrenWeight childrenWeight = new ChildrenWeight(childrenFilter, parentIds);
-        searchContext.addReleasable(childrenWeight);
-        return childrenWeight;
+            final ChildrenWeight childrenWeight = new ChildrenWeight(childrenFilter, parentIds);
+            searchContext.addReleasable(childrenWeight);
+            releaseParentIds = false;
+            return childrenWeight;
+        } finally {
+            if (releaseParentIds) {
+                Releasables.release(parentIds);
+            }
+        }
     }
 
     private final class ChildrenWeight extends Weight implements Releasable {
diff --git a/src/main/java/org/elasticsearch/index/search/child/ParentQuery.java b/src/main/java/org/elasticsearch/index/search/child/ParentQuery.java
index 6750c7cde8f69..9aa0455e889ff 100644
--- a/src/main/java/org/elasticsearch/index/search/child/ParentQuery.java
+++ b/src/main/java/org/elasticsearch/index/search/child/ParentQuery.java
@@ -133,17 +133,13 @@ public Query clone() {
     @Override
     public Weight createWeight(IndexSearcher searcher) throws IOException {
         SearchContext searchContext = SearchContext.current();
-        ParentIdAndScoreCollector collector = new ParentIdAndScoreCollector(searchContext, parentChildIndexFieldData, parentType);
+        final ParentIdAndScoreCollector collector = new ParentIdAndScoreCollector(searchContext, parentChildIndexFieldData, parentType);
         ChildWeight childWeight;
         boolean releaseCollectorResource = true;
         try {
-            final Query parentQuery;
-            if (rewrittenParentQuery == null) {
-                parentQuery = rewrittenParentQuery = searcher.rewrite(originalParentQuery);
-            } else {
-                assert rewriteIndexReader == searcher.getIndexReader() : "not equal, rewriteIndexReader=" + rewriteIndexReader + " searcher.getIndexReader()=" + searcher.getIndexReader();
-                parentQuery = rewrittenParentQuery;
-            }
+            assert rewrittenParentQuery != null;
+            assert rewriteIndexReader == searcher.getIndexReader() : "not equal, rewriteIndexReader=" + rewriteIndexReader + " searcher.getIndexReader()=" + searcher.getIndexReader();
+            final Query  parentQuery = rewrittenParentQuery;
             IndexSearcher indexSearcher = new IndexSearcher(searcher.getIndexReader());
             indexSearcher.setSimilarity(searcher.getSimilarity());
             indexSearcher.search(parentQuery, collector);

From 6d966837d17065aaa1e0b5bd0645af9715cd96c7 Mon Sep 17 00:00:00 2001
From: Boaz Leskes <b.leskes@gmail.com>
Date: Tue, 8 Apr 2014 20:53:31 +0200
Subject: [PATCH 47/54] [Test] recoverWhileRelocating: Increase timeout while
 waiting for shards to replicate.

---
 .../org/elasticsearch/recovery/RecoveryWhileUnderLoadTests.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/test/java/org/elasticsearch/recovery/RecoveryWhileUnderLoadTests.java b/src/test/java/org/elasticsearch/recovery/RecoveryWhileUnderLoadTests.java
index 537c4ef6d82fa..c39b1f4a36184 100644
--- a/src/test/java/org/elasticsearch/recovery/RecoveryWhileUnderLoadTests.java
+++ b/src/test/java/org/elasticsearch/recovery/RecoveryWhileUnderLoadTests.java
@@ -264,7 +264,7 @@ public void recoverWhileRelocating() throws Exception {
             logger.info("--> bump up number of replicas to 1 and allow all nodes to hold the index");
             allowNodes("test", 3);
             assertAcked(client().admin().indices().prepareUpdateSettings("test").setSettings(settingsBuilder().put("number_of_replicas", 1)).get());
-            assertThat(client().admin().cluster().prepareHealth().setWaitForEvents(Priority.LANGUID).setTimeout("1m").setWaitForGreenStatus().execute().actionGet().isTimedOut(), equalTo(false));
+            assertThat(client().admin().cluster().prepareHealth().setWaitForEvents(Priority.LANGUID).setTimeout("5m").setWaitForGreenStatus().execute().actionGet().isTimedOut(), equalTo(false));
 
             logger.info("--> refreshing the index");
             refreshAndAssert();

From cd0c0de088686be7b440e4308c35e1b8fadb2b49 Mon Sep 17 00:00:00 2001
From: Simon Willnauer <simonw@apache.org>
Date: Wed, 9 Apr 2014 11:37:02 +0200
Subject: [PATCH 48/54] [TEST] RecoveryWhileunderLoadTests sometimes need
 higher timeouts

In some cases when we have a lot of docs with lots of shards
recovery takes longer than 1m causing the tests to fail before all
shards are recovered. This commit raises the timeout in this test to
5m max while it's rarely needed.

This commit also adds an assertion to ElasticsearchAssertions that
ensures that the cluster health requests are not hitting a timeout.
---
 .../recovery/RecoveryWhileUnderLoadTests.java | 19 ++++++++++---------
 .../hamcrest/ElasticsearchAssertions.java     | 10 ++++++++++
 2 files changed, 20 insertions(+), 9 deletions(-)

diff --git a/src/test/java/org/elasticsearch/recovery/RecoveryWhileUnderLoadTests.java b/src/test/java/org/elasticsearch/recovery/RecoveryWhileUnderLoadTests.java
index c39b1f4a36184..ca2f8a5b05044 100644
--- a/src/test/java/org/elasticsearch/recovery/RecoveryWhileUnderLoadTests.java
+++ b/src/test/java/org/elasticsearch/recovery/RecoveryWhileUnderLoadTests.java
@@ -43,6 +43,7 @@
 import static org.elasticsearch.common.settings.ImmutableSettings.settingsBuilder;
 import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery;
 import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.*;
+import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertNoTimeout;
 import static org.hamcrest.Matchers.equalTo;
 
 public class RecoveryWhileUnderLoadTests extends ElasticsearchIntegrationTest {
@@ -87,7 +88,7 @@ public void recoverWhileUnderLoadAllocateBackupsTest() throws Exception {
 
             logger.info("--> waiting for GREEN health status ...");
             // make sure the cluster state is green, and all has been recovered
-            assertThat(client().admin().cluster().prepareHealth().setWaitForEvents(Priority.LANGUID).setTimeout("1m").setWaitForGreenStatus().setWaitForNodes(">=2").execute().actionGet().isTimedOut(), equalTo(false));
+            assertNoTimeout(client().admin().cluster().prepareHealth().setWaitForEvents(Priority.LANGUID).setTimeout("5m").setWaitForGreenStatus().setWaitForNodes(">=2"));
 
             logger.info("--> waiting for {} docs to be indexed ...", totalNumDocs);
             waitForDocs(totalNumDocs, indexer);
@@ -140,7 +141,7 @@ public void recoverWhileUnderLoadAllocateBackupsRelocatePrimariesTest() throws E
             allowNodes("test", 4);
 
             logger.info("--> waiting for GREEN health status ...");
-            assertThat(client().admin().cluster().prepareHealth().setWaitForEvents(Priority.LANGUID).setTimeout("1m").setWaitForGreenStatus().setWaitForNodes(">=4").execute().actionGet().isTimedOut(), equalTo(false));
+            assertNoTimeout(client().admin().cluster().prepareHealth().setWaitForEvents(Priority.LANGUID).setTimeout("5m").setWaitForGreenStatus().setWaitForNodes(">=4"));
 
 
             logger.info("--> waiting for {} docs to be indexed ...", totalNumDocs);
@@ -195,7 +196,7 @@ public void recoverWhileUnderLoadWithNodeShutdown() throws Exception {
             allowNodes("test", 4);
 
             logger.info("--> waiting for GREEN health status ...");
-            assertThat(client().admin().cluster().prepareHealth().setWaitForEvents(Priority.LANGUID).setTimeout("1m").setWaitForGreenStatus().setWaitForNodes(">=4").execute().actionGet().isTimedOut(), equalTo(false));
+            assertNoTimeout(client().admin().cluster().prepareHealth().setWaitForEvents(Priority.LANGUID).setTimeout("5m").setWaitForGreenStatus().setWaitForNodes(">=4"));
 
 
             logger.info("--> waiting for {} docs to be indexed ...", totalNumDocs);
@@ -207,23 +208,23 @@ public void recoverWhileUnderLoadWithNodeShutdown() throws Exception {
             logger.info("--> allow 3 nodes for index [test] ...");
             allowNodes("test", 3);
             logger.info("--> waiting for GREEN health status ...");
-            assertThat(client().admin().cluster().prepareHealth().setWaitForEvents(Priority.LANGUID).setTimeout("1m").setWaitForGreenStatus().setWaitForNodes(">=3").execute().actionGet().isTimedOut(), equalTo(false));
+            assertNoTimeout(client().admin().cluster().prepareHealth().setWaitForEvents(Priority.LANGUID).setTimeout("5m").setWaitForGreenStatus().setWaitForNodes(">=3"));
 
             logger.info("--> allow 2 nodes for index [test] ...");
             allowNodes("test", 2);
             logger.info("--> waiting for GREEN health status ...");
-            assertThat(client().admin().cluster().prepareHealth().setWaitForEvents(Priority.LANGUID).setTimeout("1m").setWaitForGreenStatus().setWaitForNodes(">=2").execute().actionGet().isTimedOut(), equalTo(false));
+            assertNoTimeout(client().admin().cluster().prepareHealth().setWaitForEvents(Priority.LANGUID).setTimeout("5m").setWaitForGreenStatus().setWaitForNodes(">=2"));
 
             logger.info("--> allow 1 nodes for index [test] ...");
             allowNodes("test", 1);
             logger.info("--> waiting for YELLOW health status ...");
-            assertThat(client().admin().cluster().prepareHealth().setWaitForEvents(Priority.LANGUID).setTimeout("1m").setWaitForYellowStatus().setWaitForNodes(">=1").execute().actionGet().isTimedOut(), equalTo(false));
+            assertNoTimeout(client().admin().cluster().prepareHealth().setWaitForEvents(Priority.LANGUID).setTimeout("5m").setWaitForYellowStatus().setWaitForNodes(">=1"));
 
             logger.info("--> marking and waiting for indexing threads to stop ...");
             indexer.stop();
             logger.info("--> indexing threads stopped");
 
-            assertThat(client().admin().cluster().prepareHealth().setWaitForEvents(Priority.LANGUID).setTimeout("1m").setWaitForYellowStatus().setWaitForNodes(">=1").execute().actionGet().isTimedOut(), equalTo(false));
+            assertNoTimeout(client().admin().cluster().prepareHealth().setWaitForEvents(Priority.LANGUID).setTimeout("5m").setWaitForYellowStatus().setWaitForNodes(">=1"));
 
             logger.info("--> refreshing the index");
             refreshAndAssert();
@@ -254,7 +255,7 @@ public void recoverWhileRelocating() throws Exception {
                 allowNodes = 2 / allowNodes;
                 allowNodes("test", allowNodes);
                 logger.info("--> waiting for GREEN health status ...");
-                assertThat(client().admin().cluster().prepareHealth().setWaitForEvents(Priority.LANGUID).setTimeout("1m").setWaitForGreenStatus().execute().actionGet().isTimedOut(), equalTo(false));
+                assertNoTimeout(client().admin().cluster().prepareHealth().setWaitForEvents(Priority.LANGUID).setTimeout("5m").setWaitForGreenStatus());
             }
 
             logger.info("--> marking and waiting for indexing threads to stop ...");
@@ -264,7 +265,7 @@ public void recoverWhileRelocating() throws Exception {
             logger.info("--> bump up number of replicas to 1 and allow all nodes to hold the index");
             allowNodes("test", 3);
             assertAcked(client().admin().indices().prepareUpdateSettings("test").setSettings(settingsBuilder().put("number_of_replicas", 1)).get());
-            assertThat(client().admin().cluster().prepareHealth().setWaitForEvents(Priority.LANGUID).setTimeout("5m").setWaitForGreenStatus().execute().actionGet().isTimedOut(), equalTo(false));
+            assertNoTimeout(client().admin().cluster().prepareHealth().setWaitForEvents(Priority.LANGUID).setTimeout("5m").setWaitForGreenStatus());
 
             logger.info("--> refreshing the index");
             refreshAndAssert();
diff --git a/src/test/java/org/elasticsearch/test/hamcrest/ElasticsearchAssertions.java b/src/test/java/org/elasticsearch/test/hamcrest/ElasticsearchAssertions.java
index 76882dadd5368..054f6c3da7cf6 100644
--- a/src/test/java/org/elasticsearch/test/hamcrest/ElasticsearchAssertions.java
+++ b/src/test/java/org/elasticsearch/test/hamcrest/ElasticsearchAssertions.java
@@ -28,6 +28,8 @@
 import org.elasticsearch.action.ActionRequest;
 import org.elasticsearch.action.ActionRequestBuilder;
 import org.elasticsearch.action.ShardOperationFailedException;
+import org.elasticsearch.action.admin.cluster.health.ClusterHealthRequestBuilder;
+import org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse;
 import org.elasticsearch.action.admin.indices.delete.DeleteIndexRequestBuilder;
 import org.elasticsearch.action.admin.indices.delete.DeleteIndexResponse;
 import org.elasticsearch.action.count.CountResponse;
@@ -74,6 +76,14 @@ public static void assertAcked(AcknowledgedRequestBuilder<?, ?, ?> builder) {
         assertAcked(builder.get());
     }
 
+    public static void assertNoTimeout(ClusterHealthRequestBuilder requestBuilder) {
+        assertNoTimeout(requestBuilder.get());
+    }
+
+    public static void assertNoTimeout(ClusterHealthResponse response) {
+        assertThat("ClusterHealthResponse has timed out", response.isTimedOut(), is(false));
+    }
+
     public static void assertAcked(AcknowledgedResponse response) {
         assertThat(response.getClass().getSimpleName() + " failed - not acked", response.isAcknowledged(), equalTo(true));
         assertVersionSerializable(response);

From a9c8624c67ab5ce7f659decce75e97fa420131e8 Mon Sep 17 00:00:00 2001
From: Simon Willnauer <simonw@apache.org>
Date: Wed, 9 Apr 2014 11:45:40 +0200
Subject: [PATCH 49/54] [TEST] MLT Rest test needs a mapping since we
 randomized number of nodes/shards

---
 rest-api-spec/test/mlt/10_basic.yaml | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/rest-api-spec/test/mlt/10_basic.yaml b/rest-api-spec/test/mlt/10_basic.yaml
index c29468133af19..f7f41b72b0696 100644
--- a/rest-api-spec/test/mlt/10_basic.yaml
+++ b/rest-api-spec/test/mlt/10_basic.yaml
@@ -7,6 +7,13 @@
          settings:
            index:
              number_of_replicas: 0
+         mappings:
+           test:
+             properties:
+               foo:
+                 type : "string"
+               title:
+                 type : "string"
 
   - do:
       index:

From 960d353dbd2d713fa66927a92f200dd30c1b2b66 Mon Sep 17 00:00:00 2001
From: Costin Leau <costin.leau@gmail.com>
Date: Mon, 17 Mar 2014 15:17:49 +0200
Subject: [PATCH 50/54] Remove plugin isolation feature for a future version

relates #5261
---
 docs/reference/cluster/nodes-info.asciidoc    |   1 -
 docs/reference/modules/plugins.asciidoc       |  14 --
 .../admin/cluster/node/info/PluginInfo.java   |  25 +--
 .../plugins/PluginClassLoader.java            | 110 ----------
 .../elasticsearch/plugins/PluginUtils.java    | 142 ------------
 .../elasticsearch/plugins/PluginsService.java | 204 ++++++++++--------
 .../rest/action/cat/RestPluginsAction.java    |   2 -
 .../nodesinfo/SimpleNodesInfoTests.java       |  36 +---
 .../plugins/IsolatedPluginTests.java          | 133 ------------
 .../plugins/PluginClassLoaderTest.java        | 103 ---------
 .../plugins/isolation/DummyClass.java         |  51 -----
 .../plugins/isolation/IsolatedPlugin.java     |  40 ----
 .../plugins/isolation/es-plugin.properties    |   1 -
 13 files changed, 120 insertions(+), 742 deletions(-)
 delete mode 100644 src/main/java/org/elasticsearch/plugins/PluginClassLoader.java
 delete mode 100644 src/main/java/org/elasticsearch/plugins/PluginUtils.java
 delete mode 100644 src/test/java/org/elasticsearch/plugins/IsolatedPluginTests.java
 delete mode 100644 src/test/java/org/elasticsearch/plugins/PluginClassLoaderTest.java
 delete mode 100644 src/test/java/org/elasticsearch/plugins/isolation/DummyClass.java
 delete mode 100644 src/test/java/org/elasticsearch/plugins/isolation/IsolatedPlugin.java
 delete mode 100644 src/test/resources/org/elasticsearch/plugins/isolation/es-plugin.properties

diff --git a/docs/reference/cluster/nodes-info.asciidoc b/docs/reference/cluster/nodes-info.asciidoc
index e6550ea740b96..61d8c1a5a6a1f 100644
--- a/docs/reference/cluster/nodes-info.asciidoc
+++ b/docs/reference/cluster/nodes-info.asciidoc
@@ -40,7 +40,6 @@ plugins per node:
 * `site`: `true` if the plugin is a site plugin
 * `jvm`: `true` if the plugin is a plugin running in the JVM
 * `url`: URL if the plugin is a site plugin
-* `isolation`: whether the plugin is loaded in isolation (`true`) or not (`false`)
 
 The result will look similar to:
 
diff --git a/docs/reference/modules/plugins.asciidoc b/docs/reference/modules/plugins.asciidoc
index 029cc627a90f9..4f3f676424721 100644
--- a/docs/reference/modules/plugins.asciidoc
+++ b/docs/reference/modules/plugins.asciidoc
@@ -142,20 +142,6 @@ bin/plugin --install mobz/elasticsearch-head --timeout 1m
 bin/plugin --install mobz/elasticsearch-head --timeout 0
 -----------------------------------
 
-added[1.1.0]
-[float]
-==== Plugins isolation
-
-Since Elasticsearch 1.1, by default, each plugin is loaded in _isolation_ (in its dedicated `ClassLoader`) to avoid class clashes between the various plugins and their associated libraries. The default can be changed through the `plugins.isolation` property in `elasticsearch.yml`, by setting it to `false`:
-
-[source,js]
---------------------------------------------------
-plugins.isolation: false
---------------------------------------------------
-
-Do note that each plugin can specify its _mandatory_ isolation through the `isolation` property in its `es-plugin.properties` configuration. In this (rare) case, the plugin setting is used, overwriting whatever default used by Elasticsearch.
-
-
 [float]
 [[known-plugins]]
 === Known Plugins
diff --git a/src/main/java/org/elasticsearch/action/admin/cluster/node/info/PluginInfo.java b/src/main/java/org/elasticsearch/action/admin/cluster/node/info/PluginInfo.java
index 9dc6e2430824e..b1b2fc2d572e4 100644
--- a/src/main/java/org/elasticsearch/action/admin/cluster/node/info/PluginInfo.java
+++ b/src/main/java/org/elasticsearch/action/admin/cluster/node/info/PluginInfo.java
@@ -41,7 +41,6 @@ static final class Fields {
         static final XContentBuilderString JVM = new XContentBuilderString("jvm");
         static final XContentBuilderString SITE = new XContentBuilderString("site");
         static final XContentBuilderString VERSION = new XContentBuilderString("version");
-        static final XContentBuilderString ISOLATION = new XContentBuilderString("isolation");
     }
 
     private String name;
@@ -49,7 +48,6 @@ static final class Fields {
     private boolean site;
     private boolean jvm;
     private String version;
-    private boolean isolation;
 
     public PluginInfo() {
     }
@@ -62,9 +60,8 @@ public PluginInfo() {
      * @param site        true if it's a site plugin
      * @param jvm         true if it's a jvm plugin
      * @param version     Version number is applicable (NA otherwise)
-     * @param isolation   true if it's an isolated plugin
      */
-    public PluginInfo(String name, String description, boolean site, boolean jvm, String version, boolean isolation) {
+    public PluginInfo(String name, String description, boolean site, boolean jvm, String version) {
         this.name = name;
         this.description = description;
         this.site = site;
@@ -74,7 +71,6 @@ public PluginInfo(String name, String description, boolean site, boolean jvm, St
         } else {
             this.version = VERSION_NOT_AVAILABLE;
         }
-        this.isolation = isolation;
     }
 
     /**
@@ -125,13 +121,6 @@ public String getVersion() {
         return version;
     }
 
-    /**
-     * @return Plugin isolation
-     */
-    public boolean isIsolation() {
-        return isolation;
-    }
-
     public static PluginInfo readPluginInfo(StreamInput in) throws IOException {
         PluginInfo info = new PluginInfo();
         info.readFrom(in);
@@ -149,11 +138,6 @@ public void readFrom(StreamInput in) throws IOException {
         } else {
             this.version = VERSION_NOT_AVAILABLE;
         }
-        if (in.getVersion().onOrAfter(Version.V_1_1_0)) {
-            this.isolation = in.readBoolean();
-        } else {
-            this.isolation = false;
-        }
     }
 
     @Override
@@ -165,9 +149,6 @@ public void writeTo(StreamOutput out) throws IOException {
         if (out.getVersion().onOrAfter(Version.V_1_0_0_RC2)) {
             out.writeString(version);
         }
-        if (out.getVersion().onOrAfter(Version.V_1_1_0)) {
-            out.writeBoolean(isolation);
-        }
     }
 
     @Override
@@ -181,7 +162,6 @@ public XContentBuilder toXContent(XContentBuilder builder, Params params) throws
         }
         builder.field(Fields.JVM, jvm);
         builder.field(Fields.SITE, site);
-        builder.field(Fields.ISOLATION, isolation);
         builder.endObject();
 
         return builder;
@@ -207,13 +187,12 @@ public int hashCode() {
 
     @Override
     public String toString() {
-        final StringBuilder sb = new StringBuilder("PluginInfo{");
+        final StringBuffer sb = new StringBuffer("PluginInfo{");
         sb.append("name='").append(name).append('\'');
         sb.append(", description='").append(description).append('\'');
         sb.append(", site=").append(site);
         sb.append(", jvm=").append(jvm);
         sb.append(", version='").append(version).append('\'');
-        sb.append(", isolation=").append(isolation);
         sb.append('}');
         return sb.toString();
     }
diff --git a/src/main/java/org/elasticsearch/plugins/PluginClassLoader.java b/src/main/java/org/elasticsearch/plugins/PluginClassLoader.java
deleted file mode 100644
index 3846d28eeb313..0000000000000
--- a/src/main/java/org/elasticsearch/plugins/PluginClassLoader.java
+++ /dev/null
@@ -1,110 +0,0 @@
-/*
- * Licensed to Elasticsearch under one or more contributor
- * license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright
- * ownership. Elasticsearch licenses this file to you under
- * the Apache License, Version 2.0 (the "License"); you may
- * not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *    http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.elasticsearch.plugins;
-
-import com.google.common.collect.Lists;
-
-import java.io.IOException;
-import java.net.URL;
-import java.net.URLClassLoader;
-import java.util.*;
-
-public class PluginClassLoader extends URLClassLoader {
-
-    private final ClassLoader system;
-    private final URL url;
-
-    PluginClassLoader(URL[] urls, ClassLoader parent) throws IOException {
-        super(urls, parent);
-        url = (urls != null && urls.length > 0 ? urls[0] : null);
-
-        ClassLoader sys = getSystemClassLoader();
-        while (sys.getParent() != null) {
-            sys = sys.getParent();
-        }
-
-        system = sys;
-    }
-
-    // load first from system class loader then fall back to this one
-    @Override
-    protected synchronized Class<?> loadClass(String name, boolean resolve) throws ClassNotFoundException {
-        Class<?> c = findLoadedClass(name);
-        if (c == null) {
-            // check system class loader (jvm & bootclasspath)
-            if (system != null) {
-                try {
-                    c = system.loadClass(name);
-                } catch (ClassNotFoundException ignored) {
-                }
-            }
-            if (c == null) {
-                try {
-                    // check plugin classloader
-                    c = findClass(name);
-                } catch (ClassNotFoundException e) {
-                    // fall back to parent
-                    c = super.loadClass(name, resolve);
-                }
-            }
-        }
-        if (resolve) {
-            resolveClass(c);
-        }
-        return c;
-    }
-
-    @Override
-    public URL getResource(String name) {
-        // apply same rules frmo loadClass
-        URL url = null;
-        if (system != null) {
-            url = system.getResource(name);
-        }
-        if (url == null) {
-            url = findResource(name);
-            if (url == null) {
-                url = super.getResource(name);
-            }
-        }
-        return url;
-    }
-
-    @Override
-    public Enumeration<URL> getResources(String name) throws IOException {
-        List<URL> urls = Lists.newArrayList();
-        if (system != null) {
-            urls.addAll(Collections.list(system.getResources(name)));
-        }
-        urls.addAll(Collections.list(findResources(name)));
-
-        ClassLoader parent = getParent();
-        if (parent != null) {
-            urls.addAll(Collections.list(parent.getResources(name)));
-        }
-
-        return Collections.enumeration(urls);
-    }
-
-    @Override
-    public String toString() {
-        return String.format(Locale.US, "PluginClassLoader for url [%s], classpath [%s], systemCL [%s]", url, Arrays.toString(getURLs()), system);
-    }
-}
\ No newline at end of file
diff --git a/src/main/java/org/elasticsearch/plugins/PluginUtils.java b/src/main/java/org/elasticsearch/plugins/PluginUtils.java
deleted file mode 100644
index 67ac0329cca71..0000000000000
--- a/src/main/java/org/elasticsearch/plugins/PluginUtils.java
+++ /dev/null
@@ -1,142 +0,0 @@
-/*
- * Licensed to Elasticsearch under one or more contributor
- * license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright
- * ownership. Elasticsearch licenses this file to you under
- * the Apache License, Version 2.0 (the "License"); you may
- * not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *    http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.elasticsearch.plugins;
-
-import com.google.common.collect.Lists;
-import org.apache.lucene.util.IOUtils;
-import org.elasticsearch.ElasticsearchException;
-import org.elasticsearch.common.Booleans;
-import org.elasticsearch.common.settings.Settings;
-
-import java.io.File;
-import java.io.IOException;
-import java.io.InputStream;
-import java.net.URL;
-import java.util.Collections;
-import java.util.List;
-import java.util.Properties;
-import java.util.jar.JarEntry;
-import java.util.jar.JarFile;
-
-public final class PluginUtils {
-
-    private PluginUtils() {}
-
-    @SuppressWarnings("unchecked")
-    static Plugin loadPlugin(String className, Settings settings, ClassLoader classLoader) {
-        try {
-            Class<? extends Plugin> pluginClass = (Class<? extends Plugin>) classLoader.loadClass(className);
-            try {
-                return pluginClass.getConstructor(Settings.class).newInstance(settings);
-            } catch (NoSuchMethodException e) {
-                try {
-                    return pluginClass.newInstance();
-                } catch (Exception e1) {
-                    throw new ElasticsearchException("No constructor for [" + pluginClass + "]. A plugin class must " +
-                            "have either an empty default constructor or a single argument constructor accepting a " +
-                            "Settings instance");
-                }
-            }
-
-        } catch (Exception e) {
-            throw new ElasticsearchException("Failed to load plugin class [" + className + "]", e);
-        }
-    }
-
-    static List<File> pluginClassPathAsFiles(File pluginFolder) throws IOException {
-        List<File> cpFiles = Lists.newArrayList();
-        cpFiles.add(pluginFolder);
-
-        List<File> libFiles = Lists.newArrayList();
-        File[] files = pluginFolder.listFiles();
-        if (files != null) {
-            Collections.addAll(libFiles, files);
-        }
-        File libLocation = new File(pluginFolder, "lib");
-        if (libLocation.exists() && libLocation.isDirectory()) {
-            files = libLocation.listFiles();
-            if (files != null) {
-                Collections.addAll(libFiles, files);
-            }
-        }
-
-        // if there are jars in it, add it as well
-        for (File libFile : libFiles) {
-            if (libFile.getName().endsWith(".jar") || libFile.getName().endsWith(".zip")) {
-                cpFiles.add(libFile);
-            }
-        }
-
-        return cpFiles;
-    }
-
-    static boolean lookupIsolation(List<URL> pluginProperties, boolean defaultIsolation) throws IOException {
-        Properties props = new Properties();
-        InputStream is = null;
-        for (URL prop : pluginProperties) {
-            try {
-                props.load(prop.openStream());
-                return Booleans.parseBoolean(props.getProperty("isolation"), defaultIsolation);
-            } finally {
-                IOUtils.closeWhileHandlingException(is);
-            }
-        }
-        return defaultIsolation;
-    }
-
-    static List<URL> lookupPluginProperties(List<File> pluginClassPath) throws Exception {
-        if (pluginClassPath.isEmpty()) {
-            return Collections.emptyList();
-        }
-
-        List<URL> found = Lists.newArrayList();
-
-        for (File file : pluginClassPath) {
-            String toString = file.getName();
-            if (toString.endsWith(".jar") || toString.endsWith(".zip")) {
-                JarFile jar = new JarFile(file);
-                try {
-                    JarEntry jarEntry = jar.getJarEntry("es-plugin.properties");
-                    if (jarEntry != null) {
-                        found.add(new URL("jar:" + file.toURI().toString() + "!/es-plugin.properties"));
-                    }
-                } finally {
-                    IOUtils.closeWhileHandlingException(jar);
-                }
-            }
-            else {
-                File props = new File(file, "es-plugin.properties");
-                if (props.exists() && props.canRead()) {
-                    found.add(props.toURI().toURL());
-                }
-            }
-        }
-
-        return found;
-    }
-
-    static URL[] convertFileToUrl(List<File> pluginClassPath) throws IOException {
-        URL[] urls = new URL[pluginClassPath.size()];
-        for (int i = 0; i < urls.length; i++) {
-            urls[i] = pluginClassPath.get(i).toURI().toURL();
-        }
-        return urls;
-    }
-}
\ No newline at end of file
diff --git a/src/main/java/org/elasticsearch/plugins/PluginsService.java b/src/main/java/org/elasticsearch/plugins/PluginsService.java
index 41ab24dffd2cf..a1b8c30de3f77 100644
--- a/src/main/java/org/elasticsearch/plugins/PluginsService.java
+++ b/src/main/java/org/elasticsearch/plugins/PluginsService.java
@@ -38,6 +38,7 @@
 
 import java.io.File;
 import java.io.FileInputStream;
+import java.io.IOException;
 import java.io.InputStream;
 import java.lang.reflect.Method;
 import java.net.URL;
@@ -88,8 +89,8 @@ public PluginsService(Settings settings, Environment environment) {
         // first we load all the default plugins from the settings
         String[] defaultPluginsClasses = settings.getAsArray("plugin.types");
         for (String pluginClass : defaultPluginsClasses) {
-            Plugin plugin = PluginUtils.loadPlugin(pluginClass, settings, settings.getClassLoader());
-            PluginInfo pluginInfo = new PluginInfo(plugin.name(), plugin.description(), hasSite(plugin.name()), true, PluginInfo.VERSION_NOT_AVAILABLE, false);
+            Plugin plugin = loadPlugin(pluginClass, settings);
+            PluginInfo pluginInfo = new PluginInfo(plugin.name(), plugin.description(), hasSite(plugin.name()), true, PluginInfo.VERSION_NOT_AVAILABLE);
             if (logger.isTraceEnabled()) {
                 logger.trace("plugin loaded from settings [{}]", pluginInfo);
             }
@@ -97,7 +98,8 @@ public PluginsService(Settings settings, Environment environment) {
         }
 
         // now, find all the ones that are in the classpath
-        tupleBuilder.addAll(loadPlugins());
+        loadPluginsIntoClassLoader();
+        tupleBuilder.addAll(loadPluginsFromClasspath(settings));
         this.plugins = tupleBuilder.build();
 
         // We need to build a List of jvm and site plugins for checking mandatory plugins
@@ -315,110 +317,103 @@ synchronized public PluginsInfo info() {
         return cachedPluginsInfo;
     }
 
-    private List<Tuple<PluginInfo,Plugin>> loadPlugins() {
-        File pluginsFile = environment.pluginsFile();
-        if (!isAccessibleDirectory(pluginsFile, logger)) {
-            return Collections.emptyList();
+    private void loadPluginsIntoClassLoader() {
+        File pluginsDirectory = environment.pluginsFile();
+        if (!isAccessibleDirectory(pluginsDirectory, logger)) {
+            return;
         }
 
-        List<Tuple<PluginInfo, Plugin>> pluginData = Lists.newArrayList();
-
-        boolean defaultIsolation = settings.getAsBoolean("plugins.isolation", Boolean.FALSE);
-        ClassLoader esClassLoader = settings.getClassLoader();
+        ClassLoader classLoader = settings.getClassLoader();
+        Class classLoaderClass = classLoader.getClass();
         Method addURL = null;
-        boolean discoveredAddUrl = false;
-
-        File[] pluginsFiles = pluginsFile.listFiles();
-
-        if (pluginsFiles != null) {
-            for (File pluginRoot : pluginsFiles) {
-                if (isAccessibleDirectory(pluginRoot, logger)) {
-                    try {
-                        logger.trace("--- adding plugin [" + pluginRoot.getAbsolutePath() + "]");
-                        // check isolation
-                        List<File> pluginClassPath = PluginUtils.pluginClassPathAsFiles(pluginRoot);
-                        List<URL> pluginProperties = PluginUtils.lookupPluginProperties(pluginClassPath);
-                        boolean isolated = PluginUtils.lookupIsolation(pluginProperties, defaultIsolation);
-
-                        if (isolated) {
-                            logger.trace("--- creating isolated space for plugin [" + pluginRoot.getAbsolutePath() + "]");
-                            PluginClassLoader pcl = new PluginClassLoader(PluginUtils.convertFileToUrl(pluginClassPath), esClassLoader);
-                            pluginData.addAll(loadPlugin(pluginClassPath, pluginProperties, pcl, true));
-                        } else {
-                            if (!discoveredAddUrl) {
-                                discoveredAddUrl = true;
-                                Class<?> esClassLoaderClass = esClassLoader.getClass();
-
-                                while (!esClassLoaderClass.equals(Object.class)) {
-                                    try {
-                                        addURL = esClassLoaderClass.getDeclaredMethod("addURL", URL.class);
-                                        addURL.setAccessible(true);
-                                        break;
-                                    } catch (NoSuchMethodException e) {
-                                        // no method, try the parent
-                                        esClassLoaderClass = esClassLoaderClass.getSuperclass();
-                                    }
-                                }
-                            }
-
-                            if (addURL == null) {
-                                logger.debug("failed to find addURL method on classLoader [" + esClassLoader + "] to add methods");
-                            }
-                            else {
-                                for (File file : pluginClassPath) {
-                                    addURL.invoke(esClassLoader, file.toURI().toURL());
-                                }
-                                pluginData.addAll(loadPlugin(pluginClassPath, pluginProperties, esClassLoader, false));
-                            }
-                        }
-                    } catch (Throwable e) {
-                        logger.warn("failed to add plugin [" + pluginRoot.getAbsolutePath() + "]", e);
-                    }
-                }
+        while (!classLoaderClass.equals(Object.class)) {
+            try {
+                addURL = classLoaderClass.getDeclaredMethod("addURL", URL.class);
+                addURL.setAccessible(true);
+                break;
+            } catch (NoSuchMethodException e) {
+                // no method, try the parent
+                classLoaderClass = classLoaderClass.getSuperclass();
             }
-        } else {
-            logger.debug("failed to list plugins from {}. Check your right access.", pluginsFile.getAbsolutePath());
+        }
+        if (addURL == null) {
+            logger.debug("failed to find addURL method on classLoader [" + classLoader + "] to add methods");
+            return;
         }
 
-        return pluginData;
-    }
+        for (File plugin : pluginsDirectory.listFiles()) {
+            // We check that subdirs are directories and readable
+            if (!isAccessibleDirectory(plugin, logger)) {
+                continue;
+            }
 
-    private Collection<? extends Tuple<PluginInfo, Plugin>> loadPlugin(List<File> pluginClassPath, List<URL> properties, ClassLoader classLoader, boolean isolation) throws Exception {
-        List<Tuple<PluginInfo, Plugin>> plugins = Lists.newArrayList();
+            logger.trace("--- adding plugin [{}]", plugin.getAbsolutePath());
 
-        Enumeration<URL> entries = Collections.enumeration(properties);
-        while (entries.hasMoreElements()) {
-            URL pluginUrl = entries.nextElement();
-            Properties pluginProps = new Properties();
-            InputStream is = null;
             try {
-                is = pluginUrl.openStream();
-                pluginProps.load(is);
-                String pluginClassName = pluginProps.getProperty("plugin");
-                if (pluginClassName == null) {
-                    throw new IllegalArgumentException("No plugin class specified");
+                // add the root
+                addURL.invoke(classLoader, plugin.toURI().toURL());
+                // gather files to add
+                List<File> libFiles = Lists.newArrayList();
+                if (plugin.listFiles() != null) {
+                    libFiles.addAll(Arrays.asList(plugin.listFiles()));
                 }
-                String pluginVersion = pluginProps.getProperty("version", PluginInfo.VERSION_NOT_AVAILABLE);
-                Plugin plugin = PluginUtils.loadPlugin(pluginClassName, settings, classLoader);
-
-                // Is it a site plugin as well? Does it have also an embedded _site structure
-                File siteFile = new File(new File(environment.pluginsFile(), plugin.name()), "_site");
-                boolean isSite = isAccessibleDirectory(siteFile, logger);
-                if (logger.isTraceEnabled()) {
-                    logger.trace("found a jvm plugin [{}], [{}]{}",
-                            plugin.name(), plugin.description(), isSite ? ": with _site structure" : "");
+                File libLocation = new File(plugin, "lib");
+                if (libLocation.exists() && libLocation.isDirectory() && libLocation.listFiles() != null) {
+                    libFiles.addAll(Arrays.asList(libLocation.listFiles()));
                 }
 
-                PluginInfo pluginInfo = new PluginInfo(plugin.name(), plugin.description(), isSite, true, pluginVersion, isolation);
-                plugins.add(new Tuple<>(pluginInfo, plugin));
+                // if there are jars in it, add it as well
+                for (File libFile : libFiles) {
+                    if (!(libFile.getName().endsWith(".jar") || libFile.getName().endsWith(".zip"))) {
+                        continue;
+                    }
+                    addURL.invoke(classLoader, libFile.toURI().toURL());
+                }
             } catch (Throwable e) {
-                logger.warn("failed to load plugin from [" + pluginUrl + "]", e);
-            } finally {
-                IOUtils.closeWhileHandlingException(is);
+                logger.warn("failed to add plugin [" + plugin + "]", e);
             }
         }
+    }
 
-        return plugins;
+    private ImmutableList<Tuple<PluginInfo,Plugin>> loadPluginsFromClasspath(Settings settings) {
+        ImmutableList.Builder<Tuple<PluginInfo, Plugin>> plugins = ImmutableList.builder();
+
+        // Trying JVM plugins: looking for es-plugin.properties files
+        try {
+            Enumeration<URL> pluginUrls = settings.getClassLoader().getResources(ES_PLUGIN_PROPERTIES);
+            while (pluginUrls.hasMoreElements()) {
+                URL pluginUrl = pluginUrls.nextElement();
+                Properties pluginProps = new Properties();
+                InputStream is = null;
+                try {
+                    is = pluginUrl.openStream();
+                    pluginProps.load(is);
+                    String pluginClassName = pluginProps.getProperty("plugin");
+                    String pluginVersion = pluginProps.getProperty("version", PluginInfo.VERSION_NOT_AVAILABLE);
+                    Plugin plugin = loadPlugin(pluginClassName, settings);
+
+                    // Is it a site plugin as well? Does it have also an embedded _site structure
+                    File siteFile = new File(new File(environment.pluginsFile(), plugin.name()), "_site");
+                    boolean isSite = isAccessibleDirectory(siteFile, logger);
+                    if (logger.isTraceEnabled()) {
+                        logger.trace("found a jvm plugin [{}], [{}]{}",
+                                plugin.name(), plugin.description(), isSite ? ": with _site structure" : "");
+                    }
+
+                    PluginInfo pluginInfo = new PluginInfo(plugin.name(), plugin.description(), isSite, true, pluginVersion);
+
+                    plugins.add(new Tuple<>(pluginInfo, plugin));
+                } catch (Throwable e) {
+                    logger.warn("failed to load plugin from [" + pluginUrl + "]", e);
+                } finally {
+                    IOUtils.closeWhileHandlingException(is);
+                }
+            }
+        } catch (IOException e) {
+            logger.warn("failed to find jvm plugins from classpath", e);
+        }
+
+        return plugins.build();
     }
 
     private ImmutableList<Tuple<PluginInfo,Plugin>> loadSitePlugins() {
@@ -471,7 +466,7 @@ private ImmutableList<Tuple<PluginInfo,Plugin>> loadSitePlugins() {
                         logger.trace("found a site plugin name [{}], version [{}], description [{}]",
                                 name, version, description);
                     }
-                    sitePlugins.add(new Tuple<PluginInfo, Plugin>(new PluginInfo(name, description, true, false, version, false), null));
+                    sitePlugins.add(new Tuple<PluginInfo, Plugin>(new PluginInfo(name, description, true, false, version), null));
                 }
             }
         }
@@ -494,4 +489,27 @@ private boolean hasSite(String name) {
         File sitePluginDir = new File(pluginsFile, name + "/_site");
         return isAccessibleDirectory(sitePluginDir, logger);
     }
-}
\ No newline at end of file
+
+    private Plugin loadPlugin(String className, Settings settings) {
+        try {
+            Class<? extends Plugin> pluginClass = (Class<? extends Plugin>) settings.getClassLoader().loadClass(className);
+            Plugin plugin;
+            try {
+                plugin = pluginClass.getConstructor(Settings.class).newInstance(settings);
+            } catch (NoSuchMethodException e) {
+                try {
+                    plugin = pluginClass.getConstructor().newInstance();
+                } catch (NoSuchMethodException e1) {
+                    throw new ElasticsearchException("No constructor for [" + pluginClass + "]. A plugin class must " +
+                            "have either an empty default constructor or a single argument constructor accepting a " +
+                            "Settings instance");
+                }
+            }
+
+            return plugin;
+
+        } catch (Throwable e) {
+            throw new ElasticsearchException("Failed to load plugin class [" + className + "]", e);
+        }
+    }
+}
diff --git a/src/main/java/org/elasticsearch/rest/action/cat/RestPluginsAction.java b/src/main/java/org/elasticsearch/rest/action/cat/RestPluginsAction.java
index cdf8efffbe709..5695adda8bf2e 100644
--- a/src/main/java/org/elasticsearch/rest/action/cat/RestPluginsAction.java
+++ b/src/main/java/org/elasticsearch/rest/action/cat/RestPluginsAction.java
@@ -84,7 +84,6 @@ Table getTableWithHeader(final RestRequest request) {
         table.addCell("component", "alias:c;desc:component");
         table.addCell("version", "alias:v;desc:component version");
         table.addCell("type", "alias:t;desc:type (j for JVM, s for Site)");
-        table.addCell("isolation", "alias:i;desc:isolation");
         table.addCell("url", "alias:u;desc:url for site plugins");
         table.addCell("description", "alias:d;default:false;desc:plugin details");
         table.endHeaders();
@@ -118,7 +117,6 @@ private Table buildTable(RestRequest req, ClusterStateResponse state, NodesInfoR
                     }
                 }
                 table.addCell(type);
-                table.addCell(pluginInfo.isIsolation() ? "x" : "");
                 table.addCell(pluginInfo.getUrl());
                 table.addCell(pluginInfo.getDescription());
                 table.endRow();
diff --git a/src/test/java/org/elasticsearch/nodesinfo/SimpleNodesInfoTests.java b/src/test/java/org/elasticsearch/nodesinfo/SimpleNodesInfoTests.java
index 01ddd7e53f39c..a17fa9313a432 100644
--- a/src/test/java/org/elasticsearch/nodesinfo/SimpleNodesInfoTests.java
+++ b/src/test/java/org/elasticsearch/nodesinfo/SimpleNodesInfoTests.java
@@ -132,43 +132,37 @@ public void testNodeInfoPlugin() throws URISyntaxException {
         logger.info("--> full json answer, status " + response.toString());
 
         assertNodeContainsPlugins(response, server1NodeId,
-                Collections.EMPTY_LIST, Collections.EMPTY_LIST, Collections.EMPTY_LIST, Collections.EMPTY_LIST,// No JVM Plugin
-                Collections.EMPTY_LIST, Collections.EMPTY_LIST, Collections.EMPTY_LIST, Collections.EMPTY_LIST);// No Site Plugin
+                Collections.EMPTY_LIST, Collections.EMPTY_LIST, Collections.EMPTY_LIST, // No JVM Plugin
+                Collections.EMPTY_LIST, Collections.EMPTY_LIST, Collections.EMPTY_LIST);// No Site Plugin
 
         assertNodeContainsPlugins(response, server2NodeId,
-                Collections.EMPTY_LIST, Collections.EMPTY_LIST, Collections.EMPTY_LIST, Collections.EMPTY_LIST, // No JVM Plugin
+                Collections.EMPTY_LIST, Collections.EMPTY_LIST, Collections.EMPTY_LIST, // No JVM Plugin
                 Lists.newArrayList(Fields.SITE_PLUGIN),                                 // Site Plugin
                 Lists.newArrayList(Fields.SITE_PLUGIN_DESCRIPTION),
-                Lists.newArrayList(Fields.SITE_PLUGIN_VERSION),
-                Lists.newArrayList(Boolean.FALSE));
+                Lists.newArrayList(Fields.SITE_PLUGIN_VERSION));
 
         assertNodeContainsPlugins(response, server3NodeId,
                 Lists.newArrayList(TestPlugin.Fields.NAME),                             // JVM Plugin
                 Lists.newArrayList(TestPlugin.Fields.DESCRIPTION),
                 Lists.newArrayList(PluginInfo.VERSION_NOT_AVAILABLE),
-                Lists.newArrayList(Boolean.FALSE),
-                Collections.EMPTY_LIST, Collections.EMPTY_LIST, Collections.EMPTY_LIST, Collections.EMPTY_LIST);// No site Plugin
+                Collections.EMPTY_LIST, Collections.EMPTY_LIST, Collections.EMPTY_LIST);// No site Plugin
 
         assertNodeContainsPlugins(response, server4NodeId,
                 Lists.newArrayList(TestNoVersionPlugin.Fields.NAME),                    // JVM Plugin
                 Lists.newArrayList(TestNoVersionPlugin.Fields.DESCRIPTION),
                 Lists.newArrayList(PluginInfo.VERSION_NOT_AVAILABLE),
-                Lists.newArrayList(Boolean.FALSE),
                 Lists.newArrayList(Fields.SITE_PLUGIN, TestNoVersionPlugin.Fields.NAME),// Site Plugin
                 Lists.newArrayList(PluginInfo.DESCRIPTION_NOT_AVAILABLE),
-                Lists.newArrayList(PluginInfo.VERSION_NOT_AVAILABLE),
-                Lists.newArrayList(Boolean.FALSE));
+                Lists.newArrayList(PluginInfo.VERSION_NOT_AVAILABLE));
     }
 
     private void assertNodeContainsPlugins(NodesInfoResponse response, String nodeId,
                                            List<String> expectedJvmPluginNames,
                                            List<String> expectedJvmPluginDescriptions,
                                            List<String> expectedJvmVersions,
-                                           List<Boolean> expectedJvmIsolations,
                                            List<String> expectedSitePluginNames,
                                            List<String> expectedSitePluginDescriptions,
-                                           List<String> expectedSiteVersions,
-                                           List<Boolean> expectedSiteIsolations) {
+                                           List<String> expectedSiteVersions) {
 
         assertThat(response.getNodesMap().get(nodeId), notNullValue());
 
@@ -190,11 +184,6 @@ private void assertNodeContainsPlugins(NodesInfoResponse response, String nodeId
             assertThat(jvmPluginVersions, hasItem(pluginVersion));
         }
 
-        List<Boolean> jvmPluginIsolations = FluentIterable.from(plugins.getInfos()).filter(jvmPluginPredicate).transform(isolationFunction).toList();
-        for (Boolean pluginIsolation : expectedJvmIsolations) {
-            assertThat(jvmPluginIsolations, hasItem(pluginIsolation));
-        }
-
         FluentIterable<String> jvmUrls = FluentIterable.from(plugins.getInfos())
                 .filter(and(jvmPluginPredicate, Predicates.not(sitePluginPredicate)))
                 .filter(isNull())
@@ -219,11 +208,6 @@ private void assertNodeContainsPlugins(NodesInfoResponse response, String nodeId
         for (String pluginVersion : expectedSiteVersions) {
             assertThat(sitePluginVersions, hasItem(pluginVersion));
         }
-
-        List<Boolean> sitePluginIsolations = FluentIterable.from(plugins.getInfos()).filter(sitePluginPredicate).transform(isolationFunction).toList();
-        for (Boolean pluginIsolation : expectedSiteIsolations) {
-            assertThat(sitePluginIsolations, hasItem(pluginIsolation));
-        }
     }
 
     private String startNodeWithPlugins(int nodeId, String ... pluginClassNames) throws URISyntaxException {
@@ -283,10 +267,4 @@ public String apply(PluginInfo pluginInfo) {
             return pluginInfo.getVersion();
         }
     };
-
-    private Function<PluginInfo, Boolean> isolationFunction = new Function<PluginInfo, Boolean>() {
-        public Boolean apply(PluginInfo pluginInfo) {
-            return pluginInfo.isIsolation();
-        }
-    };
 }
diff --git a/src/test/java/org/elasticsearch/plugins/IsolatedPluginTests.java b/src/test/java/org/elasticsearch/plugins/IsolatedPluginTests.java
deleted file mode 100644
index f8fa82a062d7c..0000000000000
--- a/src/test/java/org/elasticsearch/plugins/IsolatedPluginTests.java
+++ /dev/null
@@ -1,133 +0,0 @@
-/*
- * Licensed to Elasticsearch under one or more contributor
- * license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright
- * ownership. Elasticsearch licenses this file to you under
- * the Apache License, Version 2.0 (the "License"); you may
- * not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *    http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.elasticsearch.plugins;
-
-import com.google.common.io.Files;
-import org.elasticsearch.action.admin.cluster.node.info.NodesInfoResponse;
-import org.elasticsearch.common.Strings;
-import org.elasticsearch.common.io.FileSystemUtils;
-import org.elasticsearch.common.io.Streams;
-import org.elasticsearch.common.settings.ImmutableSettings;
-import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.test.ElasticsearchIntegrationTest;
-import org.elasticsearch.test.ElasticsearchIntegrationTest.ClusterScope;
-import org.junit.After;
-import org.junit.Before;
-import org.junit.Test;
-
-import java.io.File;
-import java.io.FileOutputStream;
-import java.io.IOException;
-import java.io.InputStream;
-import java.util.Arrays;
-import java.util.Properties;
-
-import static org.hamcrest.CoreMatchers.equalTo;
-import static org.hamcrest.CoreMatchers.notNullValue;
-import static org.hamcrest.Matchers.greaterThanOrEqualTo;
-
-// NB: the tests uses System Properties to pass the information from different plugins (loaded in separate CLs) to the test.
-// hence the use of try/finally blocks to clean these up after the test has been executed as otherwise the test framework will trigger a failure
-@ClusterScope(scope = ElasticsearchIntegrationTest.Scope.TEST, numNodes = 0)
-public class IsolatedPluginTests extends ElasticsearchIntegrationTest {
-
-    private static final Settings SETTINGS;
-    private static final File PLUGIN_DIR;
-    private Properties props;
-
-    static {
-        PLUGIN_DIR = Files.createTempDir();
-
-        SETTINGS = ImmutableSettings.settingsBuilder()
-                .put("discovery.zen.ping.multicast.enabled", false)
-                .put("force.http.enabled", true)
-                .put("plugins.isolation", true)
-                .put("path.plugins", PLUGIN_DIR.getAbsolutePath())
-                .build();
-    }
-
-    @Before
-    public void beforeTest() throws Exception {
-        props = new Properties();
-        props.putAll(System.getProperties());
-
-        logger.info("Installing plugins into folder {}", PLUGIN_DIR);
-        FileSystemUtils.mkdirs(PLUGIN_DIR);
-
-        // copy plugin
-        copyPlugin(new File(PLUGIN_DIR, "plugin-v1"));
-        copyPlugin(new File(PLUGIN_DIR, "plugin-v2"));
-    }
-
-    private void copyPlugin(File p1) throws IOException {
-        FileSystemUtils.mkdirs(p1);
-        // copy plugin
-        File dir = new File(p1, "org/elasticsearch/plugins/isolation/");
-        FileSystemUtils.mkdirs(dir);
-        copyFile(getClass().getResourceAsStream("isolation/DummyClass.class"), new File(dir, "DummyClass.class"));
-        copyFile(getClass().getResourceAsStream("isolation/IsolatedPlugin.class"), new File(dir, "IsolatedPlugin.class"));
-        copyFile(getClass().getResourceAsStream("isolation/es-plugin.properties"), new File(p1, "es-plugin.properties"));
-    }
-
-    private void copyFile(InputStream source, File out) throws IOException {
-        out.createNewFile();
-        Streams.copy(source, new FileOutputStream(out));
-    }
-
-    @After
-    public void afterTest() throws Exception {
-        FileSystemUtils.deleteRecursively(PLUGIN_DIR);
-    }
-
-    protected Settings nodeSettings(int nodeOrdinal) {
-        return SETTINGS;
-    }
-
-    @Test
-    public void testPluginNumberOfIsolatedInstances() throws Exception {
-        try {
-            NodesInfoResponse nodesInfoResponse = client().admin().cluster().prepareNodesInfo().clear().setPlugins(true).get();
-            assertThat(nodesInfoResponse.getNodes().length, equalTo(1));
-            assertThat(nodesInfoResponse.getNodes()[0].getPlugins().getInfos(), notNullValue());
-            assertThat(nodesInfoResponse.getNodes()[0].getPlugins().getInfos().size(), equalTo(2));
-        } finally {
-            System.setProperties(props);
-        }
-    }
-
-    @Test
-    public void testIsolatedPluginProperties() throws Exception {
-        try {
-            String prop = "es.test.isolated.plugin.count";
-            int count = Integer.getInteger(prop, 0);
-            client();
-            // do a >= comparison in case there are multiple tests running in the same JVM (build server)
-            assertThat(Integer.getInteger(prop), greaterThanOrEqualTo(count + 2));
-            Properties p = System.getProperties();
-            prop = p.getProperty("es.test.isolated.plugin.instantiated.hashes");
-            String[] hashes = Strings.delimitedListToStringArray(prop, " ");
-            // 2 plugins plus trailing space
-            assertThat(hashes.length, greaterThanOrEqualTo(count + 2));
-            Arrays.sort(hashes);
-            assertThat(Arrays.binarySearch(hashes, p.getProperty("es.test.isolated.plugin.instantiated")), greaterThanOrEqualTo(0));
-        } finally {
-            System.setProperties(props);
-        }
-    }
-}
\ No newline at end of file
diff --git a/src/test/java/org/elasticsearch/plugins/PluginClassLoaderTest.java b/src/test/java/org/elasticsearch/plugins/PluginClassLoaderTest.java
deleted file mode 100644
index e871006039b2a..0000000000000
--- a/src/test/java/org/elasticsearch/plugins/PluginClassLoaderTest.java
+++ /dev/null
@@ -1,103 +0,0 @@
-/*
- * Licensed to Elasticsearch under one or more contributor
- * license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright
- * ownership. Elasticsearch licenses this file to you under
- * the Apache License, Version 2.0 (the "License"); you may
- * not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *    http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.elasticsearch.plugins;
-
-import org.junit.Before;
-import org.junit.Test;
-
-import java.net.URL;
-import java.util.Properties;
-
-import static org.hamcrest.Matchers.is;
-import static org.hamcrest.Matchers.not;
-import static org.junit.Assert.*;
-
-
-public class PluginClassLoaderTest {
-
-    private static final String NAME = "es.test.isolated.plugin.name";
-    private static final String INSTANCE = "es.test.isolated.plugin.instantiated";
-    private static final String READ = "es.test.isolated.plugin.read.name";
-
-    private URL root;
-    private String clazz = getClass().getName().replace('.', '/').concat(".class");
-    private ClassLoader parent;
-
-    @Before
-    public void before() throws Exception {
-        System.getProperties().setProperty(READ, "");
-        System.getProperties().setProperty(NAME, "");
-        System.getProperties().setProperty(INSTANCE, "");
-
-        parent = getClass().getClassLoader();
-        URL url = parent.getResource(clazz);
-        root = new URL(url.toString().substring(0, url.toString().indexOf(clazz)));
-    }
-
-    @Test
-    public void testClassSpaceIsolationEvenWhenPointingToSameClass() throws Exception {
-
-        PluginClassLoader space1 = new PluginClassLoader(new URL[] { root }, null);
-        Properties props = System.getProperties();
-        assertThat(props.getProperty(READ), is(""));
-        props.setProperty(NAME, "one");
-        Class<?> class1 = Class.forName("org.elasticsearch.plugins.isolation.DummyClass", true, space1);
-        assertThat(props.getProperty(READ), is("one"));
-
-        String instance1 = props.getProperty(INSTANCE);
-
-        PluginClassLoader space2 = new PluginClassLoader(new URL[] { root }, null);
-        props.setProperty(NAME, "two");
-        Class<?> class2 = Class.forName("org.elasticsearch.plugins.isolation.DummyClass", true, space2);
-        assertThat(props.getProperty(READ), is("two"));
-
-        String instance2 = props.getProperty(INSTANCE);
-        assertThat(instance1, not(instance2));
-        assertNotSame(class1, class2);
-    }
-
-    @Test
-    public void testDelegateToParentIfResourcesNotFoundLocally() throws Exception {
-
-        PluginClassLoader space1 = new PluginClassLoader(new URL[] {}, null);
-        assertNull(space1.getResource(clazz));
-
-        PluginClassLoader space2 = new PluginClassLoader(new URL[] {}, parent);
-        assertNotNull(parent.getResource(clazz));
-        assertNotNull(space2.getResource(clazz));
-    }
-
-    @Test
-    public void testIgnoreClassesInParentIfFoundLocally() throws Exception {
-        ClassLoader parent = getClass().getClassLoader();
-
-        PluginClassLoader space1 = new PluginClassLoader(new URL[] { root }, parent);
-
-        Properties props = System.getProperties();
-        assertThat(props.getProperty(READ), is(""));
-        props.setProperty(NAME, "one");
-
-        String before = props.getProperty(NAME);
-        Class parentClass = Class.forName("org.elasticsearch.plugins.isolation.DummyClass", true, parent);
-        props.setProperty(NAME, "one");
-        Class class1 = Class.forName("org.elasticsearch.plugins.isolation.DummyClass", true, space1);
-        assertThat(props.getProperty(READ), is(before));
-        assertThat(class1, is(not(parentClass)));
-    }
-}
diff --git a/src/test/java/org/elasticsearch/plugins/isolation/DummyClass.java b/src/test/java/org/elasticsearch/plugins/isolation/DummyClass.java
deleted file mode 100644
index a3b9060d21cea..0000000000000
--- a/src/test/java/org/elasticsearch/plugins/isolation/DummyClass.java
+++ /dev/null
@@ -1,51 +0,0 @@
-/*
- * Licensed to Elasticsearch under one or more contributor
- * license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright
- * ownership. Elasticsearch licenses this file to you under
- * the Apache License, Version 2.0 (the "License"); you may
- * not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *    http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.elasticsearch.plugins.isolation;
-
-import java.util.Properties;
-
-public class DummyClass {
-
-    static final String name;
-
-    static {
-        Properties sysProps = System.getProperties();
-        // make sure to get a string even when dealing with null
-        name = "" + sysProps.getProperty("es.test.isolated.plugin.name");
-        sysProps.setProperty("es.test.isolated.plugin.instantiated", "" + DummyClass.class.hashCode());
-        Integer count = Integer.getInteger("es.test.isolated.plugin.count");
-        if (count == null) {
-            count = Integer.valueOf(0);
-        }
-
-        count = count + 1;
-
-        sysProps.setProperty("es.test.isolated.plugin.count", count.toString());
-
-        String prop = sysProps.getProperty("es.test.isolated.plugin.instantiated.hashes");
-        if (prop == null) {
-            prop = "";
-        }
-
-        prop = prop + DummyClass.class.hashCode() + " ";
-        sysProps.setProperty("es.test.isolated.plugin.instantiated.hashes", prop);
-        sysProps.setProperty("es.test.isolated.plugin.read.name",  name);
-    }
-
-}
diff --git a/src/test/java/org/elasticsearch/plugins/isolation/IsolatedPlugin.java b/src/test/java/org/elasticsearch/plugins/isolation/IsolatedPlugin.java
deleted file mode 100644
index 648d31ecd1aa7..0000000000000
--- a/src/test/java/org/elasticsearch/plugins/isolation/IsolatedPlugin.java
+++ /dev/null
@@ -1,40 +0,0 @@
-/*
- * Licensed to Elasticsearch under one or more contributor
- * license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright
- * ownership. Elasticsearch licenses this file to you under
- * the Apache License, Version 2.0 (the "License"); you may
- * not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *    http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.elasticsearch.plugins.isolation;
-
-import org.elasticsearch.plugins.AbstractPlugin;
-
-public class IsolatedPlugin extends AbstractPlugin {
-
-    private final DummyClass dummy;
-
-    public IsolatedPlugin() {
-        dummy = new DummyClass();
-    }
-
-    @Override
-    public String name() {
-        return dummy.name;
-    }
-
-    @Override
-    public String description() {
-        return "IsolatedPlugin "  + hashCode();
-    }
-}
diff --git a/src/test/resources/org/elasticsearch/plugins/isolation/es-plugin.properties b/src/test/resources/org/elasticsearch/plugins/isolation/es-plugin.properties
deleted file mode 100644
index 8b0c1759661ad..0000000000000
--- a/src/test/resources/org/elasticsearch/plugins/isolation/es-plugin.properties
+++ /dev/null
@@ -1 +0,0 @@
-plugin=org.elasticsearch.plugins.isolation.IsolatedPlugin
\ No newline at end of file

From 9aa1cb408e358e1036690c5430f93d460fe02ca1 Mon Sep 17 00:00:00 2001
From: Lee Hinman <lee@writequit.org>
Date: Tue, 8 Apr 2014 10:44:50 -0600
Subject: [PATCH 51/54] Fix format string for DiskThresholdDecider reroute
 explanation

---
 .../routing/allocation/decider/DiskThresholdDecider.java    | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/src/main/java/org/elasticsearch/cluster/routing/allocation/decider/DiskThresholdDecider.java b/src/main/java/org/elasticsearch/cluster/routing/allocation/decider/DiskThresholdDecider.java
index d928524aec6fc..c91836922065d 100644
--- a/src/main/java/org/elasticsearch/cluster/routing/allocation/decider/DiskThresholdDecider.java
+++ b/src/main/java/org/elasticsearch/cluster/routing/allocation/decider/DiskThresholdDecider.java
@@ -186,7 +186,7 @@ public Decision canAllocate(ShardRouting shardRouting, RoutingNode node, Routing
                 logger.debug("Less than the required {}% free disk threshold ({}% free) on node [{}], preventing allocation",
                         freeDiskThresholdLow, freeDiskPercentage, node.nodeId());
             }
-            return allocation.decision(Decision.NO, NAME, "less than required [%d%%] free disk on node, free: [%d%%]",
+            return allocation.decision(Decision.NO, NAME, "less than required [%s%%] free disk on node, free: [%s%%]",
                     freeDiskThresholdLow, freeDiskThresholdLow);
         }
 
@@ -204,7 +204,7 @@ public Decision canAllocate(ShardRouting shardRouting, RoutingNode node, Routing
         if (freeSpaceAfterShard < freeDiskThresholdHigh) {
             logger.warn("After allocating, node [{}] would have less than the required {}% free disk threshold ({}% free), preventing allocation",
                     node.nodeId(), freeDiskThresholdHigh, freeSpaceAfterShard);
-            return allocation.decision(Decision.NO, NAME, "after allocation less than required [%d%%] free disk on node, free: [%d%%]",
+            return allocation.decision(Decision.NO, NAME, "after allocation less than required [%s%%] free disk on node, free: [%s%%]",
                     freeDiskThresholdLow, freeSpaceAfterShard);
         }
 
@@ -266,7 +266,7 @@ public Decision canRemain(ShardRouting shardRouting, RoutingNode node, RoutingAl
                 logger.debug("Less than the required {}% free disk threshold ({}% free) on node {}, shard cannot remain",
                         freeDiskThresholdHigh, freeDiskPercentage, node.nodeId());
             }
-            return allocation.decision(Decision.NO, NAME, "after allocation less than required [%d%%] free disk on node, free: [%d%%]",
+            return allocation.decision(Decision.NO, NAME, "after allocation less than required [%s%%] free disk on node, free: [%s%%]",
                     freeDiskThresholdHigh, freeDiskPercentage);
         }
 

From a2fb4809ef9a06ff6e907594277473e042bf9cce Mon Sep 17 00:00:00 2001
From: Sean Gallagher <sean.gallagher@elasticsearch.com>
Date: Wed, 9 Apr 2014 11:51:39 -0400
Subject: [PATCH 52/54] [DOCS] Improved the upgrade doc, added upgrade table
 Author: Sean Gallagher Date: Apr 9 2014

Closes #5651
---
 docs/reference/setup/upgrade.asciidoc | 60 +++++++++++++++++++++++----
 1 file changed, 51 insertions(+), 9 deletions(-)

diff --git a/docs/reference/setup/upgrade.asciidoc b/docs/reference/setup/upgrade.asciidoc
index f3bf3df193d82..182d605acd563 100644
--- a/docs/reference/setup/upgrade.asciidoc
+++ b/docs/reference/setup/upgrade.asciidoc
@@ -1,7 +1,21 @@
 [[setup-upgrade]]
 == Upgrading
 
-Elasticsearch can usually be upgraded using a rolling upgrade process, resulting in no interruption of service.  This section details how to perform both rolling and restart upgrades.
+Elasticsearch can usually be upgraded using a rolling upgrade process, resulting in no interruption of service.  This section details how to perform both rolling and restart upgrades.  To determine whether a rolling upgrade is supported for your release, please consult this table:
+
+[cols="1,2,3",options="header",]
+|=======================================================================
+|Upgrade From |Upgrade To |Supported Upgrade Type
+|0.90.x |1.x |Restart Upgrade
+
+|< 0.90.7 |0.90.x |Restart Upgrade
+
+|>= 0.90.7 |0.90.x |Rolling Upgrade
+
+|1.x |1.x |Rolling Upgrade
+|=======================================================================
+
+Before upgrading from 0.90.x or any earlier version to 1.x or later, it is a good idea to consult the <<breaking-changes,breaking changes>> docs.
 
 [float]
 [[backup]]
@@ -58,9 +72,24 @@ To back up a running 1.0 or later system, it is simplest to use the snapshot fea
 
 [float]
 [[rolling-upgrades]]
-=== Rolling Upgrades
+=== Rolling upgrade process
 
-Within major releases after release 1.0, rolling upgrades are supported.  To perform a rolling upgrade:
+A rolling upgrade allows the ES cluster to be upgraded one node at a time, with no observable downtime for end users.  Running multiple versions of Elasticsearch in the same cluster for any length of time beyond that required for an upgrade is not supported, as shard replication from the more recent version to the previous versions will not work.
+
+Within minor or maintenance releases after release 1.0, rolling upgrades are supported.  To perform a rolling upgrade:
+
+* Disable shard reallocation (optional).  This is done to allow for a faster startup after cluster shutdown.  If this step is not performed, the nodes will immediately start trying to replicate shards to each other on startup and will spend a lot of time on wasted I/O.  With shard reallocation disabled, the nodes will join the cluster with their indices intact, without attempting to rebalance.  After startup is complete, reallocation will be turned back on.
+
+This syntax applies to Elasticsearch 1.0 and later:
+
+[source,sh]
+--------------------------------------------------
+        curl -XPUT localhost:9200/_cluster/settings -d '{
+                "transient" : {
+                    "cluster.routing.allocation.enable" : "none"
+                }
+        }'
+--------------------------------------------------
 
 * Shut down a single node within the cluster.
 
@@ -73,22 +102,35 @@ curl -XPOST 'http://localhost:9200/_cluster/nodes/_local/_shutdown'
 
 * Upgrade the stopped node.  To upgrade using a zip or compressed tarball from elasticsearch.org:
 ** Extract the zip or tarball to a new directory, usually in the same volume as the current Elasticsearch installation.  Do not overwrite the existing installation, as the downloaded archive will contain a default elasticsearch.yml file and will overwrite your existing configuration.
-** Copy the configuration files from the old Elasticsearch installation's config directory to the new Elasticsearch installation's config directory.  Copy data files if necessary.  If data files are not located within the tarball's extraction directory, they will not have to be moved.
+** Copy the configuration files from the old Elasticsearch installation's config directory to the new Elasticsearch installation's config directory.  Move data files from the old Elasticsesarch installation's data directory if necessary.  If data files are not located within the tarball's extraction directory, they will not have to be moved.
 ** The simplest solution for moving from one version to another is to have a symbolic link for 'elasticsearch' that points to the currently running version.  This link can be easily updated and will provide a stable access point to the most recent version.  Update this symbolic link if it is being used.
 
 *To upgrade using a .deb or .rpm package:
 
 ** Use rpm or deb to install the new package.  All files should be placed in their proper locations, and config files should not be overwritten.
 
-** Start the now upgraded node.  Observe that all shards are properly allocated back to the node and that it joins the cluster smoothly.
+* Start the now upgraded node.  Confirm that it joins the cluster.
+
+* Repeat this process for all remaining nodes.
+
+* When the process is complete on all nodes, you can re-enable shard reallocation:
+
+[source,sh]
+--------------------------------------------------
+        curl -XPUT localhost:9200/_cluster/settings -d '{
+                "transient" : {
+                    "cluster.routing.allocation.enable" : "all"
+                }
+        }'
+--------------------------------------------------
 
-Repeat this process for all remaining nodes.
+* Observe that all shards are properly allocated on all nodes.  Balancing may take some time.
 
 It may be possible to perform the upgrade by installing the new software while the service is running.  This would reduce downtime by ensuring the service was ready to run on the new version as soon as it is stopped on the node being upgraded.  This can be done by installing the new version in its own directory and using the symbolic link method outlined above.  It is important to test this procedure first to be sure that site-specific configuration data and production indices will not be overwritten during the upgrade process.
 
 [float]
 [[restart-upgrade]]
-=== Upgrading from 0.90.x or earlier to 1.0 or later
+=== Cluster restart upgrade process
 
 Elasticsearch releases prior to 1.0 and releases after 1.0 are not compatible with each other, so a rolling upgrade is not possible.  In order to upgrade a pre-1.0 system to 1.0 or later, a full cluster stop and start is required.  In order to perform this upgrade:
 
@@ -115,9 +157,9 @@ This syntax is from versions prior to 1.0:
 
 * After upgrading Elasticsearch on all nodes is complete, the cluster can be started by starting each node individually.
 ** Start master-eligible nodes first, one at a time.  Verify that a quorum has been reached and a master has been elected before proceeding.
-** Start data and client nodes one at a time, verifying that they successfully join the cluster.
+** Start data nodes and then client nodes one at a time, verifying that they successfully join the cluster.
 
-* When the cluster is up and running, shard reallocation can be enabled.
+* When the cluster is running and reaches a yellow state, shard reallocation can be enabled.
 
 This syntax is from release 1.0 and later:
 [source,sh]

From af0278b51bee306c48b845dfca09ccdc77548027 Mon Sep 17 00:00:00 2001
From: Nik Everett <nik9000@gmail.com>
Date: Wed, 9 Apr 2014 13:53:24 -0400
Subject: [PATCH 53/54] [Docs] Allocation setting explanation

Closes #5748
---
 .../cluster/update-settings.asciidoc          | 26 +++++++++++++------
 1 file changed, 18 insertions(+), 8 deletions(-)

diff --git a/docs/reference/cluster/update-settings.asciidoc b/docs/reference/cluster/update-settings.asciidoc
index b8b8657b12fe3..a60b9049e8b8b 100644
--- a/docs/reference/cluster/update-settings.asciidoc
+++ b/docs/reference/cluster/update-settings.asciidoc
@@ -65,22 +65,32 @@ There is a specific list of settings that can be updated, those include:
 
 [float]
 ===== Balanced Shards
+All these values are relative to one another.  The first three are used to
+compose a three separate weighting functions into one.  The cluster is balanced
+when no allowed action can bring the weights of each node closer together by
+more then the fourth setting.  Actions might not be allowed, for instance,
+due to forced awareness or allocation filtering.
 
 `cluster.routing.allocation.balance.shard`::
-     Defines the weight factor for shards allocated on a node 
-     (float). Defaults to `0.45f`.
+     Defines the weight factor for shards allocated on a node
+     (float). Defaults to `0.45f`.  Raising this raises the tendency to
+     equalize the number of shards across all nodes in the cluster.
 
 `cluster.routing.allocation.balance.index`::
-     Defines a factor to the number of shards per index allocated 
-      on a specific node (float). Defaults to `0.5f`.
+     Defines a factor to the number of shards per index allocated
+      on a specific node (float). Defaults to `0.5f`.  Raising this raises the
+      tendency to equalize the number of shards per index across all nodes in
+      the cluster.
 
 `cluster.routing.allocation.balance.primary`::
-      defines a weight factor for the number of primaries of a specific index 
-      allocated on a node (float). `0.05f`.
+     Defines a weight factor for the number of primaries of a specific index
+      allocated on a node (float). `0.05f`.  Raising this raises the tendency
+      to equalize the number of primary shards across all nodes in the cluster.
 
 `cluster.routing.allocation.balance.threshold`::
-      minimal optimization value of operations that should be performed (non 
-      negative float). Defaults to `1.0f`.
+     Minimal optimization value of operations that should be performed (non
+      negative float). Defaults to `1.0f`.  Raising this will cause the cluster
+      to be less aggressive about optimizing the shard balance.
 
 [float]
 ===== Concurrent Rebalance

From b51817c2cd6a80824b2facf42075711244f3734c Mon Sep 17 00:00:00 2001
From: Sean Gallagher <sean.gallagher@elasticsearch.com>
Date: Wed, 9 Apr 2014 15:29:06 -0400
Subject: [PATCH 54/54] [DOCS] Fixed version number in README.textfile Author:
 Sean Gallagher Date: 9 Apr 2014

---
 README.textile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/README.textile b/README.textile
index 8330e3b7b4ee7..a657b59a392fe 100644
--- a/README.textile
+++ b/README.textile
@@ -208,7 +208,7 @@ running the Elasticsearch test suite.
 
 h3. Upgrading to Elasticsearch 1.x?
 
-In order to ensure a smooth upgrade process from earlier versions of Elasticsearch (< 0.90.x), it is recommended to perform a full cluster restart. Please see the "Upgrading" section of the "setup reference":http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html.
+In order to ensure a smooth upgrade process from earlier versions of Elasticsearch (< 1.0.0), it is recommended to perform a full cluster restart. Please see the "Upgrading" section of the "setup reference":http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html.
 
 h1. License