0.18 docs

_layouts/guide.html

@@ -37,6 +37,7 @@ <h3>Indices</h3>
             <li><a href="/guide/reference/api/admin-indices-gateway-snapshot.html">Snapshot</a></li>
             <li><a href="/guide/reference/api/admin-indices-update-settings.html">Update Settings</a></li>
             <li><a href="/guide/reference/api/admin-indices-templates.html">Templates</a></li>
+            <li><a href="/guide/reference/api/admin-indices-stats.html">Stats</a></li>
             <li><a href="/guide/reference/api/admin-indices-status.html">Status</a></li>
             <li><a href="/guide/reference/api/admin-indices-segments.html">Segments</a></li>
             <li><a href="/guide/reference/api/admin-indices-clearcache.html">Clear Cache</a></li>
@@ -47,6 +48,7 @@ <h3>Cluster</h3>
         <ul>
             <li><a href="/guide/reference/api/admin-cluster-health.html">Health</a></li>
             <li><a href="/guide/reference/api/admin-cluster-state.html">State</a></li>
+            <li><a href="/guide/reference/api/admin-cluster-update-settings.html">Update Settings</a></li>
             <li><a href="/guide/reference/api/admin-cluster-nodes-info.html">Nodes Info</a></li>
             <li><a href="/guide/reference/api/admin-cluster-nodes-stats.html">Nodes Stats</a></li>
             <li><a href="/guide/reference/api/admin-cluster-nodes-shutdown.html">Nodes Shutdown</a></li>

guide/reference/api/admin-cluster-update-settings.textile

@@ -0,0 +1,48 @@
+---
+layout: guide
+title: Cluster Update Settings API
+cat: guide
+sidebar: reference_api
+---
+
+p. Allow to update cluster wide specific settings. Settings updated can either be persistent (applied cross restarts) or transient (will not survive a full cluster restart). Here is an example:
+
+
+<pre class="prettyprint">
+curl -XPUT localhost:9200/_cluster/settings -d '{
+    "persistent" : {
+        "discovery.zen.minimum_master_nodes" : 2
+    }
+}'
+</pre>
+
+p. Or:
+
+<pre class="prettyprint">
+curl -XPUT localhost:9200/_cluster/settings -d '{
+    "transient" : {
+        "discovery.zen.minimum_master_nodes" : 2
+    }
+}'
+</pre>
+
+There is a specific list of settings that can be updated, those include:
+
+* @discovery.zen.minimum_master_nodes@
+* @index.shard.recovery.concurrent_streams@
+* @cluster.routing.allocation.node_initial_primaries_recoveries@, @cluster.routing.allocation.node_concurrent_recoveries@
+* @cluster.routing.allocation.cluster_concurrent_rebalance@
+* @cluster.routing.allocation.awareness.attributes@
+* @cluster.routing.allocation.awareness.force.*@
+* @cluster.routing.allocation.disable_allocation@
+* @cluster.routing.allocation.disable_replica_allocation@
+* @cluster.routing.allocation.include.*@
+* @cluster.routing.allocation.exclude.*@
+* @indices.cache.filter.size@
+* @indices.ttl.interval@
+* @indices.recovery.file_chunk_size@, @indices.recovery.translog_ops@, @indices.recovery.translog_size@, @indices.recovery.compress@, @indices.recovery.concurrent_streams@, @indices.recovery.max_size_per_sec@.
+* @@
+
+p. Logger values can also be updated by setting @logger.@ prefix. More settings will be allowed to be updated.
+
+p. Cluster wide settings can be returned using @curl -XGET localhost:9200/_cluster/settings@.

guide/reference/api/admin-indices-clearcache.textile

@@ -13,6 +13,8 @@ $ curl -XPOST 'http://localhost:9200/twitter/_cache/clear'
 
 p. The API, by default, will clear all cached. Specific caches can cleaned explicitly by setting @filter@, @field_data@ or @bloom@ to @true@.
 
+p. All caches relating to a specific field(s) can also be cleared by specifying @fields@ parameter with a comma delimited list of the relevant fields.
+
 h1. Multi Index
 
 p. The flush API can be applied to more than one index with a single call, or even on @_all@ the indices.

guide/reference/api/admin-indices-delete-index.textile

@@ -13,4 +13,4 @@ $ curl -XDELETE 'http://localhost:9200/twitter/'
 
 p. The above example deletes an index called @twitter@.
 
-p. The delete index API can also be applied to more than one index, or on @_all@ indices (be careful!). All indices will also be deleted when no specific index is provided.
+p. The delete index API can also be applied to more than one index, or on @_all@ indices (be careful!). All indices will also be deleted when no specific index is provided. In order to disable allowing to delete all indices, set @action.disable_delete_all_indices@ setting in the config to @true@.

guide/reference/api/admin-indices-stats.textile

@@ -0,0 +1,62 @@
+---
+layout: guide
+title: Indices Stats API
+cat: guide
+sidebar: reference_api
+---
+
+p. Indices level stats provide statistics on different operations happening on an index. The API provides statistics on the index level scope (though most stats can also be retrieved using node level scope).
+
+The following returns high level aggregation and index level stats for all indices:
+
+<pre class="prettyprint">
+curl localhost:9200/_stats
+</pre>
+
+Specific index stats can be retrieved using:
+
+<pre class="prettyprint">
+curl localhost:9200/index1,index2/_stats
+</pre>
+
+By default, @docs@, @store@, and @indexing@, @get@, and @search@ stats are returned, other stats can be enabled as well:
+
+* @docs@: The number of docs / deleted docs (docs not yet merged out). Note, affected by refreshing the index.
+* @store@: The size of the index.
+* @indexing@: Indexing statistics, can be combined with a comma separated list of @types@ to provide document type level stats.
+* @get@: Get statistics, including missing stats.
+* @search@: Search statistics, including custom grouping using the @groups@ parameter (search operations can be associated with one or more groups).
+* @merge@: merge stats.
+* @flush@: flush stats.
+* @refresh@: refresh stats.
+* @clear@: Clears all the flags (first).
+
+p. Here are some samples:
+
+<pre class="prettyprint">
+# Get back stats for merge and refresh on top of the defaults
+curl 'localhost:9200/_stats?merge=true&refresh=true'
+# Get back stats just for flush
+curl 'localhost:9200/_stats?clear=true&flush=true'
+# Get back stats for type1 and type2 documents for the my_index index
+curl 'localhost:9200/my_index/_stats?clear=true&indexing=true&types=type1,type2
+</pre>
+
+p. The stats returned are aggregated on the index level, with @primaries@ and @total@ aggregations. In order to get back shard level stats, set the @level@ parameter to @shards@. 
+
+p. Note, as shards move around the cluster, their stats will be cleared as they are created on other nodes. On the other hand, even though a shard "left" a node, that node will still retain the stats that shard contributed to.
+
+h2. Specific stats endpoints
+
+Instead of using flags to indicate which stats to return, specific REST endpoints can be used, for example:
+
+<pre class="prettyprint">
+# Merge stats across all indices
+curl localhost:9200/_stats/merge
+# Merge stats for the my_index index
+curl localhost:9200/my_index/_stats/merge
+# Indexing stats for my_index
+curl localhost:9200/my_index/_stats/indexing
+# Indexing stats for my_index for my_type1 and my_type2
+curl localhost:9200/my_index/_stats/indexing/my_type1,my_type2
+</pre>

guide/reference/api/admin-indices-update-settings.textile

@@ -1,6 +1,6 @@
 ---
 layout: guide
-title: Update Settings API
+title: Indices Update Settings API
 cat: guide
 sidebar: reference_api
 ---
@@ -45,6 +45,9 @@ p. Below is the list of settings that can be changed using the update settings A
 |@index.cache.filter.expire@|The expire after access time for filter cache. Set to @-1@ to disable.|
 |@index.gateway.snapshot_interval@|The gateway snapshot interval (only applies to shared gateways).|
 |"merge policy":/guide/reference/index-modules/merge.html|All the settings for the merge policy currently configured. A different merge policy can't be set.|
+|@index.routing.allocation.include.*@|Controls which nodes the index will be allowed to be allocated on.|
+|@index.routing.allocation.exclude.*@|Controls which nodes the index will not be allowed ot be allocated on.|
+|@index.gc_deletes@||
 
 h1. Bulk Indexing Usage
 

guide/reference/api/bulk.textile

@@ -28,6 +28,8 @@ p. Because this format uses literal @\n@'s as delimiters, please be sure that th
 { "create" : { "_index" : "test", "_type" : "type1", "_id" : "3" } }
 { "field1" : "value3" }
 
+p. The endpoints are @/_bulk@, @/{index}/_bulk@, and @{index}/type/_bulk@. When the index or the index/type are provided, they will be used by default on bulk items that don't provide them explicitly.
+
 </pre>
 
 p. A note on the format. The idea here is to make processing of this as fast as possible. As some of the actions will be redirected to other shards on other nodes, only @action_meta_data@ is parsed on the receiving node side.

guide/reference/api/count.textile

@@ -29,6 +29,8 @@ p. Both examples above do the same thing, which is count the number of tweets fr
 }
 </pre>
 
+p. The query is optional, and when not provided, it will use @match_all@ to count all the docs.
+
 h1. Multiple Indices and Types
 
 p. The count API can be applied to multiple types within an index, and across multiple indices. For example, we can count all documents across all types that match a @user@ field with value @kimchy@:

guide/reference/api/get.textile

@@ -48,7 +48,7 @@ p. The get operation allows to specify a set of fields that will be returned (by
 curl -XGET 'http://localhost:9200/twitter/tweet/1?fields=title,content'
 </pre>
 
-p. The returned fields will either be loaded if they are stored, or fetched from the @_source@ (parsed and extracted). Note, certain "json sub objects" can be returned by explicitly prefixing them with @_source.@, for example: @_source.obj1.obj2@.
+p. The returned fields will either be loaded if they are stored, or fetched from the @_source@ (parsed and extracted). It also supports sub objects extraction from _source, like @obj1.obj2@.
 
 h1. Routing
 

guide/reference/api/search/fields.textile

@@ -31,4 +31,4 @@ p. An empty array will cause only the @_id@ and @_type@ for each hit to be retur
 }
 </pre>
 
-p. Script fields can also be automatically detected and used as fields, so things like @_source.obj1.obj2@ can be used.
+p. Script fields can also be automatically detected and used as fields, so things like @_source.obj1.obj2@ can be used, though not recommended, as @obj1.obj2@ will work as well.

guide/reference/api/search/index.textile

@@ -40,4 +40,17 @@ $ curl -XGET 'http://localhost:9200/twitter/tweet/_search?routing=kimchy' -d '{
 '
 </pre>
 
-p. The routing parameter can be multi valued represented as a comma separated string. This will result in hitting the relevant shards where the routing values match to.
+p. The routing parameter can be multi valued represented as a comma separated string. This will result in hitting the relevant shards where the routing values match to.
+
+h1. Stats Groups
+
+p. A search can be associated with stats groups, which maintains a statistics aggregation per group. It can later be retrieved using the indices stats API specifically. For example, here is a search body request that associate the request with two different groups:
+
+<pre class="prettyprint">
+{
+    "query" : {
+        "match_all" : {}
+    },
+    "stats" : ["group1", "group2"]
+}
+</pre>

guide/reference/api/search/preference.textile

@@ -11,4 +11,5 @@ The @preference@ can be set to:
 
 * @_primary@: The operation will go and be executed only on the primary shards.
 * @_local@: The operation will prefer to be executed on a local allocated shard is possible.
+* @_only_node:xyz@: Restricts the search to execute only on a node with the provided node id (@xyz@ in this case).
 * Custom (string) value: A custom value will be used to guarantee that the same shards will be used for the same custom value. This can help with "jumping values" when hitting different shards in different refresh states. A sample value can be something like the web session id, or the user name.

guide/reference/mapping/geo-point-type.textile

@@ -19,6 +19,12 @@ p. Mapper type called @geo_point@ to support geo based points. The declaration l
 }
 </pre>
 
+h1. Indexed Fields
+
+p. The @geo_point@ mapping will index a single field with the format of @lat,lon@. The @lat_lon@ option can be set to also index the @.lat@ and @.lon@ as numeric fields, and @geohash@ can be set to @true@ to also index @.geohash@ value.
+
+p. A good practice is to enable indexing @lat_lon@ as well, since both the geo distance and bounding box filters can either be executed using in memory checks, or using the indexed lat lon values, and it really depends on the data set which one performs better. Note though, that indexed lat lon only make sense when there is a single geo point value for the field, and not multi values.
+
 h1. Input Structure
 
 The above mapping defines a @geo_point@, which accepts different formats. The following formats are supported:
@@ -71,10 +77,6 @@ p. Format in @[lon, lat]@, note, the order of lon/lat here in order to conform w
 </pre>
 
 
-h1. Indexed Fields
-
-p. The @geo_point@ mapping will index a single field with the format of @lat,lon@. The @lat_lon@ option can be set to also index the @.lat@ and @.lon@ as numeric fields, and @geohash@ can be set to @true@ to also index @.geohash@ value. Note, those values are almost never needed to be indexed unless explicit queries against those fields are needed. All the geo components work against the default indexed field.
-
 h1. Mapping Options
 
 |_. Option |_. Description |

guide/reference/mapping/id-field.textile

@@ -19,4 +19,6 @@ p. The @_id@ field can be enabled to be indexed, and possibly stored, using:
 }
 </pre>
 
-p. In order to maintain backward compatibility, a node level setting @index.mapping._id.indexed@ can be set to @true@ to make sure that the id is indexed when upgrading to @0.16@, though its recommended to not index the id.
+p. In order to maintain backward compatibility, a node level setting @index.mapping._id.indexed@ can be set to @true@ to make sure that the id is indexed when upgrading to @0.16@, though its recommended to not index the id.
+
+p. The @_id@ mapping can also be associated with a @path@ that will be used to extract the id from a different location in the source document. This does require an additional lightweight parsing step while indexing, in order to extract the id in order to decide which shard the index operation will be executed on.

guide/reference/mapping/source-field.textile

@@ -24,3 +24,18 @@ p. The source field can be compressed (LZF) when stored in the index. This can g
 p. In order to enable compression, the @compress@ option should be set to @true@. By default it is set to @false@. Note, this can be changed on an existing index, as a mix of compressed and uncompressed sources is supported.
 
 p. Moreover, a @compress_threshold@ can be set to control when the source will be compressed. It accepts a byte size value (for example @100b@, @10kb@). Note, @compress@ should be set to @true@.
+
+h1. Includes / Excludes
+
+p. Allow to specify paths in the source that would be included / excluded when its stored, supporting @*@ as wildcard annotation. For example:
+
+<pre class="prettyprint lang-js">
+{
+    "my_type" : {
+        "_source" : {
+            "includes" : ["path1.*", "path2.*"],
+            "excludes" : ["pat3.*"]
+        }
+    }
+}
+</pre>