From 5a7adf0d8660a5c35154dcf64c0fe16067bfc28f Mon Sep 17 00:00:00 2001 From: Luca Cavanna Date: Tue, 17 Mar 2020 16:46:23 +0100 Subject: [PATCH 1/9] [DOCS] add docs for async search Relates to #49091 --- .../doc/RestTestsFromSnippetsTask.groovy | 3 - docs/reference/async-search.asciidoc | 20 ++ docs/reference/index.asciidoc | 2 + docs/reference/redirects.asciidoc | 27 +-- docs/reference/search.asciidoc | 2 + docs/reference/search/async-search.asciidoc | 197 ++++++++++++++++++ .../api/async_search.delete.json | 2 +- .../rest-api-spec/api/async_search.get.json | 2 +- .../api/async_search.submit.json | 23 +- 9 files changed, 248 insertions(+), 30 deletions(-) create mode 100644 docs/reference/async-search.asciidoc create mode 100644 docs/reference/search/async-search.asciidoc diff --git a/buildSrc/src/main/groovy/org/elasticsearch/gradle/doc/RestTestsFromSnippetsTask.groovy b/buildSrc/src/main/groovy/org/elasticsearch/gradle/doc/RestTestsFromSnippetsTask.groovy index 5b855a4d9c7b5..e271ff1a58750 100644 --- a/buildSrc/src/main/groovy/org/elasticsearch/gradle/doc/RestTestsFromSnippetsTask.groovy +++ b/buildSrc/src/main/groovy/org/elasticsearch/gradle/doc/RestTestsFromSnippetsTask.groovy @@ -314,10 +314,7 @@ class RestTestsFromSnippetsTask extends SnippetsTask { if (path == null) { path = '' // Catch requests to the root... } else { - // Escape some characters that are also escaped by sense path = path.replace('<', '%3C').replace('>', '%3E') - path = path.replace('{', '%7B').replace('}', '%7D') - path = path.replace('|', '%7C') } current.println(" - do:") if (catchPart != null) { diff --git a/docs/reference/async-search.asciidoc b/docs/reference/async-search.asciidoc new file mode 100644 index 0000000000000..a5e9771f2ab46 --- /dev/null +++ b/docs/reference/async-search.asciidoc @@ -0,0 +1,20 @@ +[role="xpack"] +[testenv="basic"] +[[async-search-intro]] +== Long running searches + +{es} generally allows to quickly search across big amounts of data. There are +situations where a search executes on many many shards, possibly against +<> and spanning multiple <>, for which +results are not expected to be returned in milliseconds. When needing to +execute long running searches, sending a search request and synchronously +waiting for its results to be returned is not ideal. Async Search allows +instead to submit a search request that gets executed asynchronously and +monitor its progress as well as retrieve its results at a later stage. +Also, partial results can be retrieved as they become available, before +the search has completed. + +An async search request can be submitted using the <> API. +The <> API allows to monitor the progress of an async search +request and retrieve its results. An async search can be deleted through the +<> API. diff --git a/docs/reference/index.asciidoc b/docs/reference/index.asciidoc index fcfc85d7184c5..bce1d615f264d 100644 --- a/docs/reference/index.asciidoc +++ b/docs/reference/index.asciidoc @@ -26,6 +26,8 @@ include::query-dsl.asciidoc[] include::modules/cross-cluster-search.asciidoc[] +include::async-search.asciidoc[] + include::scripting.asciidoc[] include::mapping.asciidoc[] diff --git a/docs/reference/redirects.asciidoc b/docs/reference/redirects.asciidoc index 9ab175ee4d8e4..6ea64b139f5f8 100644 --- a/docs/reference/redirects.asciidoc +++ b/docs/reference/redirects.asciidoc @@ -120,7 +120,7 @@ See <>. [role="exclude",id="native-settings"] ==== Native realm settings -See <>. +See <>. [role="exclude",id="configuring-saml-realm"] === Configuring a SAML realm @@ -130,27 +130,27 @@ See <>. [role="exclude",id="saml-settings"] ==== SAML realm settings -See <>. +See <>. [role="exclude",id="_saml_realm_signing_settings"] ==== SAML realm signing settings -See <>. +See <>. [role="exclude",id="_saml_realm_encryption_settings"] ==== SAML realm encryption settings -See <>. +See <>. [role="exclude",id="_saml_realm_ssl_settings"] ==== SAML realm SSL settings -See <>. +See <>. [role="exclude",id="configuring-file-realm"] === Configuring a file realm -See <>. +See <>. [role="exclude",id="ldap-user-search"] === User search mode and user DN templates mode @@ -170,7 +170,7 @@ See <>. [role="exclude",id="ldap-ssl"] === Setting up SSL between Elasticsearch and LDAP -See <>. +See <>. [role="exclude",id="configuring-kerberos-realm"] === Configuring a Kerberos realm @@ -211,7 +211,7 @@ See <>. [role="exclude",id="mapping-roles-ad"] === Mapping Active Directory users and groups to roles -See <>. +See <>. [role="exclude",id="how-security-works"] === How security works @@ -237,9 +237,9 @@ See the details in This page was deleted. [[ml-datafeed-chunking-config]] -See the details in <>, <>, +See the details in <>, <>, [[ml-datafeed-delayed-data-check-config]] -<>, +<>, [[ml-datafeed-counts]] <>. @@ -323,7 +323,7 @@ See <>. [role="exclude",id="ml-dfa-analysis-objects"] === Analysis configuration objects -This page was deleted. +This page was deleted. See <>. [role="exclude",id="slm-api-delete"] @@ -375,8 +375,3 @@ See <>. === How {ccs} works See <> and <>. - -[role="exclude",id="async-search"] -=== Asynchronous search - -coming::[7.x] diff --git a/docs/reference/search.asciidoc b/docs/reference/search.asciidoc index e8a8a42da6a00..a33807cf698ac 100644 --- a/docs/reference/search.asciidoc +++ b/docs/reference/search.asciidoc @@ -152,6 +152,8 @@ high). This default value is `5`. include::search/search.asciidoc[] +include::search/async-search.asciidoc[] + include::search/uri-request.asciidoc[] include::search/request-body.asciidoc[] diff --git a/docs/reference/search/async-search.asciidoc b/docs/reference/search/async-search.asciidoc new file mode 100644 index 0000000000000..ad5baee42d43e --- /dev/null +++ b/docs/reference/search/async-search.asciidoc @@ -0,0 +1,197 @@ +[role="xpack"] +[testenv="basic"] +[[async-search]] +=== Async Search + +Async search consists of a set of API that allow to asynchronously execute a +search request, monitor its progress as well as retrieve its partial results +as soon as they become available. + +[[submit-async-search]] +==== Submit Async Search + +Submits a search request that gets executed asynchronously. It accepts the same +request as the <>. + +[source,console,id=submit-async-search-date-histogram-example] +-------------------------------------------------- +POST /sales*/_async_search?size=0 +{ + "sort" : [ + { "date" : {"order" : "asc"} } + ], + "aggs" : { + "sale_date" : { + "date_histogram" : { + "field" : "date", + "calendar_interval": "1d" + } + } + } +} +-------------------------------------------------- +// TEST[setup:sales] +// TEST[s/size=0/size=0&wait_for_completion=0/] + +The response contains the identifier of the search being executed, to be used +to retrieve the results at a later stage. Also, the currently available search +results are returned as part of the <> object. + +[source,console-result] +-------------------------------------------------- +{ + "id" : "FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc=", <1> + "version" : 0, + "is_partial" : true, <2> + "is_running" : true, <3> + "start_time_in_millis" : 1583945890986, + "expiration_time_in_millis" : 1584377890986, + "response" : { + "took" : 1122, + "timed_out" : false, + "num_reduce_phases" : 0, + "_shards" : { + "total" : 562, <4> + "successful" : 3, <5> + "skipped" : 0, + "failed" : 0 + }, + "hits" : { + "total" : { + "value" : 157483, <6> + "relation" : "gte" + }, + "max_score" : null, + "hits" : [ ] + } + } +} +-------------------------------------------------- +// TESTRESPONSE[s/FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc=/$body.id/] +// TESTRESPONSE[s/1583945890986/$body.start_time_in_millis/] +// TESTRESPONSE[s/1584377890986/$body.expiration_time_in_millis/] +// TESTRESPONSE[s/"took" : 1122/"took": $body.response.took/] +// TESTRESPONSE[s/"total" : 562/"total": $body.response._shards.total/] +// TESTRESPONSE[s/"successful" : 3/"successful": $body.response._shards.successful/] +// TESTRESPONSE[s/"value" : 157483/"value": $body.response.hits.total.value/] + +<1> Identifier of the async search that can be used to monitor its progress, retrieve its results, and/or delete it. +<2> Whether the returned search results are partial or final +<3> Whether the search is still being executed or it has completed +<4> How many shards the search will be executed on, overall +<5> How many shards have successfully completed the search +<6> How many documents are currently matching the query, which belong to the shards that have already completed the search + +It is possible to block and wait until the search is completed up to a certain +timeout by providing the `wait_for_completion` parameter, which defaults to +`1` second. + +The Submit Async Search API supports the same <> +as the search API, though some have different default values: +`pre_filter_shard_size` defaults to `1`, `batched_reduce_size` defaults to +`5`, `request_cache` defaults to `true` and `ccs_minimize_roundtrips` +defaults to `false`. + +It is also possible to specify how long the async search needs to be +available through the `keep_alive` parameter, which defaults to `5d` (five days). +Running async searches as well as saved search results are cleared once their +corresponding `keep_alive` expires. + +WARNING: Async search does not support scroll searches, nor search requests that +only include the `suggest` section. Cross cluster searches can be executed using +async search but only with `ccs_minimize_roundtrips` disabled. + +[[get-async-search]] +==== Get Async Search + +The Get Async Search API can be used to retrieve the results of a previously +submitted async search request given its id. + +[source,console,id=get-async-search-date-histogram-example] +-------------------------------------------------- +GET /_async_search/FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc= +-------------------------------------------------- +// TEST[continued s/FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc=/\${body.id}/] + +[source,console-result] +-------------------------------------------------- +{ + "id" : "FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc=", + "version" : 2, <1> + "is_partial" : true, + "is_running" : true, + "start_time_in_millis" : 1583945890986, + "expiration_time_in_millis" : 1584377890986, + "response" : { + "took" : 12144, + "timed_out" : false, + "num_reduce_phases" : 38, + "_shards" : { + "total" : 562, + "successful" : 188, + "skipped" : 0, + "failed" : 0 + }, + "hits" : { + "total" : { + "value" : 456433, + "relation" : "eq" + }, + "max_score" : null, + "hits" : [ ] + }, + "aggregations" : { <2> + "sale_date" : { + "buckets" : [] + } + } + } +} +-------------------------------------------------- +// TESTRESPONSE[s/FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc=/$body.id/] +// TESTRESPONSE[s/"is_partial" : true/"is_partial" : false/] +// TESTRESPONSE[s/"is_running" : true/"is_running" : false/] +// TESTRESPONSE[s/1583945890986/$body.start_time_in_millis/] +// TESTRESPONSE[s/1584377890986/$body.expiration_time_in_millis/] +// TESTRESPONSE[s/"took" : 12144/"took": $body.response.took/] +// TESTRESPONSE[s/"total" : 562/"total": $body.response._shards.total/] +// TESTRESPONSE[s/"successful" : 188/"successful": $body.response._shards.successful/] +// TESTRESPONSE[s/"value" : 456433/"value": $body.response.hits.total.value/] +// TESTRESPONSE[s/"buckets" : \[\]/"buckets": $body.response.aggregations.sale_date.buckets/] +// TESTRESPONSE[s/"num_reduce_phases" : 38,//] + +<1> The returned `version` is useful to identify whether the response contains +additional results compared to previously obtained responses. If the version +stays the same, no new results have become available, otherwise a higher version +number indicates that more shards have completed their execution of the query +and their partial results are also included in the response. +<2> Partial aggregations results, coming from the shards that have already +completed the execution of the query. + +NOTE: When results are sorted by a numeric field, shards get sorted based on +minimum and maximum value that they hold for that field, hence partial +results become available following the sort criteria that was requested. + +The `wait_for_completion` parameter, which defaults to `1`, can also be provided +when calling the Get Async Search API, in order to wait for the search to be +completed up until the provided timeout. Final results will be returned if +available before the timeout expires, otherwise the currently available results +will be returned once the timeout expires. + +The `keep_alive` parameter, which defaults to `5d` (five days) can be provided +to specify how long the async search should be available in the cluster. Once +an async search expires, it is cancelled if still running and its saved results +are deleted if already completed. + +[[delete-async-search]] +==== Delete Async Search + +The Delete Async Search API can be used to manually delete a specific async +search given its id. The corresponding async search will be cancelled if it's +still running, otherwise its saved results will be deleted. + +[source,console,id=delete-async-search-date-histogram-example] +-------------------------------------------------- +DELETE /_async_search/FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc= +-------------------------------------------------- +// TEST[continued s/FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc=/\${body.id}/] diff --git a/x-pack/plugin/src/test/resources/rest-api-spec/api/async_search.delete.json b/x-pack/plugin/src/test/resources/rest-api-spec/api/async_search.delete.json index 4486b90cb9a1d..10416b04ce5c1 100644 --- a/x-pack/plugin/src/test/resources/rest-api-spec/api/async_search.delete.json +++ b/x-pack/plugin/src/test/resources/rest-api-spec/api/async_search.delete.json @@ -3,7 +3,7 @@ "documentation":{ "url":"https://www.elastic.co/guide/en/elasticsearch/reference/current/async-search.html" }, - "stability":"experimental", + "stability":"stable", "url":{ "paths":[ { diff --git a/x-pack/plugin/src/test/resources/rest-api-spec/api/async_search.get.json b/x-pack/plugin/src/test/resources/rest-api-spec/api/async_search.get.json index f5ea1424756e1..abe07c04869b7 100644 --- a/x-pack/plugin/src/test/resources/rest-api-spec/api/async_search.get.json +++ b/x-pack/plugin/src/test/resources/rest-api-spec/api/async_search.get.json @@ -3,7 +3,7 @@ "documentation":{ "url":"https://www.elastic.co/guide/en/elasticsearch/reference/current/async-search.html" }, - "stability":"experimental", + "stability":"stable", "url":{ "paths":[ { diff --git a/x-pack/plugin/src/test/resources/rest-api-spec/api/async_search.submit.json b/x-pack/plugin/src/test/resources/rest-api-spec/api/async_search.submit.json index 033939acf0703..bfd2ee8ae94e7 100644 --- a/x-pack/plugin/src/test/resources/rest-api-spec/api/async_search.submit.json +++ b/x-pack/plugin/src/test/resources/rest-api-spec/api/async_search.submit.json @@ -3,7 +3,7 @@ "documentation":{ "url":"https://www.elastic.co/guide/en/elasticsearch/reference/current/async-search.html" }, - "stability":"experimental", + "stability":"stable", "url":{ "paths":[ { @@ -32,6 +32,10 @@ "description":"Specify the time that the request should block waiting for the final response", "default": "1s" }, + "clean_on_completion":{ + "type":"boolean", + "description":"Control whether the response should not be stored in the cluster if it completed within the provided [wait_for_completion] time (default: true)" + }, "keep_alive": { "type": "time", "description": "Update the time interval in which the results (partial or final) for this search will be available" @@ -41,6 +45,15 @@ "description":"The number of shard results that should be reduced at once on the coordinating node. This value should be used as the granularity at which progress results will be made available.", "default":5 }, + "pre_filter_shard_size":{ + "type":"number", + "description":"A threshold that enforces a pre-filter roundtrip to prefilter search shards based on query rewriting if the number of shards the search request expands to exceeds the threshold. This filter roundtrip can limit the number of shards significantly if for instance a shard can not match any documents based on it's rewrite method ie. if date filters are mandatory to match but the shard bounds and the query are disjoint.", + "default":1 + }, + "request_cache":{ + "type":"boolean", + "description":"Specify if request cache should be used for this request or not, defaults to true" + }, "analyzer":{ "type":"string", "description":"The analyzer to use for the query string" @@ -204,18 +217,10 @@ "type":"boolean", "description":"Specify whether to return sequence number and primary term of the last modification of each hit" }, - "request_cache":{ - "type":"boolean", - "description":"Specify if request cache should be used for this request or not, defaults to index level setting" - }, "max_concurrent_shard_requests":{ "type":"number", "description":"The number of concurrent shard requests per node this search executes concurrently. This value should be used to limit the impact of the search on the cluster in order to limit the number of concurrent shard requests", "default":5 - }, - "clean_on_completion":{ - "type":"boolean", - "description":"Control whether the response should not be stored in the cluster if it completed within the provided [wait_for_completion] time (default: true)" } }, "body":{ From b2fe1a46b1206cb05adef085e74d1d23c778ea08 Mon Sep 17 00:00:00 2001 From: Luca Cavanna Date: Wed, 18 Mar 2020 11:52:27 +0100 Subject: [PATCH 2/9] Apply suggestions from code review Co-Authored-By: James Rodewig --- docs/reference/async-search.asciidoc | 30 +++++------ docs/reference/search/async-search.asciidoc | 55 +++++++++++---------- 2 files changed, 45 insertions(+), 40 deletions(-) diff --git a/docs/reference/async-search.asciidoc b/docs/reference/async-search.asciidoc index a5e9771f2ab46..99616bb6a412f 100644 --- a/docs/reference/async-search.asciidoc +++ b/docs/reference/async-search.asciidoc @@ -1,20 +1,22 @@ [role="xpack"] [testenv="basic"] [[async-search-intro]] -== Long running searches +== Long-running searches -{es} generally allows to quickly search across big amounts of data. There are +{es} generally allows you to quickly search across big amounts of data. There are situations where a search executes on many many shards, possibly against -<> and spanning multiple <>, for which -results are not expected to be returned in milliseconds. When needing to -execute long running searches, sending a search request and synchronously -waiting for its results to be returned is not ideal. Async Search allows -instead to submit a search request that gets executed asynchronously and -monitor its progress as well as retrieve its results at a later stage. -Also, partial results can be retrieved as they become available, before -the search has completed. +<> and spanning multiple +<>, for which +results are not expected to be returned in milliseconds. When you need to +execute long-running searches, synchronously +waiting for its results to be returned is not ideal. Instead, Async search lets +you submit a search request that gets executed _asynchronously_, +monitor the progress of the request, and retrieve results at a later stage. +You can also retrieve partial results as they become available but +before the search has completed. -An async search request can be submitted using the <> API. -The <> API allows to monitor the progress of an async search -request and retrieve its results. An async search can be deleted through the -<> API. +You can submit tn async search request using the <> API. The <> API allows you to +monitor the progress of an async search request and retrieve its results. An +ongoing async search can be deleted through the <> API. diff --git a/docs/reference/search/async-search.asciidoc b/docs/reference/search/async-search.asciidoc index ad5baee42d43e..58f974111daf7 100644 --- a/docs/reference/search/async-search.asciidoc +++ b/docs/reference/search/async-search.asciidoc @@ -1,17 +1,17 @@ [role="xpack"] [testenv="basic"] [[async-search]] -=== Async Search +=== Async search -Async search consists of a set of API that allow to asynchronously execute a -search request, monitor its progress as well as retrieve its partial results -as soon as they become available. +The async search API let you asynchronously execute a +search request, monitor its progress, and retrieve partial results +as they become available. [[submit-async-search]] -==== Submit Async Search +==== Submit async search API -Submits a search request that gets executed asynchronously. It accepts the same -request as the <>. +Executes a search request asynchronously. It accepts the same +parameters and request body as the <>. [source,console,id=submit-async-search-date-histogram-example] -------------------------------------------------- @@ -33,8 +33,9 @@ POST /sales*/_async_search?size=0 // TEST[setup:sales] // TEST[s/size=0/size=0&wait_for_completion=0/] -The response contains the identifier of the search being executed, to be used -to retrieve the results at a later stage. Also, the currently available search +The response contains an identifier of the search being executed. +You can use this ID to later retrieve the search's final results. +The currently available search results are returned as part of the <> object. [source,console-result] @@ -86,25 +87,27 @@ It is possible to block and wait until the search is completed up to a certain timeout by providing the `wait_for_completion` parameter, which defaults to `1` second. -The Submit Async Search API supports the same <> +The submit async search API supports the same <> as the search API, though some have different default values: -`pre_filter_shard_size` defaults to `1`, `batched_reduce_size` defaults to -`5`, `request_cache` defaults to `true` and `ccs_minimize_roundtrips` -defaults to `false`. -It is also possible to specify how long the async search needs to be +* `pre_filter_shard_size` defaults to `1 +* `batched_reduce_size` defaults to `5` +* `request_cache` defaults to `true` +* `ccs_minimize_roundtrips` defaults to `false`. + +You can also specify how long the async search needs to be available through the `keep_alive` parameter, which defaults to `5d` (five days). -Running async searches as well as saved search results are cleared once their -corresponding `keep_alive` expires. +Ongoing async searches and any saved search results are deleted after this +period WARNING: Async search does not support scroll searches, nor search requests that only include the `suggest` section. Cross cluster searches can be executed using async search but only with `ccs_minimize_roundtrips` disabled. [[get-async-search]] -==== Get Async Search +==== Get async search -The Get Async Search API can be used to retrieve the results of a previously +The get async search API retrieves the results of a previously submitted async search request given its id. [source,console,id=get-async-search-date-histogram-example] @@ -178,17 +181,17 @@ completed up until the provided timeout. Final results will be returned if available before the timeout expires, otherwise the currently available results will be returned once the timeout expires. -The `keep_alive` parameter, which defaults to `5d` (five days) can be provided -to specify how long the async search should be available in the cluster. Once -an async search expires, it is cancelled if still running and its saved results -are deleted if already completed. +The `keep_alive` parameter, which defaults to `5d` (five days), specifies +how long the async search should be available in the cluster. When this +period expires, the search, if still running, is cancelled. If the search is +completed, its saved results are deleted. [[delete-async-search]] -==== Delete Async Search +==== Delete async search -The Delete Async Search API can be used to manually delete a specific async -search given its id. The corresponding async search will be cancelled if it's -still running, otherwise its saved results will be deleted. +You can use the delete async search API to manually delete an async search +by ID. If the search is still running, the search request will be cancelled. +Otherwise, the saved search results are deleted. [source,console,id=delete-async-search-date-histogram-example] -------------------------------------------------- From 414c4441ebdb4a2910e15881f7d42c6b0b82d3e8 Mon Sep 17 00:00:00 2001 From: Luca Cavanna Date: Wed, 18 Mar 2020 11:54:23 +0100 Subject: [PATCH 3/9] remove pre_filter_shard_size from async search spec --- .../resources/rest-api-spec/api/async_search.submit.json | 5 ----- 1 file changed, 5 deletions(-) diff --git a/x-pack/plugin/src/test/resources/rest-api-spec/api/async_search.submit.json b/x-pack/plugin/src/test/resources/rest-api-spec/api/async_search.submit.json index bfd2ee8ae94e7..468ece1dfd2d3 100644 --- a/x-pack/plugin/src/test/resources/rest-api-spec/api/async_search.submit.json +++ b/x-pack/plugin/src/test/resources/rest-api-spec/api/async_search.submit.json @@ -45,11 +45,6 @@ "description":"The number of shard results that should be reduced at once on the coordinating node. This value should be used as the granularity at which progress results will be made available.", "default":5 }, - "pre_filter_shard_size":{ - "type":"number", - "description":"A threshold that enforces a pre-filter roundtrip to prefilter search shards based on query rewriting if the number of shards the search request expands to exceeds the threshold. This filter roundtrip can limit the number of shards significantly if for instance a shard can not match any documents based on it's rewrite method ie. if date filters are mandatory to match but the shard bounds and the query are disjoint.", - "default":1 - }, "request_cache":{ "type":"boolean", "description":"Specify if request cache should be used for this request or not, defaults to true" From b74707dcb7828ce1ae9a4bae978cf80218e53821 Mon Sep 17 00:00:00 2001 From: Luca Cavanna Date: Wed, 18 Mar 2020 13:58:56 +0100 Subject: [PATCH 4/9] small corrections --- docs/reference/async-search.asciidoc | 2 +- docs/reference/search/async-search.asciidoc | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/reference/async-search.asciidoc b/docs/reference/async-search.asciidoc index 99616bb6a412f..d51e017b0f97c 100644 --- a/docs/reference/async-search.asciidoc +++ b/docs/reference/async-search.asciidoc @@ -15,7 +15,7 @@ monitor the progress of the request, and retrieve results at a later stage. You can also retrieve partial results as they become available but before the search has completed. -You can submit tn async search request using the <> API. The <> API allows you to monitor the progress of an async search request and retrieve its results. An ongoing async search can be deleted through the <> as the search API, though some have different default values: -* `pre_filter_shard_size` defaults to `1 +* `pre_filter_shard_size` defaults to `1` * `batched_reduce_size` defaults to `5` * `request_cache` defaults to `true` * `ccs_minimize_roundtrips` defaults to `false`. @@ -98,7 +98,7 @@ as the search API, though some have different default values: You can also specify how long the async search needs to be available through the `keep_alive` parameter, which defaults to `5d` (five days). Ongoing async searches and any saved search results are deleted after this -period +period WARNING: Async search does not support scroll searches, nor search requests that only include the `suggest` section. Cross cluster searches can be executed using From bbebf3c209c5eeb86d17d430a078b9b39499582f Mon Sep 17 00:00:00 2001 From: Luca Cavanna Date: Wed, 18 Mar 2020 22:56:36 +0100 Subject: [PATCH 5/9] small corrections --- docs/reference/api-conventions.asciidoc | 6 +-- .../reference/indices/rollover-index.asciidoc | 4 +- docs/reference/search/async-search.asciidoc | 45 +++++++++++-------- 3 files changed, 32 insertions(+), 23 deletions(-) diff --git a/docs/reference/api-conventions.asciidoc b/docs/reference/api-conventions.asciidoc index 41a09c9a15ff6..3921aab0d1354 100644 --- a/docs/reference/api-conventions.asciidoc +++ b/docs/reference/api-conventions.asciidoc @@ -87,7 +87,7 @@ GET /%3Clogstash-%7Bnow%2Fd%7D%3E/_search } ---------------------------------------------------------------------- // TEST[s/^/PUT logstash-2016.09.20\n/] -// TEST[s/now/2016.09.20||/] +// TEST[s/now/2016.09.20%7C%7C/] [NOTE] .Percent encoding of date math characters @@ -141,7 +141,7 @@ GET /%3Clogstash-%7Bnow%2Fd-2d%7D%3E%2C%3Clogstash-%7Bnow%2Fd-1d%7D%3E%2C%3Clogs } ---------------------------------------------------------------------- // TEST[s/^/PUT logstash-2016.09.20\nPUT logstash-2016.09.19\nPUT logstash-2016.09.18\n/] -// TEST[s/now/2016.09.20||/] +// TEST[s/now/2016.09.20%7C%7C/] [[common-options]] === Common options @@ -367,7 +367,7 @@ GET /_search?filter_path=hits.hits._source&_source=title&sort=rating:desc [float] ==== Flat Settings -The `flat_settings` flag affects rendering of the lists of settings. When the +The `flat_settings` flag affects rendering of the lists of settings. When the `flat_settings` flag is `true`, settings are returned in a flat format: [source,console] diff --git a/docs/reference/indices/rollover-index.asciidoc b/docs/reference/indices/rollover-index.asciidoc index b75961a52559e..42c4d9a8a5b44 100644 --- a/docs/reference/indices/rollover-index.asciidoc +++ b/docs/reference/indices/rollover-index.asciidoc @@ -300,7 +300,7 @@ POST /logs_write/_rollover <2> } } -------------------------------------------------- -// TEST[s/now/2016.10.31||/] +// TEST[s/now/2016.10.31%7C%7C/] <1> Creates an index named with today's date (e.g.) `logs-2016.10.31-1` <2> Rolls over to a new index with today's date, e.g. `logs-2016.10.31-000002` if run immediately, or `logs-2016.11.01-000002` if run after 24 hours @@ -339,7 +339,7 @@ over indices created in the last three days, you could do the following: GET /%3Clogs-%7Bnow%2Fd%7D-*%3E%2C%3Clogs-%7Bnow%2Fd-1d%7D-*%3E%2C%3Clogs-%7Bnow%2Fd-2d%7D-*%3E/_search -------------------------------------------------- // TEST[continued] -// TEST[s/now/2016.10.31||/] +// TEST[s/now/2016.10.31%7C%7C/] [[rollover-index-api-dry-run-ex]] diff --git a/docs/reference/search/async-search.asciidoc b/docs/reference/search/async-search.asciidoc index 3710002e1d538..ee68276e1bfea 100644 --- a/docs/reference/search/async-search.asciidoc +++ b/docs/reference/search/async-search.asciidoc @@ -87,22 +87,33 @@ It is possible to block and wait until the search is completed up to a certain timeout by providing the `wait_for_completion` parameter, which defaults to `1` second. +You can also specify how long the async search needs to be +available through the `keep_alive` parameter, which defaults to `5d` (five days). +Ongoing async searches and any saved search results are deleted after this +period. + +NOTE: When results are sorted by a numeric field, shards get sorted based on +minimum and maximum value that they hold for that field, hence partial +results become available following the sort criteria that was requested. + The submit async search API supports the same <> as the search API, though some have different default values: -* `pre_filter_shard_size` defaults to `1` -* `batched_reduce_size` defaults to `5` +* `batched_reduce_size` defaults to `5`: this affects how often partial results +become available., which happens whenever shard results are reduced. By default +a partial reduction is performed every time the coordinating node has received +5 new shard responses. * `request_cache` defaults to `true` -* `ccs_minimize_roundtrips` defaults to `false`. +* `pre_filter_shard_size` defaults to `1`: this is to enforce the execution of +a pre-filter roundtrip to retrieve statistics from each shard so that the ones +that surely don't hold any document matching the query can be skipped. +* `ccs_minimize_roundtrips` defaults to `false`, which is also the only +permitted value -You can also specify how long the async search needs to be -available through the `keep_alive` parameter, which defaults to `5d` (five days). -Ongoing async searches and any saved search results are deleted after this -period - -WARNING: Async search does not support scroll searches, nor search requests that -only include the `suggest` section. Cross cluster searches can be executed using -async search but only with `ccs_minimize_roundtrips` disabled. +WARNING: Async search does not support <> +nor search requests that only include the <>. +{ccs} is supported only with <> +set to `false`. [[get-async-search]] ==== Get async search @@ -171,19 +182,17 @@ and their partial results are also included in the response. <2> Partial aggregations results, coming from the shards that have already completed the execution of the query. -NOTE: When results are sorted by a numeric field, shards get sorted based on -minimum and maximum value that they hold for that field, hence partial -results become available following the sort criteria that was requested. - The `wait_for_completion` parameter, which defaults to `1`, can also be provided when calling the Get Async Search API, in order to wait for the search to be completed up until the provided timeout. Final results will be returned if available before the timeout expires, otherwise the currently available results will be returned once the timeout expires. -The `keep_alive` parameter, which defaults to `5d` (five days), specifies -how long the async search should be available in the cluster. When this -period expires, the search, if still running, is cancelled. If the search is +The `keep_alive` parameter specifies how long the async search should be +available in the cluster. When not specified, the `keep_alive` set with the +corresponding submit async request will be used. Otherwise, it is possible to +override such value and extend the validity of the request. When this period +expires, the search, if still running, is cancelled. If the search is completed, its saved results are deleted. [[delete-async-search]] From 08639b96e4c48d24462fcb13bee375e3581ba254 Mon Sep 17 00:00:00 2001 From: Luca Cavanna Date: Wed, 18 Mar 2020 23:00:41 +0100 Subject: [PATCH 6/9] small corrections --- docs/reference/redirects.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/redirects.asciidoc b/docs/reference/redirects.asciidoc index f501ab250a3ba..12987a8a6c294 100644 --- a/docs/reference/redirects.asciidoc +++ b/docs/reference/redirects.asciidoc @@ -375,8 +375,8 @@ See <>. === How {ccs} works See <> and <>. -[role="exclude",id="indices-component-templates"] +[role="exclude",id="indices-component-templates"] === Component template APIs coming::[7.x] From 134b02f9cf39600a360843271954268a511baada Mon Sep 17 00:00:00 2001 From: Luca Cavanna Date: Wed, 18 Mar 2020 23:22:33 +0100 Subject: [PATCH 7/9] note on security --- docs/reference/search/async-search.asciidoc | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/reference/search/async-search.asciidoc b/docs/reference/search/async-search.asciidoc index ee68276e1bfea..e72ff979715dc 100644 --- a/docs/reference/search/async-search.asciidoc +++ b/docs/reference/search/async-search.asciidoc @@ -118,8 +118,9 @@ set to `false`. [[get-async-search]] ==== Get async search -The get async search API retrieves the results of a previously -submitted async search request given its id. +The get async search API retrieves the results of a previously submitted +async search request given its id. The access to the results of a specific +async search is restricted to the user that submitted it in the first place. [source,console,id=get-async-search-date-histogram-example] -------------------------------------------------- From d7c00738a4ed33ae8589a3e9f728d82a5ab22b2a Mon Sep 17 00:00:00 2001 From: Luca Cavanna Date: Fri, 20 Mar 2020 11:36:30 +0100 Subject: [PATCH 8/9] iter --- docs/reference/search/async-search.asciidoc | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/docs/reference/search/async-search.asciidoc b/docs/reference/search/async-search.asciidoc index e72ff979715dc..75214151682c7 100644 --- a/docs/reference/search/async-search.asciidoc +++ b/docs/reference/search/async-search.asciidoc @@ -119,8 +119,9 @@ set to `false`. ==== Get async search The get async search API retrieves the results of a previously submitted -async search request given its id. The access to the results of a specific -async search is restricted to the user that submitted it in the first place. +async search request given its id. If the {es} {security-features} are enabled. +the access to the results of a specific async search is restricted to the user +that submitted it in the first place. [source,console,id=get-async-search-date-histogram-example] -------------------------------------------------- @@ -133,10 +134,10 @@ GET /_async_search/FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsd { "id" : "FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc=", "version" : 2, <1> - "is_partial" : true, - "is_running" : true, + "is_partial" : true, <2> + "is_running" : true, <3> "start_time_in_millis" : 1583945890986, - "expiration_time_in_millis" : 1584377890986, + "expiration_time_in_millis" : 1584377890986, <4> "response" : { "took" : 12144, "timed_out" : false, @@ -155,7 +156,7 @@ GET /_async_search/FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsd "max_score" : null, "hits" : [ ] }, - "aggregations" : { <2> + "aggregations" : { <5> "sale_date" : { "buckets" : [] } @@ -180,7 +181,10 @@ additional results compared to previously obtained responses. If the version stays the same, no new results have become available, otherwise a higher version number indicates that more shards have completed their execution of the query and their partial results are also included in the response. -<2> Partial aggregations results, coming from the shards that have already +<2> Whether the returned search results are partial or final +<3> Whether the search is still being executed or it has completed +<4> When the async search will expire +<5> Partial aggregations results, coming from the shards that have already completed the execution of the query. The `wait_for_completion` parameter, which defaults to `1`, can also be provided From d804de012f36e5e775f9fb5d75d66229b103c34a Mon Sep 17 00:00:00 2001 From: Luca Cavanna Date: Fri, 20 Mar 2020 11:42:10 +0100 Subject: [PATCH 9/9] iter --- docs/reference/search/async-search.asciidoc | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/reference/search/async-search.asciidoc b/docs/reference/search/async-search.asciidoc index 75214151682c7..e58455146fb82 100644 --- a/docs/reference/search/async-search.asciidoc +++ b/docs/reference/search/async-search.asciidoc @@ -100,15 +100,15 @@ The submit async search API supports the same <> nor search requests that only include the <>.