Add search-exists API to check if any matching documents exist for a given query

[areek]

Implements a new Exists API allowing users to do fast exists check on any matched documents for a given query.

This API should be faster then using the Count API as it will:

• early terminate the search execution once any document is found to exist
• return the response as soon as the first shard reports matched documents

closes #6995

[areek]

The request format would be identical to that of the Count API:

curl -XGET 'http://localhost:9200/_exists' -d '
{
    "query" : {
        ...
    }
}'

Additionally one can specify preference, type, min_score and routing. Additionally filter on index(s) and type(s).

The Response would be as simple as:

{
    "exists": true,
    "_shards": {
       ...
    }
}

I have intentionally not integrated it with the _search endpoint, as yet another flag has to be introduced in the response, as hits would not be a good container for the exists flag.

[areek]

TODO:

• Figure out a nicer way of early termination of shard requests, any feedback on this would be greatly appreciated! (see: https://github.com/elasticsearch/elasticsearch/pull/7026/files#diff-bdd1ee7f7701d4264050053b19cacaebR207)
• Documentation
• Rest Spec/Tests

[areek]

Updated PR:

• fixed concurrency error for early termination of shard requests (thanks @kimchy)

TODO:

• Documentation
• Rest Spec/Tests

[areek]

@clintongormley would be awesome if you could take a look at the docs for the new API. The docs are quite similar to that of the count API.

TODO:

• rest spec/tests

[clintongormley]

Hi @areek

This looks good, but I think the REST API needs to change to be consistent. Exists requests in the REST layer are implemented as HEAD requests which don't return a body, just a 200 or a 404. (they can return other HTTP error codes if some other error occurs)

I think this should be:

HEAD /{index}/{type}/_search

And in the rest specs, it should probably be called search_exists (can you think of a better name?) to avoid confusion with the exists API.

[areek]

@clintongormley I changed the REST API to handle HEAD, I still have support for GET and POST due to users ability to send a request body.

Regarding the naming, I was thinking hit_exists. Thoughts?

[areek]

@clintongormley As far as I have seen, it seems like we don't have an explicit 'exists' API. I am assuming you are referring to HEAD /{index}/{type}/{id} in the GET API used to determine doc existence based on id. So I think it is ok to use _exists endpoint here. Thoughts always welcome and thanks for suggesting HTTP HEAD!

The updated REST resp/request are as follows:

curl -XHEAD -i "localhost:9200/{index|null}/{type|null}/_exists" -d '
{
  "query": ...
}
'
or 
curl -XHEAD -i "localhost:9200/{index|null}/{type|null}/_exists?q=.."

The HTTP GET and POST are also supported for having request body.

The response header is a simple:

HTTP/1.1 200 OK
Content-Type: text/plain; charset=UTF-8
Content-Length: 0

or

HTTP/1.1 404 Not Found
Content-Type: text/plain; charset=UTF-8
Content-Length: 0

The response will not have any content for any of the supported HTTP methods.

[clintongormley]

@areek Bah, I'd completely forgotten about the issue with HEAD and bodies! Apologies. And yes, by "exists" API I was referring to HEAD /{index}/{type}/{id}.

OK, so rethinking this: We already have /_search/scroll, /_search/percolate, /_search/template, wondering if this should be /_search/exists.

Also, the clients expect a body in the response for all requests except for HEAD requests, but as you've pointed out, eg the JS client won't be able to use a HEAD request here, so it looks like returning a body will be required.

Sorry for the runaround

[s1monw]

aside of @clintongormley comments this looks fantastic!

[areek]

@clintongormley I have updated the REST req/res as follows:

• Support HTTP HEAD, GET and POST methods
  • If HTTP HEAD is used, no body is returned only the status
  • else response body looks as follows:

     {
       "exists" : true | false
     }

• Added additional endpoint /_search/exists (/_exists can be used to as described in previous comment)
• Updated Docs

[clintongormley]

Call this the "Search Exists API" to make it clear which "exists" functionality it refers to in the docs search dropdown.

[areek]

renamed to Search Exists API.

[clintongormley]

Hi @areek

I'm tempted to remove support for HEAD and for /_exists completely:

• Remove HEAD because, even for clients where it works, there may be a proxy in-between the client and ES where it doesn't work, so we'll all end up implementing it as GET or POST anyway.
• Remove /_exists because it uses up a namespace which is more explicitly described by /_search/exists. /{index}/_exists looks like it would check the existence of an index.

So I think this should be exposed only as :

GET | POST /_search/exists
GET | POST /{index}/_search/exists
GET | POST /{index}/{type}/_search/exists

[clintongormley]

also, the response body looks good.

[areek]

Hi @clintongormley thanks for the review. I have updated the documentation and the REST endpoints as suggested.

[clintongormley]

LGTM

[areek]

I think this is ready. I will commit this soon, if there are no objections.