Search: Add search type scan allowing to efficiently scan large result set

[kimchy]

The scan search type allows to efficiently scroll a large result set. Its used first by executing a search request with scrolling and a query:

curl -XGET 'localhost:9200/_search?search_type=scan&scroll=10m&size=50' -d '
{
    "query" : {
        "match_all" : {}
    }
}
'

The scroll parameter control the keep alive time of the scrolling request and initiates the scrolling process. The timeout applies per round trip (i.e. between the previous scan scroll request, to the next).

The response will include no hits, with two important results, the total_hits will include the total hits that match the query, and the scroll_id that allows to start the scroll process. From this stage, the _search/scroll endpoint should be used to scroll the hits, feeding the next scroll request with the previous search result scroll_id. For example:

curl -XGET 'localhost:9200/_search/scroll?scroll=10m' -d 'c2NhbjsxOjBLMzdpWEtqU2IyZHlmVURPeFJOZnc7MzowSzM3aVhLalNiMmR5ZlVET3hSTmZ3OzU6MEszN2lYS2pTYjJkeWZVRE94Uk5mdzsyOjBLMzdpWEtqU2IyZHlmVURPeFJOZnc7NDowSzM3aVhLalNiMmR5ZlVET3hSTmZ3Ow=='

The "breaking" condition out of a scroll is when no hits has been returned. The total_hits will be maintained between scroll requests.

Note, scan search type does not support sorting (either on score or a field) or faceting.

[kimchy]

Search: Add search type scan allowing to efficiently scan large result set, closed by 818f3b4.

[karussell]

I noticed an exception: http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/f1bd216132ad4728