Remove (dfs_)query_and_fetch from the REST API

[dobariya]

According to Elastic search if working with Query_and_fetch will return size times number of shards hits.Following this rule I implemented below query using Nest having 5 shards

    int from = 0;
    int size = 1;

     while (true)
    {
        var result = client.Search<object>(x => x
       .SearchType(SearchType.QueryAndFetch)
       .Size(size)
       .From(from * size)
       .Query(qr => qr.MatchAll())
       );

        if (result.Hits.Count() == 0)
        {
            break;
        }
       from++;
    }

If I have total 17 records in index then according to the rule above query should execute 4 times returning 5,5,5,2 = 17 total records but this is not the case in the above query.
Above query executes 4 times returning 5,3,3,1 = 12 records missing 5 records.
Am I doing anything wrong or missing anything?

[clintongormley]

Hi @dobariya

Essentially, query_and_fetch doesn't support multi-shard pagination. It is an optimized query mode used internally when only a single shard is involved. In fact, we're going to remove the ability to set query_and_fetch in the REST API - being able to specify it provides no advantage at all.

[dobariya]

Hi @clintongormley

Yes that make sense.Please let me know if search type=scan will be a good practice to implement for the above mentioned code? I am having 5 million records indexed

[clintongormley]

@dobariya scanning WITH SCROLL is the answer you need, not pagination. Deep pagination is very costly. Scrolling allows you to keep pulling results until you're done, and scanning disables sorting to make that efficient.

See http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/scan-scroll.html

[clintongormley]

Didn't mean to close - we should still implement this change