Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove (dfs_)query_and_fetch from the REST API #9606

Closed
dobariya opened this issue Feb 7, 2015 · 4 comments · Fixed by #10864

Comments

@dobariya
Copy link

commented Feb 7, 2015

According to Elastic search if working with Query_and_fetch will return size times number of shards hits.Following this rule I implemented below query using Nest having 5 shards

    int from = 0;
    int size = 1;

     while (true)
    {
        var result = client.Search<object>(x => x
       .SearchType(SearchType.QueryAndFetch)
       .Size(size)
       .From(from * size)
       .Query(qr => qr.MatchAll())
       );

        if (result.Hits.Count() == 0)
        {
            break;
        }
       from++;
    }

If I have total 17 records in index then according to the rule above query should execute 4 times returning 5,5,5,2 = 17 total records but this is not the case in the above query.
Above query executes 4 times returning 5,3,3,1 = 12 records missing 5 records.
Am I doing anything wrong or missing anything?

@clintongormley

This comment has been minimized.

Copy link
Member

commented Feb 12, 2015

Hi @dobariya

Essentially, query_and_fetch doesn't support multi-shard pagination. It is an optimized query mode used internally when only a single shard is involved. In fact, we're going to remove the ability to set query_and_fetch in the REST API - being able to specify it provides no advantage at all.

@clintongormley clintongormley changed the title Pagination using search type Query_and_fetch Remove (dfs_)query_and_fetch from the REST API Feb 12, 2015
@dobariya

This comment has been minimized.

Copy link
Author

commented Feb 13, 2015

Hi @clintongormley

Yes that make sense.Please let me know if search type=scan will be a good practice to implement for the above mentioned code? I am having 5 million records indexed

@clintongormley

This comment has been minimized.

Copy link
Member

commented Feb 13, 2015

@dobariya scanning WITH SCROLL is the answer you need, not pagination. Deep pagination is very costly. Scrolling allows you to keep pulling results until you're done, and scanning disables sorting to make that efficient.

See http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/scan-scroll.html

@clintongormley

This comment has been minimized.

Copy link
Member

commented Feb 13, 2015

Didn't mean to close - we should still implement this change

clintongormley added a commit that referenced this issue Mar 2, 2015
Changed search_type docs to reflect that the `(dfs_)query_and_fetch` modes are an internal optimization and should not be specified explicitly by the user.

Relates to #9606
clintongormley added a commit that referenced this issue Mar 2, 2015
Changed search_type docs to reflect that the `(dfs_)query_and_fetch` modes are an internal optimization and should not be specified explicitly by the user.

Relates to #9606
clintongormley added a commit that referenced this issue Mar 2, 2015
Changed search_type docs to reflect that the `(dfs_)query_and_fetch` modes are an internal optimization and should not be specified explicitly by the user.

Relates to #9606
clintongormley added a commit that referenced this issue Mar 2, 2015
Changed search_type docs to reflect that the `(dfs_)query_and_fetch` modes are an internal optimization and should not be specified explicitly by the user.

Relates to #9606
mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015
Changed search_type docs to reflect that the `(dfs_)query_and_fetch` modes are an internal optimization and should not be specified explicitly by the user.

Relates to elastic#9606
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.