Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Elasticsearch Scroll API #16

Open
rmagen opened this issue May 11, 2016 · 1 comment
Open

Elasticsearch Scroll API #16

rmagen opened this issue May 11, 2016 · 1 comment

Comments

@rmagen
Copy link
Member

rmagen commented May 11, 2016

Use elasticsearch query Scroll API to iterate many results in the StartStep.
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html

According to the docs "Scrolling is not intended for real time user requests, but rather for processing large amounts of data, e.g. in order to reindex the contents of one index into a new index with a different configuration". Should we use Scroll in Unipop? If so, should we always use it? maybe only in StartStep (Assuming most times it will iterate a large amount of results).

The scroll functionality is already implemented in QueryIterator, but its currently unused.

@rmagen rmagen modified the milestones: Unipop 0.2, Unipop 0.3 May 11, 2016
@rmagen rmagen changed the title Query Scroll on StartStep Elasticsearch Scroll API May 13, 2016
@GurRonen GurRonen modified the milestone: Unipop 0.3 Jun 7, 2016
@rmagen rmagen removed the elastic2 label Jun 18, 2016
@rmagen rmagen added this to the Unipop 0.3 milestone Nov 15, 2016
@seanbarzilay
Copy link
Member

need to implement #103 first

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants