Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

change read strategy from pagination to scroll #40

Closed
costin opened this issue May 1, 2013 · 0 comments
Closed

change read strategy from pagination to scroll #40

costin opened this issue May 1, 2013 · 0 comments

Comments

@costin
Copy link
Member

costin commented May 1, 2013

The current approach of using pagination is not very effective when dealing with lots of data and should be changed to the Scroll API for better performance:
http://www.elasticsearch.org/guide/reference/api/search/search-type/

costin added a commit that referenced this issue May 8, 2013
replace pagination querying with scan/scroll

introduce per shard input split
introduce query builder

notes:
main commit that adds the core functionality and tests passing within
the same VM
integration tests are failing when running inside Hadoop (no active
shards available)
fixes #40
fixes #2
@costin costin closed this as completed in 10b9729 May 13, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant