Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use point-in-time (PIT) search instead of scroll search in OpenSearch connector #20834

Open
aravind-musigumpula opened this issue Feb 25, 2024 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@aravind-musigumpula
Copy link

Can we use PIT search instead of scroll search in opensearch connector?
It is mentioned that PIT search is preferred when deep pagination is done:
Point in Time (PIT) with search_after is the preferred pagination method in OpenSearch, especially for deep pagination. It bypasses the limitations of all other methods because it operates on a dataset that is frozen in time, it is not bound to a query, and it supports consistent pagination going forward and backward.
https://opensearch.org/docs/latest/search-plugins/searching-data/paginate/

It is mentioned to use PIT in elastic discussion also: https://discuss.elastic.co/t/sliced-scroll-with-sort/284874
It would be nice to have a feature to enable search using PIT to gain performance benefits when searching on huge datasets.

@wendigo
Copy link
Contributor

wendigo commented Feb 25, 2024

Sounds like a good idea. Contributions are welcome!

@hashhar hashhar added the enhancement New feature or request label Feb 26, 2024
@hashhar hashhar changed the title Can we use PIT search instead of scroll search in opensearch connector? Use point-in-time (PIT) search instead of scroll search in OpenSearch connector Feb 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Development

Successfully merging a pull request may close this issue.

3 participants