Elasticsearch enable Point In Time based searches #30824
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Run Java_ElasticSearch_IO_Direct PreCommit |
…ion entry points for PIT sort properties
|
Run Java_ElasticSearch_IO_Direct PreCommit |
|
Run Java_ElasticSearch_IO_Direct PreCommit |
|
Assigning reviewers. If you would like to opt out of this review, comment R: @Abacn for label java. Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
Reminder, please take a look at this pr: @Abacn @johnjcasey |
|
sorry for delay, taking a look |
Abacn
reviewed
Apr 18, 2024
...sts-common/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTestCommon.java
Show resolved
Hide resolved
...ava/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java
Outdated
Show resolved
Hide resolved
Abacn
reviewed
Apr 30, 2024
...ava/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java
Outdated
Show resolved
Hide resolved
...ava/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java
Outdated
Show resolved
Hide resolved
Abacn
approved these changes
Apr 30, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change adds a BoundedReader implementation based on the newer Point In Time search API (see doc). This API is the recommended mode when reading data from large indexes that imply retrieving documents using deep iteration (more than 10000 hits per slice).
This mode should be available for Elasticsearch clusters with version 8 and up.
When using the default read implementation, based on Scrolls, when the index is big enough and the number of documents is large, the creation of slices with more than 10000 elements to be iterated would make the read operations to fail. Now, with the PIT implementation, this is not a problem since ES does not have to keep track of the whole state of the index while iterating, only the current window for the PIT iterator.
Adding @lord-skinner from Elastic.co for visibility.