You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
As a user of the OpenSearch source, I would like to only do a one-time list of the indices rather than periodically pull new indices.
Describe the solution you'd like
An option in the OpenSearch source to configure the number of cat/indices calls to make.
Would only list indexes once, process those indexes, and then stop listing and working completely.
This can be done in a similar way as the S3 Source, which will only has a configurable scan count that will block subsequent scans once that scan count is reached (
Is your feature request related to a problem? Please describe.
As a user of the OpenSearch source, I would like to only do a one-time list of the indices rather than periodically pull new indices.
Describe the solution you'd like
An option in the OpenSearch source to configure the number of
cat/indices
calls to make.Would only list indexes once, process those indexes, and then stop listing and working completely.
This can be done in a similar way as the S3 Source, which will only has a configurable scan count that will block subsequent scans once that scan count is reached (
data-prepper/data-prepper-plugins/s3-source/src/main/java/org/opensearch/dataprepper/plugins/source/s3/S3ScanPartitionCreationSupplier.java
Line 73 in 2f9bed8
data-prepper/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/source/opensearch/worker/OpenSearchIndexPartitionCreationSupplier.java
Line 65 in 2f9bed8
Describe alternatives you've considered (Optional)
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
The text was updated successfully, but these errors were encountered: