Generic Source Plugin

[ashoktelukuntla]

Is your feature request related to a problem? Please describe.

Users are looking to migrate data in to OpenSearch. Source of data is from existing self managed, managed clusters of OpenSearch. The plugin should enable users read data, transform and write to OpenSearch clusters. This feature is useful in many ways not limiting to migration of data but also replaying , reindexing.

Describe the solution you'd like

Create a source plugin which would enable users to bulk read, bulk write on a scheduled manner to a given OpenSearch cluster. This plugin should be extendable to take user defined additional sources. Users should be able to create/schedule pipeline for migration of data by

• Auto discovery i.e. Listing all the indexes or Take given Index
• Iterate over a index , read/fetch complete data
• Enrich/transform Data (Optional)
• Sink to OpenSearch using Data Prepper
• Reconcile/report comparing source and sink data

Cron can be used to schedule the migration of data. Example: schedule: "* * * * *" ' will load data every minute

Additional context

Plugin should be able to take configurations data related to cluster including hostname:port, user credentials, optional - index and query (e.g. match_all) .

I would envision the following sequence of steps

1. cat indices - https://opensearch.org/docs/1.2/opensearch/rest-api/cat/cat-indices/
2. Iterate over an index
3. Query index i.e. match_all or scroll query for a large indices
4. Enrich/transform Data (Optional)
5. Data Prepper pipeline to ingest data in to opensearch
6. Report on data from sink and source

**References: **

This should be similar to the logstash-input-opensearch-plugin provided in the OpenSearch project.

https://opensearch.org/blog/community/2022/05/introducing-logstash-input-opensearch-plugin-for-opensearch/
https://github.com/opensearch-project/logstash-input-opensearch

[dlvenable]

@ashoktelukuntla , Thank you for this issue. This is very similar to the #1985 issue. Can you add your inputs into that issue so that we can incorporate it into the final solution?