[Feature][jira] Jira changelog extractor and converter support Incremental Mode

[klesh]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Use case

Reduce the time to crunching massive amounts of data

Description

Currently, Extractors and Converters operate exclusively in Full Sync mode, which involves deleting all target data and regenerating it via a Delete + Insert process. While effective, this approach poses several issues:

1. Scalability Concerns: As the volume of records increases in the source tables, the time required for conversion scales linearly. In particular, operations such as the jira issue changelogs extraction and conversion have been reported to take up to half an hour. This is significantly slower than the data collection phase, impacting overall efficiency.
2. Database Efficiency: Running in Full Sync mode tends to cause database bloat, particularly in databases like PostgreSQL. This bloat is evidenced by the table size being disproportionately large compared to the actual data stored — in some cases, as extreme as 18GB of space used for 1GB of actual data.

Proposed Solution:

I propose that extractors and converters should be enhanced to support Incremental Mode. This mode would enable the components to only process and insert new or changed data since the last collection, rather than performing a full refresh each time. This would likely yield the following benefits:

1. Reduced Processing Time: Incremental updates would significantly reduce the time required for data conversion, as only new or changed records would be processed.
2. Improved Database Performance: By avoiding the deletion and re-insertion of large volumes of data, we can prevent database bloat, leading to better utilization of resources and potentially lower storage costs.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct