Destination S3: add delta lake/delta table support

[mustafa-rmd]

My current requirement is to have the following data pipeline:
PostgreSQL (Source)
Air byte
Minio - S3 storage (Destination)
Apache spark configure with (Minio and Delta lake formatting) since spark doesn’t support ACID transactions.

The goals to have air bye move data from PostgreSQL (Source) to Minio storage (Destination) saved in delta format. Spark then will come and read data from S3 expected to be with delta format.

My main issue with the output format for Air bye S3 connector. Currently is only supports 3 data types: CSV, Avro and JSON Lines (JSONL).

What is the recommend way to solve this problem? since I think, many companies are trying to build this data pipeline.
Is there plan to have this feature released in upcoming releases?
Should we implement this feature? If so, is there a good documentation of how to start about it?
Or, is there another method of going about it?

Thanks,

[natalyjazzviolin]

Hi Mustafa M Al-Ramadan (@mustafa-rmd) , could you please edit your request to follow our feature request template? This will ensure all details are understood clearly. I've copied it below. Thank you!

Tell us about the problem you're trying to solve

What are you trying to do, and why is it hard? A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you’d like

A clear and concise description of what you want to see happen, or the change you would like to see

Describe the alternative you’ve considered or used

A clear and concise description of any alternative solutions or features you've considered or are using today.

Additional context

Add any other context or screenshots about the feature request here.

Are you willing to submit a PR?

Remove this with your answer :-)