[Epic] Support schema evolution

[grishick]

User story

As an Operator/Analytics Engineer/Data Scientist/Data Analyst when i set up a new connection, I would like to be notified when source schemas are changing but my syncs should keep working, migrating data accordingly. Since data is tied to my business, I would like to be notified and be able to provide custom directives on how to migrate my old data and how to backfill accordingly (if needed).

Furthermore, to improve UX, Airbyte should support default actions for some typical scenarios, such as:

• add column
• remove column
• column data type change
• column renamed (if this can be detected)

[persunde]

It would be amazing if we could add extra columns without deleting all the existing data.
We transport data from database A to database B.
But A does not keep all the data, and deletes old rows. But we want to save all original rows in B. When we change the columns on table in database A, then the sync to B delete all rows in B and that is an issue for us.
Is it possible to solve this?
Keep the original data in destination when you change the columns on the source table.
Or at least keep rows that exists in destination, but no longer exists in source.

Edit: Sorry. I think I was testing during a sync and the normalized table was truncated, but the data was in the raw table and eventually the data deleted in source was available in destination.

[Eloisee]

It would be great if we could detect and add new tables in the source automatically .
In the bigquery source, we connect with the dataset and there may be new tables day by day , and we would like to sync it without any manual work.

[alepietrobon]

just checking, is this what is listed in the roadmap under "schema evolution"? I'm really keen on understanding progress on this work. Thanks!

[shaidar]

Would be interested in hearing more about if this is still in progress.

[andrevus]

I'm interested in too. Do you have any update?

[phdmohamedali]

any updates?

[brendavarguez]

Is this still in progress?

[malikdiarra]

Hi, this is now available in v0.50.1+, you can choose how you want schema to be replicated in the replication tab of your connection