Destination Redshift: Implement refreshes

[edgao]

new test class (RedshiftS3StagingStorageOperationTest) verifies some of the error handling behavior:

• creating+writing to temp raw tables
• overwriting the real raw table from the temp raw table
• copying records from the temp raw table to the real raw table

tested manually in https://cloud.airbyte.com/workspaces/b61bc266-ef3c-460c-af2f-f70da4a2993c/connections/e1e1f841-ca63-4524-9c87-9da81f928a79/job-history with a merge refresh + normal sync:

• no airbyte_tmp tables after successful sync
• generation IDs correctly incremented, OVERWRITE stream correctly truncated

sample queries:

SELECT 'products' as stream, _airbyte_generation_id, count(1) FROM "integrationtests"."v2_staging_inserts"."products" group by _airbyte_generation_id
UNION ALL
SELECT 'purchases' as stream, _airbyte_generation_id, count(1) FROM "integrationtests"."v2_staging_inserts"."purchases" group by _airbyte_generation_id
UNION ALL
SELECT 'users' as stream, _airbyte_generation_id, count(1) FROM "integrationtests"."v2_staging_inserts"."users" group by _airbyte_generation_id
order by stream;


SELECT 'products' as stream, _airbyte_generation_id, count(1) FROM "integrationtests"."airbyte_internal"."v2_staging_inserts_raw__stream_products" group by _airbyte_generation_id
UNION ALL
SELECT 'purchases' as stream, _airbyte_generation_id, count(1) FROM "integrationtests"."airbyte_internal"."v2_staging_inserts_raw__stream_purchases" group by _airbyte_generation_id
UNION ALL
SELECT 'users' as stream, _airbyte_generation_id, count(1) FROM "integrationtests"."airbyte_internal"."v2_staging_inserts_raw__stream_users" group by _airbyte_generation_id
order by stream;

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 12, 2024 6:03pm

[edgao]

• Destination Redshift: Implement refreshes #40567 : 2 dependent PRs (#40689 , #40712 ) 👈
• Destination Redshift: Add generation_id and sync_id #40201
• Destination redshift: Use new interfaces #39008 : 1 other dependent PR (#39334 )
• Destination redshift: Remove encrypted staging option #39141
• Destination redshift: convert test classes to kotlin #38893
• Destination redshift: remove standard inserts #38886
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @edgao and the rest of your teammates on Graphite

[edgao]

/publish-java-cdk

🕑 https://github.com/airbytehq/airbyte/actions/runs/9912026151
✅ Successfully published Java CDK version=0.41.2!