Skip to content

Commit

Permalink
Destination Redshift: Update CDK, revert signatures to add back unuse…
Browse files Browse the repository at this point in the history 
…d methods and vars (airbytehq#34467)

## What
* Reverting few signature changes from airbytehq#34186
  • Loading branch information
@gisripa @jatinyadav-cc
gisripa authored and jatinyadav-cc committed Feb 26, 2024
1 parent 32a316c commit 10539e6
Show file tree
Hide file tree
Showing 5 changed files with 25 additions and 6 deletions.
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/destination-redshift/build.gradle
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ plugins {
}

airbyteJavaConnector {
cdkVersionRequired = '0.13.3'
cdkVersionRequired = '0.14.0'
features = ['db-destinations', 's3-destinations', 'typing-deduping']
useLocalCdk = false
}
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/destination-redshift/metadata.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ data:
connectorSubtype: database
connectorType: destination
definitionId: f7a7d195-377f-cf5b-70a5-be6b819019dc
dockerImageTag: 2.0.0
dockerImageTag: 2.1.0
dockerRepository: airbyte/destination-redshift
documentationUrl: https://docs.airbyte.com/integrations/destinations/redshift
githubIssueLabel: destination-redshift
Expand Down
14 changes: 11 additions & 3 deletions .../airbyte/integrations/destination/redshift/operations/RedshiftS3StagingSqlOperations.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -71,14 +71,20 @@ public String getStagingPath(final UUID connectionId,
}

@Override
public void createStageIfNotExists() throws Exception {
public String getStageName(String namespace, String streamName) {
return "garbage-unused";
}

@Override
public void createStageIfNotExists(final JdbcDatabase database, final String stageName) throws Exception {
s3StorageOperations.createBucketIfNotExists();
}

@Override
public String uploadRecordsToStage(final JdbcDatabase database,
final SerializableBuffer recordsData,
final String schemaName,
final String stageName,
final String stagingPath)
throws Exception {
return s3StorageOperations.uploadRecordsToBucket(recordsData, schemaName, stagingPath);
Expand All @@ -92,6 +98,7 @@ private String putManifest(final String manifestContents, final String stagingPa

@Override
public void copyIntoTableFromStage(final JdbcDatabase database,
final String stageName,
final String stagingPath,
final List<String> stagedFiles,
final String tableName,
Expand Down Expand Up @@ -160,8 +167,9 @@ private static String getManifestPath(final String s3BucketName, final String s3
}

@Override
public void dropStageIfExists(final JdbcDatabase database, final String stageName) throws Exception {
s3StorageOperations.dropBucketObject(stageName);
public void dropStageIfExists(final JdbcDatabase database, final String stageName, final String stagingPath) throws Exception {
// stageName is unused here but used in Snowflake. This interface needs to be fixed.
s3StorageOperations.dropBucketObject(stagingPath);
}

}
10 changes: 10 additions & 0 deletions ...airbyte/integrations/destination/redshift/typing_deduping/RedshiftDestinationHandler.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,16 @@ public void execute(final Sql sql) throws Exception {
}
}

/**
* Issuing a select 1 limit 1 query can be expensive, so relying on SVV_TABLE_INFO system table.
* EXPLAIN of the select 1 from table limit 1 query: (seq scan and then limit is applied, read from
* bottom to top) XN Lim it (co st=0. 0 .0.01 rows=1 width=0) -> XN Seq Scan on _airbyte_raw_ users
* (cost=0.00..1000.00 rows=100000 width=0)
*
* @param id
* @return
* @throws Exception
*/
@Override
public boolean isFinalTableEmpty(final StreamId id) throws Exception {
// Redshift doesn't have an information_schema.tables table, so we have to use SVV_TABLE_INFO.
Expand Down
3 changes: 2 additions & 1 deletion docs/integrations/destinations/redshift.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -237,6 +237,7 @@ Each stream will be output into its own raw table in Redshift. Each table will c

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:-----------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 2.1.0 | 2024-01-24 | [34467](https://github.com/airbytehq/airbyte/pull/34467) | Upgrade CDK to 0.14.0 |
| 2.0.0 | 2024-01-23 | [\#34077](https://github.com/airbytehq/airbyte/pull/34077) | Destinations V2 |
| 0.8.0 | 2024-01-18 | [\#34236](https://github.com/airbytehq/airbyte/pull/34236) | Upgrade CDK to 0.13.0 |
| 0.7.15 | 2024-01-11 | [\#34186](https://github.com/airbytehq/airbyte/pull/34186) | Update check method with svv_table_info permission check, fix bug where s3 staging files were not being deleted. |
Expand Down Expand Up @@ -317,4 +318,4 @@ Each stream will be output into its own raw table in Redshift. Each table will c
| 0.3.14 | 2021-10-08 | [\#5924](https://github.com/airbytehq/airbyte/pull/5924) | Fixed AWS S3 Staging COPY is writing records from different table in the same raw table |
| 0.3.13 | 2021-09-02 | [\#5745](https://github.com/airbytehq/airbyte/pull/5745) | Disable STATUPDATE flag when using S3 staging to speed up performance |
| 0.3.12 | 2021-07-21 | [\#3555](https://github.com/airbytehq/airbyte/pull/3555) | Enable partial checkpointing for halfway syncs |
| 0.3.11 | 2021-07-20 | [\#4874](https://github.com/airbytehq/airbyte/pull/4874) | allow `additionalProperties` in connector spec |
| 0.3.11 | 2021-07-20 | [\#4874](https://github.com/airbytehq/airbyte/pull/4874) | allow `additionalProperties` in connector spec |

0 comments on commit 10539e6

Please sign in to comment.