From 411115b7350e19e2459e56a1668d1f2962c6bc3c Mon Sep 17 00:00:00 2001 From: Gireesh Sreepathi Date: Tue, 19 Mar 2024 11:39:50 -0700 Subject: [PATCH] Destination Redshift: Certification & docs update (#36255) --- .../connectors/destination-redshift/metadata.yaml | 6 +++--- docs/integrations/destinations/redshift.md | 13 +++++++++---- 2 files changed, 12 insertions(+), 7 deletions(-) diff --git a/airbyte-integrations/connectors/destination-redshift/metadata.yaml b/airbyte-integrations/connectors/destination-redshift/metadata.yaml index ca3f84239371b..9a9b637d8b73d 100644 --- a/airbyte-integrations/connectors/destination-redshift/metadata.yaml +++ b/airbyte-integrations/connectors/destination-redshift/metadata.yaml @@ -5,7 +5,7 @@ data: connectorSubtype: database connectorType: destination definitionId: f7a7d195-377f-cf5b-70a5-be6b819019dc - dockerImageTag: 2.3.0 + dockerImageTag: 2.3.1 dockerRepository: airbyte/destination-redshift documentationUrl: https://docs.airbyte.com/integrations/destinations/redshift githubIssueLabel: destination-redshift @@ -24,14 +24,14 @@ data: This version introduces [Destinations V2](https://docs.airbyte.com/release_notes/upgrading_to_destinations_v2/#what-is-destinations-v2), which provides better error handling, incremental delivery of data for large syncs, and improved final table structures. To review the breaking changes, and how to upgrade, see [here](https://docs.airbyte.com/release_notes/upgrading_to_destinations_v2/#quick-start-to-upgrading). These changes will likely require updates to downstream dbt / SQL models, which we walk through [here](https://docs.airbyte.com/release_notes/upgrading_to_destinations_v2/#updating-downstream-transformations). Selecting `Upgrade` will upgrade **all** connections using this destination at their next sync. You can manually sync existing connections prior to the next scheduled sync to start the upgrade early. upgradeDeadline: "2024-03-15" - releaseStage: beta resourceRequirements: jobSpecific: - jobType: sync resourceRequirements: memory_limit: 1Gi memory_request: 1Gi - supportLevel: community + releaseStage: generally_available + supportLevel: certified supportsDbt: true tags: - language:java diff --git a/docs/integrations/destinations/redshift.md b/docs/integrations/destinations/redshift.md index 6495e287e6deb..9c50c40517268 100644 --- a/docs/integrations/destinations/redshift.md +++ b/docs/integrations/destinations/redshift.md @@ -187,10 +187,14 @@ characters. ### Data Size Limitations Redshift specifies a maximum limit of 16MB (and 65535 bytes for any VARCHAR fields within the JSON -record) to store the raw JSON record data. Thus, when a row is too big to fit, the Redshift -destination fails to load such data and currently ignores that record. See docs for -[SUPER](https://docs.aws.amazon.com/redshift/latest/dg/r_SUPER_type.html) and -[SUPER limitations](https://docs.aws.amazon.com/redshift/latest/dg/limitations-super.html). +record) to store the raw JSON record data. Thus, when a row is too big to fit, the destination connector will +do one of the following. +1. Null the value if the varchar size > 65535, The corresponding key information is added to `_airbyte_meta`. +2. Null the whole record while trying to preserve the Primary Keys and cursor field declared as part of your stream configuration, if the total record size is > 16MB. + * For DEDUPE sync mode, if we do not find Primary key(s), we fail the sync. + * For OVERWRITE and APPEND mode, syncs will succeed with empty records emitted, if we fail to find Primary key(s). + +See AWS docs for [SUPER](https://docs.aws.amazon.com/redshift/latest/dg/r_SUPER_type.html) and [SUPER limitations](https://docs.aws.amazon.com/redshift/latest/dg/limitations-super.html). ### Encryption @@ -232,6 +236,7 @@ Each stream will be output into its own raw table in Redshift. Each table will c | Version | Date | Pull Request | Subject | |:--------|:-----------|:-----------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| 2.3.1 | 2024-03-18 | [\#36255](https://github.com/airbytehq/airbyte/pull/36255) | Mark as Certified-GA | | 2.3.0 | 2024-03-18 | [\#36203](https://github.com/airbytehq/airbyte/pull/36203) | CDK 0.25.0; Record nulling for VARCHAR > 64K & record > 16MB (super limit) | | 2.2.0 | 2024-03-14 | [\#35981](https://github.com/airbytehq/airbyte/pull/35981) | CDK 0.24.0; `_airbyte_meta` in Raw table for tracking upstream data modifications. | | 2.1.10 | 2024-03-07 | [\#35899](https://github.com/airbytehq/airbyte/pull/35899) | Adopt CDK 0.23.18; Null safety check in state parsing |