Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Destination Redshift: Certification & docs update #36255

Merged
merged 2 commits into from
Mar 19, 2024
Merged

Destination Redshift: Certification & docs update #36255

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d6d0fee
certification & docs update
gisripa Mar 18, 2024
4353df2
fix PR link in changelog
gisripa Mar 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions airbyte-integrations/connectors/destination-redshift/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ data:
connectorSubtype: database
connectorType: destination
definitionId: f7a7d195-377f-cf5b-70a5-be6b819019dc
dockerImageTag: 2.3.0
dockerImageTag: 2.3.1
dockerRepository: airbyte/destination-redshift
documentationUrl: https://docs.airbyte.com/integrations/destinations/redshift
githubIssueLabel: destination-redshift
Expand All @@ -24,14 +24,14 @@ data:
This version introduces [Destinations V2](https://docs.airbyte.com/release_notes/upgrading_to_destinations_v2/#what-is-destinations-v2), which provides better error handling, incremental delivery of data for large syncs, and improved final table structures. To review the breaking changes, and how to upgrade, see [here](https://docs.airbyte.com/release_notes/upgrading_to_destinations_v2/#quick-start-to-upgrading). These changes will likely require updates to downstream dbt / SQL models, which we walk through [here](https://docs.airbyte.com/release_notes/upgrading_to_destinations_v2/#updating-downstream-transformations).
Selecting `Upgrade` will upgrade **all** connections using this destination at their next sync. You can manually sync existing connections prior to the next scheduled sync to start the upgrade early.
upgradeDeadline: "2024-03-15"
releaseStage: beta
resourceRequirements:
jobSpecific:
- jobType: sync
resourceRequirements:
memory_limit: 1Gi
memory_request: 1Gi
supportLevel: community
releaseStage: generally_available
supportLevel: certified
supportsDbt: true
tags:
- language:java
Expand Down
13 changes: 9 additions & 4 deletions docs/integrations/destinations/redshift.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -187,10 +187,14 @@ characters.
### Data Size Limitations

Redshift specifies a maximum limit of 16MB (and 65535 bytes for any VARCHAR fields within the JSON
record) to store the raw JSON record data. Thus, when a row is too big to fit, the Redshift
destination fails to load such data and currently ignores that record. See docs for
[SUPER](https://docs.aws.amazon.com/redshift/latest/dg/r_SUPER_type.html) and
[SUPER limitations](https://docs.aws.amazon.com/redshift/latest/dg/limitations-super.html).
record) to store the raw JSON record data. Thus, when a row is too big to fit, the destination connector will
do one of the following.
1. Null the value if the varchar size > 65535, The corresponding key information is added to `_airbyte_meta`.
2. Null the whole record while trying to preserve the Primary Keys and cursor field declared as part of your stream configuration, if the total record size is > 16MB.
* For DEDUPE sync mode, if we do not find Primary key(s), we fail the sync.
* For OVERWRITE and APPEND mode, syncs will succeed with empty records emitted, if we fail to find Primary key(s).

See AWS docs for [SUPER](https://docs.aws.amazon.com/redshift/latest/dg/r_SUPER_type.html) and [SUPER limitations](https://docs.aws.amazon.com/redshift/latest/dg/limitations-super.html).
Comment on lines +190 to +197
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️


### Encryption

Expand Down Expand Up @@ -232,6 +236,7 @@ Each stream will be output into its own raw table in Redshift. Each table will c

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:-----------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 2.3.1 | 2024-03-18 | [\#36255](https://github.com/airbytehq/airbyte/pull/36255) | Mark as Certified-GA |
| 2.3.0 | 2024-03-18 | [\#36203](https://github.com/airbytehq/airbyte/pull/36203) | CDK 0.25.0; Record nulling for VARCHAR > 64K & record > 16MB (super limit) |
| 2.2.0 | 2024-03-14 | [\#35981](https://github.com/airbytehq/airbyte/pull/35981) | CDK 0.24.0; `_airbyte_meta` in Raw table for tracking upstream data modifications. |
| 2.1.10 | 2024-03-07 | [\#35899](https://github.com/airbytehq/airbyte/pull/35899) | Adopt CDK 0.23.18; Null safety check in state parsing |
Expand Down
Loading