`destination-redshift` should fail syncs if records or properties are too large, rather than silently skipping records and succeeding #27993

evantahler · 2023-07-05T22:35:06Z

Closes #19990, albiet in a less-than-ideal way. At least now, users will be notified when source data won't fit in the destination. From there, a view or other filtering mechanism on the source can be used to adjust the incoming records to fix within Redshift.

…oo large, rather than silently skipping records and succeding

github-actions · 2023-07-05T22:35:38Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan and you've followed all steps in the Breaking Changes Checklist
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
The connector tests are passing in CI
You've updated the connector's metadata.yaml file (new!)
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

octavia-squidington-iii · 2023-07-06T00:03:37Z

destination-redshift test report (commit `116f297c83`) - ❌

⏲️ Total pipeline duration: 30mn19s

Step	Result
Validate airbyte-integrations/connectors/destination-redshift/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-redshift docker image for platform linux/x86_64	✅
Build airbyte/normalization-redshift:dev	✅
./gradlew :airbyte-integrations:connectors:destination-redshift:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-redshift test

octavia-squidington-iii · 2023-07-06T08:07:25Z

destination-redshift test report (commit `8398de38f4`) - ❌

⏲️ Total pipeline duration: 28mn55s

Step	Result
Validate airbyte-integrations/connectors/destination-redshift/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-redshift docker image for platform linux/x86_64	✅
Build airbyte/normalization-redshift:dev	✅
./gradlew :airbyte-integrations:connectors:destination-redshift:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-redshift test

edgao

what do you think of keeping this behavior available as a config option? I'm nervous about sources that don't have the ability to do that sort of filtering

if not: We can also delete implementsRecordSizeLimitChecks and testSyncVeryBigRecords from DestinationAcceptanceTest / RedshiftStagingS3DestinationAcceptanceTest

either way - this diff lgtm

evantahler · 2023-07-06T17:34:26Z

Thanks for the review @edgao!

what do you think of keeping this behavior available as a config option? I'm nervous about sources that don't have the ability to do that sort of filtering

Give the choice, I didn't want to go with a config option, because whenever possible I want the behavior of our destinations to match. Snowflake and BQ have the same problem today, and one day we should holistically tackle this problem, rather than a piecemeal per-destination solution.

For example, once we get one-stream-one-table released (#26028), we could add a feature to every destination that handles per-field and per-record size limitations. We could just write the PK and cursor for the record, and then include an _airbyte_medata.error = ['data for column "foo" too large']. This would work for every destination, and not require any config settings. 1s1t solves every problem ;)

octavia-squidington-iii · 2023-07-06T18:56:00Z

destination-oracle-strict-encrypt test report (commit `8e65341d0e`) - ❌

⏲️ Total pipeline duration: 12mn27s

Step	Result
Validate airbyte-integrations/connectors/destination-oracle-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-oracle-strict-encrypt docker image for platform linux/x86_64	✅
Build airbyte/normalization-oracle:dev	✅
./gradlew :airbyte-integrations:connectors:destination-oracle-strict-encrypt:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-oracle-strict-encrypt test

...rc/main/java/io/airbyte/integrations/standardtest/destination/DestinationAcceptanceTest.java

edgao · 2023-07-06T18:59:52Z

sgtm

octavia-squidington-iii · 2023-07-06T19:02:55Z

destination-pubsub test report (commit `8e65341d0e`) - ❌

⏲️ Total pipeline duration: 06mn46s

Step	Result
Validate airbyte-integrations/connectors/destination-pubsub/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-pubsub docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-pubsub:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-pubsub test

octavia-squidington-iii · 2023-07-06T19:04:35Z

destination-vertica test report (commit `8e65341d0e`) - ❌

⏲️ Total pipeline duration: 01mn31s

Step	Result
Validate airbyte-integrations/connectors/destination-vertica/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-vertica test

octavia-squidington-iii · 2023-07-06T19:12:29Z

destination-mqtt test report (commit `8e65341d0e`) - ❌

⏲️ Total pipeline duration: 07mn44s

Step	Result
Validate airbyte-integrations/connectors/destination-mqtt/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-mqtt docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-mqtt:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mqtt test

octavia-squidington-iii · 2023-07-06T19:19:07Z

destination-azure-blob-storage test report (commit `8e65341d0e`) - ❌

⏲️ Total pipeline duration: 06mn30s

Step	Result
Validate airbyte-integrations/connectors/destination-azure-blob-storage/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-azure-blob-storage docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-azure-blob-storage:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-azure-blob-storage test

octavia-squidington-iii · 2023-07-06T19:24:03Z

destination-keen test report (commit `8e65341d0e`) - ❌

⏲️ Total pipeline duration: 04mn47s

Step	Result
Validate airbyte-integrations/connectors/destination-keen/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-keen docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-keen:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-keen test

octavia-squidington-iii · 2023-07-06T19:36:14Z

destination-starburst-galaxy test report (commit `8e65341d0e`) - ❌

⏲️ Total pipeline duration: 12mn02s

Step	Result
Validate airbyte-integrations/connectors/destination-starburst-galaxy/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-starburst-galaxy docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-starburst-galaxy:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-starburst-galaxy test

octavia-squidington-iii · 2023-07-06T19:47:59Z

destination-yugabytedb test report (commit `8e65341d0e`) - ❌

⏲️ Total pipeline duration: 11mn28s

Step	Result
Validate airbyte-integrations/connectors/destination-yugabytedb/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-yugabytedb docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-yugabytedb:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-yugabytedb test

octavia-squidington-iii · 2023-07-06T19:55:05Z

destination-exasol test report (commit `8e65341d0e`) - ❌

⏲️ Total pipeline duration: 06mn57s

Step	Result
Validate airbyte-integrations/connectors/destination-exasol/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌
Build connector tar	✅
Build destination-exasol docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-exasol:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-exasol test

octavia-squidington-iii · 2023-07-06T19:58:47Z

destination-dev-null test report (commit `8e65341d0e`) - ❌

⏲️ Total pipeline duration: 03mn33s

Step	Result
Validate airbyte-integrations/connectors/destination-dev-null/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-dev-null docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-dev-null:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-dev-null test

octavia-squidington-iii · 2023-07-13T23:13:01Z

destination-redshift test report (commit `c6b4ef0732`) - ✅

⏲️ Total pipeline duration: 01mn33s

Step	Result
Validate airbyte-integrations/connectors/destination-redshift/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-redshift docker image for platform linux/x86_64	✅
Build airbyte/normalization-redshift:dev	✅
./gradlew :airbyte-integrations:connectors:destination-redshift:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-redshift test

octavia-squidington-iii · 2023-07-13T23:17:25Z

destination-doris test report (commit `c6b4ef0732`) - ❌

⏲️ Total pipeline duration: 04mn14s

Step	Result
Validate airbyte-integrations/connectors/destination-doris/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌
Build connector tar	✅
Build destination-doris docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-doris:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-doris test

octavia-squidington-iii · 2023-07-13T23:33:34Z

destination-dynamodb test report (commit `c6b4ef0732`) - ✅

⏲️ Total pipeline duration: 16mn00s

Step	Result
Validate airbyte-integrations/connectors/destination-dynamodb/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-dynamodb docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-dynamodb:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-dynamodb test

octavia-squidington-iii · 2023-07-13T23:42:59Z

destination-gcs test report (commit `c6b4ef0732`) - ✅

⏲️ Total pipeline duration: 09mn16s

Step	Result
Validate airbyte-integrations/connectors/destination-gcs/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-gcs docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-gcs:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-gcs test

octavia-squidington-iii · 2023-07-13T23:49:09Z

destination-r2 test report (commit `c6b4ef0732`) - ❌

⏲️ Total pipeline duration: 06mn01s

Step	Result
Validate airbyte-integrations/connectors/destination-r2/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-r2 docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-r2:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-r2 test

octavia-squidington-iii · 2023-07-13T23:58:02Z

destination-azure-blob-storage test report (commit `c6b4ef0732`) - ❌

⏲️ Total pipeline duration: 08mn43s

Step	Result
Validate airbyte-integrations/connectors/destination-azure-blob-storage/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-azure-blob-storage docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-azure-blob-storage:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-azure-blob-storage test

octavia-squidington-iii · 2023-07-14T00:07:09Z

destination-s3-glue test report (commit `c6b4ef0732`) - ✅

⏲️ Total pipeline duration: 08mn58s

Step	Result
Validate airbyte-integrations/connectors/destination-s3-glue/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-s3-glue docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-s3-glue:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-s3-glue test

octavia-squidington-iii · 2023-07-14T00:48:33Z

destination-mysql test report (commit `c6b4ef0732`) - ❌

⏲️ Total pipeline duration: 41mn14s

Step	Result
Validate airbyte-integrations/connectors/destination-mysql/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-mysql docker image for platform linux/x86_64	✅
Build airbyte/normalization-mysql:dev	✅
./gradlew :airbyte-integrations:connectors:destination-mysql:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mysql test

octavia-squidington-iii · 2023-07-14T00:57:23Z

destination-redpanda test report (commit `c6b4ef0732`) - ❌

⏲️ Total pipeline duration: 08mn40s

Step	Result
Validate airbyte-integrations/connectors/destination-redpanda/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-redpanda docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-redpanda:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-redpanda test

octavia-squidington-iii · 2023-07-14T09:50:40Z

destination-iceberg test report (commit `c6b4ef0732`) - ❌

⏲️ Total pipeline duration: 533mn08s

Step	Result
Validate airbyte-integrations/connectors/destination-iceberg/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌
Build connector tar	✅
Build destination-iceberg docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-iceberg:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-iceberg test

octavia-squidington-iii · 2023-07-14T10:18:21Z

destination-bigquery test report (commit `c6b4ef0732`) - ✅

⏲️ Total pipeline duration: 27mn31s

Step	Result
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Build airbyte/normalization:dev	✅
./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

octavia-squidington-iii · 2023-07-14T10:57:17Z

destination-snowflake test report (commit `c6b4ef0732`) - ❌

⏲️ Total pipeline duration: 38mn46s

Step	Result
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Build airbyte/normalization-snowflake:dev	✅
./gradlew :airbyte-integrations:connectors:destination-snowflake:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

octavia-squidington-iii · 2023-07-14T11:07:05Z

destination-kinesis test report (commit `c6b4ef0732`) - ✅

⏲️ Total pipeline duration: 09mn37s

Step	Result
Validate airbyte-integrations/connectors/destination-kinesis/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-kinesis docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-kinesis:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-kinesis test

octavia-squidington-iii · 2023-07-14T11:12:47Z

destination-csv test report (commit `c6b4ef0732`) - ✅

⏲️ Total pipeline duration: 05mn33s

Step	Result
Validate airbyte-integrations/connectors/destination-csv/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-csv docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-csv:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-csv test

octavia-squidington-iii · 2023-07-14T11:22:42Z

destination-kafka test report (commit `c6b4ef0732`) - ❌

⏲️ Total pipeline duration: 09mn46s

Step	Result
Validate airbyte-integrations/connectors/destination-kafka/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-kafka docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-kafka:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-kafka test

evantahler · 2023-07-14T18:20:59Z

Enough of the tests for other connectors are passing, and the builds all seem to be OK... so... Merging!

evantahler · 2023-07-14T19:26:26Z

/approve-and-merge reason="enough of the connector tests are passing"

octavia-approvington · 2023-07-14T19:27:09Z

It's time

… too large, rather than silently skipping records and succeeding (airbytehq#27993) * `destination-redshift` will fail syncs if records or properties are too large, rather than silently skipping records and succeding * Bump version * remove tests that don't matter any more * more test removal * more test removal --------- Co-authored-by: Augustin <augustin@airbyte.io>

alexnikitchuk · 2023-07-27T11:40:01Z

Thanks for the review @edgao!

what do you think of keeping this behavior available as a config option? I'm nervous about sources that don't have the ability to do that sort of filtering

Give the choice, I didn't want to go with a config option, because whenever possible I want the behavior of our destinations to match. Snowflake and BQ have the same problem today, and one day we should holistically tackle this problem, rather than a piecemeal per-destination solution.

For example, once we get one-stream-one-table released (#26028), we could add a feature to every destination that handles per-field and per-record size limitations. We could just write the PK and cursor for the record, and then include an _airbyte_medata.error = ['data for column "foo" too large']. This would work for every destination, and not require any config settings. 1s1t solves every problem ;)

This looks like a breaking change. IMO better to have the option to switch back to the old behavior.

jablonskijakub · 2023-08-21T17:53:25Z

thanks @evantahler for your PR. tbh I was not aware of the issue with Redshift's limits on SUPER type until I upgraded my connector to your version.
Unfortunately it is not quite a solution I would prefer as I need to load all rows from a source db and I was wondering if it would be possible to allow for copying if a row is < 1mb but varchar fields exceed the limit and blocking normalisation so that the data remains in the raw format? what could be a solution here otherwise?

ok, I already know that it is not possible. https://docs.aws.amazon.com/redshift/latest/dg/limitations-super.html#

a workaround would be appreciated though :)

evantahler · 2023-08-21T18:46:18Z

We've got a feature on the roadmap to use our new features in Destinations V2 (#26028) to have a work around! We'll import the columns we can, and include an error message, per-record, for any column which has a filed that is too large for the destination.

Would you prefer the too-large field with the error to have NULL or a truncated value?

jablonskijakub · 2023-08-23T09:47:31Z

sounds great! personally I would prefer to have it as NULL as then it would indicate that given field was too big for a destination and with a truncated value I suspect it might sl without being noticed.

If I understood you correctly Destinations V2 would be used instead on the normalisation step which is being used atm. I believe my issue is slightly different as my payload field is in a jsonb format and being read as a single string value (please correct me if I'm wrong) thus exceeding 65535 bytes for a single value within a SUPER data type.

destination-redshift will fail syncs if records or properties are t…

ed9cae8

…oo large, rather than silently skipping records and succeding

evantahler marked this pull request as ready for review July 5, 2023 22:35

evantahler requested a review from a team as a code owner July 5, 2023 22:35

octavia-squidington-iii added area/connectors Connector related issues connectors/destination/redshift labels Jul 5, 2023

Bump version

443c3e5

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Jul 5, 2023

remove tests that don't matter any more

116f297

Merge branch 'master' into evan/redshift-fails-on-big-data

8398de3

edgao approved these changes Jul 6, 2023

View reviewed changes

more test removal

8e65341

edgao reviewed Jul 6, 2023

View reviewed changes

...rc/main/java/io/airbyte/integrations/standardtest/destination/DestinationAcceptanceTest.java Outdated Show resolved Hide resolved

more test removal

05a0e56

octavia-approvington approved these changes Jul 14, 2023

View reviewed changes

octavia-approvington merged commit b81cc03 into master Jul 14, 2023
26 of 70 checks passed

octavia-approvington deleted the evan/redshift-fails-on-big-data branch July 14, 2023 19:27

This was referenced Oct 10, 2023

Redshift connector now crashes if the ingested data is over Redshift's limits #31216

Open

[Epic] - Destinations produce airbyte_error when record or column is too large, and sync succeeds #28541

Closed

nataliekwong mentioned this pull request Feb 6, 2024

[Roadmap] Sync large records gracefully #34887

Open

destination-redshift should fail syncs if records or properties are too large, rather than silently skipping records and succeeding #27993

destination-redshift should fail syncs if records or properties are too large, rather than silently skipping records and succeeding #27993

Conversation

evantahler commented Jul 5, 2023 • edited

github-actions bot commented Jul 5, 2023 • edited by evantahler

Before Merging a Connector Pull Request

octavia-squidington-iii commented Jul 6, 2023

destination-redshift test report (commit 116f297c83) - ❌

octavia-squidington-iii commented Jul 6, 2023

destination-redshift test report (commit 8398de38f4) - ❌

edgao left a comment

Choose a reason for hiding this comment

evantahler commented Jul 6, 2023 • edited

octavia-squidington-iii commented Jul 6, 2023

destination-oracle-strict-encrypt test report (commit 8e65341d0e) - ❌

edgao commented Jul 6, 2023

octavia-squidington-iii commented Jul 6, 2023

destination-pubsub test report (commit 8e65341d0e) - ❌

octavia-squidington-iii commented Jul 6, 2023

destination-vertica test report (commit 8e65341d0e) - ❌

octavia-squidington-iii commented Jul 6, 2023

destination-mqtt test report (commit 8e65341d0e) - ❌

octavia-squidington-iii commented Jul 6, 2023

destination-azure-blob-storage test report (commit 8e65341d0e) - ❌

octavia-squidington-iii commented Jul 6, 2023

destination-keen test report (commit 8e65341d0e) - ❌

octavia-squidington-iii commented Jul 6, 2023

destination-starburst-galaxy test report (commit 8e65341d0e) - ❌

octavia-squidington-iii commented Jul 6, 2023

destination-yugabytedb test report (commit 8e65341d0e) - ❌

octavia-squidington-iii commented Jul 6, 2023

destination-exasol test report (commit 8e65341d0e) - ❌

octavia-squidington-iii commented Jul 6, 2023

destination-dev-null test report (commit 8e65341d0e) - ❌

octavia-squidington-iii commented Jul 13, 2023

destination-redshift test report (commit c6b4ef0732) - ✅

octavia-squidington-iii commented Jul 13, 2023

destination-doris test report (commit c6b4ef0732) - ❌

octavia-squidington-iii commented Jul 13, 2023

destination-dynamodb test report (commit c6b4ef0732) - ✅

octavia-squidington-iii commented Jul 13, 2023

destination-gcs test report (commit c6b4ef0732) - ✅

octavia-squidington-iii commented Jul 13, 2023

destination-r2 test report (commit c6b4ef0732) - ❌

octavia-squidington-iii commented Jul 13, 2023

destination-azure-blob-storage test report (commit c6b4ef0732) - ❌

octavia-squidington-iii commented Jul 14, 2023

destination-s3-glue test report (commit c6b4ef0732) - ✅

octavia-squidington-iii commented Jul 14, 2023

destination-mysql test report (commit c6b4ef0732) - ❌

octavia-squidington-iii commented Jul 14, 2023

destination-redpanda test report (commit c6b4ef0732) - ❌

octavia-squidington-iii commented Jul 14, 2023

destination-iceberg test report (commit c6b4ef0732) - ❌

octavia-squidington-iii commented Jul 14, 2023

destination-bigquery test report (commit c6b4ef0732) - ✅

octavia-squidington-iii commented Jul 14, 2023

destination-snowflake test report (commit c6b4ef0732) - ❌

octavia-squidington-iii commented Jul 14, 2023

destination-kinesis test report (commit c6b4ef0732) - ✅

octavia-squidington-iii commented Jul 14, 2023

destination-csv test report (commit c6b4ef0732) - ✅

octavia-squidington-iii commented Jul 14, 2023

destination-kafka test report (commit c6b4ef0732) - ❌

evantahler commented Jul 14, 2023

evantahler commented Jul 14, 2023

octavia-approvington commented Jul 14, 2023

alexnikitchuk commented Jul 27, 2023

jablonskijakub commented Aug 21, 2023 • edited

evantahler commented Aug 21, 2023

jablonskijakub commented Aug 23, 2023

`destination-redshift` should fail syncs if records or properties are too large, rather than silently skipping records and succeeding #27993

`destination-redshift` should fail syncs if records or properties are too large, rather than silently skipping records and succeeding #27993

evantahler commented Jul 5, 2023 •

edited

github-actions bot commented Jul 5, 2023 •

edited by evantahler

destination-redshift test report (commit `116f297c83`) - ❌

destination-redshift test report (commit `8398de38f4`) - ❌

evantahler commented Jul 6, 2023 •

edited

destination-oracle-strict-encrypt test report (commit `8e65341d0e`) - ❌

destination-pubsub test report (commit `8e65341d0e`) - ❌

destination-vertica test report (commit `8e65341d0e`) - ❌

destination-mqtt test report (commit `8e65341d0e`) - ❌

destination-azure-blob-storage test report (commit `8e65341d0e`) - ❌

destination-keen test report (commit `8e65341d0e`) - ❌

destination-starburst-galaxy test report (commit `8e65341d0e`) - ❌

destination-yugabytedb test report (commit `8e65341d0e`) - ❌

destination-exasol test report (commit `8e65341d0e`) - ❌

destination-dev-null test report (commit `8e65341d0e`) - ❌

destination-redshift test report (commit `c6b4ef0732`) - ✅

destination-doris test report (commit `c6b4ef0732`) - ❌

destination-dynamodb test report (commit `c6b4ef0732`) - ✅

destination-gcs test report (commit `c6b4ef0732`) - ✅

destination-r2 test report (commit `c6b4ef0732`) - ❌

destination-azure-blob-storage test report (commit `c6b4ef0732`) - ❌

destination-s3-glue test report (commit `c6b4ef0732`) - ✅

destination-mysql test report (commit `c6b4ef0732`) - ❌

destination-redpanda test report (commit `c6b4ef0732`) - ❌

destination-iceberg test report (commit `c6b4ef0732`) - ❌

destination-bigquery test report (commit `c6b4ef0732`) - ✅

destination-snowflake test report (commit `c6b4ef0732`) - ❌

destination-kinesis test report (commit `c6b4ef0732`) - ✅

destination-csv test report (commit `c6b4ef0732`) - ✅

destination-kafka test report (commit `c6b4ef0732`) - ❌

jablonskijakub commented Aug 21, 2023 •

edited