🐛 AWS S3 Staging COPY is writing records from different table in the same raw table #6949

andriikorotkov · 2021-10-11T13:17:24Z

What

AWS S3 Staging COPY is writing records from different table in the same raw table

How

For the latest changes that were made for destination-jdbc working well, but destination-snowflake and destination-redshift versions have not been changed. I have updated the prefix size for S3 and GCS files to avoid duplication with a large number of files. I also added new destination-snowflake and destination-redshift versions.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

…s3-copy-fix � Conflicts: � docs/integrations/destinations/redshift.md � docs/integrations/destinations/snowflake.md

andriikorotkov · 2021-10-11T13:18:50Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1329123121
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1329123121
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    12      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     120      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 11      0   100%
	 normalization/transform_catalog/stream_processor.py                 370    218    41%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         151     39    74%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1073    413    62%

andriikorotkov · 2021-10-11T13:42:47Z

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1329211571
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1329211571
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    12      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     120      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 11      0   100%
	 normalization/transform_catalog/stream_processor.py                 370    218    41%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         151     39    74%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1073    413    62%

tuliren

Looks good.

I have two minor comments:

You can extract the suffix length as a constant at the beginning of the class for easy tracking.
Would it be more straightforward to use incrementing integers as file suffix instead of random characters? That seems easier to debugging.

…s3-copy-fix � Conflicts: � airbyte-integrations/connectors/destination-jdbc/src/main/java/io/airbyte/integrations/destination/jdbc/copy/s3/S3StreamCopier.java � airbyte-integrations/connectors/destination-redshift/src/main/java/io/airbyte/integrations/destination/redshift/RedshiftStreamCopier.java

andriikorotkov · 2021-10-13T19:38:22Z

/publish connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1338988082
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1338988082

andriikorotkov · 2021-10-13T20:06:56Z

/publish connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1339077488
❌ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1339077488

andriikorotkov · 2021-10-13T20:46:12Z

/publish connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1339203562
❌ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1339203562

andriikorotkov · 2021-10-13T21:10:22Z

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1339282551
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1339282551
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    12      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     120      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 11      0   100%
	 normalization/transform_catalog/stream_processor.py                 370    218    41%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         151     39    74%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1073    413    62%

andriikorotkov · 2021-10-13T21:25:17Z

/publish connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1339328826
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1339328826

…same raw table (airbytehq#6949) * updated jdbc destination * updated snowflake and redshift destination version * updated documentation * updated documentation * updated prefix length for snowflake and redshift streams * fixed remarks * updated new redshift version

andriikorotkov added 4 commits October 8, 2021 18:09

updated jdbc destination

5f76fb9

updated snowflake and redshift destination version

9868a36

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

6a9289f

…s3-copy-fix � Conflicts: � docs/integrations/destinations/redshift.md � docs/integrations/destinations/snowflake.md

updated documentation

a4c8b51

andriikorotkov requested review from etsybaev, irynakruk and alexandr-shegeda October 11, 2021 13:17

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Oct 11, 2021

updated documentation

2dcc1f8

andriikorotkov temporarily deployed to more-secrets October 11, 2021 13:19 Inactive

jrhizor temporarily deployed to more-secrets October 11, 2021 13:21 Inactive

jrhizor temporarily deployed to more-secrets October 11, 2021 13:45 Inactive

etsybaev approved these changes Oct 11, 2021

View reviewed changes

andriikorotkov requested a review from tuliren October 11, 2021 13:55

andriikorotkov marked this pull request as ready for review October 11, 2021 13:55

updated prefix length for snowflake and redshift streams

50f0ab3

andriikorotkov temporarily deployed to more-secrets October 12, 2021 07:24 Inactive

irynakruk approved these changes Oct 12, 2021

View reviewed changes

tuliren approved these changes Oct 13, 2021

View reviewed changes

alexandr-shegeda linked an issue Oct 13, 2021 that may be closed by this pull request

AWS S3 Staging COPY is writing records from different table in the same raw table #5664

Closed

fixed remarks

65c2741

andriikorotkov temporarily deployed to more-secrets October 13, 2021 19:18 Inactive

andriikorotkov temporarily deployed to more-secrets October 13, 2021 19:21 Inactive

jrhizor temporarily deployed to more-secrets October 13, 2021 19:40 Inactive

jrhizor temporarily deployed to more-secrets October 13, 2021 20:08 Inactive

jrhizor temporarily deployed to more-secrets October 13, 2021 20:48 Inactive

updated new redshift version

65174ba

andriikorotkov temporarily deployed to more-secrets October 13, 2021 21:11 Inactive

jrhizor temporarily deployed to more-secrets October 13, 2021 21:12 Inactive

jrhizor temporarily deployed to more-secrets October 13, 2021 21:27 Inactive

andriikorotkov merged commit 80c9ed2 into master Oct 13, 2021

andriikorotkov deleted the akorotkov/s3-copy-fix branch October 13, 2021 22:04

andriikorotkov removed a link to an issue Oct 15, 2021

AWS S3 Staging COPY is writing records from different table in the same raw table #5664

Closed

tuliren mentioned this pull request Oct 15, 2021

🐛 fixed OOM error when splitting a stream into several files #7074

Merged

38 tasks

jrhizor mentioned this pull request Oct 19, 2021

Bump Airbyte version from 0.30.20-alpha to 0.30.21-alpha #7145

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 AWS S3 Staging COPY is writing records from different table in the same raw table #6949

🐛 AWS S3 Staging COPY is writing records from different table in the same raw table #6949

andriikorotkov commented Oct 11, 2021

andriikorotkov commented Oct 11, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Oct 11, 2021 •

edited by github-actions bot

Loading

tuliren left a comment

andriikorotkov commented Oct 13, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Oct 13, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Oct 13, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Oct 13, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Oct 13, 2021 •

edited by github-actions bot

Loading

🐛 AWS S3 Staging COPY is writing records from different table in the same raw table #6949

🐛 AWS S3 Staging COPY is writing records from different table in the same raw table #6949

Conversation

andriikorotkov commented Oct 11, 2021

What

How

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

andriikorotkov commented Oct 11, 2021 • edited by github-actions bot Loading

andriikorotkov commented Oct 11, 2021 • edited by github-actions bot Loading

tuliren left a comment

Choose a reason for hiding this comment

andriikorotkov commented Oct 13, 2021 • edited by github-actions bot Loading

andriikorotkov commented Oct 13, 2021 • edited by github-actions bot Loading

andriikorotkov commented Oct 13, 2021 • edited by github-actions bot Loading

andriikorotkov commented Oct 13, 2021 • edited by github-actions bot Loading

andriikorotkov commented Oct 13, 2021 • edited by github-actions bot Loading

andriikorotkov commented Oct 11, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Oct 11, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Oct 13, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Oct 13, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Oct 13, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Oct 13, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Oct 13, 2021 •

edited by github-actions bot

Loading