🐛Destination S3 and GCS - Fixed connector's bug that prevent writing streams with more than 50GB #5890

etsybaev · 2021-09-07T18:10:12Z

What

Currently, we can't write to S3 and GCS objects with more than 50GB of each

How

The reason is that servers allow 10,000 blocks max and currently we have block size hardcoded as 5MB. but servers allow to have it up to 5GB each.
To fix the described issue - added a new arg that allows to configure it. Basically, to reach the max object limit of 5TB - the max block's size of 525 (MB) is enough. The new arg supports size from 5MB (min for server) to 525MB.

https://cloud.google.com/storage/quotas
https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html

Tested locally:

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

…ance destination issues reproducing and debugging

etsybaev · 2021-09-07T19:16:07Z

/test connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1210702460
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1210702460

…-for-gcs-and-s3

etsybaev · 2021-09-11T17:09:06Z

/test connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1224756646
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1224756646
🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1224756646
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1224756646
🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1224756646
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1224756646

etsybaev · 2021-09-11T17:45:50Z

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1224812601
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1224812601
🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1224812601
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1224812601
🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1224812601
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1224812601
🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1224812601
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1224812601
🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1224812601
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1224812601

...-gcs/src/test/java/io/airbyte/integrations/destination/gcs/avro/GcsAvroFormatConfigTest.java

...src/main/java/io/airbyte/integrations/destination/s3/util/S3StreamTransferManagerHelper.java

alexandr-shegeda · 2021-09-13T15:51:39Z

LGTM

…-for-gcs-and-s3 # Conflicts: # docs/integrations/destinations/s3.md

…-for-gcs-and-s3

davinchia

Great stuff! Appreciate the screen shots. Appreciate all the test you added.

Small feedback from me - it would have been easier to review if the PR was split into the base S3 changes, and a follow up PR for GCS changes.

Please also go through the updating connector checklist in the future so it's easy for me to know what is/what is not done.

Thanks!

airbyte-integrations/connectors/destination-gcs/src/main/resources/spec.json

...src/main/java/io/airbyte/integrations/destination/s3/util/S3StreamTransferManagerHelper.java

etsybaev · 2021-09-17T07:42:45Z

/publish connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1244622918
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1244622918

etsybaev · 2021-09-17T07:56:17Z

/publish connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1244656389
❌ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1244656389
🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1244656389
❌ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1244656389
🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1244656389
❌ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1244656389

etsybaev · 2021-09-17T08:46:22Z

Released new S3 connector, but can't release GCS due to other blocking bug - #6134

…-for-gcs-and-s3

etsybaev · 2021-09-17T11:51:31Z

Merging the latest master with a fix didn't help. It seems like we've got new issues in master that prevent merging. So still can't publish GCS and merge this

…-for-gcs-and-s3

sherifnada · 2021-09-17T19:01:49Z

/publish connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1246666581
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1246666581

PS-istarostenko · 2021-09-20T23:52:43Z

@etsybaev can we please get the same fix, but for BigQuery and Snowflake when using GCS and S3 as a staging loading method?
#5720

ievgeniit added 2 commits September 6, 2021 20:50

Added basic performance\stress test to speed-up an edge cases perform…

c11d1d8

…ance destination issues reproducing and debugging

Added optional partSize config for Destination-S3 connector

2109088

github-actions bot added the area/connectors Connector related issues label Sep 7, 2021

Fixed checkstyle

35c1e69

jrhizor temporarily deployed to more-secrets September 7, 2021 19:18 Inactive

ievgeniit added 5 commits September 9, 2021 15:48

Merge branch 'master' into etsybaev/made-partsize-config-configurable…

4d7c484

…-for-gcs-and-s3

Merge branch 'master' into etsybaev/made-partsize-config-configurable…

14f619e

…-for-gcs-and-s3

Merge branch 'master' into etsybaev/made-partsize-config-configurable…

2a33d19

…-for-gcs-and-s3

added block size processing to gcs

be18356

Updated docs

8067d9c

github-actions bot added the area/documentation Improvements or additions to documentation label Sep 11, 2021

jrhizor temporarily deployed to more-secrets September 11, 2021 17:11 Inactive

etsybaev changed the title ~~[DRAFT_PR_IGNORE_IT] Made partsize config configurable for gcs and s3~~ [DRAFT_PR_IGNORE_IT] 🐛Destination S3 and GCS - Fixed connector's bug that prevent writing streams with more than 50GB Sep 11, 2021

etsybaev linked an issue Sep 11, 2021 that may be closed by this pull request

S3 and GCS writes are limited to 50GB #5720

Closed

jrhizor temporarily deployed to more-secrets September 11, 2021 17:48 Inactive

alexandr-shegeda reviewed Sep 12, 2021

View reviewed changes

...-gcs/src/test/java/io/airbyte/integrations/destination/gcs/avro/GcsAvroFormatConfigTest.java Outdated Show resolved Hide resolved

alexandr-shegeda reviewed Sep 12, 2021

View reviewed changes

...-gcs/src/test/java/io/airbyte/integrations/destination/gcs/avro/GcsAvroFormatConfigTest.java Outdated Show resolved Hide resolved

alexandr-shegeda reviewed Sep 12, 2021

View reviewed changes

...src/main/java/io/airbyte/integrations/destination/s3/util/S3StreamTransferManagerHelper.java Outdated Show resolved Hide resolved

Fixed as per comments in PR

5d07fd6

Merge branch 'master' into etsybaev/made-partsize-config-configurable…

099634d

…-for-gcs-and-s3 # Conflicts: # docs/integrations/destinations/s3.md

etsybaev changed the title ~~[DRAFT_PR_IGNORE_IT] 🐛Destination S3 and GCS - Fixed connector's bug that prevent writing streams with more than 50GB~~ 🐛Destination S3 and GCS - Fixed connector's bug that prevent writing streams with more than 50GB Sep 14, 2021

etsybaev requested a review from alexandr-shegeda September 14, 2021 07:43

alexandr-shegeda approved these changes Sep 14, 2021

View reviewed changes

etsybaev marked this pull request as ready for review September 14, 2021 08:39

etsybaev requested a review from davinchia September 14, 2021 08:40

Merge branch 'master' into etsybaev/made-partsize-config-configurable…

3d44c0c

…-for-gcs-and-s3

davinchia approved these changes Sep 17, 2021

View reviewed changes

airbyte-integrations/connectors/destination-gcs/src/main/resources/spec.json Show resolved Hide resolved

...src/main/java/io/airbyte/integrations/destination/s3/util/S3StreamTransferManagerHelper.java Outdated Show resolved Hide resolved

bumped versions for s3 ang gcs

b70c42b

jrhizor temporarily deployed to more-secrets September 17, 2021 07:45 Inactive

jrhizor temporarily deployed to more-secrets September 17, 2021 07:58 Inactive

jrhizor temporarily deployed to more-secrets September 17, 2021 08:24 Inactive

Merge branch 'master' into etsybaev/made-partsize-config-configurable…

fd70690

…-for-gcs-and-s3

Merge branch 'master' into etsybaev/made-partsize-config-configurable…

05b2e34

…-for-gcs-and-s3

jrhizor temporarily deployed to more-secrets September 17, 2021 19:04 Inactive

sherifnada merged commit 64ce6fd into master Sep 17, 2021

sherifnada deleted the etsybaev/made-partsize-config-configurable-for-gcs-and-s3 branch September 17, 2021 21:21

jrhizor mentioned this pull request Sep 18, 2021

Bump Airbyte version from 0.29.19-alpha to 0.29.20-alpha #6243

Merged

PS-istarostenko mentioned this pull request Sep 20, 2021

S3 and GCS writes are limited to 50GB #5720

Closed

yurii-bidiuk mentioned this pull request Dec 16, 2021

When replicating very large tables to s3, sync fails when 10000 part limit is exceeded. Add part size config. #6090

Closed

karinakuz added connectors/destinations-files connectors/destination/gcs labels Jan 14, 2022

rx007 mentioned this pull request Oct 28, 2023

[Snyk] Security upgrade axios from 0.21.1 to 1.6.0 rx007/airbyte#2130

Open

mresposito mentioned this pull request Oct 28, 2023

[Snyk] Security upgrade axios from 0.21.1 to 1.6.0 candulabs/airbyte#104

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛Destination S3 and GCS - Fixed connector's bug that prevent writing streams with more than 50GB #5890

🐛Destination S3 and GCS - Fixed connector's bug that prevent writing streams with more than 50GB #5890

etsybaev commented Sep 7, 2021 •

edited

Loading

etsybaev commented Sep 7, 2021 •

edited by github-actions bot

Loading

etsybaev commented Sep 11, 2021 •

edited by github-actions bot

Loading

etsybaev commented Sep 11, 2021 •

edited by github-actions bot

Loading

alexandr-shegeda commented Sep 13, 2021

davinchia left a comment

etsybaev commented Sep 17, 2021 •

edited by github-actions bot

Loading

etsybaev commented Sep 17, 2021 •

edited by github-actions bot

Loading

etsybaev commented Sep 17, 2021 •

edited

Loading

etsybaev commented Sep 17, 2021

sherifnada commented Sep 17, 2021 •

edited by github-actions bot

Loading

PS-istarostenko commented Sep 20, 2021

🐛Destination S3 and GCS - Fixed connector's bug that prevent writing streams with more than 50GB #5890

🐛Destination S3 and GCS - Fixed connector's bug that prevent writing streams with more than 50GB #5890

Conversation

etsybaev commented Sep 7, 2021 • edited Loading

What

How

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

etsybaev commented Sep 7, 2021 • edited by github-actions bot Loading

etsybaev commented Sep 11, 2021 • edited by github-actions bot Loading

etsybaev commented Sep 11, 2021 • edited by github-actions bot Loading

alexandr-shegeda commented Sep 13, 2021

davinchia left a comment

Choose a reason for hiding this comment

etsybaev commented Sep 17, 2021 • edited by github-actions bot Loading

etsybaev commented Sep 17, 2021 • edited by github-actions bot Loading

etsybaev commented Sep 17, 2021 • edited Loading

etsybaev commented Sep 17, 2021

sherifnada commented Sep 17, 2021 • edited by github-actions bot Loading

PS-istarostenko commented Sep 20, 2021

etsybaev commented Sep 7, 2021 •

edited

Loading

etsybaev commented Sep 7, 2021 •

edited by github-actions bot

Loading

etsybaev commented Sep 11, 2021 •

edited by github-actions bot

Loading

etsybaev commented Sep 11, 2021 •

edited by github-actions bot

Loading

etsybaev commented Sep 17, 2021 •

edited by github-actions bot

Loading

etsybaev commented Sep 17, 2021 •

edited by github-actions bot

Loading

etsybaev commented Sep 17, 2021 •

edited

Loading

sherifnada commented Sep 17, 2021 •

edited by github-actions bot

Loading