Make Redshift part size configurable. #3053

davinchia · 2021-04-26T12:07:52Z

What

The Redshift Copy strategy currently has it's part size set to 10 MB. Since S3 allows a file to be broken into max of 10k parts, this results in a 100GB table limit. A user is trying to sync a table of 115GB and running into this issue.

This makes the part size configurable so users can increase this size if needed.

How

Add part_size as part of the spec.json.
Use part_size if present. Otherwise, use the default of 10 MB.

Pre-merge Checklist

Run integration tests
Publish Docker images

Recommended reading order

spec.json
RedshiftCopier

davinchia · 2021-04-26T12:16:38Z

airbyte-integrations/connectors/destination-redshift/src/main/resources/spec.json

@@ -102,6 +102,14 @@
        "description": "The corresponding secret to the above access key id.",
        "title": "S3 Access Key",
        "airbyte_secret": true
+      },


davinchia · 2021-04-26T12:28:02Z

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/785542618
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/785542618

davinchia · 2021-04-26T12:40:55Z

I tested this locally with a part size of 30MB to confirm this works:

jrhizor

approved % comments

jrhizor · 2021-04-26T17:03:17Z

...n/resources/config/STANDARD_DESTINATION_DEFINITION/f7a7d195-377f-cf5b-70a5-be6b819019dc.json

@@ -2,7 +2,7 @@
  "destinationDefinitionId": "f7a7d195-377f-cf5b-70a5-be6b819019dc",
  "name": "Redshift",
  "dockerRepository": "airbyte/destination-redshift",
-  "dockerImageTag": "0.3.0",
+  "dockerImageTag": "0.3.1",


Heads up, you'll need to merge master and do 0.3.2

jrhizor · 2021-04-26T17:03:22Z

airbyte-config/init/src/main/resources/seed/destination_definitions.yaml

@@ -27,7 +27,7 @@
 - destinationDefinitionId: f7a7d195-377f-cf5b-70a5-be6b819019dc
  name: Redshift
  dockerRepository: airbyte/destination-redshift
-  dockerImageTag: 0.3.0
+  dockerImageTag: 0.3.1


jrhizor · 2021-04-26T17:03:43Z

...tion-redshift/src/main/java/io/airbyte/integrations/destination/redshift/RedshiftCopier.java

@@ -63,7 +63,7 @@
  // The smallest part size is 5MB. An S3 upload can be maximally formed of 10,000 parts. This gives
  // us an upper limit of 10,000 * 10 / 1000 = 100 GB per table with a 10MB part size limit.
  // WARNING: Too large a part size can cause potential OOM errors.
-  private static final int PART_SIZE_MB = 10;
+  public static final int DEFAULT_PART_SIZE_MB = 10;


Can we make this default larger? 10MB seems really small.

jrhizor · 2021-04-26T17:04:40Z

airbyte-integrations/connectors/destination-redshift/src/main/resources/spec.json

+      "part_size": {
+        "type": "integer",
+        "minimum": 10,
+        "maximum": 100,


Why 100mb maximum? Is that the Redshift maximum?

cgardens · 2021-04-26T20:43:22Z

Do we need to make this configurable? Could we just make the part size really big? What do we lose by doing that?

davinchia · 2021-04-26T21:43:38Z

@jrhizor @cgardens replying both of ya together since you had the same question. The tradeoff is the typical latency vs throughput with big data. A bigger part size gives us overall larger throughput (since we can send more data overall) and a larger destination table size, however it means slower latency, since the data is buffered in memory until it hits the part size before being sent off. i.e. the same sync takes longer to begin since it now needs to read more data per part. This also increases memory requirements. 100 MB part size with 10 upload threads (currently configured) in theory gives us a minimum requirement of 1GB RAM (I haven't tested this, this is from my theoretical understanding of S3.)

I figured if we made this really large, we'll probably still eventually run into a case where a user requires even larger part size, and making it configurable is best since users can decide for themselves. For this reason, think we should keep this configurable.

cgardens

I think you need to add a sections in the redshift docs explaining this field in more detail. sounds like there are some serious tradeoffs here and if you start messing with it, you're likely to run into memory limits unless you also adjust container memory size as well. we should make it as safe as possible to play with this new value.

cgardens · 2021-04-26T22:17:26Z

airbyte-integrations/connectors/destination-redshift/src/main/resources/spec.json

+        "minimum": 10,
+        "maximum": 100,
+        "examples": ["10"],
+        "description": "Files are streamed to S3 in parts. This determines the size of each part, in MBs. As S3 has a limit of 10,000 parts per file, part size affects the table size. This is 10MB by default, resulting in a default limit of 100GB tables. Increase this for larger tables. A larger part size will result in larger memory requirements. Only relevant for COPY.",


can you add in the description that this otptional. the average user shouldn't need to worry about setting this. dunno if we should make it optional or set a default? but i guess it should also be clear what the default is.

I'll make it optional. The default is 10MB and is handled in the Copier code. Will leave comments and all explaining this.

…re-part-size

davinchia · 2021-04-27T04:27:35Z

/test connector=destination-redshift

🕑 destination-redshift https://github.com/airbytehq/airbyte/actions/runs/787866068
✅ destination-redshift https://github.com/airbytehq/airbyte/actions/runs/787866068

davinchia · 2021-04-27T04:27:49Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/787866608
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/787866608

davinchia · 2021-04-27T04:43:54Z

Manual testing confirms this works:

davinchia · 2021-04-27T04:45:07Z

/publish connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/787896754
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/787896754

davinchia · 2021-04-27T05:01:39Z

/publish connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/787926542
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/787926542

davinchia added 2 commits April 26, 2021 19:24

Checkpoint: initial implementation.

f8d3a5b

Checkpoint: initial implementation.

38d4326

davinchia commented Apr 26, 2021

View reviewed changes

Correctly retrieve part size.

e43fdd8

davinchia changed the title ~~Davinchia/redshift configure part size~~ Make Redshift part size configurable. Apr 26, 2021

davinchia requested review from cgardens and jrhizor April 26, 2021 12:41

davinchia marked this pull request as ready for review April 26, 2021 12:41

auto-assign bot requested a review from michel-tricot April 26, 2021 12:41

davinchia removed the request for review from michel-tricot April 26, 2021 12:41

Bump redshift destination version.

a685cf7

jrhizor approved these changes Apr 26, 2021

View reviewed changes

cgardens approved these changes Apr 26, 2021

View reviewed changes

davinchia added 4 commits April 27, 2021 11:28

Merge remote-tracking branch 'origin' into davinchia/redshift-configu…

901ff47

…re-part-size

Changes after merging in master. Bump version to 0.3.2.

bd561e0

Changes after pulling from master.

3e65e24

Bump snowflake to 0.3.2.

5ae5b45

Better comments.

379b30c

davinchia merged commit e63ab84 into master Apr 27, 2021

davinchia deleted the davinchia/redshift-configure-part-size branch April 27, 2021 05:35

karinakuz added the connectors/destinations-warehouse label Jan 18, 2022

karinakuz added the connectors/destination/snowflake label Jan 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make Redshift part size configurable. #3053

Make Redshift part size configurable. #3053

davinchia commented Apr 26, 2021 •

edited

Loading

davinchia Apr 26, 2021

davinchia commented Apr 26, 2021 •

edited by github-actions bot

Loading

davinchia commented Apr 26, 2021

jrhizor left a comment

jrhizor Apr 26, 2021

jrhizor Apr 26, 2021

jrhizor Apr 26, 2021

jrhizor Apr 26, 2021

cgardens commented Apr 26, 2021

davinchia commented Apr 26, 2021 •

edited

Loading

cgardens left a comment

cgardens Apr 26, 2021

davinchia Apr 27, 2021

davinchia commented Apr 27, 2021 •

edited by github-actions bot

Loading

davinchia commented Apr 27, 2021 •

edited by github-actions bot

Loading

davinchia commented Apr 27, 2021

davinchia commented Apr 27, 2021 •

edited by github-actions bot

Loading

davinchia commented Apr 27, 2021 •

edited by github-actions bot

Loading

Make Redshift part size configurable. #3053

Make Redshift part size configurable. #3053

Conversation

davinchia commented Apr 26, 2021 • edited Loading

What

How

Pre-merge Checklist

Recommended reading order

davinchia Apr 26, 2021

Choose a reason for hiding this comment

davinchia commented Apr 26, 2021 • edited by github-actions bot Loading

davinchia commented Apr 26, 2021

jrhizor left a comment

Choose a reason for hiding this comment

jrhizor Apr 26, 2021

Choose a reason for hiding this comment

jrhizor Apr 26, 2021

Choose a reason for hiding this comment

jrhizor Apr 26, 2021

Choose a reason for hiding this comment

jrhizor Apr 26, 2021

Choose a reason for hiding this comment

cgardens commented Apr 26, 2021

davinchia commented Apr 26, 2021 • edited Loading

cgardens left a comment

Choose a reason for hiding this comment

cgardens Apr 26, 2021

Choose a reason for hiding this comment

davinchia Apr 27, 2021

Choose a reason for hiding this comment

davinchia commented Apr 27, 2021 • edited by github-actions bot Loading

davinchia commented Apr 27, 2021 • edited by github-actions bot Loading

davinchia commented Apr 27, 2021

davinchia commented Apr 27, 2021 • edited by github-actions bot Loading

davinchia commented Apr 27, 2021 • edited by github-actions bot Loading

davinchia commented Apr 26, 2021 •

edited

Loading

davinchia commented Apr 26, 2021 •

edited by github-actions bot

Loading

davinchia commented Apr 26, 2021 •

edited

Loading

davinchia commented Apr 27, 2021 •

edited by github-actions bot

Loading

davinchia commented Apr 27, 2021 •

edited by github-actions bot

Loading

davinchia commented Apr 27, 2021 •

edited by github-actions bot

Loading

davinchia commented Apr 27, 2021 •

edited by github-actions bot

Loading