Skip to content

S3 destinations: use STANDARD retry mode - #24736

Merged
Edward Gao (edgao) merged 7 commits into
masterfrom
edgao/s3_retries
Mar 31, 2023
Merged

Edward Gao (edgao) merged 7 commits into
masterfrom
edgao/s3_retries

Conversation

@edgao

@edgao Edward Gao (edgao) commented Mar 30, 2023 •

Copy link
Copy Markdown
Contributor

What

Try to improve how we respond to s3 rate-limiting errors (https://github.com/airbytehq/oncall/issues/1758#issuecomment-1490412590)

How

As title. According to the docs (https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html), we were previously defaulting to LEGACY mode. The aws-java-sdk-core javadoc has this description for STANDARD:

Standard mode is built on top of legacy mode and has throttled retry enabled for throttling errors apart from transient errors

which sounds like better handling for a 503 SlowDown response?

I have no idea how to actually test this though.

@github-actions

github-actions Bot commented Mar 30, 2023 •

Copy link
Copy Markdown
Contributor

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

  • Run integration tests
  • Bump connector or module version
  • Add changelog
  • Publish the new version

✅ Sources (0)

Connector Version Changelog Publish
  • See "Actionable Items" below for how to resolve warnings and errors.

✅ Destinations (10)

Connector Version Changelog Publish
destination-bigquery 1.2.19 ✅ ✅
destination-bigquery-denormalized 1.2.19 ✅ ✅
destination-databricks 1.0.0 ✅ ✅
destination-gcs 0.2.16 ✅ ✅
destination-jdbc 0.3.14 🔵
(ignored)
🔵
(ignored)
destination-r2 0.1.0 ✅ ✅
destination-redshift 0.4.5 ✅ ✅
destination-s3 0.3.23 ✅ ✅
destination-s3-glue 0.1.4 ✅ ✅
destination-snowflake 0.4.61 ✅ ✅
  • See "Actionable Items" below for how to resolve warnings and errors.

👀 Other Modules (1)

  • base-normalization

Actionable Items

(click to expand)

Category Status Actionable Item
Version ❌
mismatch
The version of the connector is different from its normal variant. Please bump the version of the connector.
⚠
doc not found
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.
Changelog ⚠
doc not found
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.
❌
changelog missing
There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish ⚠
not in seed
The connector is not in the seed file (e.g. source_definitions.yaml), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
❌
diff seed version
The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the /publish command to publish the latest version.

@edgao

Edward Gao (edgao) commented Mar 30, 2023 •

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/4569280079
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/4569280079
No Python unittests run

Build Passed

Test summary info:

All Passed

@edgao
Edward Gao (edgao) marked this pull request as ready for review March 30, 2023 21:43
@edgao
Edward Gao (edgao) requested a review from a team as a code owner March 30, 2023 21:43
@edgao

Edward Gao (edgao) commented Mar 30, 2023 •

Copy link
Copy Markdown
Contributor Author

/publish connector=connectors/destination-s3

🕑 Publishing the following connectors:
connectors/destination-s3
https://github.com/airbytehq/airbyte/actions/runs/4570101014


Connector Did it publish? Were definitions generated?
connectors/destination-s3 ✅ ❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@edgao

Edward Gao (edgao) commented Mar 30, 2023 •

Copy link
Copy Markdown
Contributor Author

/publish connector=connectors/destination-snowflake

🕑 Publishing the following connectors:
connectors/destination-snowflake
https://github.com/airbytehq/airbyte/actions/runs/4570101337


Connector Did it publish? Were definitions generated?
connectors/destination-snowflake ✅ ❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@edgao

Edward Gao (edgao) commented Mar 30, 2023 •

Copy link
Copy Markdown
Contributor Author

/publish connector=connectors/destination-redshift

🕑 Publishing the following connectors:
connectors/destination-redshift
https://github.com/airbytehq/airbyte/actions/runs/4570101942


Connector Did it publish? Were definitions generated?
connectors/destination-redshift ✅ ✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@edgao
Edward Gao (edgao) enabled auto-merge (squash) March 31, 2023 14:27
@edgao
Edward Gao (edgao) merged commit 0974a05 into master Mar 31, 2023
@edgao
Edward Gao (edgao) deleted the edgao/s3_retries branch March 31, 2023 14:50
@ryankfu

Copy link
Copy Markdown
Contributor

/test connector=connectors/destination-snowflake

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants