New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🐛 Source Notion: refactor backoff time for 500 errors #31612
Conversation
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Ignored Deployment
|
Before Merging a Connector Pull RequestWow! What a great pull request you have here! 🎉 To merge this PR, ensure the following has been done/considered for each connector added or updated:
If the checklist is complete, but the CI check is failing,
|
…airbytehq/airbyte into christo/notion-backoff-strategy
source-notion test report (commit
|
Step | Result |
---|---|
Build source-notion docker image for platform(s) linux/x86_64 | ✅ |
Unit tests | ✅ |
Acceptance tests | ✅ |
Code format checks | ✅ |
Validate metadata for source-notion | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-notion test
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good.
Co-authored-by: ChristoGrab <ChristoGrab@users.noreply.github.com>
Co-authored-by: ChristoGrab <ChristoGrab@users.noreply.github.com>
What
Refactored the backoff_time method to use exponential backoff time for 500-level errors.
Hoping this will resolve oncall #3268
How
In the case of a 429 error, we simply use the retry-after header as before with no exponential backoff, since Notion has a very generous rate limiting policy.
For 500 level errors we use the default exponential backoff provided in the CDK with a retry_factor of 8. This should provide more leeway for customers encountering sync failures due to lingering Notion server issues by substantially increasing the total backoff_time before failure.
The existing logic for the retryable
invalid_start_cursor
400 level error has been left as is.User Impact
None, not a breaking change.