Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix/requests error handling #477

Merged
merged 7 commits into from
Jul 8, 2023
Merged

Fix/requests error handling #477

merged 7 commits into from
Jul 8, 2023

Conversation

steinitzu
Copy link
Collaborator

Fixes #473

  • ChunkedLoadingException added to retry list. Seems this almost always due to some kind of dropped connection
  • requests Timeout and ConnectionError are already retried and all server errors inherit those
  • Added some of the retry settings to RunConfiguration

@steinitzu steinitzu requested a review from rudolfix July 7, 2023 00:36
@netlify
Copy link

netlify bot commented Jul 7, 2023

Deploy Preview for dlt-hub-docs canceled.

Name Link
🔨 Latest commit 2c44656
🔍 Latest deploy log https://app.netlify.com/sites/dlt-hub-docs/deploys/64a88eec4cf0830008dd381d

Copy link
Collaborator

@rudolfix rudolfix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks excellent! +1 for removing the tuples. pls. see one proposed improvement in the review.
what is missing is a short explanation how the timeout and retries are configured. please add a new section to performance.md in the docs - give an example how to configure retries and what are the defaults

@@ -146,27 +148,29 @@ class Client:
"""
def __init__(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should be able to decorate this @with_config(RuntimeConfiguration). and the right arguments will be injected. why I think it is cool:

  • if someone creates a session from a source, the configuration may be defined in the scope of the source ("sources", "pipedrive", "runtime")
  • maybe if at some point we'll do a lazy instantiation of default client, even the default config will be scoped

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. Added this and some docs!

@steinitzu steinitzu force-pushed the fix/requests-error-handling branch from d0978ca to 60b4a46 Compare July 7, 2023 22:01
@steinitzu steinitzu force-pushed the fix/requests-error-handling branch from 60b4a46 to cf0734f Compare July 7, 2023 22:06
@rudolfix rudolfix closed this Jul 8, 2023
@rudolfix rudolfix reopened this Jul 8, 2023
@rudolfix rudolfix merged commit 23427f1 into devel Jul 8, 2023
39 of 43 checks passed
@rudolfix rudolfix deleted the fix/requests-error-handling branch July 8, 2023 15:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

add more retry exceptions and configurable timeout to requests' session
2 participants