-
Notifications
You must be signed in to change notification settings - Fork 3.8k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* checkout files from test branch * read_incremental works * reset to master * remove dead code * comment * fix * Add test * comments * utc * format * small fix * Add test with rfc3339 * remove unused param * fix test * configurable state checkpointing * update test * start working on retrier * retry predicate * return response status * look in error message * cleanup test * constant backoff strategy * chain backoff strategy * chain retrier * Add to class types registry * extract backoff time from header * wait until * update * split file * parse_records * classmethod * delete dead code * comment * comment * comments * fix * test for instantiating chain retrier * fix parsing * cleanup * fix * reset * never raise on http error * remove print * comment * comment * comment * comment * remove prints * add declarative stream to registry * Delete dead code * Add docstrings * quick fix * exponential backoff * fix test * fix * delete unused properties * fix * missing unit tests * uppercase * docstrings * rename to success * compare full request instead of just url * renmae module * rename test file * rename interface * rename default retrier * rename to compositeerrorhandler * fix missing renames * move action to filter * str -> minmaxdatetime * small fixes * plural * add example * handle header variations * also fix wait time from * allow using a regex to extract the value * group() * docstring * add docs * update comment * docstrings * update comment * Update airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/http_requester.py Co-authored-by: Sherif A. Nada <snadalive@gmail.com> * version: Update Parquet library to latest release (#14502) The upstream Parquet library that is currently pinned for use in the S3 destination plugin is over a year old. The current version is generating invalid schemas for date-time with time-zone fields which appears to be addressed in the `1.12.3` release of the library in commit apache/parquet-java@c72862b * merge * 🎉 Source Github: improve schema for stream `pull_request_commits` added "null" (#14613) Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * Docs: Fixed broken links (#14622) * fixing broken links * more broken links * source-hubspot: change mentioning of Mailchimp into HubSpot doc (#14620) * Helm Chart: Add external temporal option (#14597) * conflict env configmap and chart lock * reverting lock * add eof lines and documentation on values yaml * conflict json file * rollback json * solve conflict * correct minio with new version Co-authored-by: Guy Feldman <gfeldman@86labs.com> * 🎉 Add YAML format to source-file reader (#14588) * Add yaml reader * Update docs * Bumpversion of connector * bump docs * Update pyarrow dependency * Upgrade pandas dependency * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * 🎉 Source Okta: add GroupMembers stream (#14380) * add Group_Members stream to okta source - Group_Members return a list of users, the same schema of Users stream. - Create a shared schema users, and both group_members and users sechema use it as a reference. - Add Group_Members stream to source connector * add tests and fix logs schema - fix the test error: None is not one of enums though the enum type includes both string and null, it comes from json schema validator https://github.com/python-jsonschema/jsonschema/blob/ddb87afad8f5d5c40600b5ede0ab96e4d4bdf7d3/jsonschema/_validators.py#L279-L285 - change grouop_members to use id as the cursor field since `filter` is not supported in the query string - fix the abnormal state test on logs stream, when since is abnormally large, until has to defined, an equal or a larger value - remove logs stream from full sync test, because 2 full sync always has a gap -- at least a new log about users or groups api. * last polish before submit the PR - bump docker version - update changelog - add the right abnormal value for logs stream - correct the sample catalog * address comments:: - improve comments for until parameter under the logs stream - add use_cache on groupMembers * add use_cache to Group_Members * change configured_catalog to test * auto-bump connector version Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * split test files * renames * missing unit test * add missing unit tests * rename * assert isinstance * start extracting to their own files * use final instead of classmethod * assert we retry 429 errors * Add log * replace asserts with valueexceptions * delete superfluous print statement * fix factory so we don't need to union everything with strings * get class_name from type * remove from class types registry * process error handlers one at a time * sort * delete print statement * comment * comment * format * delete unused file Co-authored-by: Sherif A. Nada <snadalive@gmail.com> Co-authored-by: Tobias Macey <tmacey@boundlessnotions.com> Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com> Co-authored-by: Amruta Ranade <11484018+Amruta-Ranade@users.noreply.github.com> Co-authored-by: Bas Beelen <bjgbeelen@gmail.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: Guy Feldman <gfeldman@86labs.com> Co-authored-by: Christophe Duong <christophe.duong@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: Yiyang Li <yiyangli2010@gmail.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
- Loading branch information
12 people
committed
Jul 14, 2022
1 parent
c622dd0
commit 09aa685
Showing
45 changed files
with
1,274 additions
and
213 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
3 changes: 3 additions & 0 deletions
3
airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/error_handlers/__init__.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| # | ||
| # Copyright (c) 2022 Airbyte, Inc., all rights reserved. | ||
| # |
3 changes: 3 additions & 0 deletions
3
.../airbyte_cdk/sources/declarative/requesters/error_handlers/backoff_strategies/__init__.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| # | ||
| # Copyright (c) 2022 Airbyte, Inc., all rights reserved. | ||
| # |
23 changes: 23 additions & 0 deletions
23
...ces/declarative/requesters/error_handlers/backoff_strategies/constant_backoff_strategy.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| # | ||
| # Copyright (c) 2022 Airbyte, Inc., all rights reserved. | ||
| # | ||
|
|
||
| from typing import Optional | ||
|
|
||
| import requests | ||
| from airbyte_cdk.sources.declarative.requesters.error_handlers.backoff_strategy import BackoffStrategy | ||
|
|
||
|
|
||
| class ConstantBackoffStrategy(BackoffStrategy): | ||
| """ | ||
| Backoff strategy with a constant backoff interval | ||
| """ | ||
|
|
||
| def __init__(self, backoff_time_in_seconds: float): | ||
| """ | ||
| :param backoff_time_in_seconds: time to backoff before retrying a retryable request | ||
| """ | ||
| self._backoff_time_in_seconds = backoff_time_in_seconds | ||
|
|
||
| def backoff(self, response: requests.Response, attempt_count: int) -> Optional[float]: | ||
| return self._backoff_time_in_seconds |
23 changes: 23 additions & 0 deletions
23
.../declarative/requesters/error_handlers/backoff_strategies/exponential_backoff_strategy.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| # | ||
| # Copyright (c) 2022 Airbyte, Inc., all rights reserved. | ||
| # | ||
|
|
||
| from typing import Optional | ||
|
|
||
| import requests | ||
| from airbyte_cdk.sources.declarative.requesters.error_handlers.backoff_strategy import BackoffStrategy | ||
|
|
||
|
|
||
| class ExponentialBackoffStrategy(BackoffStrategy): | ||
| """ | ||
| Backoff strategy with an exponential backoff interval | ||
| """ | ||
|
|
||
| def __init__(self, factor: float = 5): | ||
| """ | ||
| :param factor: multiplicative factor | ||
| """ | ||
| self._factor = factor | ||
|
|
||
| def backoff(self, response: requests.Response, attempt_count: int) -> Optional[float]: | ||
| return self._factor * 2**attempt_count |
39 changes: 39 additions & 0 deletions
39
...yte_cdk/sources/declarative/requesters/error_handlers/backoff_strategies/header_helper.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| # | ||
| # Copyright (c) 2022 Airbyte, Inc., all rights reserved. | ||
| # | ||
|
|
||
| import numbers | ||
| from re import Pattern | ||
| from typing import Optional | ||
|
|
||
| import requests | ||
|
|
||
|
|
||
| def get_numeric_value_from_header(response: requests.Response, header: str, regex: Optional[Pattern]) -> Optional[float]: | ||
| """ | ||
| Extract a header value from the response as a float | ||
| :param response: response the extract header value from | ||
| :param header: Header to extract | ||
| :param regex: optional regex to apply on the header to obtain the value | ||
| :return: header value as float if it's a number. None otherwise | ||
| """ | ||
| header_value = response.headers.get(header, None) | ||
| if not header_value: | ||
| return None | ||
| if isinstance(header_value, str): | ||
| if regex: | ||
| match = regex.match(header_value) | ||
| if match: | ||
| header_value = match.group() | ||
| return _as_float(header_value) | ||
| elif isinstance(header_value, numbers.Number): | ||
| return float(header_value) | ||
| else: | ||
| return None | ||
|
|
||
|
|
||
| def _as_float(s: str) -> Optional[float]: | ||
| try: | ||
| return float(s) | ||
| except ValueError: | ||
| return None |
28 changes: 28 additions & 0 deletions
28
...ve/requesters/error_handlers/backoff_strategies/wait_time_from_header_backoff_strategy.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| # | ||
| # Copyright (c) 2022 Airbyte, Inc., all rights reserved. | ||
| # | ||
|
|
||
| import re | ||
| from typing import Optional | ||
|
|
||
| import requests | ||
| from airbyte_cdk.sources.declarative.requesters.error_handlers.backoff_strategies.header_helper import get_numeric_value_from_header | ||
| from airbyte_cdk.sources.declarative.requesters.error_handlers.backoff_strategy import BackoffStrategy | ||
|
|
||
|
|
||
| class WaitTimeFromHeaderBackoffStrategy(BackoffStrategy): | ||
| """ | ||
| Extract wait time from http header | ||
| """ | ||
|
|
||
| def __init__(self, header: str, regex: Optional[str] = None): | ||
| """ | ||
| :param header: header to read wait time from | ||
| :param regex: optional regex to apply on the header to extract its value | ||
| """ | ||
| self._header = header | ||
| self._regex = re.compile(regex) if regex else None | ||
|
|
||
| def backoff(self, response: requests.Response, attempt_count: int) -> Optional[float]: | ||
| header_value = get_numeric_value_from_header(response, self._header, self._regex) | ||
| return header_value |
45 changes: 45 additions & 0 deletions
45
...uesters/error_handlers/backoff_strategies/wait_until_time_from_header_backoff_strategy.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| # | ||
| # Copyright (c) 2022 Airbyte, Inc., all rights reserved. | ||
| # | ||
|
|
||
| import numbers | ||
| import re | ||
| import time | ||
| from typing import Optional | ||
|
|
||
| import requests | ||
| from airbyte_cdk.sources.declarative.requesters.error_handlers.backoff_strategies.header_helper import get_numeric_value_from_header | ||
| from airbyte_cdk.sources.declarative.requesters.error_handlers.backoff_strategy import BackoffStrategy | ||
|
|
||
|
|
||
| class WaitUntilTimeFromHeaderBackoffStrategy(BackoffStrategy): | ||
| """ | ||
| Extract time at which we can retry the request from response header | ||
| and wait for the difference between now and that time | ||
| """ | ||
|
|
||
| def __init__(self, header: str, min_wait: Optional[float] = None, regex: Optional[str] = None): | ||
| """ | ||
| :param header: header to read wait time from | ||
| :param min_wait: minimum time to wait for safety | ||
| :param regex: optional regex to apply on the header to extract its value | ||
| """ | ||
| self._header = header | ||
| self._min_wait = min_wait | ||
| self._regex = re.compile(regex) if regex else None | ||
|
|
||
| def backoff(self, response: requests.Response, attempt_count: int) -> Optional[float]: | ||
| now = time.time() | ||
| wait_until = get_numeric_value_from_header(response, self._header, self._regex) | ||
| if wait_until is None or not wait_until: | ||
| return self._min_wait | ||
| if (isinstance(wait_until, str) and wait_until.isnumeric()) or isinstance(wait_until, numbers.Number): | ||
| wait_time = float(wait_until) - now | ||
| else: | ||
| return self._min_wait | ||
| if self._min_wait: | ||
| return max(wait_time, self._min_wait) | ||
| elif wait_time < 0: | ||
| return None | ||
| return wait_time |
19 changes: 19 additions & 0 deletions
19
...-cdk/python/airbyte_cdk/sources/declarative/requesters/error_handlers/backoff_strategy.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| # | ||
| # Copyright (c) 2022 Airbyte, Inc., all rights reserved. | ||
| # | ||
|
|
||
| from abc import abstractmethod | ||
| from typing import Optional | ||
|
|
||
| import requests | ||
|
|
||
|
|
||
| class BackoffStrategy: | ||
| @abstractmethod | ||
| def backoff(self, response: requests.Response, attempt_count: int) -> Optional[float]: | ||
| """ | ||
| Return time to wait before retrying the request. | ||
| :param response: response received for the request to retry | ||
| :param attempt_count: number of attempts to submit the request | ||
| :return: time to wait in seconds | ||
| """ |
Oops, something went wrong.