🎉 Source Recharge: improve 'backoff' for HTTP requests (#7626)

* backoff for successful but incomplete responses Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
airbytehq · Nov 8, 2021 · af903f2 · af903f2
1 parent 154ecce
commit af903f2
Show file tree

Hide file tree

Showing 5 changed files with 13 additions and 3 deletions.
diff --git a/...ain/resources/config/STANDARD_SOURCE_DEFINITION/45d2e135-2ede-49e1-939f-3e3ec357a65e.json b/...ain/resources/config/STANDARD_SOURCE_DEFINITION/45d2e135-2ede-49e1-939f-3e3ec357a65e.json
@@ -2,6 +2,6 @@
   "sourceDefinitionId": "45d2e135-2ede-49e1-939f-3e3ec357a65e",
   "name": "Recharge",
   "dockerRepository": "airbyte/source-recharge",
-  "dockerImageTag": "0.1.3",
+  "dockerImageTag": "0.1.4",
   "documentationUrl": "https://docs.airbyte.io/integrations/sources/recharge"
 }
diff --git a/airbyte-config/init/src/main/resources/seed/source_definitions.yaml b/airbyte-config/init/src/main/resources/seed/source_definitions.yaml
@@ -436,7 +436,7 @@
 - name: Recharge
   sourceDefinitionId: 45d2e135-2ede-49e1-939f-3e3ec357a65e
   dockerRepository: airbyte/source-recharge
-  dockerImageTag: 0.1.3
+  dockerImageTag: 0.1.4
   documentationUrl: https://docs.airbyte.io/integrations/sources/recharge
   sourceType: api
 - name: Recurly

diff --git a/airbyte-integrations/connectors/source-recharge/Dockerfile b/airbyte-integrations/connectors/source-recharge/Dockerfile
@@ -12,5 +12,5 @@ RUN pip install .
 ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
 ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
 
-LABEL io.airbyte.version=0.1.3
+LABEL io.airbyte.version=0.1.4
 LABEL io.airbyte.name=airbyte/source-recharge
diff --git a/airbyte-integrations/connectors/source-recharge/source_recharge/api.py b/airbyte-integrations/connectors/source-recharge/source_recharge/api.py
@@ -56,6 +56,15 @@ def get_stream_data(self, response_data: Any) -> List[dict]:
         else:
             return [response_data]
 
+    def should_retry(self, response: requests.Response) -> bool:
+        res = super().should_retry(response)
+        if res:
+            return res
+
+        # For some reason, successful responses contains incomplete data
+        content_length = int(response.headers.get("Content-Length", 0))
+        return response.status_code == 200 and content_length > len(response.content)
+
 
 class IncrementalRechargeStream(RechargeStream, ABC):
 

diff --git a/docs/integrations/sources/recharge.md b/docs/integrations/sources/recharge.md
@@ -50,6 +50,7 @@ Please read [How to generate your API token](https://support.rechargepayments.co
 
 | Version | Date | Pull Request | Subject |
 | :--- | :--- | :--- | :--- |
+| 0.1.4 | 2021-11-05 | [7626](https://github.com/airbytehq/airbyte/pull/7626) | Improve 'backoff' for HTTP requests |
 | 0.1.3 | 2021-09-17 | [6149](https://github.com/airbytehq/airbyte/pull/6149) | Update `discount` and `order` schema |
 | 0.1.2 | 2021-09-17 | [6149](https://github.com/airbytehq/airbyte/pull/6149) | Change `cursor_field` for Incremental streams |
 |  |  |  |  |