🐛 Source Slack: Fix sync operations hang forever issue (#5830)

* Add user-defined backoff * Update airbyte-integrations/connectors/source-slack/source_slack/source.py * Bumped version Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
airbytehq · Sep 3, 2021 · 898e2c8 · 898e2c8
1 parent 7f18cec
commit 898e2c8
Show file tree

Hide file tree

Showing 5 changed files with 17 additions and 3 deletions.
diff --git a/...ain/resources/config/STANDARD_SOURCE_DEFINITION/c2281cee-86f9-4a86-bb48-d23286b4c7bd.json b/...ain/resources/config/STANDARD_SOURCE_DEFINITION/c2281cee-86f9-4a86-bb48-d23286b4c7bd.json
@@ -2,7 +2,7 @@
   "sourceDefinitionId": "c2281cee-86f9-4a86-bb48-d23286b4c7bd",
   "name": "Slack",
   "dockerRepository": "airbyte/source-slack",
-  "dockerImageTag": "0.1.10",
+  "dockerImageTag": "0.1.11",
   "documentationUrl": "https://docs.airbyte.io/integrations/sources/slack",
   "icon": "slack.svg"
 }
diff --git a/airbyte-config/init/src/main/resources/seed/source_definitions.yaml b/airbyte-config/init/src/main/resources/seed/source_definitions.yaml
@@ -355,7 +355,7 @@
 - sourceDefinitionId: c2281cee-86f9-4a86-bb48-d23286b4c7bd
   name: Slack
   dockerRepository: airbyte/source-slack
-  dockerImageTag: 0.1.10
+  dockerImageTag: 0.1.11
   documentationUrl: https://docs.airbyte.io/integrations/sources/slack
   icon: slack.svg
 - sourceDefinitionId: 6ff047c0-f5d5-4ce5-8c81-204a830fa7e1

diff --git a/airbyte-integrations/connectors/source-slack/Dockerfile b/airbyte-integrations/connectors/source-slack/Dockerfile
@@ -16,5 +16,5 @@ RUN pip install .
 ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
 ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
 
-LABEL io.airbyte.version=0.1.10
+LABEL io.airbyte.version=0.1.11
 LABEL io.airbyte.name=airbyte/source-slack
diff --git a/airbyte-integrations/connectors/source-slack/source_slack/source.py b/airbyte-integrations/connectors/source-slack/source_slack/source.py
@@ -79,6 +79,19 @@ def parse_response(
         json_response = response.json()
         yield from json_response.get(self.data_field, [])
 
+    def backoff_time(self, response: requests.Response) -> Optional[float]:
+        """This method is called if we run into the rate limit.
+        Slack puts the retry time in the `Retry-After` response header so we
+        we return that value. If the response is anything other than a 429 (e.g: 5XX)
+        fall back on default retry behavior.
+        Rate Limits Docs: https://api.slack.com/docs/rate-limits#web"""
+
+        if "Retry-After" in response.headers: 
+            return int(response.headers["Retry-After"])
+        else: 
+            self.logger.info("Retry-after header not found. Using default backoff value")
+            return 5
+
     @property
     @abstractmethod
     def data_field(self) -> str:

diff --git a/docs/integrations/sources/slack.md b/docs/integrations/sources/slack.md
@@ -101,6 +101,7 @@ We recommend creating a restricted, read-only key specifically for Airbyte acces
 
 | Version | Date       | Pull Request | Subject |
 | :------ | :--------  | :-----       | :------ |
+| 0.1.11   | 2021-08-27 | [5830](https://github.com/airbytehq/airbyte/pull/5830) | Fixed sync operations hang forever issue |
 | 0.1.10   | 2021-08-27 | [5697](https://github.com/airbytehq/airbyte/pull/5697) | Fixed max retries issue |
 | 0.1.9   | 2021-07-20 | [4860](https://github.com/airbytehq/airbyte/pull/4860) | Fixed reading threads issue |
 | 0.1.8   | 2021-07-14 | [4683](https://github.com/airbytehq/airbyte/pull/4683) | Add float_ts primary key |