🐛 Source Facebook Marketing: fix DATA_RETENTION_PERIOD validation a…

…nd schema data type `failed_delivery_checks` issues (#15012) * Fix date validation and schema issues * Updated PR number * Updated to review * Updated to review * Update airbyte-integrations/connectors/source-facebook-marketing/unit_tests/test_utils.py Co-authored-by: Pedro S. Lopez <pedroslopez@me.com> * Update airbyte-integrations/connectors/source-facebook-marketing/unit_tests/test_utils.py Co-authored-by: Pedro S. Lopez <pedroslopez@me.com> * Fix to linter * Fix typo * Updated Docker version * auto-bump connector version [ci skip] Co-authored-by: Pedro S. Lopez <pedroslopez@me.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
airbytehq · Jul 26, 2022 · 5d18fed · 5d18fed
1 parent b01c731
commit 5d18fed
Show file tree

Hide file tree

Showing 7 changed files with 70 additions and 12 deletions.
diff --git a/airbyte-config/init/src/main/resources/seed/source_definitions.yaml b/airbyte-config/init/src/main/resources/seed/source_definitions.yaml
@@ -248,7 +248,7 @@
 - name: Facebook Marketing
   sourceDefinitionId: e7778cfc-e97c-4458-9ecb-b4f2bba8946c
   dockerRepository: airbyte/source-facebook-marketing
-  dockerImageTag: 0.2.57
+  dockerImageTag: 0.2.58
   documentationUrl: https://docs.airbyte.io/integrations/sources/facebook-marketing
   icon: facebook.svg
   sourceType: api

diff --git a/airbyte-config/init/src/main/resources/seed/source_specs.yaml b/airbyte-config/init/src/main/resources/seed/source_specs.yaml
@@ -1823,7 +1823,7 @@
     supportsNormalization: false
     supportsDBT: false
     supported_destination_sync_modes: []
-- dockerImage: "airbyte/source-facebook-marketing:0.2.57"
+- dockerImage: "airbyte/source-facebook-marketing:0.2.58"
   spec:
     documentationUrl: "https://docs.airbyte.io/integrations/sources/facebook-marketing"
     changelogUrl: "https://docs.airbyte.io/integrations/sources/facebook-marketing"

diff --git a/airbyte-integrations/connectors/source-facebook-marketing/Dockerfile b/airbyte-integrations/connectors/source-facebook-marketing/Dockerfile
@@ -13,5 +13,5 @@ ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
 ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
 
 
-LABEL io.airbyte.version=0.2.57
+LABEL io.airbyte.version=0.2.58
 LABEL io.airbyte.name=airbyte/source-facebook-marketing
diff --git a/...ns/connectors/source-facebook-marketing/source_facebook_marketing/schemas/ad_account.json b/...ns/connectors/source-facebook-marketing/source_facebook_marketing/schemas/ad_account.json
@@ -78,7 +78,21 @@
       "type": ["null", "number"]
     },
     "failed_delivery_checks": {
-      "type": ["null", "number"]
+      "type": ["null", "array"],
+       "items": {
+          "type": ["null", "object"],
+          "properties": {
+            "summary": {
+              "type": ["null", "string"]
+            },
+            "description": {
+              "type": ["null", "string"]
+            },
+            "check_name": {
+              "type": ["null", "string"]
+            }
+          }
+        }
     },
     "fb_entity": {
       "type": ["null", "number"]

diff --git a/airbyte-integrations/connectors/source-facebook-marketing/source_facebook_marketing/utils.py b/airbyte-integrations/connectors/source-facebook-marketing/source_facebook_marketing/utils.py
@@ -4,11 +4,18 @@
 
 from datetime import datetime
 
+import pendulum
 
-class FutureDateException(Exception):
-    def __init__(self, field_name, *args, **kwargs):
-        self.field_name = field_name
-        self.message = f"{self.field_name} cannot be in the future. Please set today's date or later"
+# Facebook store metrics maximum of 37 months old. Any time range that
+# older that 37 months from current date would result in 400 Bad request
+# HTTP response.
+# https://developers.facebook.com/docs/marketing-api/reference/ad-account/insights/#overview
+DATA_RETENTION_PERIOD = pendulum.duration(months=37)
+
+
+class ValidationDateException(Exception):
+    def __init__(self, message, *args, **kwargs):
+        self.message = message
         super().__init__(self.message, *args, **kwargs)
 
     def __str__(self):
@@ -18,7 +25,12 @@ def __repr__(self):
         return self.__str__()
 
 
-def validate_date_field(field, date):
-    if date.timestamp() > datetime.now().timestamp():
-        raise FutureDateException(field)
+def validate_date_field(field_name: str, date: datetime) -> datetime:
+    pendulum_date = pendulum.instance(date)
+    if pendulum_date.timestamp() > pendulum.now().timestamp():
+        message = f"{field_name} cannot be in the future. Please set today's date or later."
+        raise ValidationDateException(message)
+    elif pendulum_date.timestamp() < (pendulum.now() - DATA_RETENTION_PERIOD).timestamp():
+        message = f"{field_name} cannot be beyond {DATA_RETENTION_PERIOD.months} months from the current date."
+        raise ValidationDateException(message)
     return date
diff --git a/airbyte-integrations/connectors/source-facebook-marketing/unit_tests/test_utils.py b/airbyte-integrations/connectors/source-facebook-marketing/unit_tests/test_utils.py
@@ -0,0 +1,31 @@
+#
+# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
+#
+
+import pendulum
+import pytest
+from source_facebook_marketing.utils import DATA_RETENTION_PERIOD, ValidationDateException, validate_date_field
+
+
+@pytest.mark.parametrize(
+    "date, expected_message, raise_error",
+    [
+        (pendulum.now(), "", False),
+        (
+            pendulum.now() - pendulum.duration(months=DATA_RETENTION_PERIOD.months + 1),
+            f" cannot be beyond {DATA_RETENTION_PERIOD.months} months from the current date.",
+            True,
+        ),
+        (pendulum.now() + pendulum.duration(months=1), " cannot be in the future. Please set today's date or later.", True),
+    ],
+    ids=["valid_date", f"date in the past by {DATA_RETENTION_PERIOD.months} months", "date in future"],
+)
+def test_validate_date_field(date, expected_message, raise_error):
+    field_name = "test_field_name"
+
+    if raise_error:
+        with pytest.raises(ValidationDateException) as error:
+            assert validate_date_field(field_name, date)
+        assert str(error.value) == field_name + expected_message
+    else:
+        assert validate_date_field(field_name, date)
diff --git a/docs/integrations/sources/facebook-marketing.md b/docs/integrations/sources/facebook-marketing.md
@@ -120,7 +120,8 @@ Please be informed that the connector uses the `lookback_window` parameter to pe
 
 | Version | Date       | Pull Request                                             | Subject                                                                                                                                                                                                                                                                                           |
 |:--------|:-----------|:---------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| 0.2.57  | 2022-07-25 | [14831](https://github.com/airbytehq/airbyte/pull/14831) | Update Facebook SDK to version 14.0.0                                                                                                                                                            | 
+| 0.2.58  | 2022-07-25 | [15012](https://github.com/airbytehq/airbyte/pull/15012) | Add `DATA_RETENTION_PERIOD`validation and fix `failed_delivery_checks` field schema type issue                                                                                                                                                                                                    |     
+| 0.2.57  | 2022-07-25 | [14831](https://github.com/airbytehq/airbyte/pull/14831) | Update Facebook SDK to version 14.0.0                                                                                                                                                                                                                                                             | 
 | 0.2.56  | 2022-07-19 | [14831](https://github.com/airbytehq/airbyte/pull/14831) | Add future `start_date` and `end_date` validation                                                                                                                                                                                                                                                 |     
 | 0.2.55  | 2022-07-18 | [14786](https://github.com/airbytehq/airbyte/pull/14786) | Check if the authorized user has the "MANAGE" task permission when getting the `funding_source_details` field in the ad\_account stream                                                                                                                                                           |
 | 0.2.54  | 2022-06-29 | [14267](https://github.com/airbytehq/airbyte/pull/14267) | Make MAX_BATCH_SIZE available in config                                                                                                                                                                                                                                                           |