🎉 New Source: Facebook Pages #5158

gaart · 2021-08-03T14:30:40Z

What

Facebook Pages connector.
#1247

Pre-merge Checklist

Expand the checklist which is relevant for this PR.

Connector checklist

Connector Generator checklist

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

gaart · 2021-08-03T14:31:02Z

/test connector=connectors/source-facebook-pages

🕑 connectors/source-facebook-pages https://github.com/airbytehq/airbyte/actions/runs/1094300626
❌ connectors/source-facebook-pages https://github.com/airbytehq/airbyte/actions/runs/1094300626

vovavovavovavova · 2021-08-05T20:42:52Z

airbyte-integrations/connectors/source-facebook-pages/acceptance-test-config.yml

+#        extra_fields: no
+#        exact_order: no
+#        extra_records: yes
+#  incremental: # TODO if your connector does not implement incremental sync, remove this block


please add a comment why it is commented / not supported ( complex state or something else?)

a bit complicated task, need further investigation how to implement it using graphql endpoints

vovavovavovavova · 2021-08-05T20:43:50Z

airbyte-integrations/connectors/source-facebook-pages/build.gradle

+
+dependencies {
+    implementation files(project(':airbyte-integrations:bases:source-acceptance-test').airbyteDocker.outputs)
+    implementation files(project(':airbyte-integrations:bases:base-python').airbyteDocker.outputs)


this line (13th) is deprecated but still exists in template-generator, need to be removed (look on another connectors for example)

vovavovavovavova · 2021-08-05T20:47:36Z

airbyte-integrations/connectors/source-facebook-pages/integration_tests/sample_state.json

+  "todo-stream-name": {
+    "todo-field-name": "value"
+  }
+}


fields should be similar to your state.json, not todo-x

vovavovavovavova · 2021-08-05T20:54:50Z

airbyte-integrations/connectors/source-facebook-pages/source_facebook_pages/source.py

+            "access_token": config["access_token"],
+        }
+
+        response = requests.get("https://graph.facebook.com/v11.0/me", params=params)


usually some stream is used here last times (examples https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/source.py#L65, https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-posthog/source_posthog/source.py#L55 etc )

vovavovavovavova · 2021-08-05T20:58:43Z

airbyte-integrations/connectors/source-facebook-pages/source_facebook_pages/streams.py

+from .metrics import PAGE_FIELDS, PAGE_METRICS, POST_FIELDS, POST_METRICS
+
+
+class FacebookStream(HttpStream, ABC):


probably FacebookPagesStream is more clear, what do you think

vovavovavovavova · 2021-08-05T20:59:56Z

airbyte-integrations/connectors/source-facebook-pages/source_facebook_pages/streams.py

+            return f"{self._page_id}/posts"
+
+    def parse_response(self, response: requests.Response, **kwargs) -> Iterable[Mapping]:
+        records = response.json().get(self.data_field) or []


records = response.json().get(self.data_field, [])

in case self.data_field is empty (None, False, etc) - it's better to have [] as a result

vovavovavovavova · 2021-08-05T21:02:18Z

airbyte-integrations/connectors/source-facebook-pages/source_facebook_pages/streams.py

+    """
+
+    def path(self, **kwargs) -> str:
+        if self._page_id:


is not this parameter mandatory or what happen if it would not be provided?

vovavovavovavova · 2021-08-05T21:07:58Z

there are few questions in comments
you need to ask Eugene / (anyone else with rights) to add credentials from lastpass to github to avoid missing secrets, like git CI shows now.
you will need to upgrade source-definitions. See this comment and files changed ( 4 first files changed) as an example 🎉 Posthog: New Source #3768 (review)
also you will need to upgrade documentation (details are present in connector checklist and examples)

vovavovavovavova · 2021-08-10T19:26:29Z

/test connector=connectors/source-facebook-pages

🕑 connectors/source-facebook-pages https://github.com/airbytehq/airbyte/actions/runs/1117793779
❌ connectors/source-facebook-pages https://github.com/airbytehq/airbyte/actions/runs/1117793779

gaart · 2021-08-11T20:07:34Z

/test connector=connectors/source-facebook-pages

🕑 connectors/source-facebook-pages https://github.com/airbytehq/airbyte/actions/runs/1121468832
❌ connectors/source-facebook-pages https://github.com/airbytehq/airbyte/actions/runs/1121468832

yevhenii-ldv · 2021-08-11T21:27:54Z

/test connector=connectors/source-facebook-pages

🕑 connectors/source-facebook-pages https://github.com/airbytehq/airbyte/actions/runs/1121696239
❌ connectors/source-facebook-pages https://github.com/airbytehq/airbyte/actions/runs/1121696239

gaart · 2021-08-12T07:11:08Z

/test connector=connectors/source-facebook-pages

🕑 connectors/source-facebook-pages https://github.com/airbytehq/airbyte/actions/runs/1122967786
❌ connectors/source-facebook-pages https://github.com/airbytehq/airbyte/actions/runs/1122967786

yevhenii-ldv · 2021-08-12T13:03:20Z

/test connector=connectors/source-facebook-marketing

🕑 connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/1123974721
❌ connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/1123974721

gaart · 2021-08-13T06:26:20Z

/test connector=connectors/source-facebook-marketing

🕑 connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/1126684195
❌ connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/1126684195

sherifnada · 2021-08-25T07:47:33Z

.github/workflows/publish-command.yml

@@ -89,7 +89,7 @@ jobs:
          DRIFT_INTEGRATION_TEST_CREDS: ${{ secrets.DRIFT_INTEGRATION_TEST_CREDS }}
          EXCHANGE_RATES_TEST_CREDS: ${{ secrets.EXCHANGE_RATES_TEST_CREDS }}
          FACEBOOK_MARKETING_TEST_INTEGRATION_CREDS: ${{ secrets.FACEBOOK_MARKETING_TEST_INTEGRATION_CREDS }}
-          FACEBOOK_MARKETING_API_TEST_INTEGRATION_CREDS: ${{ secrets.FACEBOOK_MARKETING_API_TEST_INTEGRATION_CREDS }}


this one was unused?

It's a merged change made by @keu , commit #24f310de

airbyte-integrations/connectors/source-facebook-pages/source_facebook_pages/streams.py

sherifnada · 2021-08-26T05:56:40Z

airbyte-integrations/connectors/source-facebook-pages/source_facebook_pages/streams.py

+
+class Post(FacebookPagesStream):
+    """
+    API docs: https://developers.facebook.com/docs/graph-api/reference/post/,


this link mentions /{post-id} but doesn't mention {page-id}/posts - is that link documented anywhere? is it same as feed? if so then we need to handle pagination

Also seems to be a paginated

Yes, it's the same as the feed endpoint.
Updated, pagination added

sherifnada · 2021-08-26T06:01:31Z

airbyte-integrations/connectors/source-facebook-pages/source_facebook_pages/streams.py

+    def parse_response(self, response: requests.Response, **kwargs) -> Iterable[Mapping]:
+        records = response.json().get(self.data_field) or []
+
+        for insights in records:


data contains insightResult nodes which don't seem to have an insights field. Why do we have this block?

insightResult contains a list of dicts with "insights" field which contains data

apologies if I'm being silly here but I don't see this in the insightResults documentation. Where is it defined?

sherifnada · 2021-08-26T06:02:34Z

airbyte-integrations/connectors/source-facebook-pages/source_facebook_pages/streams.py

+    def path(self, **kwargs) -> str:
+        return f'{self._page_id}/posts/?fields=insights.metric({",".join(POST_METRICS)})'
+
+    def request_params(


use DRY, this should be defined at the base class

sherifnada · 2021-08-26T06:04:30Z

airbyte-integrations/connectors/source-facebook-pages/source_facebook_pages/streams.py

+        return {"access_token": self._access_token}
+
+    def parse_response(self, response: requests.Response, **kwargs) -> Iterable[Mapping]:
+        data = response.json().get("insights")


the logic around data_field can be significantly simplified and DRY'd at the base class level. Data field should be the "path" in the record where the data field can be found. Then the top level parse_response should have a consistent impl that recursively goes into the record until it gets the desired field.

Updated.
PostInsigths class has a different logic for data_field, so for this class we have the parse_response method overridden

sherifnada · 2021-08-26T06:05:25Z

airbyte-integrations/connectors/source-facebook-pages/source_facebook_pages/streams.py

+    ):
+        super().__init__(**kwargs)
+        self._access_token = access_token
+        self._start_date = start_date


we are not using this parameter.

airbyte-integrations/connectors/source-facebook-pages/source_facebook_pages/streams.py

sherifnada · 2021-08-27T18:22:02Z

airbyte-integrations/connectors/source-facebook-pages/source_facebook_pages/streams.py

+    def parse_response(self, response: requests.Response, **kwargs) -> Iterable[Mapping]:
+        records = response.json().get(self.data_field) or []
+
+        for insights in records:


apologies if I'm being silly here but I don't see this in the insightResults documentation. Where is it defined?

airbyte-integrations/connectors/source-facebook-pages/unit_tests/unit_test.py

sherifnada · 2021-08-27T18:24:04Z

airbyte-integrations/connectors/source-facebook-pages/source_facebook_pages/streams.py

+            yield response.json()
+
+        else:
+            data_fields = self.data_field.split("_")


why don't we make data_field an array instead of doing string parsing? while reading the code it's not clear that data_field = "insights_data" is actually referring to a nested field. It's also problematic if the data field ever has an underscore in the name

i think this was not addressed

gaart · 2021-08-30T14:34:19Z

@sherifnada

apologies if I'm being silly here but I don't see this in the insightResults documentation. Where is it defined?

In insightResults, we can see the result for 1 metric. So, if we have 1 metric and 1 post, the result will be:

{
  "data": [
    {
      "insights":
        {
          "data": [insightResults]
          "paging": [],
          "id": "112704783733939_361771145493967"
    }

]

But in the request for Post Insights we have many posts and many metrics, so the INSIGHTS.DATA field contains a list of insightsResult objects for each metric, and the DATA field contains a list of all posts.

sherifnada · 2021-08-30T19:01:17Z

@gaart looks good, can you address the last comment about data field and run tests?

gaart · 2021-08-31T06:29:01Z

/test connector=connectors/source-facebook-pages

🕑 connectors/source-facebook-pages https://github.com/airbytehq/airbyte/actions/runs/1185032948
❌ connectors/source-facebook-pages https://github.com/airbytehq/airbyte/actions/runs/1185032948

gaart · 2021-08-31T10:12:27Z

/test connector=connectors/source-facebook-pages

🕑 connectors/source-facebook-pages https://github.com/airbytehq/airbyte/actions/runs/1185706769
✅ connectors/source-facebook-pages https://github.com/airbytehq/airbyte/actions/runs/1185706769

gaart · 2021-08-31T10:31:38Z

@sherifnada done

gaart · 2021-09-01T15:44:41Z

/test connector=connectors/source-facebook-pages

🕑 connectors/source-facebook-pages https://github.com/airbytehq/airbyte/actions/runs/1190847203
✅ connectors/source-facebook-pages https://github.com/airbytehq/airbyte/actions/runs/1190847203

gaart · 2021-09-01T15:56:32Z

/publish connector=connectors/source-facebook-pages

🕑 connectors/source-facebook-pages https://github.com/airbytehq/airbyte/actions/runs/1190885083
✅ connectors/source-facebook-pages https://github.com/airbytehq/airbyte/actions/runs/1190885083

tuliren · 2021-09-04T18:11:20Z

airbyte-integrations/connectors/source-facebook-pages/source_facebook_pages/spec.json

@@ -0,0 +1,21 @@
+{
+  "documentationUrl": "https://docs.airbyte.io/integrations/sources/facebook-pages",


By the way, this documentation is missing.

I added a placeholder doc in #5847. Please populate the content later on.

gaart added 2 commits July 30, 2021 00:11

[WIP] Add basic facebook pages support

6d399d5

Upd schema

a89f593

github-actions bot added the area/connectors Connector related issues label Aug 3, 2021

Hardcode metrics to reduce API call amount

389deac

gaart linked an issue Aug 5, 2021 that may be closed by this pull request

Facebook Pages #1247

Closed

gaart requested review from vovavovavovavova, vitaliizazmic and keu August 5, 2021 13:00

vovavovavovavova reviewed Aug 5, 2021

View reviewed changes

gaart added 2 commits August 9, 2021 08:31

Refactoring

fb58607

Upd test configs

e6673ae

sherifnada modified the milestone: Connectors August 20th Aug 11, 2021

gaart added 2 commits August 11, 2021 22:15

Add secrets to CI

9e8027b

Fix typo

f8bb038

Add a source definition

9172375

gaart requested review from Phlair and tuliren August 25, 2021 10:45

sherifnada suggested changes Aug 26, 2021

View reviewed changes

Rm start_date, add test

62eee6f

gaart requested a review from sherifnada August 27, 2021 05:46

VasylLazebnyk removed request for keu and vitaliizazmic August 27, 2021 08:58

sherifnada reviewed Aug 27, 2021

View reviewed changes

Upd schema

8a76880

gaart requested a review from sherifnada August 30, 2021 14:34

jrhizor temporarily deployed to more-secrets August 31, 2021 06:31 Inactive

Cleanup

c0e6f48

jrhizor temporarily deployed to more-secrets August 31, 2021 10:15 Inactive

sherifnada approved these changes Aug 31, 2021

View reviewed changes

jrhizor temporarily deployed to more-secrets September 1, 2021 15:47 Inactive

jrhizor temporarily deployed to more-secrets September 1, 2021 15:58 Inactive

gaart merged commit 7361263 into master Sep 1, 2021

gaart deleted the gaart/1247-facebook-pages branch September 1, 2021 16:11

jrhizor mentioned this pull request Sep 3, 2021

Bump Airbyte version from 0.29.13-alpha to 0.29.14-alpha #5831

Merged

tuliren reviewed Sep 4, 2021

View reviewed changes

igrankova added connectors/sources-api connectors/source/facebook-pages labels Jan 6, 2022

		from .metrics import PAGE_FIELDS, PAGE_METRICS, POST_FIELDS, POST_METRICS


		class FacebookStream(HttpStream, ABC):

		@@ -0,0 +1,21 @@
		{
		"documentationUrl": "https://docs.airbyte.io/integrations/sources/facebook-pages",

🎉 New Source: Facebook Pages #5158

🎉 New Source: Facebook Pages #5158

Conversation

gaart commented Aug 3, 2021 • edited Loading

What

Pre-merge Checklist

gaart commented Aug 3, 2021 • edited by github-actions bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vovavovavovavova Aug 5, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vovavovavovavova Aug 5, 2021 • edited Loading

Choose a reason for hiding this comment

vovavovavovavova commented Aug 5, 2021 • edited Loading

vovavovavovavova commented Aug 10, 2021 • edited by github-actions bot Loading

gaart commented Aug 11, 2021 • edited by github-actions bot Loading

yevhenii-ldv commented Aug 11, 2021 • edited by github-actions bot Loading

gaart commented Aug 12, 2021 • edited by github-actions bot Loading

yevhenii-ldv commented Aug 12, 2021 • edited by github-actions bot Loading

gaart commented Aug 13, 2021 • edited by github-actions bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gaart commented Aug 30, 2021

sherifnada commented Aug 30, 2021

gaart commented Aug 31, 2021 • edited by github-actions bot Loading

gaart commented Aug 31, 2021 • edited by github-actions bot Loading

gaart commented Aug 31, 2021

gaart commented Sep 1, 2021 • edited by github-actions bot Loading

gaart commented Sep 1, 2021 • edited by github-actions bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gaart commented Aug 3, 2021 •

edited

Loading

gaart commented Aug 3, 2021 •

edited by github-actions bot

Loading

vovavovavovavova Aug 5, 2021 •

edited

Loading

vovavovavovavova Aug 5, 2021 •

edited

Loading

vovavovavovavova commented Aug 5, 2021 •

edited

Loading

vovavovavovavova commented Aug 10, 2021 •

edited by github-actions bot

Loading

gaart commented Aug 11, 2021 •

edited by github-actions bot

Loading

yevhenii-ldv commented Aug 11, 2021 •

edited by github-actions bot

Loading

gaart commented Aug 12, 2021 •

edited by github-actions bot

Loading

yevhenii-ldv commented Aug 12, 2021 •

edited by github-actions bot

Loading

gaart commented Aug 13, 2021 •

edited by github-actions bot

Loading

gaart commented Aug 31, 2021 •

edited by github-actions bot

Loading

gaart commented Aug 31, 2021 •

edited by github-actions bot

Loading

gaart commented Sep 1, 2021 •

edited by github-actions bot

Loading

gaart commented Sep 1, 2021 •

edited by github-actions bot

Loading