Source Facebook Marketing: Add Ad Insights streams #26720

artem1205 · 2023-05-29T16:08:14Z

What

Resolve #17238
Add Custom Ad Insights streams

How

Add new Reports using custom Ad Insights breakdowns and action_breakdowns combinations.
See Docs update for more info.

🚨 User Impact 🚨

No Breaking Changes

Pre-merge Actions

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Unit & integration tests added

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

github-actions · 2023-05-29T16:08:41Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan and you've followed all steps in the Breaking Changes Checklist
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
You, or an Airbyter, have run /test successfully on this PR - or on a non-forked branch
You've updated the connector's metadata.yaml file (new!)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

artem1205 · 2023-05-29T16:37:07Z

/test connector=connectors/source-facebook-marketing

🕑 connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/5113769641
❌ connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/5113769641
🐛 https://gradle.com/s/chrdvbrer63dc

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 FAILED unit_tests/test_source.py::TestSourceFacebookMarketing::test_streams
	 �[31m=================== �[31m�[1m1 failed�[0m, �[32m117 passed�[0m, �[33m1 warning�[0m�[31m in 1.70s�[0m�[31m ===================�[0m

artem1205 · 2023-05-29T19:31:28Z

/test connector=connectors/source-facebook-marketing

🕑 connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/5114805878
❌ connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/5114805878
🐛 https://gradle.com/s/ncpbmk73vtqyc

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Stream ads_i...
ERROR test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
======== 1 failed, 37 passed, 1 skipped, 1 error in 4539.50s (1:15:39) =========

artem1205 · 2023-05-30T13:54:36Z

/test connector=connectors/source-facebook-marketing

🕑 connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/5122373089

…ams-17238

…ights-streams-17238' into artem1205/source-facebook-ad-insights-streams-17238

artem1205 · 2023-05-30T16:14:14Z

/test connector=connectors/source-facebook-marketing

🕑 connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/5123836646
✅ connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/5123836646
Python tests coverage:

Name                                                        Stmts   Miss  Cover
-------------------------------------------------------------------------------
source_facebook_marketing/utils.py                             26      0   100%
source_facebook_marketing/streams/common.py                    44      0   100%
source_facebook_marketing/streams/async_job.py                217      0   100%
source_facebook_marketing/streams/__init__.py                   2      0   100%
source_facebook_marketing/spec.py                              39      0   100%
source_facebook_marketing/__init__.py                           5      0   100%
source_facebook_marketing/streams/async_job_manager.py         78      3    96%
source_facebook_marketing/streams/patches.py                   22      2    91%
source_facebook_marketing/source.py                            62      6    90%
source_facebook_marketing/api.py                               99     10    90%
source_facebook_marketing/streams/base_insight_streams.py     145     16    89%
source_facebook_marketing/streams/base_streams.py             143     22    85%
source_facebook_marketing/streams/streams.py                  160     29    82%
-------------------------------------------------------------------------------
TOTAL                                                        1042     88    92%
Name                                                        Stmts   Miss  Cover
-------------------------------------------------------------------------------
source_facebook_marketing/streams/__init__.py                   2      0   100%
source_facebook_marketing/spec.py                              39      0   100%
source_facebook_marketing/__init__.py                           5      0   100%
source_facebook_marketing/api.py                               99     13    87%
source_facebook_marketing/streams/patches.py                   22      3    86%
source_facebook_marketing/streams/base_streams.py             143     28    80%
source_facebook_marketing/streams/streams.py                  160     50    69%
source_facebook_marketing/streams/common.py                    44     16    64%
source_facebook_marketing/utils.py                             26     10    62%
source_facebook_marketing/source.py                            62     24    61%
source_facebook_marketing/streams/base_insight_streams.py     145     64    56%
source_facebook_marketing/streams/async_job.py                217    136    37%
source_facebook_marketing/streams/async_job_manager.py         78     60    23%
-------------------------------------------------------------------------------
TOTAL                                                        1042    404    61%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
================== 39 passed, 1 skipped in 4554.02s (1:15:54) ==================

lazebnyi

Nice!

girarda · 2023-05-30T19:17:40Z

airbyte-integrations/connectors/source-facebook-marketing/acceptance-test-config.yml

@@ -27,14 +27,44 @@ acceptance_tests:
            bypass_reason: "Thumbnail urls changes permanently"
          - name: "ad_creatives"
            bypass_reason: "Thumbnail urls changes permanently"
+          - name: "ads_insights_action_carousel_card"


how are we testing the new streams if there's no data in our test account?

We do have some data for all these streams. Or, more precisely, all ad Insight streams make requests to the same data, but using different "slicing" columns (like in pivot tables).
I've disable them for basic_read test, because we use our "live" account for testing.

We still test all these streams in incremental and full_refresh tests

* Source Facebook Marketing: Add Ad Insights streams * Source Facebook Marketing: update docs; bump version * Source Facebook Marketing: update tests config * Source Facebook Marketing: fix unit test * Source Facebook Marketing: typo fix * Source Facebook Marketing: update future_state * Source Facebook Marketing: fix expected records

Source Facebook Marketing: Add Ad Insights streams

e67c31e

octavia-squidington-iii added area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/facebook-marketing labels May 29, 2023

artem1205 added 2 commits May 29, 2023 18:09

Source Facebook Marketing: update docs; bump version

4f58417

Source Facebook Marketing: update tests config

572d1c5

Source Facebook Marketing: fix unit test

284b9b7

artem1205 requested review from bazarnov and lazebnyi May 29, 2023 18:26

Source Facebook Marketing: typo fix

5fcd286

artem1205 added 2 commits May 30, 2023 14:21

Source Facebook Marketing: update future_state

a181845

Source Facebook Marketing: fix expected records

37a8bed

artem1205 and others added 3 commits May 30, 2023 17:39

Merge branch 'master' into artem1205/source-facebook-ad-insights-stre…

cd05c41

…ams-17238

Merge branch 'master' into artem1205/source-facebook-ad-insights-stre…

8b59f36

…ams-17238

Merge remote-tracking branch 'origin/artem1205/source-facebook-ad-ins…

8540661

…ights-streams-17238' into artem1205/source-facebook-ad-insights-streams-17238

artem1205 requested a review from a team May 30, 2023 18:54

bazarnov approved these changes May 30, 2023

View reviewed changes

lazebnyi approved these changes May 30, 2023

View reviewed changes

lazebnyi requested a review from girarda May 30, 2023 19:03

girarda reviewed May 30, 2023

View reviewed changes

artem1205 requested a review from girarda May 30, 2023 19:59

girarda approved these changes May 30, 2023

View reviewed changes

artem1205 merged commit bf8bb6a into master May 30, 2023
30 of 33 checks passed

artem1205 deleted the artem1205/source-facebook-ad-insights-streams-17238 branch May 30, 2023 21:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Facebook Marketing: Add Ad Insights streams #26720

Source Facebook Marketing: Add Ad Insights streams #26720

artem1205 commented May 29, 2023 •

edited

github-actions bot commented May 29, 2023 •

edited by artem1205

artem1205 commented May 29, 2023 •

edited by github-actions bot

artem1205 commented May 29, 2023 •

edited by github-actions bot

artem1205 commented May 30, 2023 •

edited by github-actions bot

artem1205 commented May 30, 2023 •

edited by github-actions bot

lazebnyi left a comment

girarda May 30, 2023

artem1205 May 30, 2023 •

edited

artem1205 May 30, 2023

Source Facebook Marketing: Add Ad Insights streams #26720

Source Facebook Marketing: Add Ad Insights streams #26720

Conversation

artem1205 commented May 29, 2023 • edited

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Actions

Community member or Airbyter

Airbyter

github-actions bot commented May 29, 2023 • edited by artem1205

Before Merging a Connector Pull Request

artem1205 commented May 29, 2023 • edited by github-actions bot

Build Failed

artem1205 commented May 29, 2023 • edited by github-actions bot

Build Failed

artem1205 commented May 30, 2023 • edited by github-actions bot

artem1205 commented May 30, 2023 • edited by github-actions bot

Build Passed

lazebnyi left a comment

Choose a reason for hiding this comment

girarda May 30, 2023

Choose a reason for hiding this comment

artem1205 May 30, 2023 • edited

Choose a reason for hiding this comment

artem1205 May 30, 2023

Choose a reason for hiding this comment

artem1205 commented May 29, 2023 •

edited

github-actions bot commented May 29, 2023 •

edited by artem1205

artem1205 commented May 29, 2023 •

edited by github-actions bot

artem1205 commented May 29, 2023 •

edited by github-actions bot

artem1205 commented May 30, 2023 •

edited by github-actions bot

artem1205 commented May 30, 2023 •

edited by github-actions bot

artem1205 May 30, 2023 •

edited