Add the new Facebook API v10.0 fields to the ads_insights schema. #3646 #3693

keu · 2021-05-28T06:46:16Z

What

original PR #3646
closes #3645
all credits to @zestyping

How

Describe the solution

Pre-merge Checklist

Run integration tests
Publish Docker images

Recommended reading order

test.java
component.ts
the rest

…keys=True).

…sted.

…rsions" fields and add other available fields that have been tested and confirmed to work.

keu · 2021-05-28T11:17:38Z

/test connector=source-facebook-marketing

🕑 source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/885392474
❌ source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/885392474

keu · 2021-05-28T12:58:04Z

/publish connector=connectors/source-facebook-marketing

🕑 connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/885649230
❌ connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/885649230

keu · 2021-05-28T14:50:58Z

/publish connector=connectors/source-facebook-marketing

🕑 connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/885975898
❌ connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/885975898

keu · 2021-05-30T02:25:22Z

/publish connector=connectors/source-facebook-marketing

🕑 connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/889380623
❌ connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/889380623

keu · 2021-05-30T11:49:44Z

/publish connector=connectors/source-facebook-marketing

🕑 connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/890168025
❌ connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/890168025

keu · 2021-05-30T12:54:24Z

/publish connector=connectors/source-facebook-marketing

🕑 connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/890267885
❌ connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/890267885

keu · 2021-05-30T22:41:20Z

Unfortunately, after hours of debugging, I can only confirm that there is still a problem with Insight schemas, API simply returns an error (Job Failed) for all streams with breakdowns. This error has nothing to do with call rate handling. My other PR #3743 has fixes for call rate handling but it is blocked because of the above issues with this PR.

@zestyping I can see that there is quite a big difference in requested fields between your script and this PR.

sherifnada

@eugene-kulak are the issues appearing for all streams? insights? insight with breakdown?

sherifnada · 2021-06-01T22:34:45Z

airbyte-integrations/connectors/source-facebook-marketing/CHANGELOG.md

@@ -8,3 +8,6 @@ Allow configuring insights lookback window (#3396)

 ## 0.2.6
 Fix handling call rate limit (#3525) 
+
+## 0.2.7
+Add missing fields to AdInsights streams (#3693) 


wrong order in the changelog. should be reverse chrono

zestyping · 2021-06-01T23:54:01Z

Hey @keu, are we possibly looking at different files?

Here's the Python script that I tested, that works and downloads the data quickly (for context, see #3525):
ping-ad-insights.zip

Here's the JSON schema definition in this PR for the plain ads_insights stream, without segmentation:
https://github.com/airbytehq/airbyte/blob/7894984ef34eb5cc520bd8b72b67a8d14d318f6a/airbyte-integrations/connectors/source-facebook-marketing/source_facebook_marketing/schemas/ads_insights.json

And here's how I convinced myself that the field lists are identical.

Step 1. Get the list from the first file:

23:47:56 love[1933] unzip ping-ad-insights.zip
Archive:  ping-ad-insights.zip
  inflating: fb.py
  inflating: query_insights
  inflating: fetch_results
2s 23:47:58 love[1934] tail +31 query_insights | head -115
AD_FIELDS = [
    'account_currency',
    'account_id',
    'account_name',
    'action_values',
    'actions',
    'ad_click_actions',
    'ad_id',
    'ad_impression_actions',
    'ad_name',
    'adset_id',
    'adset_name',
    'age_targeting',
    'attribution_setting',
    'auction_bid',
    'auction_competitiveness',
    'auction_max_competitor_bid',
    'buying_type',
    'campaign_id',
    'campaign_name',
    'canvas_avg_view_percent',
    'canvas_avg_view_time',
    'catalog_segment_actions',
    'catalog_segment_value',
    'catalog_segment_value_mobile_purchase_roas',
    'catalog_segment_value_omni_purchase_roas',
    'catalog_segment_value_website_purchase_roas',
    'clicks',
    'conversion_rate_ranking',
    'conversion_values',
    'conversions',
    'converted_product_quantity',
    'converted_product_value',
    'cost_per_15_sec_video_view',
    'cost_per_2_sec_continuous_video_view',
    'cost_per_action_type',
    'cost_per_ad_click',
    'cost_per_conversion',
    'cost_per_estimated_ad_recallers',
    'cost_per_inline_link_click',
    'cost_per_inline_post_engagement',
    'cost_per_outbound_click',
    'cost_per_store_visit_action',
    'cost_per_thruplay',
    'cost_per_unique_action_type',
    'cost_per_unique_click',
    'cost_per_unique_inline_link_click',
    'cost_per_unique_outbound_click',
    'cpc',
    'cpm',
    'cpp',
    'created_time',
    'ctr',
    'date_start',
    'date_stop',
    'engagement_rate_ranking',
    'estimated_ad_recall_rate',
    'estimated_ad_recall_rate_lower_bound',
    'estimated_ad_recall_rate_upper_bound',
    'estimated_ad_recallers',
    'estimated_ad_recallers_lower_bound',
    'estimated_ad_recallers_upper_bound',
    'frequency',
    'full_view_impressions',
    'full_view_reach',
    'gender_targeting',
    'impressions',
    'inline_link_click_ctr',
    'inline_link_clicks',
    'inline_post_engagement',
    'instant_experience_clicks_to_open',
    'instant_experience_clicks_to_start',
    'instant_experience_outbound_clicks',
    'labels',
    'location',
    'mobile_app_purchase_roas',
    'objective',
    'optimization_goal',
    'outbound_clicks',
    'outbound_clicks_ctr',
    'purchase_roas',
    'qualifying_question_qualify_answer_rate',
    'quality_ranking',
    'reach',
    'social_spend',
    'spend',
    'store_visit_actions',
    'unique_actions',
    'unique_clicks',
    'unique_ctr',
    'unique_inline_link_click_ctr',
    'unique_inline_link_clicks',
    'unique_link_clicks_ctr',
    'unique_outbound_clicks',
    'unique_outbound_clicks_ctr',
    'updated_time',
    'video_15_sec_watched_actions',
    'video_30_sec_watched_actions',
    'video_avg_time_watched_actions',
    'video_continuous_2_sec_watched_actions',
    'video_p100_watched_actions',
    'video_p25_watched_actions',
    'video_p50_watched_actions',
    'video_p75_watched_actions',
    'video_p95_watched_actions',
    'video_play_actions',
    'video_play_curve_actions',
    'video_play_retention_0_to_15s_actions',
    'video_play_retention_20_to_60s_actions',
    'video_play_retention_graph_actions',
    'video_time_watched_actions',
    'website_ctr',
    'website_purchase_roas',
    'wish_bid',
]
5s 23:48:04 love[1935] !! > a.txt
tail +31 query_insights | head -115  > a.txt

Step 2. Get the list from the second file:

11s 23:48:14 love[1936] head pr-3693-ads_insights.json
{
  "properties": {
    "account_currency": {
      "type": ["null", "string"]
    },
    "account_id": {
      "type": ["null", "string"]
    },
    "account_name": {
      "type": ["null", "string"]
11s 23:48:25 love[1937] grep '^    "' pr-3693-ads_insights.json
    "account_currency": {
    "account_id": {
    "account_name": {
    "action_values": {
    "actions": {
    "ad_click_actions": {
    "ad_id": {
    "ad_impression_actions": {
    "ad_name": {
    "adset_id": {
    "adset_name": {
    "age_targeting": {
    "attribution_setting": {
    "auction_bid": {
    "auction_competitiveness": {
    "auction_max_competitor_bid": {
    "buying_type": {
    "campaign_id": {
    "campaign_name": {
    "canvas_avg_view_percent": {
    "canvas_avg_view_time": {
    "catalog_segment_actions": {
    "catalog_segment_value": {
    "catalog_segment_value_mobile_purchase_roas": {
    "catalog_segment_value_omni_purchase_roas": {
    "catalog_segment_value_website_purchase_roas": {
    "clicks": {
    "conversion_rate_ranking": {
    "conversion_values": {
    "conversions": {
    "converted_product_quantity": {
    "converted_product_value": {
    "cost_per_15_sec_video_view": {
    "cost_per_2_sec_continuous_video_view": {
    "cost_per_action_type": {
    "cost_per_ad_click": {
    "cost_per_conversion": {
    "cost_per_estimated_ad_recallers": {
    "cost_per_inline_link_click": {
    "cost_per_inline_post_engagement": {
    "cost_per_outbound_click": {
    "cost_per_store_visit_action": {
    "cost_per_thruplay": {
    "cost_per_unique_action_type": {
    "cost_per_unique_click": {
    "cost_per_unique_inline_link_click": {
    "cost_per_unique_outbound_click": {
    "cpc": {
    "cpm": {
    "cpp": {
    "created_time": {
    "ctr": {
    "date_start": {
    "date_stop": {
    "engagement_rate_ranking": {
    "estimated_ad_recall_rate": {
    "estimated_ad_recall_rate_lower_bound": {
    "estimated_ad_recall_rate_upper_bound": {
    "estimated_ad_recallers": {
    "estimated_ad_recallers_lower_bound": {
    "estimated_ad_recallers_upper_bound": {
    "frequency": {
    "full_view_impressions": {
    "full_view_reach": {
    "gender_targeting": {
    "impressions": {
    "inline_link_click_ctr": {
    "inline_link_clicks": {
    "inline_post_engagement": {
    "instant_experience_clicks_to_open": {
    "instant_experience_clicks_to_start": {
    "instant_experience_outbound_clicks": {
    "labels": {
    "location": {
    "mobile_app_purchase_roas": {
    "objective": {
    "optimization_goal": {
    "outbound_clicks": {
    "outbound_clicks_ctr": {
    "purchase_roas": {
    "qualifying_question_qualify_answer_rate": {
    "quality_ranking": {
    "reach": {
    "social_spend": {
    "spend": {
    "store_visit_actions": {
    "unique_actions": {
    "unique_clicks": {
    "unique_ctr": {
    "unique_inline_link_click_ctr": {
    "unique_inline_link_clicks": {
    "unique_link_clicks_ctr": {
    "unique_outbound_clicks": {
    "unique_outbound_clicks_ctr": {
    "updated_time": {
    "video_15_sec_watched_actions": {
    "video_30_sec_watched_actions": {
    "video_avg_time_watched_actions": {
    "video_continuous_2_sec_watched_actions": {
    "video_p100_watched_actions": {
    "video_p25_watched_actions": {
    "video_p50_watched_actions": {
    "video_p75_watched_actions": {
    "video_p95_watched_actions": {
    "video_play_actions": {
    "video_play_curve_actions": {
    "video_play_retention_0_to_15s_actions": {
    "video_play_retention_20_to_60s_actions": {
    "video_play_retention_graph_actions": {
    "video_time_watched_actions": {
    "website_ctr": {
    "website_purchase_roas": {
    "wish_bid": {
9s 23:48:34 love[1938] !! | sed -e s/:.*/,/ -e s/\"/\'/g > b.txt
grep '^    "' pr-3693-ads_insights.json | sed -e s/:.*/,/ -e s/\"/\'/g > b.txt

Step 3. Compare.

15s 23:48:49 love[1939] diff a.txt b.txt
1d0
< AD_FIELDS = [
115d113
< ]
5s (1) 23:48:54 love[1940]

Let me know if I've missed something or made an error, or if there is anything I can provide to help unblock you. Thanks!

keu · 2021-06-03T18:58:07Z

@zestyping you are absolutely right, I misinterpreted your scripts. I nailed the problem to the single stream - ads_insights_platform_and_device. It works for you but it seems that is not the case when we use action_breakdowns. The job just fails after ~ 60 sec of running. For now, I will merge this and publish it as 0.2.7 to unblock my other PR #3743 with fixes and improvements.

zestyping · 2021-06-03T19:02:29Z

Interesting! Maybe Facebook just doesn't like combining those breakdowns together. Thanks so much for carrying this forward!

keu · 2021-06-03T19:20:28Z

/publish connector=connectors/source-facebook-marketing

🕑 connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/903941045
✅ connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/903941045

sherifnada

brilliant -- thanks @eugene-kulak. Great work 👍🏼

zestyping added 5 commits May 27, 2021 00:20

Add all the missing fields except AdAsset and AdInsightsResult fields.

7ffa945

Sort and format ad_insights.json with json.dumps(..., indent=2, sort_…

c202e21

…keys=True).

Remove fields from ads_insights.json that have not been externally te…

5c61eaf

…sted.

Facebook source: Remove the offending "unique_conversions" and "conve…

69b1fb9

…rsions" fields and add other available fields that have been tested and confirmed to work.

Update all the other ads_insights schemas for the various breakdowns.

4d19cb2

auto-assign bot requested review from jrhizor and sherifnada May 28, 2021 06:46

eugene-kulak added 2 commits May 28, 2021 11:44

format

b9cea63

bump version, update changelog

590f52c

keu removed the request for review from jrhizor May 28, 2021 11:20

keu force-pushed the ping/facebook-ads-insights-v10-schema branch from 8d4712c to 590f52c Compare May 30, 2021 02:18

fix async job handling

337f504

fix async job handling, add missing dockerignore

7894984

sherifnada reviewed Jun 1, 2021

View reviewed changes

replace platform_and_device stream in integration tests because it fails

5c4ea1b

keu requested a review from sherifnada June 3, 2021 20:21

sherifnada approved these changes Jun 3, 2021

View reviewed changes

keu merged commit e602133 into master Jun 3, 2021

keu deleted the ping/facebook-ads-insights-v10-schema branch July 20, 2021 00:11

karinakuz added connectors/sources-api connectors/source/facebook-marketing labels Jan 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add the new Facebook API v10.0 fields to the ads_insights schema. #3646 #3693

Add the new Facebook API v10.0 fields to the ads_insights schema. #3646 #3693

keu commented May 28, 2021 •

edited

keu commented May 28, 2021 •

edited by github-actions bot

keu commented May 28, 2021 •

edited by github-actions bot

keu commented May 28, 2021 •

edited by github-actions bot

keu commented May 30, 2021 •

edited by github-actions bot

keu commented May 30, 2021 •

edited by github-actions bot

keu commented May 30, 2021 •

edited by github-actions bot

keu commented May 30, 2021 •

edited

sherifnada left a comment

sherifnada Jun 1, 2021

zestyping commented Jun 1, 2021

keu commented Jun 3, 2021

zestyping commented Jun 3, 2021

keu commented Jun 3, 2021 •

edited by github-actions bot

sherifnada left a comment

Add the new Facebook API v10.0 fields to the ads_insights schema. #3646 #3693

Add the new Facebook API v10.0 fields to the ads_insights schema. #3646 #3693

Conversation

keu commented May 28, 2021 • edited

What

How

Pre-merge Checklist

Recommended reading order

keu commented May 28, 2021 • edited by github-actions bot

keu commented May 28, 2021 • edited by github-actions bot

keu commented May 28, 2021 • edited by github-actions bot

keu commented May 30, 2021 • edited by github-actions bot

keu commented May 30, 2021 • edited by github-actions bot

keu commented May 30, 2021 • edited by github-actions bot

keu commented May 30, 2021 • edited

sherifnada left a comment

Choose a reason for hiding this comment

sherifnada Jun 1, 2021

Choose a reason for hiding this comment

zestyping commented Jun 1, 2021

keu commented Jun 3, 2021

zestyping commented Jun 3, 2021

keu commented Jun 3, 2021 • edited by github-actions bot

sherifnada left a comment

Choose a reason for hiding this comment

keu commented May 28, 2021 •

edited

keu commented May 28, 2021 •

edited by github-actions bot

keu commented May 28, 2021 •

edited by github-actions bot

keu commented May 28, 2021 •

edited by github-actions bot

keu commented May 30, 2021 •

edited by github-actions bot

keu commented May 30, 2021 •

edited by github-actions bot

keu commented May 30, 2021 •

edited by github-actions bot

keu commented May 30, 2021 •

edited

keu commented Jun 3, 2021 •

edited by github-actions bot