Merge pull request #90 from fivetran/release/v1.4.0

release/v1.4.0

CHANGELOG.md

@@ -1,9 +1,39 @@
-# dbt_ad_reporting v1.UPDATE.UPDATE
+# dbt_ad_reporting v1.4.0
 
- ## Under the Hood:
+## 🎉 Feature Enhancement 🎉
+- Added `ad_reporting__<report>_passthrough_metrics` variables to easily add common metrics across all platforms into the `ad_reporting` models! This allows metrics other than the standard `clicks`, `impressions`, and `cost` to be included in the final ad reporting models. See below for a full list of new variables and example metrics to passthrough. ([PR #85](https://github.com/fivetran/dbt_ad_reporting/pull/84))
+  - It is important to call out that this is only possible if the relevant upstream Ad platform variables have the same metric to be unioned in the roll up model. Please see the [README](https://github.com/fivetran/dbt_ad_reporting#optional-step-6-additional-configurations) section for details around how to configure the passthrough metrics.
+  - Please ensure you exercised due diligence when adding metrics to these models. The metrics added by default (`clicks`, `impressions`, and `cost`) have been vetted by the Fivetran team maintaining this package for accuracy. There are metrics included within the source reports, for example metric averages, which may be inaccurately represented at the grain for reports created in this package. You will want to ensure whichever metrics you pass through are indeed appropriate to aggregate at the respective reporting levels provided in this package.
+```yml
+vars:
+  ad_reporting__account_passthrough_metrics:
+    - name: conversions
+    - name: view_through_conversions
+  ad_reporting__campaign_passthrough_metrics: 
+    - name: total_shares
+    - name: conversions
+  ad_reporting__ad_group_passthrough_metrics:
+    - name: conversions
+    - name: interactions
+  ad_reporting__ad_passthrough_metrics: ## For both Ad and URL reports
+    - name: conversions
+    - name: video_views_captured
+  ad_reporting__keyword_passthrough_metrics:
+    - name: interactions
+  ad_reporting__search_passthrough_metrics:
+    - name: conversions
+    - name: local_spend_amount
+```
+- Addition of the `pinterest__using_keywords` (default=`true`) variable that allows users to disable the relevant keyword reports in the downstream Pinterest models if they are not used. ([PR #89](https://github.com/fivetran/dbt_ad_reporting/pull/89))
+
+## Under the Hood:
+
+- Incorporated the new `fivetran_utils.drop_schemas_automation` macro into the end of each Buildkite integration test job. ([PR #86](https://github.com/fivetran/dbt_ad_reporting/pull/86))
+- Updated the pull request [templates](/.github). ([PR #86](https://github.com/fivetran/dbt_ad_reporting/pull/86))
+
+## Contributors
+- [@aleix-cd](https://github.com/aleix-cd) ([PR #85](https://github.com/fivetran/dbt_ad_reporting/pull/84))
 
-- Incorporated the new `fivetran_utils.drop_schemas_automation` macro into the end of each Buildkite integration test job.
-- Updated the pull request [templates](/.github).
 # dbt_ad_reporting v1.3.1
 
 ## Updates

README.md

@@ -77,7 +77,7 @@ Include the following github package version in your `packages.yml`
 ```yaml
 packages:
   - package: fivetran/ad_reporting
-    version: [">=1.3.0", "<1.4.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=1.4.0", "<1.5.0"] # we recommend using ranges to capture non-breaking changes automatically
 ```
 
 Do NOT include the individual ad platform packages in this file. The ad reporting package itself has dependencies on these packages and will install them as well.
@@ -145,14 +145,17 @@ vars:
 ### Enable/Disable Specific Reports within Platforms
 For **Apple Search Ads**, if you are not utilizing the search functionality, you may choose to update the respective variable below.
 
-For **Twitter Ads**, if you are tracking keyword performance, you may choose to update the corresponding variable below.
+For **Twitter Ads**, if you are not tracking keyword performance, you may choose to update the corresponding variable below.
+
+For **Pinterest Ads**, if you are not tracking keyword performance, you may choose to update the corresponding variable below.
 
 Add the following variables to your dbt_project.yml file
 
 ```yml
 vars:
   apple_search_ads__using_search_terms: False # by default this is assumed to be True
   twitter_ads__using_keywords: False # by default this is assumed to be True
+  pinterest__using_keywords: False # by default this is assumed to be True
 ```
 
 ## (Recommended) Step 5: Change the Build Schema
@@ -225,6 +228,210 @@ models:
 <details><summary>Expand for details</summary>
 <br>
 
+## Adding custom metrics to final reports
+
+By default, this package selects `clicks`, `impressions`, and `cost` metrics from the upstream Ad platform reports. Additionally, each specific upstream Ad platform package allows for custom passthrough metrics to be included in the individual platform's final reports. You can find a complete list of available passthrough metric variables for each platform by referring to the relevant links below and inspecting the additional configurations for each platform:
+    - [Amazon Ads](https://github.com/fivetran/dbt_amazon_ads#optional-step-5-additional-configurations)
+    - [Apple Search Ads](https://github.com/fivetran/dbt_apple_search_ads#optional-step-4-additional-configurations)
+    - [Facebook Ads](https://github.com/fivetran/dbt_facebook_ads#optional-step-4-additional-configurations)
+    - [Google Ads](https://github.com/fivetran/dbt_google_ads#optional-step-4-additional-configurations)
+    - [LinkedIn Ad Analytics](https://github.com/fivetran/dbt_linkedin#optional-step-4-additional-configurations)
+    - [Microsoft Advertising](https://github.com/fivetran/dbt_microsoft_ads#optional-step-4-additional-configurations)
+    - [Pinterest Ads](https://github.com/fivetran/dbt_pinterest#optional-step-4-additional-configurations)
+    - [Snapchat Ads](https://github.com/fivetran/dbt_snapchat_ads#optional-step-4-additional-configurations)
+    - [TikTok Ads](https://github.com/fivetran/dbt_tiktok_ads#optional-step-4-additional-configurations)
+    - [Twitter Ads](https://github.com/fivetran/dbt_twitter#optional-step-5-additional-configurations)
+    - [Reddit Ads](https://github.com/fivetran/dbt_reddit_ads#optional-step-4-additional-configurations)
+
+Furthermore, this package allows you to include these configured upstream passthrough metrics in the final roll-up models of the combined Ad Reporting package. To include passthrough metrics in the respective final models, you need to define the following `ad_reporting__*` variables in your `dbt_project.yml` file:
+
+```yml
+vars:
+  ad_reporting__account_passthrough_metrics:
+    - name: conversions
+    - name: view_through_conversions
+  ad_reporting__campaign_passthrough_metrics: 
+    - name: total_shares
+    - name: conversions
+  ad_reporting__ad_group_passthrough_metrics:
+    - name: conversions
+    - name: interactions
+  ad_reporting__ad_passthrough_metrics: ## For both Ad and URL reports
+    - name: conversions
+    - name: video_views_captured
+  ad_reporting__keyword_passthrough_metrics:
+    - name: interactions
+  ad_reporting__search_passthrough_metrics:
+    - name: conversions
+    - name: local_spend_amount
+```
+It is important to ensure that if you want to configure a passthrough metric for an ad reporting end model, that metric **must** be included in all of your upstream variables. Additionally, the name of the metric **must** be consistent across platforms. If a certain upstream platform does not include the metric you **must** include a `transform_sql` argument to pass a null value through (see below for examples). The following configuration is an example when using the Microsoft Ads, Apple Search Ads, Google Ads, Snapchat Ads, TikTok Ads, and Reddit Ads platforms within a `dbt_project.yml` file:
+
+>**Note**: Please ensure you exercised due diligence when adding metrics to these models. The metrics added by default (`clicks`, `impressions`, and `cost`) have been vetted by the Fivetran team maintaining this package for accuracy. There are metrics included within the source reports, for example metric averages, which may be inaccurately represented at the grain for reports created in this package. You will want to ensure whichever metrics you pass through are indeed appropriate to aggregate at the respective reporting levels provided in this package.
+
+>**Note**: While the below configuration is only for a subset of Ad platforms, the same strategy will be used for all other possible combinations of upstream Ad platform dependencies.
+
+```yml
+vars:
+  ## Account Report Passthrough Metrics
+  microsoft_ads__account_passthrough_metrics:
+    - name: conversions
+    - name: view_through_conversions
+      transform_sql: "null"
+  apple_search_ads__campaign_passthrough_metrics:
+    - name: conversions
+    - name: view_through_conversions
+      transform_sql: "null"
+    - name: total_shares
+      transform_sql: "null"
+  google_ads__account_stats_passthrough_metrics:
+    - name: conversions
+    - name: view_through_conversions
+  # snapchat_ads__ad_hourly_passthrough_metrics: # Defined below in the ad/url metrics therefore, not needed here but kept for documentation.
+  #   - name: conversion_view_content
+  #     alias: view_through_conversions
+  #   - name: conversion_sign_ups
+  #     alias: conversions
+  tiktok_ads__ad_hourly_passthrough_metrics:
+    - name: conversion
+      alias: conversions
+    - name: view_through_conversions
+      transform_sql: "null"
+  reddit_ads__account_passthrough_metrics:
+    - name: conversion_roas
+      alias: conversions
+    - name: legacy_view_conversions_attribution_window_day
+      alias: view_through_conversions
+  ad_reporting__account_passthrough_metrics:
+    - name: conversions
+    - name: view_through_conversions
+
+  ## Campaign Report Passthrough Metrics
+  microsoft_ads__campaign_passthrough_metrics:
+    - name: conversions
+    - name: total_shares
+      transform_sql: "null"
+  google_ads__campaign_stats_passthrough_metrics:
+    - name: conversions
+    - name: total_shares
+      transform_sql: cast(total_shares as int)
+  snapchat_ads__campaign_hourly_report_passthrough_metrics:
+    - name: conversion_sign_ups
+      alias: conversions
+    - name: shares
+      alias: total_shares
+  tiktok_ads__campaign_hourly_passthrough_metrics:
+    - name: conversion
+      alias: conversions
+    - name: shares
+      alias: total_shares
+  reddit_ads__campaign_passthrough_metrics:
+    - name: conversions
+      transform_sql: "null"
+    - name: total_shares
+      transform_sql: "null"
+  ad_reporting__campaign_passthrough_metrics: 
+    - name: total_shares
+    - name: conversions
+
+  ## Ad Group Report Passthrough Metrics
+  microsoft_ads__ad_group_passthrough_metrics:
+    - name: conversions
+    - name: phone_calls
+      alias: interactions
+  apple_search_ads__ad_group_passthrough_metrics:
+    - name: conversions
+    - name: new_downloads
+      alias: interactions
+  google_ads__ad_group_stats_passthrough_metrics:
+    - name: conversions
+    - name: interactions
+  snapchat_ads__ad_squad_hourly_passthrough_metrics:
+    - name: conversion_add_cart
+      alias: conversions
+    - name: saves
+      alias: interactions
+  tiktok_ads__ad_group_hourly_passthrough_metrics:
+    - name: conversion
+      alias: conversions
+    - name: likes
+      alias: interactions
+  reddit_ads__ad_group_passthrough_metrics:
+    - name: conversion_roas
+      alias: conversions
+    - name: video_started
+      alias: interactions
+  ad_reporting__ad_group_passthrough_metrics:
+    - name: conversions
+    - name: interactions
+
+## Ad and URL Report Passthrough Metrics
+  microsoft_ads__ad_passthrough_metrics:
+    - name: conversions
+    - name: video_views_captured
+      transform_sql: "null"
+  apple_search_ads__ad_passthrough_metrics:
+    - name: conversions
+    - name: video_views_captured
+      transform_sql: "null"
+  google_ads__ad_stats_passthrough_metrics:
+    - name: video_views
+      alias: video_views_captured
+      transform_sql: cast(video_views_captured as int64)
+    - name: conversions
+  snapchat_ads__ad_hourly_passthrough_metrics:
+    - name: conversion_view_content
+      alias: view_through_conversions
+    - name: conversion_sign_ups
+      alias: conversions
+    - name: video_views
+      alias: video_views_captured
+      transform_sql: cast(video_views_captured as int64)
+  tiktok_ads__ad_hourly_passthrough_metrics:
+    - name: conversion
+      alias: conversions
+    - name: view_through_conversions
+      transform_sql: "null"
+    - name: video_watched_2_s
+      alias: video_views_captured
+      transform_sql: cast(video_views_captured as int64)
+  reddit_ads__ad_passthrough_metrics:
+    - name: conversion_roas
+      alias: conversions
+    - name: video_watched_3_seconds
+      alias: video_views_captured
+      transform_sql: cast(video_views_captured as int64)
+  ad_reporting__ad_passthrough_metrics:
+    - name: conversions
+    - name: video_views_captured
+
+  # Keyword Report Passthrough Metrics
+  microsoft_ads__keyword_passthrough_metrics:
+    - name: interactions
+      transform_sql: "null"
+  apple_search_ads__keyword_passthrough_metrics:
+    - name: new_downloads
+      alias: interactions
+  google_ads__keyword_stats_passthrough_metrics:
+    - name: interactions
+  ad_reporting__keyword_passthrough_metrics:
+    - name: interactions
+
+  # Search Report Passthrough Metrics
+  microsoft_ads__search_passthrough_metrics:
+    - name: conversions
+    - name: local_spend_amount
+      transform_sql: "null"
+  apple_search_ads__search_term_passthrough_metrics:
+    - name: local_spend_amount
+      transform_sql: "cast(local_spend_amount as int64)"
+    - name: conversions
+      transform_sql: "null"
+  ad_reporting__search_passthrough_metrics:
+    - name: conversions
+    - name: local_spend_amount
+```
+
 ## Disabling null URL filtering from URL reports
 The default behavior for the `ad_reporting__url_report` end model is to filter out records having null URL fields, however, you are able to turn off this filter if needed. To turn off the filter, include the below in your `dbt_project.yml` file. This variable will affect ALL Fivetran platform packages enabled in Ad Reporting, therefore either all URL reports will have null URLs filtered, or all URL reports will have null URLs included.
 

dbt_project.yml

@@ -1,14 +1,23 @@
 name: 'ad_reporting'
-version: '1.3.1'
+version: '1.4.0'
 
 config-version: 2
 
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 
+vars:
+  ad_reporting:
+    ad_reporting__account_passthrough_metrics: []
+    ad_reporting__campaign_passthrough_metrics: []
+    ad_reporting__ad_group_passthrough_metrics: []
+    ad_reporting__ad_passthrough_metrics: []
+    ad_reporting__keyword_passthrough_metrics: []
+    ad_reporting__search_passthrough_metrics: []
+
 models:
   ad_reporting:
     +materialized: table 
     +schema: ad_reporting 
     intermediate:
       +materialized: ephemeral
-      +schema: int_ad_reporting 
+      +schema: int_ad_reporting

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'ad_reporting_integration_tests'
-version: '1.3.1'
+version: '1.4.0'
 profile: 'integration_tests'
 config-version: 2