-
Notifications
You must be signed in to change notification settings - Fork 4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🎉 Source Amazon Ads : Add attribution reports #16342
Changes from 3 commits
8ac8cee
e8998ed
4d86c53
57c3808
d12a9bd
e213fc4
3634a83
08dfd90
4a6f9e5
1b4b833
91d0682
8764d9f
108b81c
02c5b61
1b817b8
7744ca4
26e1694
d65e7dc
e96b497
75baea2
7d7cd06
2abdeab
f0a78f4
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,40 @@ | ||
{ | ||
"streams": [ | ||
{ | ||
"stream": { | ||
"name": "attribution_report_products", | ||
"json_schema": {}, | ||
"supported_sync_modes": ["full_refresh"] | ||
}, | ||
"sync_mode": "full_refresh", | ||
"destination_sync_mode": "overwrite" | ||
}, | ||
{ | ||
"stream": { | ||
"name": "attribution_report_performance_adgroup", | ||
"json_schema": {}, | ||
"supported_sync_modes": ["full_refresh"] | ||
}, | ||
"sync_mode": "full_refresh", | ||
"destination_sync_mode": "overwrite" | ||
}, | ||
{ | ||
"stream": { | ||
"name": "attribution_report_performance_campaign", | ||
"json_schema": {}, | ||
"supported_sync_modes": ["full_refresh"] | ||
}, | ||
"sync_mode": "full_refresh", | ||
"destination_sync_mode": "overwrite" | ||
}, | ||
{ | ||
"stream": { | ||
"name": "attribution_report_performance_creative", | ||
"json_schema": {}, | ||
"supported_sync_modes": ["full_refresh"] | ||
}, | ||
"sync_mode": "full_refresh", | ||
"destination_sync_mode": "overwrite" | ||
} | ||
] | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
# | ||
# Copyright (c) 2022 Airbyte, Inc., all rights reserved. | ||
# | ||
|
||
from typing import List | ||
|
||
from .common import CatalogModel | ||
|
||
|
||
class Report(CatalogModel): | ||
date: str | ||
brandName: str | ||
marketplace: str | ||
campaignId: str | ||
productAsin: str | ||
productConversionType: str | ||
advertiserName: str | ||
adGroupId: str | ||
creativeId: str | ||
productName: str | ||
productCategory: str | ||
productSubcategory: str | ||
productGroup: str | ||
publisher: str | ||
|
||
|
||
class AttributionReportModel(CatalogModel): | ||
reports: List[Report] | ||
size: int | ||
cursorId: str |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,172 @@ | ||
# | ||
# Copyright (c) 2022 Airbyte, Inc., all rights reserved. | ||
# | ||
|
||
from typing import Any, Iterable, Mapping, MutableMapping, Optional | ||
|
||
import pendulum | ||
import requests | ||
from source_amazon_ads.schemas import AttributionReportModel, Profile | ||
from source_amazon_ads.streams.common import AmazonAdsStream | ||
|
||
BRAND_REFERRAL_BONUS = "brb_bonus_amount" | ||
|
||
METRICS_MAP = { | ||
"PERFORMANCE": [ | ||
"Click-throughs", | ||
"attributedDetailPageViewsClicks14d", | ||
"attributedAddToCartClicks14d", | ||
"attributedPurchases14d", | ||
"unitsSold14d", | ||
"attributedSales14d", | ||
"attributedTotalDetailPageViewsClicks14d", | ||
"attributedTotalAddToCartClicks14d", | ||
"attributedTotalPurchases14d", | ||
"totalUnitsSold14d", | ||
"totalAttributedSales14d", | ||
], | ||
"PRODUCTS": [ | ||
"attributedDetailPageViewsClicks14d", | ||
"attributedAddToCartClicks14d", | ||
"attributedPurchases14d", | ||
"unitsSold14d", | ||
"attributedSales14d", | ||
"brandHaloDetailPageViewsClicks14d", | ||
"brandHaloAttributedAddToCartClicks14d", | ||
"brandHaloAttributedPurchases14d", | ||
"brandHaloUnitsSold14d", | ||
"brandHaloAttributedSales14d", | ||
"attributedNewToBrandPurchases14d", | ||
"attributedNewToBrandUnitsSold14d", | ||
"attributedNewToBrandSales14d", | ||
"brandHaloNewToBrandPurchases14d", | ||
"brandHaloNewToBrandUnitsSold14d", | ||
"brandHaloNewToBrandSales14d", | ||
], | ||
} | ||
|
||
|
||
class AttributionReport(AmazonAdsStream): | ||
""" | ||
This stream corresponds to Amazon Advertising API - Attribution Reports | ||
https://advertising.amazon.com/API/docs/en-us/amazon-attribution-prod-3p/#/ | ||
""" | ||
|
||
model = AttributionReportModel | ||
primary_key = None | ||
data_field = "reports" | ||
page_size = 300 | ||
|
||
report_type = "" | ||
metrics = "" | ||
group_by = "" | ||
|
||
_next_page_token_field = "cursorId" | ||
_current_profile_id = "" | ||
|
||
REPORT_DATE_FORMAT = "YYYYMMDD" | ||
CONFIG_DATE_FORMAT = "YYYY-MM-DD" | ||
REPORTING_PERIOD = 90 | ||
|
||
def __init__(self, config: Mapping[str, Any], *args, **kwargs): | ||
self._start_date = config.get("start_date") | ||
self._req_start_date = "" | ||
self._req_end_date = "" | ||
|
||
super().__init__(config, *args, **kwargs) | ||
|
||
def _set_dates(self, profile: Profile): | ||
new_start_date = pendulum.now(tz=profile.timezone).subtract(days=1).date() | ||
new_end_date = pendulum.now(tz=profile.timezone).date() | ||
|
||
if self._start_date: | ||
new_start_date = max(self._start_date, new_end_date.subtract(days=self.REPORTING_PERIOD)) | ||
|
||
self._req_start_date = new_start_date.format(self.REPORT_DATE_FORMAT) | ||
self._req_end_date = new_end_date.format(self.REPORT_DATE_FORMAT) | ||
|
||
@property | ||
def http_method(self) -> str: | ||
return "POST" | ||
|
||
def read_records(self, *args, **kvargs) -> Iterable[Mapping[str, Any]]: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I finally solved my local issue by adding a few profile Ids, though our dev account doesn't have any attribute reports. So, earlier, under the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. current implementation
if There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. can you do me a favor? Can you unset the
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. No error when When there are no user provided profiles, API call is made to fetch all profiles. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. thanks for the feedback, that's really helpful. The acceptance test is not passed. And the reason is that not all profiles are authorized to fetch reports, and the read command doesn't fail gracefully, which blocks the read on other streams. I read the codes again, looks like a profile is not independent of the others. In our dev test account, there are about 10 profiles fetched from the API, and only 4 of them are authorized to read the attribute reports. If I pass just these 4 profiles, the tests are good. But if I include another, the read command fails on that profile and stops fetching reports on other profiles. And under the Similarly, if I set a collection of profiles to Under a You may consider to use a stream slice to store profiles, and fetch reports for each profile in that slice. The GitHub connector provides a good example in fetching users in an org. Let me know if you have a question. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can you help me in providing a way to reproduce the error? I tried running this command
It's returning 0 records for all the streams except
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sure, be happy to! firstly, check
looks like you have 3 profiles. I guess all 3 profiles are authorized to read attribute_reports_*. Can you create another profile that is not authorized and add it to the array of profiles in There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Any ideas on why I am not getting any error with fake profile IDs in
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I am not sure, maybe on the api side, the validation handles it before the authorization module is run? |
||
""" | ||
Iterate through self._profiles list and send read all records for each profile. | ||
""" | ||
for profile in self._profiles: | ||
self._set_dates(profile) | ||
self._current_profile_id = profile.profileId | ||
yield from super().read_records(*args, **kvargs) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. try:
self._set_dates(profile)
self._current_profile_id = profile.profileId
yield from super().read_records(*args, **kvargs)
except Exception:
# This profile doesn't have any records/access
self.logger.info(
"This profile doesn't have any records/access") There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Have implemented above changes. Updated the unit tests also. PTAL There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks a lot. |
||
|
||
def request_headers(self, *args, **kvargs) -> MutableMapping[str, Any]: | ||
headers = super().request_headers(*args, **kvargs) | ||
headers["Amazon-Advertising-API-Scope"] = str(self._current_profile_id) | ||
return headers | ||
|
||
def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]: | ||
stream_data = response.json() | ||
next_page_token = stream_data.get(self._next_page_token_field) | ||
if next_page_token: | ||
return {self._next_page_token_field: next_page_token} | ||
|
||
def path(self, **kvargs) -> str: | ||
return "/attribution/report" | ||
|
||
def request_body_json( | ||
self, | ||
stream_state: Mapping[str, Any], | ||
stream_slice: Mapping[str, Any] = None, | ||
next_page_token: Mapping[str, Any] = None, | ||
) -> Optional[Mapping]: | ||
body = { | ||
"reportType": self.report_type, | ||
"count": self.page_size, | ||
"metrics": self.metrics, | ||
"startDate": self._req_start_date, | ||
"endDate": self._req_end_date, | ||
self._next_page_token_field: "", | ||
} | ||
|
||
if self.group_by: | ||
body["groupBy"] = self.group_by | ||
|
||
if next_page_token: | ||
body[self._next_page_token_field] = next_page_token[self._next_page_token_field] | ||
|
||
return body | ||
|
||
|
||
class AttributionReportProducts(AttributionReport): | ||
report_type = "PRODUCTS" | ||
|
||
metrics = ",".join(METRICS_MAP[report_type]) | ||
|
||
group_by = "" | ||
|
||
|
||
class AttributionReportPerformanceCreative(AttributionReport): | ||
report_type = "PERFORMANCE" | ||
|
||
metrics = ",".join(METRICS_MAP[report_type]) | ||
|
||
group_by = "CREATIVE" | ||
|
||
|
||
class AttributionReportPerformanceAdgroup(AttributionReport): | ||
report_type = "PERFORMANCE" | ||
|
||
metrics_list = METRICS_MAP[report_type] | ||
metrics_list.append(BRAND_REFERRAL_BONUS) | ||
metrics = ",".join(metrics_list) | ||
|
||
group_by = "ADGROUP" | ||
|
||
|
||
class AttributionReportPerformanceCampaign(AttributionReport): | ||
report_type = "PERFORMANCE" | ||
|
||
metrics_list = METRICS_MAP[report_type] | ||
metrics_list.append(BRAND_REFERRAL_BONUS) | ||
metrics = ",".join(metrics_list) | ||
|
||
group_by = "CAMPAIGN" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so,
secrets/config_report.json
looks like a secret for report, a little different than the one forintegration_tests/configured_catalog.json
, can you merge the configured catalog to eitherconfigured_catalog.json
orconfigured_catalog_report.json
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you please differentiate between contents of files
secrets/config_report.json
andsecrets/config.json
.Merging them with existing catalogue would still use the same secret file.
I would like to keep it segregated for attribution reports unless they are causing any issues with integration test suite.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
configured_catalog.json
usessecrets/config.json
, andconfigured_catalog_report.json
usessecrets/config_report.json
, ifconfigured_catalog_attribution_report.json
also usessecrets/config.json
, can you merge the json tointegration_tests/configured_catalog.json
?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
have merged it, but if you can share the difference between contents of files
secrets/config_report.json
andsecrets/config.json
, it will be helpful to understand. ThanksThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The CI may use multiple secret sets, but I only got one set of secrets. I will ask IT to share it with me. In my local environment, I use the same content for the both.
You mentioned that you were stuck in 70%, I am able to reach 73% after fixing the
test_backward_compatibility
, here's my change: YiyangLi@9341fc6 since we are using different accounts, the fix doesn't help you too much. Can you try again by removingexpect_records
?I am currently stuck at 73%,
TestBasicRead.test_read
, even if I removebasic_read
section, I will be stuck atincremental read
. I wonder if that's different on your side, and I may fix the dev account on our side.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After pulling the latest from master it's going till 72%, before getting stuck at
TestBasicRead.test_read[inputs0]
I tried with removing
expect_records
section, but same results.When I removed
basic_read
section, success % reached till 82% before getting stuck atincremental read
.