🎉 Source Amazon Ads : Add attribution reports #16342
Conversation
github-actions
bot
added
area/connectors
|
Need help with acceptance tests. With my version of |
sajarin
added
bounty-M
| @@ -38,3 +38,5 @@ tests: | |||
| ignored_fields: | |||
| "sponsored_products_report_stream": ["updatedAt"] | |||
| timeout_seconds: 3600 | |||
| - config_path: "secrets/config.json" | |||
| configured_catalog_path: "integration_tests/configured_catalog_attribution_report.json" | |||
There was a problem hiding this comment.
so, secrets/config_report.json looks like a secret for report, a little different than the one for integration_tests/configured_catalog.json, can you merge the configured catalog to either configured_catalog.json or configured_catalog_report.json
There was a problem hiding this comment.
Can you please differentiate between contents of files secrets/config_report.json and secrets/config.json.
Merging them with existing catalogue would still use the same secret file.
I would like to keep it segregated for attribution reports unless they are causing any issues with integration test suite.
There was a problem hiding this comment.
configured_catalog.json uses secrets/config.json, and configured_catalog_report.json uses secrets/config_report.json, if configured_catalog_attribution_report.json also uses secrets/config.json, can you merge the json to integration_tests/configured_catalog.json?
There was a problem hiding this comment.
have merged it, but if you can share the difference between contents of files secrets/config_report.json and secrets/config.json, it will be helpful to understand. Thanks
There was a problem hiding this comment.
The CI may use multiple secret sets, but I only got one set of secrets. I will ask IT to share it with me. In my local environment, I use the same content for the both.
You mentioned that you were stuck in 70%, I am able to reach 73% after fixing the test_backward_compatibility, here's my change: YiyangLi@9341fc6 since we are using different accounts, the fix doesn't help you too much. Can you try again by removing expect_records?
I am currently stuck at 73%, TestBasicRead.test_read, even if I remove basic_read section, I will be stuck at incremental read. I wonder if that's different on your side, and I may fix the dev account on our side.
There was a problem hiding this comment.
After pulling the latest from master it's going till 72%, before getting stuck at TestBasicRead.test_read[inputs0]
I tried with removing expect_records section, but same results.
When I removed basic_read section, success % reached till 82% before getting stuck at incremental read.
…azon-ads-attribution-reports
sajarin
added
bounty-XL
|
regarding the test being stuck at 70%, can you remove the |
|
@sajarin can you help cherry-pick this commit to the topic branch? YiyangLi@9341fc6 It's irrelevant to this topic branch. The acceptance test fails at I don't know why, in my local test, the test is stuck at To reproduce the issue, you can use the credential, but change the region from |
…azon-ads-attribution-reports
|
/test connector=connectors/source-amazon-ads |
|
/test connector=connectors/source-amazon-ads
Build FailedTest summary info: |
|
@ganpatagarwal The CI throws an error of Thanks. |
…azon-ads-attribution-reports
|
The amazon ads account I am using for testing is not given me any issues to fetch attribution report. We had been using the attribution feature on that account and used the same to fetch the reports via API. If you are authorizing your developer application with My guess would be limitations with the amazon ads account your are using with CI I have already added the link for amazon attribution reports in |
…azon-ads-attribution-reports
Thank you, I will get the dev account and try to get a new token. Overall, the goal is to get acceptance tests passed at 100%, then we are good to merge. So, are you able to complete the acceptance test? If not, you could bypass the expect_records by removing the section locally? It would be great if you could share the pass rate on your end. |
|
I started executing acceptance test iteratively on my local env. First Error Applied below fix Second Error How to fix this? |
|
it looks like |
| def http_method(self) -> str: | ||
| return "POST" | ||
|
|
||
| def read_records(self, *args, **kvargs) -> Iterable[Mapping[str, Any]]: |
There was a problem hiding this comment.
I finally solved my local issue by adding a few profile Ids, though our dev account doesn't have any attribute reports. So, earlier, under the secrets/spec.json, when the profiles is null or empty, it continues to fetch. Can you add an if-statement around the line to skip the read_records if _profiles is empty?
There was a problem hiding this comment.
current implementation
def read_records(self, *args, **kvargs) -> Iterable[Mapping[str, Any]]:
"""
Iterate through self._profiles list and send read all records for each profile.
"""
for profile in self._profiles:
self._set_dates(profile)
self._current_profile_id = profile.profileId
yield from super().read_records(*args, **kvargs)
if self._profiles is empty, it should not trigger any fetches
There was a problem hiding this comment.
can you do me a favor? Can you unset the profiles attribute in secrets/config.json, see if you get an error by running the read command?
python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
{
"client_id": "amzn1.application-oa2-client.client-id",
"client_secret": "secret",
"scope": "advertising::campaign_management",
"refresh_token": "a very long token",
"region": "NA"
}
There was a problem hiding this comment.
No error when profiles attribute is removed from config.json file.
When there are no user provided profiles, API call is made to fetch all profiles.
There was a problem hiding this comment.
thanks for the feedback, that's really helpful.
The acceptance test is not passed. And the reason is that not all profiles are authorized to fetch reports, and the read command doesn't fail gracefully, which blocks the read on other streams.
I read the codes again, looks like a profile is not independent of the others. In our dev test account, there are about 10 profiles fetched from the API, and only 4 of them are authorized to read the attribute reports.
If I pass just these 4 profiles, the tests are good. But if I include another, the read command fails on that profile and stops fetching reports on other profiles. And under the configured_catalog.json, if I shuffle the streams collection to fetch attribute reports first, the following streams will stop.
Similarly, if I set a collection of profiles to secrets/config.json, but one of the profiles is authorized, the same error is thrown and not handled gracefully.
Under a read command, we want to fetch as much data as possible, and reports from one profile shouldn't impact another profile or other streams. Can you adjust the code to satisfy that?
You may consider to use a stream slice to store profiles, and fetch reports for each profile in that slice. The GitHub connector provides a good example in fetching users in an org.
Let me know if you have a question.
There was a problem hiding this comment.
Can you help me in providing a way to reproduce the error?
I tried running this command
python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
config.json
{
"region": "NA",
"client_id": "amzn1.application-oa2-client.",
"start_date": "2022-09-13",
"client_secret": "secret",
"refresh_token": "secret",
"report_wait_timeout": 30,
"report_generation_max_retries": 5,
"profiles": [1, 2]
}
It's returning 0 records for all the streams except profiles
{"type": "LOG", "log": {"level": "INFO", "message": "Read 3 records from profiles stream"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing profiles"}}
{"type": "LOG", "log": {"level": "INFO", "message": "SourceAmazonAds runtimes:\nSyncing stream profiles 0:00:00.005063"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Syncing stream: sponsored_display_campaigns "}}
{"type": "LOG", "log": {"level": "INFO", "message": "Read 0 records from sponsored_display_campaigns stream"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing sponsored_display_campaigns"}}
{"type": "LOG", "log": {"level": "INFO", "message": "SourceAmazonAds runtimes:\nSyncing stream profiles 0:00:00.005063\nSyncing stream sponsored_display_campaigns 0:00:00.000228"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Syncing stream: sponsored_display_ad_groups "}}
{"type": "LOG", "log": {"level": "INFO", "message": "Read 0 records from sponsored_display_ad_groups stream"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing sponsored_display_ad_groups"}}
{"type": "LOG", "log": {"level": "INFO", "message": "SourceAmazonAds runtimes:\nSyncing stream profiles 0:00:00.005063\nSyncing stream sponsored_display_ad_groups 0:00:00.000135\nSyncing stream sponsored_display_campaigns 0:00:00.000228"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Syncing stream: sponsored_display_product_ads "}}
{"type": "LOG", "log": {"level": "INFO", "message": "Read 0 records from sponsored_display_product_ads stream"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing sponsored_display_product_ads"}}
{"type": "LOG", "log": {"level": "INFO", "message": "SourceAmazonAds runtimes:\nSyncing stream profiles 0:00:00.005063\nSyncing stream sponsored_display_ad_groups 0:00:00.000135\nSyncing stream sponsored_display_campaigns 0:00:00.000228\nSyncing stream sponsored_display_product_ads 0:00:00.000272"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Syncing stream: sponsored_display_targetings "}}
{"type": "LOG", "log": {"level": "INFO", "message": "Read 0 records from sponsored_display_targetings stream"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing sponsored_display_targetings"}}
{"type": "LOG", "log": {"level": "INFO", "message": "SourceAmazonAds runtimes:\nSyncing stream profiles 0:00:00.005063\nSyncing stream sponsored_display_ad_groups 0:00:00.000135\nSyncing stream sponsored_display_campaigns 0:00:00.000228\nSyncing stream sponsored_display_product_ads 0:00:00.000272\nSyncing stream sponsored_display_targetings 0:00:00.000179"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Syncing stream: sponsored_product_campaigns "}}
{"type": "LOG", "log": {"level": "INFO", "message": "Read 0 records from sponsored_product_campaigns stream"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing sponsored_product_campaigns"}}
{"type": "LOG", "log": {"level": "INFO", "message": "SourceAmazonAds runtimes:\nSyncing stream profiles 0:00:00.005063\nSyncing stream sponsored_display_ad_groups 0:00:00.000135\nSyncing stream sponsored_display_campaigns 0:00:00.000228\nSyncing stream sponsored_display_product_ads 0:00:00.000272\nSyncing stream sponsored_display_targetings 0:00:00.000179\nSyncing stream sponsored_product_campaigns 0:00:00.000159"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Syncing stream: sponsored_product_ad_groups "}}
{"type": "LOG", "log": {"level": "INFO", "message": "Read 0 records from sponsored_product_ad_groups stream"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing sponsored_product_ad_groups"}}
{"type": "LOG", "log": {"level": "INFO", "message": "SourceAmazonAds runtimes:\nSyncing stream profiles 0:00:00.005063\nSyncing stream sponsored_display_ad_groups 0:00:00.000135\nSyncing stream sponsored_display_campaigns 0:00:00.000228\nSyncing stream sponsored_display_product_ads 0:00:00.000272\nSyncing stream sponsored_display_targetings 0:00:00.000179\nSyncing stream sponsored_product_ad_groups 0:00:00.000161\nSyncing stream sponsored_product_campaigns 0:00:00.000159"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Syncing stream: sponsored_product_keywords "}}
{"type": "LOG", "log": {"level": "INFO", "message": "Read 0 records from sponsored_product_keywords stream"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing sponsored_product_keywords"}}
{"type": "LOG", "log": {"level": "INFO", "message": "SourceAmazonAds runtimes:\nSyncing stream profiles 0:00:00.005063\nSyncing stream sponsored_display_ad_groups 0:00:00.000135\nSyncing stream sponsored_display_campaigns 0:00:00.000228\nSyncing stream sponsored_display_product_ads 0:00:00.000272\nSyncing stream sponsored_display_targetings 0:00:00.000179\nSyncing stream sponsored_product_ad_groups 0:00:00.000161\nSyncing stream sponsored_product_campaigns 0:00:00.000159\nSyncing stream sponsored_product_keywords 0:00:00.000143"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Syncing stream: sponsored_product_negative_keywords "}}
{"type": "LOG", "log": {"level": "INFO", "message": "Read 0 records from sponsored_product_negative_keywords stream"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing sponsored_product_negative_keywords"}}
{"type": "LOG", "log": {"level": "INFO", "message": "SourceAmazonAds runtimes:\nSyncing stream profiles 0:00:00.005063\nSyncing stream sponsored_display_ad_groups 0:00:00.000135\nSyncing stream sponsored_display_campaigns 0:00:00.000228\nSyncing stream sponsored_display_product_ads 0:00:00.000272\nSyncing stream sponsored_display_targetings 0:00:00.000179\nSyncing stream sponsored_product_ad_groups 0:00:00.000161\nSyncing stream sponsored_product_campaigns 0:00:00.000159\nSyncing stream sponsored_product_keywords 0:00:00.000143\nSyncing stream sponsored_product_negative_keywords 0:00:00.000143"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Syncing stream: sponsored_product_ads "}}
{"type": "LOG", "log": {"level": "INFO", "message": "Read 0 records from sponsored_product_ads stream"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing sponsored_product_ads"}}
{"type": "LOG", "log": {"level": "INFO", "message": "SourceAmazonAds runtimes:\nSyncing stream profiles 0:00:00.005063\nSyncing stream sponsored_display_ad_groups 0:00:00.000135\nSyncing stream sponsored_display_campaigns 0:00:00.000228\nSyncing stream sponsored_display_product_ads 0:00:00.000272\nSyncing stream sponsored_display_targetings 0:00:00.000179\nSyncing stream sponsored_product_ad_groups 0:00:00.000161\nSyncing stream sponsored_product_ads 0:00:00.000361\nSyncing stream sponsored_product_campaigns 0:00:00.000159\nSyncing stream sponsored_product_keywords 0:00:00.000143\nSyncing stream sponsored_product_negative_keywords 0:00:00.000143"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Syncing stream: sponsored_product_targetings "}}
{"type": "LOG", "log": {"level": "INFO", "message": "Read 0 records from sponsored_product_targetings stream"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing sponsored_product_targetings"}}
{"type": "LOG", "log": {"level": "INFO", "message": "SourceAmazonAds runtimes:\nSyncing stream profiles 0:00:00.005063\nSyncing stream sponsored_display_ad_groups 0:00:00.000135\nSyncing stream sponsored_display_campaigns 0:00:00.000228\nSyncing stream sponsored_display_product_ads 0:00:00.000272\nSyncing stream sponsored_display_targetings 0:00:00.000179\nSyncing stream sponsored_product_ad_groups 0:00:00.000161\nSyncing stream sponsored_product_ads 0:00:00.000361\nSyncing stream sponsored_product_campaigns 0:00:00.000159\nSyncing stream sponsored_product_keywords 0:00:00.000143\nSyncing stream sponsored_product_negative_keywords 0:00:00.000143\nSyncing stream sponsored_product_targetings 0:00:00.000144"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Syncing stream: sponsored_brands_campaigns "}}
{"type": "LOG", "log": {"level": "INFO", "message": "Read 0 records from sponsored_brands_campaigns stream"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing sponsored_brands_campaigns"}}
{"type": "LOG", "log": {"level": "INFO", "message": "SourceAmazonAds runtimes:\nSyncing stream profiles 0:00:00.005063\nSyncing stream sponsored_brands_campaigns 0:00:00.000144\nSyncing stream sponsored_display_ad_groups 0:00:00.000135\nSyncing stream sponsored_display_campaigns 0:00:00.000228\nSyncing stream sponsored_display_product_ads 0:00:00.000272\nSyncing stream sponsored_display_targetings 0:00:00.000179\nSyncing stream sponsored_product_ad_groups 0:00:00.000161\nSyncing stream sponsored_product_ads 0:00:00.000361\nSyncing stream sponsored_product_campaigns 0:00:00.000159\nSyncing stream sponsored_product_keywords 0:00:00.000143\nSyncing stream sponsored_product_negative_keywords 0:00:00.000143\nSyncing stream sponsored_product_targetings 0:00:00.000144"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Syncing stream: sponsored_brands_ad_groups "}}
{"type": "LOG", "log": {"level": "INFO", "message": "Read 0 records from sponsored_brands_ad_groups stream"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing sponsored_brands_ad_groups"}}
{"type": "LOG", "log": {"level": "INFO", "message": "SourceAmazonAds runtimes:\nSyncing stream profiles 0:00:00.005063\nSyncing stream sponsored_brands_ad_groups 0:00:00.000145\nSyncing stream sponsored_brands_campaigns 0:00:00.000144\nSyncing stream sponsored_display_ad_groups 0:00:00.000135\nSyncing stream sponsored_display_campaigns 0:00:00.000228\nSyncing stream sponsored_display_product_ads 0:00:00.000272\nSyncing stream sponsored_display_targetings 0:00:00.000179\nSyncing stream sponsored_product_ad_groups 0:00:00.000161\nSyncing stream sponsored_product_ads 0:00:00.000361\nSyncing stream sponsored_product_campaigns 0:00:00.000159\nSyncing stream sponsored_product_keywords 0:00:00.000143\nSyncing stream sponsored_product_negative_keywords 0:00:00.000143\nSyncing stream sponsored_product_targetings 0:00:00.000144"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Syncing stream: sponsored_brands_keywords "}}
{"type": "LOG", "log": {"level": "INFO", "message": "Read 0 records from sponsored_brands_keywords stream"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing sponsored_brands_keywords"}}
{"type": "LOG", "log": {"level": "INFO", "message": "SourceAmazonAds runtimes:\nSyncing stream profiles 0:00:00.005063\nSyncing stream sponsored_brands_ad_groups 0:00:00.000145\nSyncing stream sponsored_brands_campaigns 0:00:00.000144\nSyncing stream sponsored_brands_keywords 0:00:00.000170\nSyncing stream sponsored_display_ad_groups 0:00:00.000135\nSyncing stream sponsored_display_campaigns 0:00:00.000228\nSyncing stream sponsored_display_product_ads 0:00:00.000272\nSyncing stream sponsored_display_targetings 0:00:00.000179\nSyncing stream sponsored_product_ad_groups 0:00:00.000161\nSyncing stream sponsored_product_ads 0:00:00.000361\nSyncing stream sponsored_product_campaigns 0:00:00.000159\nSyncing stream sponsored_product_keywords 0:00:00.000143\nSyncing stream sponsored_product_negative_keywords 0:00:00.000143\nSyncing stream sponsored_product_targetings 0:00:00.000144"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Syncing stream: attribution_report_products "}}
{"type": "LOG", "log": {"level": "INFO", "message": "Read 0 records from attribution_report_products stream"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing attribution_report_products"}}
{"type": "LOG", "log": {"level": "INFO", "message": "SourceAmazonAds runtimes:\nSyncing stream attribution_report_products 0:00:00.007100\nSyncing stream profiles 0:00:00.005063\nSyncing stream sponsored_brands_ad_groups 0:00:00.000145\nSyncing stream sponsored_brands_campaigns 0:00:00.000144\nSyncing stream sponsored_brands_keywords 0:00:00.000170\nSyncing stream sponsored_display_ad_groups 0:00:00.000135\nSyncing stream sponsored_display_campaigns 0:00:00.000228\nSyncing stream sponsored_display_product_ads 0:00:00.000272\nSyncing stream sponsored_display_targetings 0:00:00.000179\nSyncing stream sponsored_product_ad_groups 0:00:00.000161\nSyncing stream sponsored_product_ads 0:00:00.000361\nSyncing stream sponsored_product_campaigns 0:00:00.000159\nSyncing stream sponsored_product_keywords 0:00:00.000143\nSyncing stream sponsored_product_negative_keywords 0:00:00.000143\nSyncing stream sponsored_product_targetings 0:00:00.000144"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Syncing stream: attribution_report_performance_adgroup "}}
{"type": "LOG", "log": {"level": "INFO", "message": "Read 0 records from attribution_report_performance_adgroup stream"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing attribution_report_performance_adgroup"}}
{"type": "LOG", "log": {"level": "INFO", "message": "SourceAmazonAds runtimes:\nSyncing stream attribution_report_performance_adgroup 0:00:00.000172\nSyncing stream attribution_report_products 0:00:00.007100\nSyncing stream profiles 0:00:00.005063\nSyncing stream sponsored_brands_ad_groups 0:00:00.000145\nSyncing stream sponsored_brands_campaigns 0:00:00.000144\nSyncing stream sponsored_brands_keywords 0:00:00.000170\nSyncing stream sponsored_display_ad_groups 0:00:00.000135\nSyncing stream sponsored_display_campaigns 0:00:00.000228\nSyncing stream sponsored_display_product_ads 0:00:00.000272\nSyncing stream sponsored_display_targetings 0:00:00.000179\nSyncing stream sponsored_product_ad_groups 0:00:00.000161\nSyncing stream sponsored_product_ads 0:00:00.000361\nSyncing stream sponsored_product_campaigns 0:00:00.000159\nSyncing stream sponsored_product_keywords 0:00:00.000143\nSyncing stream sponsored_product_negative_keywords 0:00:00.000143\nSyncing stream sponsored_product_targetings 0:00:00.000144"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Syncing stream: attribution_report_performance_campaign "}}
{"type": "LOG", "log": {"level": "INFO", "message": "Read 0 records from attribution_report_performance_campaign stream"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing attribution_report_performance_campaign"}}
{"type": "LOG", "log": {"level": "INFO", "message": "SourceAmazonAds runtimes:\nSyncing stream attribution_report_performance_adgroup 0:00:00.000172\nSyncing stream attribution_report_performance_campaign 0:00:00.000294\nSyncing stream attribution_report_products 0:00:00.007100\nSyncing stream profiles 0:00:00.005063\nSyncing stream sponsored_brands_ad_groups 0:00:00.000145\nSyncing stream sponsored_brands_campaigns 0:00:00.000144\nSyncing stream sponsored_brands_keywords 0:00:00.000170\nSyncing stream sponsored_display_ad_groups 0:00:00.000135\nSyncing stream sponsored_display_campaigns 0:00:00.000228\nSyncing stream sponsored_display_product_ads 0:00:00.000272\nSyncing stream sponsored_display_targetings 0:00:00.000179\nSyncing stream sponsored_product_ad_groups 0:00:00.000161\nSyncing stream sponsored_product_ads 0:00:00.000361\nSyncing stream sponsored_product_campaigns 0:00:00.000159\nSyncing stream sponsored_product_keywords 0:00:00.000143\nSyncing stream sponsored_product_negative_keywords 0:00:00.000143\nSyncing stream sponsored_product_targetings 0:00:00.000144"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Syncing stream: attribution_report_performance_creative "}}
{"type": "LOG", "log": {"level": "INFO", "message": "Read 0 records from attribution_report_performance_creative stream"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing attribution_report_performance_creative"}}
{"type": "LOG", "log": {"level": "INFO", "message": "SourceAmazonAds runtimes:\nSyncing stream attribution_report_performance_adgroup 0:00:00.000172\nSyncing stream attribution_report_performance_campaign 0:00:00.000294\nSyncing stream attribution_report_performance_creative 0:00:00.007223\nSyncing stream attribution_report_products 0:00:00.007100\nSyncing stream profiles 0:00:00.005063\nSyncing stream sponsored_brands_ad_groups 0:00:00.000145\nSyncing stream sponsored_brands_campaigns 0:00:00.000144\nSyncing stream sponsored_brands_keywords 0:00:00.000170\nSyncing stream sponsored_display_ad_groups 0:00:00.000135\nSyncing stream sponsored_display_campaigns 0:00:00.000228\nSyncing stream sponsored_display_product_ads 0:00:00.000272\nSyncing stream sponsored_display_targetings 0:00:00.000179\nSyncing stream sponsored_product_ad_groups 0:00:00.000161\nSyncing stream sponsored_product_ads 0:00:00.000361\nSyncing stream sponsored_product_campaigns 0:00:00.000159\nSyncing stream sponsored_product_keywords 0:00:00.000143\nSyncing stream sponsored_product_negative_keywords 0:00:00.000143\nSyncing stream sponsored_product_targetings 0:00:00.000144"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing SourceAmazonAds"}}
There was a problem hiding this comment.
Sure, be happy to!
firstly, check integration_tests/configured_catalog.json, if profiles is one of the streams, the output will include the profile detail.
"Read 3 records from profiles stream"
looks like you have 3 profiles.
I guess all 3 profiles are authorized to read attribute_reports_*. Can you create another profile that is not authorized and add it to the array of profiles in secrets/config.json? For our case, if one of profiles in the array is not authorized, we will get the error of This profileID is not authorized to use Amazon Attribution, and the read stops.
There was a problem hiding this comment.
Any ideas on why I am not getting any error with fake profile IDs in config.json
{
"region": "NA",
"client_id": "amzn1.application-oa2-client.",
"start_date": "2022-09-13",
"client_secret": "secret",
"refresh_token": "secret",
"report_wait_timeout": 30,
"report_generation_max_retries": 5,
"profiles": [1, 2]
}
There was a problem hiding this comment.
I am not sure, maybe on the api side, the validation handles it before the authorization module is run?
|
/test connector=connectors/source-amazon-ads
Build FailedTest summary info: |
…azon-ads-attribution-reports
According to the log, it looks like the Integration test timeouts at incremental test, can you help to run the test again? Thanks. |
|
/test connector=connectors/source-amazon-ads
Build FailedTest summary info: |
|
#17202 there is a patch merged to master today, @ganpatagarwal can you rebase and try again? sorry for the inconvenience.
The conflict is related to the doc, hope it's quick. |
…azon-ads-attribution-reports
|
Done !! |
|
/test connector=connectors/source-amazon-ads
Build PassedTest summary info: |
| "sponsored_brands_keywords", | ||
| "sponsored_product_keywords", | ||
| "sponsored_product_negative_keywords", | ||
| "sponsored_product_targetings", |
There was a problem hiding this comment.
Why this streams: sponsored_product_keywords, sponsored_product_negative_keywords, sponsored_product_targetings
added as empty ?
There was a problem hiding this comment.
it comes from me, irrelevant to the topic branch. It was about 2 weeks ago, the integration test says these streams are empty in the test account when I run it locally. And I have the login credential, but I can't access the OTP token, so I can't login.
…azon-ads-attribution-reports
…azon-ads-attribution-reports
…azon-ads-attribution-reports
* use data field for json response * add attribution reports * update changelog * add atrribution report integration test * clean up expected_records add empty streams to acceptance test config * handle exception for profile * update tests Co-authored-by: Yiyang Li <yiyangli2010@gmail.com> Co-authored-by: Harshith Mullapudi <harshithmullapudi@gmail.com> Co-authored-by: Sajarin <sajarindider@gmail.com>
What
*Add attribution reports
How
*Call attribution report API for different report types
Recommended reading order
source.pystreams/attribution_report.py🚨 User Impact 🚨
No impact to existing functionality. Not a breaking change
Pre-merge Checklist
Updating a connector
Community member or Airbyter
airbyte_secret./gradlew :airbyte-integrations:connectors:<name>:integrationTest.README.mdbootstrap.md. See description and examplesdocs/integrations/<source or destination>/<name>.mdincluding changelog. See changelog exampleAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>command is passing/publishcommand described hereTests
Unit
Integration
Put your integration tests output here.
Acceptance
Put your acceptance tests output here.