Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Source Amazon Ads: fix reports stream records primary keys #21677

Conversation

roman-yermilov-gl
Copy link
Collaborator

What

Because of non-unique values are defined as a composite primary key of the report streams, we are missing a bunch of records per each sync if dedup+history strategy chosen. This PR contains a fix but also adds some possibly incompatible changes that needs to be discussed. At this point I splitted asins report to two reports asins_keywords and asins_targets. We need this separation for everything to be consistent but maybe it require some preparation before delivery to cloud.

@roman-yermilov-gl
Copy link
Collaborator Author

roman-yermilov-gl commented Jan 20, 2023

/test connector=connectors/source-amazon-ads

🕑 connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/3971099219
✅ connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/3971099219
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_amazon_ads/utils.py                                           22      0   100%
source_amazon_ads/streams/sponsored_products.py                      41      0   100%
source_amazon_ads/streams/sponsored_display.py                       31      0   100%
source_amazon_ads/streams/sponsored_brands.py                        26      0   100%
source_amazon_ads/streams/report_streams/products_report.py          20      0   100%
source_amazon_ads/streams/report_streams/display_report.py           18      0   100%
source_amazon_ads/streams/report_streams/brands_video_report.py      12      0   100%
source_amazon_ads/streams/report_streams/brands_report.py            12      0   100%
source_amazon_ads/streams/report_streams/__init__.py                  5      0   100%
source_amazon_ads/streams/profiles.py                                21      0   100%
source_amazon_ads/streams/attribution_report.py                      81      0   100%
source_amazon_ads/streams/__init__.py                                 7      0   100%
source_amazon_ads/schemas/sponsored_products.py                      37      0   100%
source_amazon_ads/schemas/sponsored_display.py                       31      0   100%
source_amazon_ads/schemas/sponsored_brands.py                        22      0   100%
source_amazon_ads/schemas/profile.py                                 16      0   100%
source_amazon_ads/schemas/attribution_report.py                      21      0   100%
source_amazon_ads/schemas/__init__.py                                 7      0   100%
source_amazon_ads/constants.py                                        1      0   100%
source_amazon_ads/__init__.py                                         2      0   100%
source_amazon_ads/streams/common.py                                  79      1    99%
source_amazon_ads/schemas/common.py                                  52      1    98%
source_amazon_ads/source.py                                          44      1    98%
source_amazon_ads/streams/report_streams/report_streams.py          243     19    92%
-------------------------------------------------------------------------------------
TOTAL                                                               851     22    97%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       160     14    91%   58-65, 70-83, 246
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1609    339    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: The previous and actual specifications are identical.
================== 33 passed, 1 skipped in 549.10s (0:09:09) ===================

@octavia-squidington-iv octavia-squidington-iv added the area/documentation Improvements or additions to documentation label Jan 20, 2023
@erohmensing
Copy link
Contributor

This PR contains a fix but also adds some possibly incompatible changes that needs to be discussed.

Can you lay out the possible incompatible changes?

@roman-yermilov-gl
Copy link
Collaborator Author

This PR contains a fix but also adds some possibly incompatible changes that needs to be discussed.

Can you lay out the possible incompatible changes?

It's on description. Not sure what else can I add more, unless concrete question

@erohmensing
Copy link
Contributor

erohmensing commented Jan 25, 2023

We need this separation for everything to be consistent but maybe it require some preparation before delivery to cloud.

What is the preparation needed - is it on our end or on the customer's end? If it's on the customer's end we should figure out how many people are impacted by this so that TCS can coordinate

some possibly incompatible changes that needs to be discussed.

It's unclear to me who needs to discuss/agree on what changes - does this only refer to the asins report?

@davydov-d davydov-d self-assigned this Jan 26, 2023
@davydov-d
Copy link
Collaborator

Hey guys, Roman is unavailable for a couple of days so I will pick up this issue. Serhii and I discussed this problem and the suggested solution and decided that it needs to be thinked over one more time and probably reimplemented. Suggested solution fixes existing problem but most probably introduces a new one.
Will keep you updated

@erohmensing
Copy link
Contributor

erohmensing commented Jan 26, 2023

Thank you for the update @davydov-d!

@davydov-d
Copy link
Collaborator

davydov-d commented Jan 30, 2023

/test connector=connectors/source-amazon-ads

🕑 connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/4043321045
❌ connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/4043321045
🐛 https://gradle.com/s/3evgfvr7ipuhu

Build Failed

Test summary info:

Could not find result summary

@davydov-d
Copy link
Collaborator

@erohmensing PR updated, please review

@octavia-squidington-iv octavia-squidington-iv added the area/frontend Related to the Airbyte webapp label Jan 30, 2023
@davydov-d
Copy link
Collaborator

davydov-d commented Jan 30, 2023

/test connector=connectors/source-amazon-ads

🕑 connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/4044884437
✅ connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/4044884437
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_amazon_ads/utils.py                                           22      0   100%
source_amazon_ads/streams/sponsored_products.py                      41      0   100%
source_amazon_ads/streams/sponsored_display.py                       31      0   100%
source_amazon_ads/streams/sponsored_brands.py                        26      0   100%
source_amazon_ads/streams/report_streams/products_report.py          20      0   100%
source_amazon_ads/streams/report_streams/display_report.py           18      0   100%
source_amazon_ads/streams/report_streams/brands_video_report.py      12      0   100%
source_amazon_ads/streams/report_streams/brands_report.py            12      0   100%
source_amazon_ads/streams/report_streams/__init__.py                  5      0   100%
source_amazon_ads/streams/profiles.py                                21      0   100%
source_amazon_ads/streams/attribution_report.py                      81      0   100%
source_amazon_ads/streams/__init__.py                                 7      0   100%
source_amazon_ads/schemas/sponsored_products.py                      37      0   100%
source_amazon_ads/schemas/sponsored_display.py                       31      0   100%
source_amazon_ads/schemas/sponsored_brands.py                        22      0   100%
source_amazon_ads/schemas/profile.py                                 16      0   100%
source_amazon_ads/schemas/attribution_report.py                      21      0   100%
source_amazon_ads/schemas/__init__.py                                 7      0   100%
source_amazon_ads/constants.py                                        1      0   100%
source_amazon_ads/__init__.py                                         2      0   100%
source_amazon_ads/streams/common.py                                  79      1    99%
source_amazon_ads/schemas/common.py                                  50      1    98%
source_amazon_ads/source.py                                          47      1    98%
source_amazon_ads/streams/report_streams/report_streams.py          243     19    92%
-------------------------------------------------------------------------------------
TOTAL                                                               852     22    97%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              476    117    75%   53, 58, 97-108, 113-120, 124-125, 129-130, 380, 400, 438, 476-493, 506-517, 521-526, 532, 565-570, 608-615, 658-660, 663, 728-736, 748-751, 756, 812-813, 819, 822, 858-868, 881-906
	 source_acceptance_test/tests/test_incremental.py       160     14    91%   58-65, 70-83, 246
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     114     13    89%   31-32, 39, 42, 66-69, 97, 121, 203-205
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1690    341    80%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:98: The previous and actual specifications are identical.
================== 37 passed, 1 skipped in 447.15s (0:07:27) ===================

@erohmensing
Copy link
Contributor

erohmensing commented Jan 30, 2023

@davydov-d would you please rebase? There are lots of unrelated changes in the diff. Otherwise I believe it looks good

@davydov-d davydov-d force-pushed the ryermilov/source-amazon-ads-fix-report-stream-record-primary-keys branch from ed6fd30 to b8e71d0 Compare January 30, 2023 18:11
@davydov-d davydov-d force-pushed the ryermilov/source-amazon-ads-fix-report-stream-record-primary-keys branch from b8e71d0 to 4e67027 Compare January 30, 2023 18:17
@octavia-squidington-iv octavia-squidington-iv removed the area/frontend Related to the Airbyte webapp label Jan 30, 2023
@davydov-d
Copy link
Collaborator

@erohmensing Sorry, I overlooked it, must have been pre-commit hooks. Removed non related changes.

@davydov-d
Copy link
Collaborator

davydov-d commented Jan 30, 2023

/test connector=connectors/source-amazon-ads

🕑 connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/4046935053
✅ connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/4046935053
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_amazon_ads/utils.py                                           22      0   100%
source_amazon_ads/streams/sponsored_products.py                      41      0   100%
source_amazon_ads/streams/sponsored_display.py                       31      0   100%
source_amazon_ads/streams/sponsored_brands.py                        26      0   100%
source_amazon_ads/streams/report_streams/products_report.py          20      0   100%
source_amazon_ads/streams/report_streams/display_report.py           18      0   100%
source_amazon_ads/streams/report_streams/brands_video_report.py      12      0   100%
source_amazon_ads/streams/report_streams/brands_report.py            12      0   100%
source_amazon_ads/streams/report_streams/__init__.py                  5      0   100%
source_amazon_ads/streams/profiles.py                                21      0   100%
source_amazon_ads/streams/attribution_report.py                      81      0   100%
source_amazon_ads/streams/__init__.py                                 7      0   100%
source_amazon_ads/schemas/sponsored_products.py                      37      0   100%
source_amazon_ads/schemas/sponsored_display.py                       31      0   100%
source_amazon_ads/schemas/sponsored_brands.py                        22      0   100%
source_amazon_ads/schemas/profile.py                                 16      0   100%
source_amazon_ads/schemas/attribution_report.py                      21      0   100%
source_amazon_ads/schemas/__init__.py                                 7      0   100%
source_amazon_ads/constants.py                                        1      0   100%
source_amazon_ads/__init__.py                                         2      0   100%
source_amazon_ads/schemas/common.py                                  50      1    98%
source_amazon_ads/source.py                                          47      1    98%
source_amazon_ads/streams/common.py                                  83      2    98%
source_amazon_ads/streams/report_streams/report_streams.py          243     19    92%
-------------------------------------------------------------------------------------
TOTAL                                                               856     23    97%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              476    117    75%   53, 58, 97-108, 113-120, 124-125, 129-130, 380, 400, 438, 476-493, 506-517, 521-526, 532, 565-570, 608-615, 658-660, 663, 728-736, 748-751, 756, 812-813, 819, 822, 858-868, 881-906
	 source_acceptance_test/tests/test_incremental.py       160     14    91%   58-65, 70-83, 246
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     114     13    89%   31-32, 39, 42, 66-69, 97, 121, 203-205
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1690    341    80%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:98: The previous and actual specifications are identical.
================== 37 passed, 1 skipped in 459.90s (0:07:39) ===================

Copy link
Contributor

@erohmensing erohmensing left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please make sure to update TCS with any changes customers will have to make and get their OK before merging (they may want to schedule it if it is still breaking) - thanks for adopting this PR!

Copy link
Contributor

@girarda girarda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. Reminder there is a breaking change so we'll need to coordinate with TCS to release this to Cloud

@davydov-d
Copy link
Collaborator

davydov-d commented Feb 6, 2023

/publish connector=connectors/source-amazon-ads

🕑 Publishing the following connectors:
connectors/source-amazon-ads
https://github.com/airbytehq/airbyte/actions/runs/4102425543


Connector Did it publish? Were definitions generated?
connectors/source-amazon-ads

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@airbyteio airbyteio temporarily deployed to more-secrets February 6, 2023 11:21 — with GitHub Actions Inactive
@airbyteio airbyteio temporarily deployed to more-secrets February 6, 2023 11:21 — with GitHub Actions Inactive
@github-actions
Copy link
Contributor

github-actions bot commented Feb 6, 2023

Airbyte Code Coverage

There is no coverage information present for the Files changed

Total Project Coverage 24.52%

@davydov-d davydov-d merged commit 7d13ad5 into master Feb 6, 2023
@davydov-d davydov-d deleted the ryermilov/source-amazon-ads-fix-report-stream-record-primary-keys branch February 6, 2023 16:53
letiescanciano added a commit that referenced this pull request Feb 6, 2023
* master:
  Discover worker starts to use API to write schema result (#21875)
  🪟 🎉  Connector Builder Landing Page (#22122)
  Fix pnpm cache path (#22418)
  Add additional shorter setup guides (#22318)
  Source Amazon Ads: fix reports stream records primary keys (#21677)
  Connector acceptance test: Fix discovered catalog caching for different configs (#22301)
  🪟🐛 Make modal scrollable (#21973)
  only compute diff if the schema discovery actually succeeded (#22377)
  Source Klaviyo: fix schema (#22071)
  🪟 🔧 Switch to `pnpm` for package managing (#22053)
  Source Sentry: turn on default availability strategy (#22303)
  Source freshdesk: deduplicate table names (#22164)
letiescanciano added a commit that referenced this pull request Feb 6, 2023
* master: (86 commits)
  Discover worker starts to use API to write schema result (#21875)
  🪟 🎉  Connector Builder Landing Page (#22122)
  Fix pnpm cache path (#22418)
  Add additional shorter setup guides (#22318)
  Source Amazon Ads: fix reports stream records primary keys (#21677)
  Connector acceptance test: Fix discovered catalog caching for different configs (#22301)
  🪟🐛 Make modal scrollable (#21973)
  only compute diff if the schema discovery actually succeeded (#22377)
  Source Klaviyo: fix schema (#22071)
  🪟 🔧 Switch to `pnpm` for package managing (#22053)
  Source Sentry: turn on default availability strategy (#22303)
  Source freshdesk: deduplicate table names (#22164)
  Update connector-acceptance-tests-reference.md (#22370)
  Update the default security groups for the EC2 runner (#22347)
  Trace refresh schema operations (#22326)
  Remove manual docker upgrades from workflows (#22344)
  Update CODEOWNERS for connector acceptance tests to connector ops (#22341)
  🐛 source: airtable - handle singleSelect types (#22311)
  Source tiktok: chunk advertiser IDs (#22309)
  🪟 🧪 E2E Tests for auto-detect schema changes (#20682)
  ...
letiescanciano added a commit that referenced this pull request Feb 6, 2023
* master: (24 commits)
  Discover worker starts to use API to write schema result (#21875)
  🪟 🎉  Connector Builder Landing Page (#22122)
  Fix pnpm cache path (#22418)
  Add additional shorter setup guides (#22318)
  Source Amazon Ads: fix reports stream records primary keys (#21677)
  Connector acceptance test: Fix discovered catalog caching for different configs (#22301)
  🪟🐛 Make modal scrollable (#21973)
  only compute diff if the schema discovery actually succeeded (#22377)
  Source Klaviyo: fix schema (#22071)
  🪟 🔧 Switch to `pnpm` for package managing (#22053)
  Source Sentry: turn on default availability strategy (#22303)
  Source freshdesk: deduplicate table names (#22164)
  Update connector-acceptance-tests-reference.md (#22370)
  Update the default security groups for the EC2 runner (#22347)
  Trace refresh schema operations (#22326)
  Remove manual docker upgrades from workflows (#22344)
  Update CODEOWNERS for connector acceptance tests to connector ops (#22341)
  🐛 source: airtable - handle singleSelect types (#22311)
  Source tiktok: chunk advertiser IDs (#22309)
  🪟 🧪 E2E Tests for auto-detect schema changes (#20682)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/amazon-ads
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

9 participants