🎉 Source Amazon Ads: add option to customize look back window value #19491

dandpz · 2022-11-16T17:07:40Z

resolve #18929

What

Add "look_back_window" internal settings
Add "Look Back Window" to source settings in order to let the user customize this value

🚨 User Impact 🚨

The User will find a new field to set inside the resource settings. No breaking change, default value is the old one.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

CLAassistant · 2022-11-16T17:07:46Z

All committers have signed the CLA.

marcosmarxm · 2022-11-22T20:25:49Z

Hello 👋, first thank you for this amazing contribution.

We really appreciate the effort you've made to improve the project.
We ask you patience for the code review. Last month our team was focused on Hacktoberfest event and that probably left some PR without the proper feedback. And this week, due to the Thanksgiving US Holiday, most our team is out of office with their families. Another important piece of information why code won't be merge this week is: as a safety measure the core team has decided to freeze merging code to main branch to keep the release stable. Next week we'll return to you with the proper code review and update the status of your contribution.

If you have any questions feel free to send me a message in Slack!
Thanks!

marcosmarxm · 2022-12-20T19:52:03Z

Hello 👋:skin-tone-2: and thank you for your contribution!

Airbyte has instituted a code freeze between 19 and 30 December, to make sure there are no disruptions during the holidays.
Because of this, reviewing and merging your contribution may take longer than usual.
We apologize for the delay, but we want everyone to have a quiet and happy holiday.

If you have any questions or need further clarification, please don't hesitate to ping via Slack.

marcosmarxm

Thanks @dandpz I only need the final approval from the connector team.

...ions/connectors/source-amazon-ads/source_amazon_ads/streams/report_streams/report_streams.py

grubberr · 2023-01-19T09:36:02Z

@dandpz can you please merge master branch and resolve conflicts?

airbyte-config/init/src/main/resources/seed/source_definitions.yaml

airbyte-config/init/src/main/resources/seed/source_specs.yaml

grubberr

sorry I missed this

can you please pre-load default value
like did here for "region"

https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-amazon-ads/source_amazon_ads/source.py#L52

grubberr · 2023-01-20T09:34:35Z

@dandpz please fully revert source_specs.yaml not only version.
it's updated automatically

...ions/connectors/source-amazon-ads/source_amazon_ads/streams/report_streams/report_streams.py

grubberr · 2023-01-20T10:17:51Z

/test connector=connectors/source-amazon-ads

🕑 connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/3966718555
❌ connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/3966718555
🐛 https://gradle.com/s/n2dilmk23hvjw

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 FAILED unit_tests/test_report_streams.py::test_display_report_stream - KeyErr...
	 FAILED unit_tests/test_report_streams.py::test_products_report_stream - KeyEr...
	 FAILED unit_tests/test_report_streams.py::test_brands_report_stream - KeyErro...
	 FAILED unit_tests/test_report_streams.py::test_brands_video_report_stream - K...
	 FAILED unit_tests/test_report_streams.py::test_display_report_stream_init_failure
	 FAILED unit_tests/test_report_streams.py::test_display_report_stream_init_http_exception
	 FAILED unit_tests/test_report_streams.py::test_display_report_stream_init_too_many_requests
	 FAILED unit_tests/test_report_streams.py::test_display_report_stream_backoff[modifiers0-5]
	 FAILED unit_tests/test_report_streams.py::test_display_report_stream_backoff[modifiers1-10]
	 FAILED unit_tests/test_report_streams.py::test_display_report_stream_backoff[modifiers2-ReportGenerationInProgress]
	 FAILED unit_tests/test_report_streams.py::test_display_report_stream_backoff[modifiers3-15]
	 FAILED unit_tests/test_report_streams.py::test_display_report_stream_backoff[modifiers4-ReportGenerationFailure]
	 FAILED unit_tests/test_report_streams.py::test_display_report_stream_slices_full_refresh
	 FAILED unit_tests/test_report_streams.py::test_display_report_stream_slices_incremental
	 FAILED unit_tests/test_report_streams.py::test_get_start_date - KeyError: 'lo...
	 FAILED unit_tests/test_report_streams.py::test_stream_slices_different_timezones
	 FAILED unit_tests/test_report_streams.py::test_stream_slices_lazy_evaluation
	 FAILED unit_tests/test_report_streams.py::test_read_incremental_without_records
	 FAILED unit_tests/test_report_streams.py::test_read_incremental_with_records
	 FAILED unit_tests/test_report_streams.py::test_read_incremental_without_records_start_date
	 FAILED unit_tests/test_report_streams.py::test_read_incremental_with_records_start_date
	 �[31m================= �[31m�[1m21 failed�[0m, �[32m48 passed�[0m, �[33m1364 warnings�[0m�[31m in 4.68s�[0m�[31m =================�[0m

grubberr · 2023-01-20T11:03:40Z

@dandpz it seems unit-tests failed

grubberr

it seems unit-tests failed

grubberr · 2023-01-20T11:48:30Z

/test connector=connectors/source-amazon-ads

🕑 connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/3967342628
✅ connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/3967342628
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_amazon_ads/utils.py                                           22      0   100%
source_amazon_ads/streams/sponsored_products.py                      41      0   100%
source_amazon_ads/streams/sponsored_display.py                       31      0   100%
source_amazon_ads/streams/sponsored_brands.py                        26      0   100%
source_amazon_ads/streams/report_streams/products_report.py          18      0   100%
source_amazon_ads/streams/report_streams/display_report.py           16      0   100%
source_amazon_ads/streams/report_streams/brands_video_report.py      10      0   100%
source_amazon_ads/streams/report_streams/brands_report.py            10      0   100%
source_amazon_ads/streams/report_streams/__init__.py                  5      0   100%
source_amazon_ads/streams/profiles.py                                21      0   100%
source_amazon_ads/streams/attribution_report.py                      81      0   100%
source_amazon_ads/streams/__init__.py                                 7      0   100%
source_amazon_ads/schemas/sponsored_products.py                      37      0   100%
source_amazon_ads/schemas/sponsored_display.py                       31      0   100%
source_amazon_ads/schemas/sponsored_brands.py                        22      0   100%
source_amazon_ads/schemas/profile.py                                 16      0   100%
source_amazon_ads/schemas/attribution_report.py                      21      0   100%
source_amazon_ads/schemas/__init__.py                                 7      0   100%
source_amazon_ads/constants.py                                        1      0   100%
source_amazon_ads/__init__.py                                         2      0   100%
source_amazon_ads/streams/common.py                                  79      1    99%
source_amazon_ads/schemas/common.py                                  51      1    98%
source_amazon_ads/source.py                                          47      1    98%
source_amazon_ads/streams/report_streams/report_streams.py          240     19    92%
-------------------------------------------------------------------------------------
TOTAL                                                               842     22    97%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       160     14    91%   58-65, 70-83, 246
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1609    339    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:377: The previous and actual discovered catalogs are identical.
================== 33 passed, 1 skipped in 5222.69s (1:27:02) ==================

marcosmarxm · 2023-01-26T16:36:12Z

/publish connector=connectors/source-amazon-ads

🕑 Publishing the following connectors:
connectors/source-amazon-ads
https://github.com/airbytehq/airbyte/actions/runs/4017181253

Connector	Did it publish?	Were definitions generated?
connectors/source-amazon-ads	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

octavia-squidington-iv added area/connectors Connector related issues bounty community connectors/source/amazon-ads area/documentation Improvements or additions to documentation labels Nov 16, 2022

dandpz force-pushed the customizable-lookback-window branch from 007c4f3 to f6f6d06 Compare November 16, 2022 17:21

dandpz mentioned this pull request Nov 16, 2022

Amazon Ads Source - make lookback window customizable #18929

Closed

dandpz added 3 commits November 16, 2022 18:52

add new configurable field

43b1175

update docker image version

66eb96c

amazon-ads.md updated

5e6f604

dandpz force-pushed the customizable-lookback-window branch from f6f6d06 to 5e6f604 Compare November 16, 2022 17:52

dandpz added 2 commits November 17, 2022 09:54

Merge branch 'master' into customizable-lookback-window

f828c2e

Merge branch 'master' into customizable-lookback-window

949a002

Merge branch 'master' into customizable-lookback-window

bea3214

sajarin added internal and removed bounty labels Dec 5, 2022

sajarin requested a review from grubberr December 5, 2022 17:01

marcosmarxm approved these changes Jan 18, 2023

View reviewed changes

marcosmarxm self-assigned this Jan 18, 2023

grubberr reviewed Jan 19, 2023

View reviewed changes

...ions/connectors/source-amazon-ads/source_amazon_ads/streams/report_streams/report_streams.py Show resolved Hide resolved

dandpz and others added 3 commits January 19, 2023 10:41

remove unnecessary constant

d8c46fa

Merge branch 'master' into customizable-lookback-window

4eecbd0

Merge branch 'master' into customizable-lookback-window

b6e74e7

grubberr reviewed Jan 20, 2023

View reviewed changes

airbyte-config/init/src/main/resources/seed/source_definitions.yaml Show resolved Hide resolved

grubberr reviewed Jan 20, 2023

View reviewed changes

airbyte-config/init/src/main/resources/seed/source_specs.yaml Show resolved Hide resolved

grubberr reviewed Jan 20, 2023

View reviewed changes

airbyte-config/init/src/main/resources/seed/source_specs.yaml Show resolved Hide resolved

grubberr suggested changes Jan 20, 2023

View reviewed changes

dandpz added 2 commits January 20, 2023 10:38

fully revert changes

d2386e5

pre load lbw default value

0a04945

dandpz requested a review from grubberr January 20, 2023 09:43

grubberr reviewed Jan 20, 2023

View reviewed changes

...ions/connectors/source-amazon-ads/source_amazon_ads/streams/report_streams/report_streams.py Outdated Show resolved Hide resolved

fix value access

509da21

grubberr approved these changes Jan 20, 2023

View reviewed changes

grubberr suggested changes Jan 20, 2023

View reviewed changes

update config fixture

7817006

dandpz requested a review from grubberr January 20, 2023 13:33

grubberr approved these changes Jan 20, 2023

View reviewed changes

dandpz added 8 commits January 23, 2023 16:26

Merge branch 'master' into customizable-lookback-window

45ce570

Merge branch 'master' into customizable-lookback-window

c95ece0

Merge branch 'master' into customizable-lookback-window

f6ac47c

Merge branch 'master' into customizable-lookback-window

83bf40e

Merge branch 'master' into customizable-lookback-window

5eaaa21

Merge branch 'master' into customizable-lookback-window

067b981

Merge branch 'master' into customizable-lookback-window

868bb74

Merge branch 'master' into customizable-lookback-window

88335e3

octavia-squidington-iii and others added 2 commits January 26, 2023 17:14

auto-bump connector version

eb48a41

Merge branch 'master' into customizable-lookback-window

241b00c

marcosmarxm merged commit 13eac98 into airbytehq:master Jan 27, 2023

This was referenced Jan 27, 2023

Bump Airbyte version from 0.40.29 to 0.40.30 #21991

Closed

Bump Airbyte version from 0.40.29 to 0.40.30 #22045

Merged

dandpz mentioned this pull request Mar 13, 2023

🐛 Source Amazon Ads: enable applying look back window also to incremental sync mode #23979

Closed

37 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Source Amazon Ads: add option to customize look back window value #19491

🎉 Source Amazon Ads: add option to customize look back window value #19491

dandpz commented Nov 16, 2022 •

edited

Loading

CLAassistant commented Nov 16, 2022 •

edited

Loading

marcosmarxm commented Nov 22, 2022

marcosmarxm commented Dec 20, 2022

marcosmarxm left a comment

grubberr commented Jan 19, 2023

grubberr left a comment

grubberr commented Jan 20, 2023

grubberr commented Jan 20, 2023 •

edited by github-actions bot

Loading

grubberr commented Jan 20, 2023

grubberr left a comment

grubberr commented Jan 20, 2023 •

edited by github-actions bot

Loading

marcosmarxm commented Jan 26, 2023 •

edited by github-actions bot

Loading

🎉 Source Amazon Ads: add option to customize look back window value #19491

🎉 Source Amazon Ads: add option to customize look back window value #19491

Conversation

dandpz commented Nov 16, 2022 • edited Loading

What

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Tests

CLAassistant commented Nov 16, 2022 • edited Loading

marcosmarxm commented Nov 22, 2022

marcosmarxm commented Dec 20, 2022

marcosmarxm left a comment

Choose a reason for hiding this comment

grubberr commented Jan 19, 2023

grubberr left a comment

Choose a reason for hiding this comment

grubberr commented Jan 20, 2023

grubberr commented Jan 20, 2023 • edited by github-actions bot Loading

Build Failed

grubberr commented Jan 20, 2023

grubberr left a comment

Choose a reason for hiding this comment

grubberr commented Jan 20, 2023 • edited by github-actions bot Loading

Build Passed

marcosmarxm commented Jan 26, 2023 • edited by github-actions bot Loading

dandpz commented Nov 16, 2022 •

edited

Loading

CLAassistant commented Nov 16, 2022 •

edited

Loading

grubberr commented Jan 20, 2023 •

edited by github-actions bot

Loading

grubberr commented Jan 20, 2023 •

edited by github-actions bot

Loading

marcosmarxm commented Jan 26, 2023 •

edited by github-actions bot

Loading