🐛 Source Amazon Ads - Generate slices by lazy evaluation #15637

grubberr · 2022-08-14T12:03:05Z

Signed-off-by: Sergey Chvalyuk grubberr@gmail.com

What

Amazon can pretty slowly generate reports and slice dates which was generated initially can be non-actual. It's better to generate report dates lazily.

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-08-14T12:42:12Z

/publish connector=connectors/source-amazon-ads

🕑 Publishing the following connectors:
connectors/source-amazon-ads
https://github.com/airbytehq/airbyte/actions/runs/2855787174

Connector	Did it publish?	Were definitions generated?
connectors/source-amazon-ads	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

…com:airbytehq/airbyte into grubberr/generator-slices-source-amazon-ads

grubberr · 2022-08-14T15:39:49Z

/test connector=connectors/source-amazon-ads

🕑 connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/2856293702
✅ connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/2856293702
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_amazon_ads/utils.py                                            9      0   100%
source_amazon_ads/streams/sponsored_products.py                      32      0   100%
source_amazon_ads/streams/sponsored_display.py                       22      0   100%
source_amazon_ads/streams/sponsored_brands.py                        17      0   100%
source_amazon_ads/streams/report_streams/products_report.py          18      0   100%
source_amazon_ads/streams/report_streams/display_report.py           16      0   100%
source_amazon_ads/streams/report_streams/brands_video_report.py      10      0   100%
source_amazon_ads/streams/report_streams/brands_report.py            10      0   100%
source_amazon_ads/streams/report_streams/__init__.py                  5      0   100%
source_amazon_ads/streams/profiles.py                                21      0   100%
source_amazon_ads/streams/__init__.py                                 6      0   100%
source_amazon_ads/schemas/sponsored_products.py                      37      0   100%
source_amazon_ads/schemas/sponsored_display.py                       31      0   100%
source_amazon_ads/schemas/sponsored_brands.py                        22      0   100%
source_amazon_ads/schemas/profile.py                                 16      0   100%
source_amazon_ads/schemas/__init__.py                                 6      0   100%
source_amazon_ads/constants.py                                        6      0   100%
source_amazon_ads/__init__.py                                         2      0   100%
source_amazon_ads/streams/common.py                                  76      1    99%
source_amazon_ads/schemas/common.py                                  51      1    98%
source_amazon_ads/source.py                                          34      1    97%
source_amazon_ads/streams/report_streams/report_streams.py          215     20    91%
-------------------------------------------------------------------------------------
TOTAL                                                               662     23    97%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-216
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1321    463    65%

Build Passed

Test summary info:

All Passed

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-08-15T06:02:40Z

/test connector=connectors/source-amazon-ads

🕑 connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/2858769513
✅ connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/2858769513
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_amazon_ads/utils.py                                            9      0   100%
source_amazon_ads/streams/sponsored_products.py                      32      0   100%
source_amazon_ads/streams/sponsored_display.py                       22      0   100%
source_amazon_ads/streams/sponsored_brands.py                        17      0   100%
source_amazon_ads/streams/report_streams/products_report.py          18      0   100%
source_amazon_ads/streams/report_streams/display_report.py           16      0   100%
source_amazon_ads/streams/report_streams/brands_video_report.py      10      0   100%
source_amazon_ads/streams/report_streams/brands_report.py            10      0   100%
source_amazon_ads/streams/report_streams/__init__.py                  5      0   100%
source_amazon_ads/streams/profiles.py                                21      0   100%
source_amazon_ads/streams/__init__.py                                 6      0   100%
source_amazon_ads/schemas/sponsored_products.py                      37      0   100%
source_amazon_ads/schemas/sponsored_display.py                       31      0   100%
source_amazon_ads/schemas/sponsored_brands.py                        22      0   100%
source_amazon_ads/schemas/profile.py                                 16      0   100%
source_amazon_ads/schemas/__init__.py                                 6      0   100%
source_amazon_ads/constants.py                                        6      0   100%
source_amazon_ads/__init__.py                                         2      0   100%
source_amazon_ads/streams/common.py                                  76      1    99%
source_amazon_ads/schemas/common.py                                  51      1    98%
source_amazon_ads/source.py                                          34      1    97%
source_amazon_ads/streams/report_streams/report_streams.py          215     20    91%
-------------------------------------------------------------------------------------
TOTAL                                                               662     23    97%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-216
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1321    463    65%

Build Passed

Test summary info:

All Passed

grubberr · 2022-08-16T08:34:22Z

/publish connector=connectors/source-amazon-ads

🕑 Publishing the following connectors:
connectors/source-amazon-ads
https://github.com/airbytehq/airbyte/actions/runs/2866466624

Connector	Did it publish?	Were definitions generated?
connectors/source-amazon-ads	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

grubberr · 2022-08-16T11:20:34Z

/publish connector=connectors/source-amazon-ads

🕑 Publishing the following connectors:
connectors/source-amazon-ads
https://github.com/airbytehq/airbyte/actions/runs/2867495260

Connector	Did it publish?	Were definitions generated?
connectors/source-amazon-ads	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

grubberr · 2022-08-16T15:32:00Z

/publish connector=connectors/source-amazon-ads

🕑 Publishing the following connectors:
connectors/source-amazon-ads
https://github.com/airbytehq/airbyte/actions/runs/2869194089

Connector	Did it publish?	Were definitions generated?
connectors/source-amazon-ads	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

bump version source-amazon-ads 0.1.13

ab5a193

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr self-assigned this Aug 14, 2022

github-actions bot added the area/connectors Connector related issues label Aug 14, 2022

octavia-squidington-iii added the connectors/source/amazon-ads label Aug 14, 2022

grubberr added 3 commits August 14, 2022 15:04

REPORTING_PERIOD = 60 # added

3c17aec

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

stream_slices lazy generator

70098e8

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

iterate_one_by_one added

9e0beed

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

octavia-squidington-iii and others added 3 commits August 14, 2022 13:51

auto-bump connector version [ci skip]

52bdd8a

test_stream_slices_lazy_evaluation added

824c7ad

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

Merge branch 'grubberr/generator-slices-source-amazon-ads' of github.…

e59c335

…com:airbytehq/airbyte into grubberr/generator-slices-source-amazon-ads

grubberr temporarily deployed to more-secrets August 14, 2022 15:41 Inactive

grubberr added 2 commits August 15, 2022 08:46

test_get_date_range_lazy_evaluation added

4cfc27c

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

revert versions

61f34e7

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr added the team/connectors-python label Aug 15, 2022

amazon-ads.md updated

64b1c6f

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

github-actions bot added the area/documentation Improvements or additions to documentation label Aug 15, 2022

grubberr changed the title ~~🐛 Source Amazon Ads - bump version source-amazon-ads 0.1.13~~ 🐛 Source Amazon Ads - Generate slices by lazy evaluation Aug 15, 2022

grubberr requested a review from arsenlosenko August 15, 2022 09:27

arsenlosenko approved these changes Aug 15, 2022

View reviewed changes

grubberr requested a review from Phlair August 15, 2022 14:35

Merge branch 'master' into grubberr/generator-slices-source-amazon-ads

fafcb82

auto-bump connector version [ci skip]

dbf15c8

grubberr merged commit f281f8f into master Aug 16, 2022

grubberr deleted the grubberr/generator-slices-source-amazon-ads branch August 16, 2022 16:51

octavia-squidington-iii mentioned this pull request Aug 19, 2022

Bump Airbyte version from 0.40.0-alpha to 0.40.1-alpha #15775

Closed

rodireich pushed a commit that referenced this pull request Aug 20, 2022

🐛 Source Amazon Ads - Generate slices by lazy evaluation (#15637)

b21b387

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

evantahler mentioned this pull request Aug 22, 2022

Bump Airbyte version from 0.40.0-alpha to 0.40.1 #15857

Merged

grubberr mentioned this pull request Aug 23, 2022

[EPIC] Source Amazon Ads to GA #14686

Closed

15 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Source Amazon Ads - Generate slices by lazy evaluation #15637

🐛 Source Amazon Ads - Generate slices by lazy evaluation #15637

grubberr commented Aug 14, 2022 •

edited

Loading

grubberr commented Aug 14, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 14, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 15, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 16, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 16, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 16, 2022 •

edited by github-actions bot

Loading

🐛 Source Amazon Ads - Generate slices by lazy evaluation #15637

🐛 Source Amazon Ads - Generate slices by lazy evaluation #15637

Conversation

grubberr commented Aug 14, 2022 • edited Loading

What

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Tests

grubberr commented Aug 14, 2022 • edited by github-actions bot Loading

grubberr commented Aug 14, 2022 • edited by github-actions bot Loading

Build Passed

grubberr commented Aug 15, 2022 • edited by github-actions bot Loading

Build Passed

grubberr commented Aug 16, 2022 • edited by github-actions bot Loading

grubberr commented Aug 16, 2022 • edited by github-actions bot Loading

grubberr commented Aug 16, 2022 • edited by github-actions bot Loading

grubberr commented Aug 14, 2022 •

edited

Loading

grubberr commented Aug 14, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 14, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 15, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 16, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 16, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 16, 2022 •

edited by github-actions bot

Loading