🎉 Source Amazon Ads: Skip API error "Report date is too far in the past." #15921

grubberr · 2022-08-24T10:34:50Z

Signed-off-by: Sergey Chvalyuk grubberr@gmail.com

What

Skip API error: Report date is too far in the past. Reports are only available for 60 days.

In theory, connector does not have to get such an error because the connector correctly calculates the start_date,
but from practice, we can still catch such errors from time to time.

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-08-24T10:51:56Z

/test connector=connectors/source-amazon-ads

🕑 connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/2918503304
✅ connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/2918503304
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_amazon_ads/utils.py                                            9      0   100%
source_amazon_ads/streams/sponsored_products.py                      32      0   100%
source_amazon_ads/streams/sponsored_display.py                       22      0   100%
source_amazon_ads/streams/sponsored_brands.py                        17      0   100%
source_amazon_ads/streams/report_streams/products_report.py          18      0   100%
source_amazon_ads/streams/report_streams/display_report.py           16      0   100%
source_amazon_ads/streams/report_streams/brands_video_report.py      10      0   100%
source_amazon_ads/streams/report_streams/brands_report.py            10      0   100%
source_amazon_ads/streams/report_streams/__init__.py                  5      0   100%
source_amazon_ads/streams/profiles.py                                21      0   100%
source_amazon_ads/streams/__init__.py                                 6      0   100%
source_amazon_ads/schemas/sponsored_products.py                      37      0   100%
source_amazon_ads/schemas/sponsored_display.py                       31      0   100%
source_amazon_ads/schemas/sponsored_brands.py                        22      0   100%
source_amazon_ads/schemas/profile.py                                 16      0   100%
source_amazon_ads/schemas/__init__.py                                 6      0   100%
source_amazon_ads/constants.py                                        1      0   100%
source_amazon_ads/__init__.py                                         2      0   100%
source_amazon_ads/streams/common.py                                  76      1    99%
source_amazon_ads/schemas/common.py                                  51      1    98%
source_amazon_ads/streams/report_streams/report_streams.py          233     14    94%
source_amazon_ads/source.py                                          38      8    79%
-------------------------------------------------------------------------------------
TOTAL                                                               679     24    96%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-216
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1322    463    65%

Build Passed

Test summary info:

All Passed

grubberr · 2022-08-24T12:21:29Z

/publish connector=connectors/source-amazon-ads

🕑 Publishing the following connectors:
connectors/source-amazon-ads
https://github.com/airbytehq/airbyte/actions/runs/2919048374

Connector	Did it publish?	Were definitions generated?
connectors/source-amazon-ads	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-08-24T14:53:52Z

/test connector=connectors/source-amazon-ads

🕑 connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/2920103281
❌ connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/2920103281
🐛 https://gradle.com/s/ge5dvnoj5qmtk

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_incremental.py::TestIncremental::test_two_sequential_reads[inputs0]
================== 1 failed, 31 passed in 5244.93s (1:27:24) ===================

grubberr · 2022-08-24T19:46:53Z

/test connector=connectors/source-amazon-ads

🕑 connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/2921861436
✅ connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/2921861436
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_amazon_ads/utils.py                                            9      0   100%
source_amazon_ads/streams/sponsored_products.py                      32      0   100%
source_amazon_ads/streams/sponsored_display.py                       22      0   100%
source_amazon_ads/streams/sponsored_brands.py                        17      0   100%
source_amazon_ads/streams/report_streams/products_report.py          18      0   100%
source_amazon_ads/streams/report_streams/display_report.py           16      0   100%
source_amazon_ads/streams/report_streams/brands_video_report.py      10      0   100%
source_amazon_ads/streams/report_streams/brands_report.py            10      0   100%
source_amazon_ads/streams/report_streams/__init__.py                  5      0   100%
source_amazon_ads/streams/profiles.py                                21      0   100%
source_amazon_ads/streams/__init__.py                                 6      0   100%
source_amazon_ads/schemas/sponsored_products.py                      37      0   100%
source_amazon_ads/schemas/sponsored_display.py                       31      0   100%
source_amazon_ads/schemas/sponsored_brands.py                        22      0   100%
source_amazon_ads/schemas/profile.py                                 16      0   100%
source_amazon_ads/schemas/__init__.py                                 6      0   100%
source_amazon_ads/constants.py                                        1      0   100%
source_amazon_ads/__init__.py                                         2      0   100%
source_amazon_ads/streams/common.py                                  76      1    99%
source_amazon_ads/schemas/common.py                                  51      1    98%
source_amazon_ads/streams/report_streams/report_streams.py          232     15    94%
source_amazon_ads/source.py                                          38      8    79%
-------------------------------------------------------------------------------------
TOTAL                                                               678     25    96%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-216
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1322    463    65%

Build Passed

Test summary info:

All Passed

grubberr · 2022-08-24T20:53:34Z

/publish connector=connectors/source-amazon-ads

🕑 Publishing the following connectors:
connectors/source-amazon-ads
https://github.com/airbytehq/airbyte/actions/runs/2922224112

Connector	Did it publish?	Were definitions generated?
connectors/source-amazon-ads	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

…st." (#15921) * _check_report_date_error added Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * amazon-ads.md updated Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * improve docs Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * improve logging, improve type hints Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * remove ReportInitFatal Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * amazon-ads.md updated Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * auto-bump connector version [ci skip] Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

_check_report_date_error added

bebb57f

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

octavia-squidington-iii added the connectors/source/amazon-ads label Aug 24, 2022

github-actions bot added the area/connectors Connector related issues label Aug 24, 2022

amazon-ads.md updated

0823a85

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

github-actions bot added the area/documentation Improvements or additions to documentation label Aug 24, 2022

grubberr added 2 commits August 24, 2022 13:40

improve docs

2604f3f

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

improve logging, improve type hints

a482b06

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr self-assigned this Aug 24, 2022

grubberr requested review from bazarnov, lazebnyi and davydov-d August 24, 2022 10:52

lazebnyi approved these changes Aug 24, 2022

View reviewed changes

bazarnov approved these changes Aug 24, 2022

View reviewed changes

grubberr added 2 commits August 24, 2022 17:49

remove ReportInitFatal

fe2f296

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

amazon-ads.md updated

b841e81

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr changed the title ~~🎉 Source Amazon Ads: don't re-try on "Report date is too far in the past" error~~ 🎉 Source Amazon Ads: Skip API error "Report date is too far in the past." Aug 24, 2022

auto-bump connector version [ci skip]

095bab1

grubberr merged commit f952ace into master Aug 25, 2022

grubberr deleted the grubberr/source-amazon-ads-improve-error-handling branch August 25, 2022 05:38

grubberr mentioned this pull request Aug 25, 2022

[EPIC] Source Amazon Ads to GA #14686

Closed

15 tasks

octavia-squidington-iii mentioned this pull request Aug 26, 2022

Bump Airbyte version from 0.40.2 to 0.40.3 #16028

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Source Amazon Ads: Skip API error "Report date is too far in the past." #15921

🎉 Source Amazon Ads: Skip API error "Report date is too far in the past." #15921

grubberr commented Aug 24, 2022 •

edited

Loading

grubberr commented Aug 24, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 24, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 24, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 24, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 24, 2022 •

edited by github-actions bot

Loading

🎉 Source Amazon Ads: Skip API error "Report date is too far in the past." #15921

🎉 Source Amazon Ads: Skip API error "Report date is too far in the past." #15921

Conversation

grubberr commented Aug 24, 2022 • edited Loading

What

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Tests

grubberr commented Aug 24, 2022 • edited by github-actions bot Loading

Build Passed

grubberr commented Aug 24, 2022 • edited by github-actions bot Loading

grubberr commented Aug 24, 2022 • edited by github-actions bot Loading

Build Failed

grubberr commented Aug 24, 2022 • edited by github-actions bot Loading

Build Passed

grubberr commented Aug 24, 2022 • edited by github-actions bot Loading

grubberr commented Aug 24, 2022 •

edited

Loading

grubberr commented Aug 24, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 24, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 24, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 24, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 24, 2022 •

edited by github-actions bot

Loading