🎉 Source Amazon Ads: Implement OAuth2.0 #11430

grubberr · 2022-03-25T12:22:40Z

Signed-off-by: Sergey Chvalyuk grubberr@gmail.com

What

How

Describe the solution

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

codecov · 2022-03-25T12:29:46Z

Codecov Report

❗ No coverage uploaded for pull request base (master@245e617). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #11430   +/-   ##
=========================================
  Coverage          ?   98.78%           
=========================================
  Files             ?       22           
  Lines             ?      660           
  Branches          ?        0           
=========================================
  Hits              ?      652           
  Misses            ?        8           
  Partials          ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 245e617...0c1ed16. Read the comment docs.

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

https://advertising.amazon.com/API/docs/en-us/concepts/authorization/access-tokens#generating-an-access-token-using-a-refresh-token Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-03-27T05:23:52Z

/test connector=connectors/source-amazon-ads

🕑 connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/2046802299
❌ connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/2046802299
🐛 https://gradle.com/s/epux2vgk4s7ko
Python short test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_match_expected[inputs0] - AssertionError:...
FAILED test_core.py::TestBasicRead::test_read[inputs1] - docker.errors.Contai...
FAILED test_core.py::TestBasicRead::test_read[inputs2] - docker.errors.Contai...
FAILED test_core.py::TestBasicRead::test_read[inputs3] - Failed: Stream profi...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
SKIPPED [1] ../usr/local/lib/python3.7/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
=================== 5 failed, 18 passed, 1 skipped in 15.64s ===================

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-03-27T13:28:20Z

/test connector=connectors/source-amazon-ads

🕑 connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/2047900121
❌ connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/2047900121
🐛 https://gradle.com/s/hydo5qik6nz74
Python short test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs1] - docker.errors.Contai...
FAILED test_core.py::TestBasicRead::test_read[inputs2] - docker.errors.Contai...
FAILED test_core.py::TestBasicRead::test_read[inputs3] - Failed: Stream profi...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
SKIPPED [1] ../usr/local/lib/python3.7/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
=================== 4 failed, 19 passed, 1 skipped in 13.99s ===================

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

sherifnada · 2022-04-13T04:43:21Z

airbyte-integrations/connectors/source-amazon-ads/source_amazon_ads/spec.py

@@ -28,19 +31,6 @@ class Config:
        airbyte_secret=True,
    )

-    # Amazon docs don't describe which of the below scopes to use under what circumstances so


make sure additionalproperties=true so existing configs don't break

sherifnada · 2022-04-13T04:44:03Z

airbyte-oauth/src/main/java/io/airbyte/oauth/flows/AmazonAdsOAuthFlow.java

+import java.util.function.Supplier;
+import org.apache.http.client.utils.URIBuilder;
+
+public class AmazonAdsOAuthFlow extends BaseOAuth2Flow {


can you add a unit test for this please?

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-04-15T18:20:35Z

/test connector=connectors/source-amazon-ads

🕑 connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/2173832352
❌ connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/2173832352
🐛 https://gradle.com/s/okgdovl6eqtxs
Python short test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs1] - docker.errors.Contai...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs1]
SKIPPED [1] ../usr/local/lib/python3.7/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
============= 2 failed, 21 passed, 1 skipped in 1457.84s (0:24:17) =============

grubberr · 2022-04-15T19:10:14Z

/test connector=connectors/source-amazon-ads

🕑 connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/2174010372
❌ connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/2174010372
🐛 https://gradle.com/s/qcxzj4yypblvs
Python short test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs1] - docker.errors.Contai...
SKIPPED [1] ../usr/local/lib/python3.7/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
============= 1 failed, 22 passed, 1 skipped in 1441.76s (0:24:01) =============

grubberr · 2022-04-16T06:01:05Z

/test connector=connectors/source-amazon-ads

🕑 connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/2175746766
✅ connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/2175746766
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              285    106    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  886    259    71%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_amazon_ads/streams/sponsored_products.py                      32      0   100%
source_amazon_ads/streams/sponsored_display.py                       22      0   100%
source_amazon_ads/streams/sponsored_brands.py                        17      0   100%
source_amazon_ads/streams/report_streams/products_report.py          17      0   100%
source_amazon_ads/streams/report_streams/display_report.py           16      0   100%
source_amazon_ads/streams/report_streams/brands_video_report.py      10      0   100%
source_amazon_ads/streams/report_streams/brands_report.py            10      0   100%
source_amazon_ads/streams/report_streams/__init__.py                  5      0   100%
source_amazon_ads/streams/profiles.py                                21      0   100%
source_amazon_ads/streams/__init__.py                                 6      0   100%
source_amazon_ads/spec.py                                            27      0   100%
source_amazon_ads/schemas/sponsored_products.py                      37      0   100%
source_amazon_ads/schemas/sponsored_display.py                       31      0   100%
source_amazon_ads/schemas/sponsored_brands.py                        22      0   100%
source_amazon_ads/schemas/profile.py                                 16      0   100%
source_amazon_ads/schemas/common.py                                  45      0   100%
source_amazon_ads/schemas/__init__.py                                 6      0   100%
source_amazon_ads/constants.py                                        7      0   100%
source_amazon_ads/__init__.py                                         2      0   100%
source_amazon_ads/streams/common.py                                  77      1    99%
source_amazon_ads/source.py                                          34      1    97%
source_amazon_ads/streams/report_streams/report_streams.py          200      6    97%
-------------------------------------------------------------------------------------
TOTAL                                                               660      8    99%

Python short test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.7/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
================== 23 passed, 1 skipped in 465.27s (0:07:45) ===================

Signed-off-by: Serhii Chvaliuk <grubberr@gmail.com>

grubberr · 2022-04-16T06:36:12Z

/publish connector=connectors/source-amazon-ads

🕑 connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/2175827417
❌ Failed to publish connectors/source-amazon-ads
❌ Couldn't auto-bump version for connectors/source-amazon-ads

grubberr · 2022-04-16T07:09:31Z

/publish connector=connectors/source-amazon-ads

🕑 connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/2175896901
🚀 Successfully published connectors/source-amazon-ads
🚀 Auto-bumped version for connectors/source-amazon-ads
✅ connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/2175896901

* Added support OAuth2.0 Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

Added support OAuth2.0

1359d44

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr self-assigned this Mar 25, 2022

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Mar 25, 2022

grubberr linked an issue Mar 25, 2022 that may be closed by this pull request

🎉 Source Amazon Ads: Implement OAuth2.0 #11417

Closed

grubberr temporarily deployed to more-secrets March 25, 2022 12:24 Inactive

fix amazon-ads.md

ec12083

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets March 25, 2022 14:51 Inactive

grubberr temporarily deployed to more-secrets March 25, 2022 14:52 Inactive

grubberr added 2 commits March 25, 2022 19:58

fix oauth2.0 scope -> "advertising::campaign_management"

710b897

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

we don't need scope on exchange refresh -> access token

bf18c4b

https://advertising.amazon.com/API/docs/en-us/concepts/authorization/access-tokens#generating-an-access-token-using-a-refresh-token Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets March 27, 2022 05:25 Inactive

spec.json fixed

7c07ba9

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets March 27, 2022 13:30 Inactive

Merge branch 'master' into grubberr/11417-source-amazon-ads

01cac75

grubberr temporarily deployed to more-secrets March 28, 2022 21:13 Inactive

grubberr added 2 commits March 29, 2022 21:12

Merge branch 'master' into grubberr/11417-source-amazon-ads

b754c3c

Merge branch 'master' into grubberr/11417-source-amazon-ads

ac5491b

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets April 8, 2022 16:41 Inactive

enum removed

bd530ce

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets April 8, 2022 19:01 Inactive

sherifnada reviewed Apr 13, 2022

View reviewed changes

grubberr added 2 commits April 13, 2022 08:19

Merge branch 'master' into grubberr/11417-source-amazon-ads

ce417c5

"additionalProperties": true

b246b29

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets April 13, 2022 07:29 Inactive

AmazonAdsOAuthFlowTest.java added

9022d2e

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr requested review from sherifnada and midavadim April 13, 2022 08:44

grubberr temporarily deployed to more-secrets April 13, 2022 08:44 Inactive

sherifnada approved these changes Apr 15, 2022

View reviewed changes

Merge branch 'master' into grubberr/11417-source-amazon-ads

ff4b28f

grubberr temporarily deployed to more-secrets April 15, 2022 18:21 Inactive

grubberr added 2 commits April 16, 2022 09:24

Merge branch 'master' into grubberr/11417-source-amazon-ads

76bbc88

revert version to check auto-bump

e18d4ae

Signed-off-by: Serhii Chvaliuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets April 16, 2022 06:37 Inactive

auto-bump connector version

0c1ed16

octavia-squidington-iii temporarily deployed to more-secrets April 16, 2022 07:26 Inactive

octavia-squidington-iii temporarily deployed to more-secrets April 16, 2022 07:27 Inactive

grubberr merged commit 189ce99 into master Apr 16, 2022

grubberr deleted the grubberr/11417-source-amazon-ads branch April 16, 2022 07:35

octavia-squidington-iii mentioned this pull request Apr 19, 2022

Bump Airbyte version from 0.36.0-alpha to 0.36.1-alpha #12148

Merged

suhomud pushed a commit that referenced this pull request May 23, 2022

🎉 Source Amazon Ads: Implement OAuth2.0 (#11430)

2fb410b

* Added support OAuth2.0 Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Source Amazon Ads: Implement OAuth2.0 #11430

🎉 Source Amazon Ads: Implement OAuth2.0 #11430

grubberr commented Mar 25, 2022 •

edited

Loading

codecov bot commented Mar 25, 2022 •

edited

Loading

grubberr commented Mar 27, 2022 •

edited by github-actions bot

Loading

grubberr commented Mar 27, 2022 •

edited by github-actions bot

Loading

sherifnada Apr 13, 2022

sherifnada Apr 13, 2022

grubberr commented Apr 15, 2022 •

edited by github-actions bot

Loading

grubberr commented Apr 15, 2022 •

edited by github-actions bot

Loading

grubberr commented Apr 16, 2022 •

edited by github-actions bot

Loading

grubberr commented Apr 16, 2022 •

edited by github-actions bot

Loading

grubberr commented Apr 16, 2022 •

edited by github-actions bot

Loading

🎉 Source Amazon Ads: Implement OAuth2.0 #11430

🎉 Source Amazon Ads: Implement OAuth2.0 #11430

Conversation

grubberr commented Mar 25, 2022 • edited Loading

What

How

Pre-merge Checklist

Community member or Airbyter

Airbyter

Tests

codecov bot commented Mar 25, 2022 • edited Loading

Codecov Report

grubberr commented Mar 27, 2022 • edited by github-actions bot Loading

grubberr commented Mar 27, 2022 • edited by github-actions bot Loading

sherifnada Apr 13, 2022

Choose a reason for hiding this comment

sherifnada Apr 13, 2022

Choose a reason for hiding this comment

grubberr commented Apr 15, 2022 • edited by github-actions bot Loading

grubberr commented Apr 15, 2022 • edited by github-actions bot Loading

grubberr commented Apr 16, 2022 • edited by github-actions bot Loading

grubberr commented Apr 16, 2022 • edited by github-actions bot Loading

grubberr commented Apr 16, 2022 • edited by github-actions bot Loading

grubberr commented Mar 25, 2022 •

edited

Loading

codecov bot commented Mar 25, 2022 •

edited

Loading

grubberr commented Mar 27, 2022 •

edited by github-actions bot

Loading

grubberr commented Mar 27, 2022 •

edited by github-actions bot

Loading

grubberr commented Apr 15, 2022 •

edited by github-actions bot

Loading

grubberr commented Apr 15, 2022 •

edited by github-actions bot

Loading

grubberr commented Apr 16, 2022 •

edited by github-actions bot

Loading

grubberr commented Apr 16, 2022 •

edited by github-actions bot

Loading

grubberr commented Apr 16, 2022 •

edited by github-actions bot

Loading