source-amazon-seller-partner: allow to use IAM user arn or IAM role arn #12523

pecalleja · 2022-05-02T20:25:24Z

What

The amazon sp-api integration allows to use IAM user entity or IAM role entity. amazon documentation
if we use a user based arn we need to use another function from the boto3 client to get the required credentials role
This will allow to integrate with airbyte apps configured with user arn entity in the amazon seller central app.

How

just use the required function for differents IAM entities

🚨 User Impact 🚨

None

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

CLAassistant · 2022-05-02T20:25:29Z

All committers have signed the CLA.

alafanechere · 2022-05-03T13:53:34Z

Thank you @pecalleja for this contribution. I'm running the tests and will get back for a first review once they pass.

alafanechere · 2022-05-03T13:53:58Z

/test connector=connectors/source-amazon-seller-partner

🕑 connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/2264220922
✅ connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/2264220922
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              285    106    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  896    259    71%
Name                                        Stmts   Miss  Cover
---------------------------------------------------------------
source_amazon_seller_partner/__init__.py        2      0   100%
source_amazon_seller_partner/constants.py      37      1    97%
source_amazon_seller_partner/source.py         68     10    85%
source_amazon_seller_partner/streams.py       353    163    54%
source_amazon_seller_partner/auth.py           61     36    41%
---------------------------------------------------------------
TOTAL                                         521    210    60%

Python short test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestBasicRead.test_read because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestFullRefresh.test_sequential_reads because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
======================== 17 passed, 3 skipped in 6.05s =========================

alafanechere

Minor suggestions

...-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/source.py

airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_source.py

pecalleja · 2022-05-03T14:31:02Z

do you want I include more unit tests in this PR to fully cover the connector source code?

…ource_amazon_seller_partner/source.py use a better exception text message Co-authored-by: Augustin <augustin@airbyte.io>

…ource_amazon_seller_partner/source.py use better variable name Co-authored-by: Augustin <augustin@airbyte.io>

alafanechere · 2022-05-03T14:46:31Z

do you want I include more unit tests in this PR to fully cover the connector source code?

I'd prefer you only add tests for the code you add. Feel free to open another PR if you want to increase the coverage of other functions.

alafanechere · 2022-05-04T07:52:43Z

/test connector=connectors/source-amazon-seller-partner

🕑 connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/2268452681
❌ connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/2268452681
🐛 https://gradle.com/s/3ftqpzn7vgzeq
Python short test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_match_expected[inputs0] - AssertionError:...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestBasicRead.test_read because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestFullRefresh.test_sequential_reads because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
=================== 1 failed, 16 passed, 3 skipped in 5.96s ====================

alafanechere

Thank you @pecalleja, we are close to having a mergeable PR 😄
I need to ask you some extra work to make your change available in the published connector:

Bump the connector version in
- Dockerfile
- airbyte-config/init/src/main/resources/seed/source_definitions.yaml
- airbyte-config/init/src/main/resources/seed/source_specs.yaml

Update the changelog in docs/integrations/sources/amazon-seller-partner.md

I can't do these changes myself as I've no permission to push on your branch.

alafanechere · 2022-05-04T08:06:53Z

Could you also please merge master into your branch? This is probably the reason the latest test fail.

pecalleja

requested changes implemented 👍. Thank you.

alafanechere · 2022-05-04T12:44:38Z

/test connector=connectors/source-amazon-seller-partner

🕑 connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/2269823429
❌ connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/2269823429
🐛 https://gradle.com/s/4qbn3b22bdywy
Python short test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_match_expected[inputs0] - AssertionError:...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestBasicRead.test_read because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestFullRefresh.test_sequential_reads because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
=================== 1 failed, 16 passed, 3 skipped in 5.84s ====================

airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_source.py

pecalleja

LGTM

pecalleja · 2022-05-04T13:59:34Z

/test connector=connectors/source-amazon-seller-partner

alafanechere · 2022-05-04T14:35:28Z

/test connector=connectors/source-amazon-seller-partner

🕑 connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/2270436193
✅ connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/2270436193
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              285    106    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  896    259    71%
Name                                        Stmts   Miss  Cover
---------------------------------------------------------------
source_amazon_seller_partner/__init__.py        2      0   100%
source_amazon_seller_partner/constants.py      37      1    97%
source_amazon_seller_partner/source.py         64      9    86%
source_amazon_seller_partner/streams.py       353    163    54%
source_amazon_seller_partner/auth.py           61     36    41%
---------------------------------------------------------------
TOTAL                                         517    209    60%

Python short test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestBasicRead.test_read because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestFullRefresh.test_sequential_reads because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
======================== 17 passed, 3 skipped in 5.76s =========================

alafanechere · 2022-05-04T15:11:32Z

/publish connector=connectors/source-amazon-seller-partner auto-bump-version=false

🕑 connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/2270647087
🚀 Successfully published connectors/source-amazon-seller-partner
✅ connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/2270647087

alafanechere

Thank you @pecalleja for your changes!

…rn (#12523)

allow to use IAM user arn or IAM role arn. This resolve #5942

ad849a5

octavia-squidington-iii added the community label May 2, 2022

github-actions bot added the area/connectors Connector related issues label May 2, 2022

pecalleja changed the title ~~allow to use IAM user arn or IAM role arn. This resolve #5942~~ allow to use IAM user arn or IAM role arn in the amazon seller partner connector May 2, 2022

pecalleja added 2 commits May 2, 2022 18:20

add type hints

5749c52

add tests

c3d7bbd

alafanechere self-assigned this May 3, 2022

alafanechere requested changes May 3, 2022

View reviewed changes

pecalleja and others added 2 commits May 3, 2022 10:33

Update airbyte-integrations/connectors/source-amazon-seller-partner/s…

f73d225

…ource_amazon_seller_partner/source.py use a better exception text message Co-authored-by: Augustin <augustin@airbyte.io>

Update airbyte-integrations/connectors/source-amazon-seller-partner/s…

dbe6003

…ource_amazon_seller_partner/source.py use better variable name Co-authored-by: Augustin <augustin@airbyte.io>

include minor sugestions

96d7420

alafanechere changed the title ~~allow to use IAM user arn or IAM role arn in the amazon seller partner connector~~ source-amazon-seller-partner: allow to use IAM user arn or IAM role arn May 3, 2022

alafanechere added the connectors/source/amazon-seller-partner label May 3, 2022

pecalleja added 6 commits May 3, 2022 10:52

use a better function name

10add5e

use a better variable name

42578d8

refactor the test for the new function name

9653e21

update the helper text for the conector config

424d581

add test for a bad arn value

4114379

refactor test for better readability

98f5362

pecalleja requested a review from alafanechere May 3, 2022 15:52

alafanechere requested changes May 4, 2022

View reviewed changes

pecalleja added 2 commits May 4, 2022 08:33

Merge branch 'master' into feature/amazon-user-role-authentication

1161519

bump connector versions

1cd8800

github-actions bot added the area/documentation Improvements or additions to documentation label May 4, 2022

pecalleja commented May 4, 2022

View reviewed changes

pecalleja requested a review from alafanechere May 4, 2022 12:43

alafanechere reviewed May 4, 2022

View reviewed changes

airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_source.py Show resolved Hide resolved

refactor test to assert calling the right function

4027de9

pecalleja commented May 4, 2022

View reviewed changes

pecalleja requested a review from alafanechere May 4, 2022 13:48

update the helper text on the connector spec

0422eb6

Merge branch 'master' into feature/amazon-user-role-authentication

7dd6eb5

alafanechere approved these changes May 4, 2022

View reviewed changes

alafanechere merged commit 4a13b60 into airbytehq:master May 4, 2022

octavia-squidington-iii mentioned this pull request May 4, 2022

Bump Airbyte version from 0.36.8-alpha to 0.36.9-alpha #12585

Merged

suhomud pushed a commit that referenced this pull request May 23, 2022

source amazon-seller-partner: allow to use IAM user arn or IAM role a…

b492598

…rn (#12523)

pecalleja mentioned this pull request Aug 16, 2022

allow to use IAM user arn or IAM role arn. saleweaver/python-amazon-sp-api#692

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

source-amazon-seller-partner: allow to use IAM user arn or IAM role arn #12523

source-amazon-seller-partner: allow to use IAM user arn or IAM role arn #12523

pecalleja commented May 2, 2022

CLAassistant commented May 2, 2022 •

edited

Loading

alafanechere commented May 3, 2022

alafanechere commented May 3, 2022 •

edited by github-actions bot

Loading

alafanechere left a comment

pecalleja commented May 3, 2022

alafanechere commented May 3, 2022

alafanechere commented May 4, 2022 •

edited by github-actions bot

Loading

alafanechere left a comment

alafanechere commented May 4, 2022

pecalleja left a comment

alafanechere commented May 4, 2022 •

edited by github-actions bot

Loading

pecalleja left a comment

pecalleja commented May 4, 2022

alafanechere commented May 4, 2022 •

edited by github-actions bot

Loading

alafanechere commented May 4, 2022 •

edited by github-actions bot

Loading

alafanechere left a comment

source-amazon-seller-partner: allow to use IAM user arn or IAM role arn #12523

source-amazon-seller-partner: allow to use IAM user arn or IAM role arn #12523

Conversation

pecalleja commented May 2, 2022

What

How

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Tests

CLAassistant commented May 2, 2022 • edited Loading

alafanechere commented May 3, 2022

alafanechere commented May 3, 2022 • edited by github-actions bot Loading

alafanechere left a comment

Choose a reason for hiding this comment

pecalleja commented May 3, 2022

alafanechere commented May 3, 2022

alafanechere commented May 4, 2022 • edited by github-actions bot Loading

alafanechere left a comment

Choose a reason for hiding this comment

alafanechere commented May 4, 2022

pecalleja left a comment

Choose a reason for hiding this comment

alafanechere commented May 4, 2022 • edited by github-actions bot Loading

pecalleja left a comment

Choose a reason for hiding this comment

pecalleja commented May 4, 2022

alafanechere commented May 4, 2022 • edited by github-actions bot Loading

alafanechere commented May 4, 2022 • edited by github-actions bot Loading

alafanechere left a comment

Choose a reason for hiding this comment

CLAassistant commented May 2, 2022 •

edited

Loading

alafanechere commented May 3, 2022 •

edited by github-actions bot

Loading

alafanechere commented May 4, 2022 •

edited by github-actions bot

Loading

alafanechere commented May 4, 2022 •

edited by github-actions bot

Loading

alafanechere commented May 4, 2022 •

edited by github-actions bot

Loading

alafanechere commented May 4, 2022 •

edited by github-actions bot

Loading