Source Amazon Seller Partner: fix GET_SELLER_FEEDBACK_DATA report Date cursor field format #8810
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
area/connectors
test resultspython -m pytest integration_tests -p integration_tests.acceptance
Test session starts (platform: darwin, Python 3.9.8, pytest 6.2.5, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/lizdeika/Projects/airbyte, configfile: pytest.ini
plugins: sugar-0.9.4, mock-3.6.1, timeout-1.4.2
collecting ... {"type": "LOG", "log": {"level": "INFO", "message": "Docker run: \nspec\ninput: /private/var/folders/pd/rd8643xs7xz4fnhp4f4wlnk40000gn/T/pytest-of-lizdeika/pytest-16/test_match_expected_inputs0_0/run_1/input\noutput: /private/var/folders/pd/rd8643xs7xz4fnhp4f4wlnk40000gn/T/pytest-of-lizdeika/pytest-16/test_match_expected_inputs0_0/run_1/output"}}
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_match_expected[inputs0] ✓ 10% █
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_required[inputs0] ✓ 20% ██
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_optional[inputs0] ✓ 30% ███
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_has_secret[inputs0] ✓ 40% ████
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_secret_never_in_the_output[inputs0] ✓ 50% █████ {"type": "LOG", "log": {"level": "INFO", "message": "Docker run: \nspec\ninput: /private/var/folders/pd/rd8643xs7xz4fnhp4f4wlnk40000gn/T/pytest-of-lizdeika/pytest-16/test_oauth_flow_parameters_inp0/run_1/input\noutput: /private/var/folders/pd/rd8643xs7xz4fnhp4f4wlnk40000gn/T/pytest-of-lizdeika/pytest-16/test_oauth_flow_parameters_inp0/run_1/output"}}
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_oauth_flow_parameters[inputs0] ✓ 60% ██████ {"type": "LOG", "log": {"level": "INFO", "message": "Docker run: \ncheck --config tap_config.json\ninput: /private/var/folders/pd/rd8643xs7xz4fnhp4f4wlnk40000gn/T/pytest-of-lizdeika/pytest-16/test_check_inputs0_0/run_1/input\noutput: /private/var/folders/pd/rd8643xs7xz4fnhp4f4wlnk40000gn/T/pytest-of-lizdeika/pytest-16/test_check_inputs0_0/run_1/output"}}
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection.test_check[inputs0] ✓ 70% ███████ {"type": "LOG", "log": {"level": "INFO", "message": "Docker run: \ndiscover --config tap_config.json\ninput: /private/var/folders/pd/rd8643xs7xz4fnhp4f4wlnk40000gn/T/pytest-of-lizdeika/pytest-16/test_discover_inputs0_0/run_1/input\noutput: /private/var/folders/pd/rd8643xs7xz4fnhp4f4wlnk40000gn/T/pytest-of-lizdeika/pytest-16/test_discover_inputs0_0/run_1/output"}}
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_discover[inputs0] ✓ 80% ████████ {"type": "LOG", "log": {"level": "INFO", "message": "Docker run: \ndiscover --config tap_config.json\ninput: /private/var/folders/pd/rd8643xs7xz4fnhp4f4wlnk40000gn/T/pytest-of-lizdeika/pytest-16/test_defined_cursors_exist_in_0/run_1/input\noutput: /private/var/folders/pd/rd8643xs7xz4fnhp4f4wlnk40000gn/T/pytest-of-lizdeika/pytest-16/test_defined_cursors_exist_in_0/run_1/output"}}
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_cursors_exist_in_schema[inputs0] ✓90% █████████
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_refs_exist_in_schema[inputs0] ✓100% ██████████
=================================================================== short test summary info ====================================================================
SKIPPED [1] ../../bases/source-acceptance-test/source_acceptance_test/plugin.py:56: Skipping TestBasicRead.test_read because not found in the config
SKIPPED [1] ../../bases/source-acceptance-test/source_acceptance_test/plugin.py:56: Skipping TestFullRefresh.test_sequential_reads because not found in the config
SKIPPED [1] ../../bases/source-acceptance-test/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
Results (9.63s):
10 passed
python -m pytest integration_tests
Test session starts (platform: darwin, Python 3.9.8, pytest 6.2.5, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/lizdeika/Projects/airbyte, configfile: pytest.ini
plugins: sugar-0.9.4, mock-3.6.1, timeout-1.4.2
collecting ...
Results (0.02s):
python -m pytest unit_tests
Test session starts (platform: darwin, Python 3.9.8, pytest 6.2.5, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/lizdeika/Projects/airbyte, configfile: pytest.ini
plugins: sugar-0.9.4, mock-3.6.1, timeout-1.4.2
collecting ...
airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_repots_streams_rate_limits.py::test_reports_stream_should_retry ✓ 33% ███▍
airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_repots_streams_rate_limits.py::test_reports_stream_send_request ✓ 67% ██████▋ {"type": "LOG", "log": {"level": "INFO", "message": "Backing off _send_request(...) for 5.0s (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException)"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Caught retryable error '' after 1 tries. Waiting 5 seconds then retrying..."}}
{"type": "LOG", "log": {"level": "INFO", "message": "Backing off _send_request(...) for 10.0s (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException)"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Caught retryable error '' after 2 tries. Waiting 10 seconds then retrying..."}}
{"type": "LOG", "log": {"level": "INFO", "message": "Backing off _send_request(...) for 20.0s (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException)"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Caught retryable error '' after 3 tries. Waiting 20 seconds then retrying..."}}
{"type": "LOG", "log": {"level": "INFO", "message": "Backing off _send_request(...) for 40.0s (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException)"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Caught retryable error '' after 4 tries. Waiting 40 seconds then retrying..."}}
{"type": "LOG", "log": {"level": "ERROR", "message": "Giving up _send_request(...) after 5 tries (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException)"}}
airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_repots_streams_rate_limits.py::test_reports_stream_send_request_backoff_exception ✓100% ██████████
======================================================================= warnings summary =======================================================================
source_amazon_seller_partner/streams.py:154
/Users/lizdeika/Projects/airbyte/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/streams.py:154: DeprecationWarning: Call to deprecated class NoAuth. (Set `authenticator=None` instead) -- Deprecated since version 0.1.20.
authenticator: HttpAuthenticator = NoAuth(),
.venv/lib/python3.9/site-packages/deprecated/classic.py:173
airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_repots_streams_rate_limits.py::test_reports_stream_should_retry
airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_repots_streams_rate_limits.py::test_reports_stream_send_request
airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_repots_streams_rate_limits.py::test_reports_stream_send_request_backoff_exception
/Users/lizdeika/Projects/airbyte/airbyte-integrations/connectors/source-amazon-seller-partner/.venv/lib/python3.9/site-packages/deprecated/classic.py:173: DeprecationWarning: Call to deprecated class HttpAuthenticator. (Use requests.auth.AuthBase instead) -- Deprecated since version 0.1.20.
return old_new1(cls, *args, **kwargs)
airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_repots_streams_rate_limits.py::test_reports_stream_should_retry
airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_repots_streams_rate_limits.py::test_reports_stream_send_request
airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_repots_streams_rate_limits.py::test_reports_stream_send_request_backoff_exception
/Users/lizdeika/Projects/airbyte/airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_repots_streams_rate_limits.py:29: DeprecationWarning: Call to deprecated class NoAuth. (Set `authenticator=None` instead) -- Deprecated since version 0.1.20.
authenticator=NoAuth(),
-- Docs: https://docs.pytest.org/en/stable/warnings.html
Results (0.50s):
3 passed |
marcosmarxm
reviewed
Dec 15, 2021
...integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/streams.py
Outdated
Show resolved
Hide resolved
marcosmarxm
requested changes
Dec 15, 2021
...integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/streams.py
Outdated
Show resolved
Hide resolved
...integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/streams.py
Show resolved
Hide resolved
…ource_amazon_seller_partner/streams.py Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
…ource_amazon_seller_partner/streams.py Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
…LLER_FEEDBACK_DATA
|
@marcosmarxm I have added a unit test for transform function unit test results |
…LLER_FEEDBACK_DATA
…LLER_FEEDBACK_DATA
|
Hi @lizdeika thanks for the changes, we'll go for a final review before the end of the week! |
alafanechere
approved these changes
Dec 28, 2021
|
@alafanechere do I need todo something more to get this merged into master? |
|
Hi @lizdeika I'm sorry but we are currently facing some trouble with our CI (the connectors base build step), which is not related to your PR. I'll merge as soon as we fix this. |
|
@lizdeika merged 🎉 , sorry for the delay. |
lizdeika
deleted the
fix-amazon-seller-partner-GET_SELLER_FEEDBACK_DATA
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Report returns
Datefield(used for cursor) in different formats in different marketplaces.Now the format is static value
M/D/YYwhich is valid only for some marketplaces and yields to errors for other ones.How
Added a map
marketplace_id:date_format.Map is located in report class because it is report specific, unreusable.
Refactored transform function registration according to Augustin's suggestion: #8711
Country date format source: https://en.wikipedia.org/wiki/Date_format_by_country
Recommended reading order
🚨 User Impact 🚨
Pre-merge Checklist
Updating a connector
Community member or Airbyter
airbyte_secret./gradlew :airbyte-integrations:connectors:<name>:integrationTest.README.mdbootstrap.md. See description and examplesdocs/integrations/<source or destination>/<name>.mdincluding changelog. See changelog exampleAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>command is passing./publishcommand described here