Source Amazon Seller Partner: extract REPORTS_MAX_WAIT_SECONDS to configurable parameter

[lizdeika]

What

Extract the REPORTS_MAX_WAIT_SECONDS constant to configurable parameter

How

Recommended reading order

1. streams.py
2. source.py

🚨 User Impact 🚨

Users will be able to configure max wait for reports by their will.

Pre-merge Checklist

Updating a connector

Community member or Airbyter

• Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• Credentials added to Github CI. Instructions.
• /test connector=connectors/<name> command is passing.
• New Connector version released on Dockerhub by running the /publish command described here
• After the new connector version is published, connector version bumped in the seed directory as described here
• Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

[lizdeika]

Test results

» python -m pytest integration_tests -p integration_tests.acceptance
Test session starts (platform: darwin, Python 3.9.8, pytest 6.2.5, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/lizdeika/Projects/airbyte, configfile: pytest.ini
plugins: sugar-0.9.4, mock-3.6.1, timeout-1.4.2
collecting ...
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_match_expected[inputs0] ✓              9% ▉
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_required[inputs0] ✓                   18% █▊
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_optional[inputs0] ✓                   27% ██▊
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_has_secret[inputs0] ✓                 36% ███▋
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_secret_never_in_the_output[inputs0] ✓ 45% ████▋
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_defined_refs_exist_in_json_spec_file[inputs0] ✓55% █████▌
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_oauth_flow_parameters[inputs0] ✓      64% ██████▍
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection.test_check[inputs0] ✓                73% ███████▍
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_discover[inputs0] ✓              82% ████████▎
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_cursors_exist_in_schema[inputs0] ✓91% █████████▏
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_refs_exist_in_schema[inputs0] ✓100% ██████████
=================================================================== short test summary info ====================================================================
SKIPPED [1] ../../bases/source-acceptance-test/source_acceptance_test/plugin.py:56: Skipping TestBasicRead.test_read because not found in the config
SKIPPED [1] ../../bases/source-acceptance-test/source_acceptance_test/plugin.py:56: Skipping TestFullRefresh.test_sequential_reads because not found in the config
SKIPPED [1] ../../bases/source-acceptance-test/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config

Results (11.72s):
      11 passed

» python -m pytest integration_tests
Test session starts (platform: darwin, Python 3.9.8, pytest 6.2.5, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/lizdeika/Projects/airbyte, configfile: pytest.ini
plugins: sugar-0.9.4, mock-3.6.1, timeout-1.4.2
collecting ...

Results (0.04s):

» python -m pytest unit_tests
Test session starts (platform: darwin, Python 3.9.8, pytest 6.2.5, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/lizdeika/Projects/airbyte, configfile: pytest.ini
plugins: sugar-0.9.4, mock-3.6.1, timeout-1.4.2
collecting ...
 airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_repots_streams_rate_limits.py::test_reports_stream_should_retry ✓   33% ███▍
 airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_repots_streams_rate_limits.py::test_reports_stream_send_request ✓   67% ██████▋   {"type": "LOG", "log": {"level": "INFO", "message": "Backing off _send_request(...) for 5.0s (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException)"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Caught retryable error '' after 1 tries. Waiting 5 seconds then retrying..."}}
{"type": "LOG", "log": {"level": "INFO", "message": "Backing off _send_request(...) for 10.0s (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException)"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Caught retryable error '' after 2 tries. Waiting 10 seconds then retrying..."}}
{"type": "LOG", "log": {"level": "INFO", "message": "Backing off _send_request(...) for 20.0s (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException)"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Caught retryable error '' after 3 tries. Waiting 20 seconds then retrying..."}}
{"type": "LOG", "log": {"level": "INFO", "message": "Backing off _send_request(...) for 40.0s (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException)"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Caught retryable error '' after 4 tries. Waiting 40 seconds then retrying..."}}
{"type": "LOG", "log": {"level": "ERROR", "message": "Giving up _send_request(...) after 5 tries (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException)"}}

 airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_repots_streams_rate_limits.py::test_reports_stream_send_request_backoff_exception ✓100% ██████████
======================================================================= warnings summary =======================================================================
source_amazon_seller_partner/streams.py:155
  /Users/lizdeika/Projects/airbyte/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/streams.py:155: DeprecationWarning: Call to deprecated class NoAuth. (Set `authenticator=None` instead) -- Deprecated since version 0.1.20.
    authenticator: HttpAuthenticator = NoAuth(),

.venv/lib/python3.9/site-packages/deprecated/classic.py:173
airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_repots_streams_rate_limits.py::test_reports_stream_should_retry
airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_repots_streams_rate_limits.py::test_reports_stream_send_request
airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_repots_streams_rate_limits.py::test_reports_stream_send_request_backoff_exception
  /Users/lizdeika/Projects/airbyte/airbyte-integrations/connectors/source-amazon-seller-partner/.venv/lib/python3.9/site-packages/deprecated/classic.py:173: DeprecationWarning: Call to deprecated class HttpAuthenticator. (Use requests.auth.AuthBase instead) -- Deprecated since version 0.1.20.
    return old_new1(cls, *args, **kwargs)

airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_repots_streams_rate_limits.py::test_reports_stream_should_retry
airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_repots_streams_rate_limits.py::test_reports_stream_send_request
airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_repots_streams_rate_limits.py::test_reports_stream_send_request_backoff_exception
  /Users/lizdeika/Projects/airbyte/airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_repots_streams_rate_limits.py:29: DeprecationWarning: Call to deprecated class NoAuth. (Set `authenticator=None` instead) -- Deprecated since version 0.1.20.
    authenticator=NoAuth(),

-- Docs: https://docs.pytest.org/en/stable/warnings.html

Results (0.64s):
       3 passed

[alafanechere]

Thanks @lizdeika for this addition! I'll review this before end of the week!