-
Notifications
You must be signed in to change notification settings - Fork 4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🐛Source Amazon Seller Partner: add integration tests #33996
Merged
askarpets
merged 30 commits into
master
from
source-amazon-seller-partner-integration-tests
Feb 13, 2024
Merged
Changes from 25 commits
Commits
Show all changes
30 commits
Select commit
Hold shift + click to select a range
3c0d898
Source Amazon Seller Partner: add integration tests
097ba8a
Add first test for VendorDirectFulfillmentShipping
027c137
Merge branch 'master' into source-amazon-seller-partner-integration-t…
56a2656
Merge branch 'master' into source-amazon-seller-partner-integration-t…
44b5775
Add full refresh mode tests for report-based streams
04ea558
Fix migration test configs
dd009aa
Enable brand analytics streams
d3b9eb0
Merge branch 'master' into source-amazon-seller-partner-integration-t…
5c5826a
Merge branch 'master' into source-amazon-seller-partner-integration-t…
60aa7f5
Replace subTest with pytest parametrize
2f3b21a
Merge branch 'master' into source-amazon-seller-partner-integration-t…
0f4f7be
Add warnings assertion
d4661c6
Merge branch 'master' into source-amazon-seller-partner-integration-t…
7cf2366
Add TestIncremental for report based streams
fcd83a7
Merge branch 'master' into source-amazon-seller-partner-integration-t…
4105076
Add tests for VendorDirectFulfillmentShipping stream
847a407
Add tests for VendorDirectFulfillmentShipping stream
7d1a654
Merge branch 'master' into source-amazon-seller-partner-integration-t…
5919883
Merge branch 'master' into source-amazon-seller-partner-integration-t…
2c4b7dd
Update docs
7441e58
Merge branch 'master' into source-amazon-seller-partner-integration-t…
78c8f8b
Updates per comments
64dd6bb
Updates per comments
e800ce2
Update assert_message_in_log_output method
566f16c
Update assert_message_in_log_output method
076a61f
Merge branch 'master' into source-amazon-seller-partner-integration-t…
59b7598
Update bypass reason for empty streams
b122846
Merge branch 'master' into source-amazon-seller-partner-integration-t…
c40db8e
Small refactor
2d75cba
Update expected records
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Empty file.
46 changes: 46 additions & 0 deletions
46
...yte-integrations/connectors/source-amazon-seller-partner/unit_tests/integration/config.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,46 @@ | ||
# | ||
# Copyright (c) 2024 Airbyte, Inc., all rights reserved. | ||
# | ||
|
||
|
||
from __future__ import annotations | ||
|
||
from datetime import datetime | ||
from typing import Dict | ||
|
||
import pendulum | ||
|
||
ACCESS_TOKEN = "test_access_token" | ||
LWA_APP_ID = "amazon_app_id" | ||
LWA_CLIENT_SECRET = "amazon_client_secret" | ||
MARKETPLACE_ID = "ATVPDKIKX0DER" | ||
REFRESH_TOKEN = "amazon_refresh_token" | ||
|
||
CONFIG_START_DATE = "2023-01-01T00:00:00Z" | ||
CONFIG_END_DATE = "2023-01-30T00:00:00Z" | ||
NOW = pendulum.now(tz="utc") | ||
|
||
|
||
class ConfigBuilder: | ||
def __init__(self) -> None: | ||
self._config: Dict[str, str] = { | ||
"refresh_token": REFRESH_TOKEN, | ||
"lwa_app_id": LWA_APP_ID, | ||
"lwa_client_secret": LWA_CLIENT_SECRET, | ||
"replication_start_date": CONFIG_START_DATE, | ||
"replication_end_date": CONFIG_END_DATE, | ||
"aws_environment": "PRODUCTION", | ||
"region": "US", | ||
"account_type": "Seller", | ||
} | ||
|
||
def with_start_date(self, start_date: datetime) -> ConfigBuilder: | ||
self._config["replication_start_date"] = start_date.isoformat()[:-13] + "Z" | ||
return self | ||
|
||
def with_end_date(self, end_date: datetime) -> ConfigBuilder: | ||
self._config["replication_end_date"] = end_date.isoformat()[:-13] + "Z" | ||
return self | ||
|
||
def build(self) -> Dict[str, str]: | ||
return self._config |
16 changes: 16 additions & 0 deletions
16
...integrations/connectors/source-amazon-seller-partner/unit_tests/integration/pagination.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
# | ||
# Copyright (c) 2024 Airbyte, Inc., all rights reserved. | ||
# | ||
|
||
|
||
from typing import Any, Dict | ||
|
||
from airbyte_cdk.test.mock_http.response_builder import PaginationStrategy | ||
|
||
NEXT_TOKEN_STRING = "MDAwMDAwMDAwMQ==" | ||
|
||
|
||
class VendorDirectFulfillmentShippingPaginationStrategy(PaginationStrategy): | ||
def update(self, response: Dict[str, Any]) -> None: | ||
response["payload"]["pagination"] = {} | ||
response["payload"]["pagination"]["nextToken"] = NEXT_TOKEN_STRING |
83 changes: 83 additions & 0 deletions
83
...rations/connectors/source-amazon-seller-partner/unit_tests/integration/request_builder.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,83 @@ | ||
# | ||
# Copyright (c) 2024 Airbyte, Inc., all rights reserved. | ||
# | ||
|
||
|
||
from __future__ import annotations | ||
|
||
import json | ||
from typing import Any, List, Mapping, Optional, Union | ||
|
||
from airbyte_cdk.test.mock_http.request import ANY_QUERY_PARAMS, HttpRequest | ||
|
||
from .config import ACCESS_TOKEN, LWA_APP_ID, LWA_CLIENT_SECRET, MARKETPLACE_ID, NOW, REFRESH_TOKEN | ||
|
||
|
||
class RequestBuilder: | ||
|
||
@classmethod | ||
def auth_endpoint(cls) -> RequestBuilder: | ||
request_headers = {"Content-Type": "application/x-www-form-urlencoded"} | ||
request_body = f"grant_type=refresh_token&client_id={LWA_APP_ID}&client_secret={LWA_CLIENT_SECRET}&refresh_token={REFRESH_TOKEN}" | ||
return cls("auth/o2/token").with_base_url("https://api.amazon.com").with_headers(request_headers).with_body(request_body) | ||
|
||
@classmethod | ||
def create_report_endpoint(cls, report_name: str) -> RequestBuilder: | ||
request_body = { | ||
"reportType": report_name, | ||
"marketplaceIds": [MARKETPLACE_ID], | ||
"dataStartTime": "2023-01-01T00:00:00Z", | ||
"dataEndTime": "2023-01-30T00:00:00Z", | ||
} | ||
return cls("reports/2021-06-30/reports").with_body(json.dumps(request_body)) | ||
|
||
@classmethod | ||
def check_report_status_endpoint(cls, report_id: str) -> RequestBuilder: | ||
return cls(f"reports/2021-06-30/reports/{report_id}") | ||
|
||
@classmethod | ||
def get_document_download_url_endpoint(cls, document_id: str) -> RequestBuilder: | ||
return cls(f"reports/2021-06-30/documents/{document_id}") | ||
|
||
@classmethod | ||
def download_document_endpoint(cls, url: str) -> RequestBuilder: | ||
return cls("").with_base_url(url).with_headers(None) | ||
|
||
@classmethod | ||
def vendor_direct_fulfillment_shipping_endpoint(cls) -> RequestBuilder: | ||
return cls("vendor/directFulfillment/shipping/v1/shippingLabels") | ||
|
||
def __init__(self, resource: str) -> None: | ||
self._resource = resource | ||
self._base_url = "https://sellingpartnerapi-na.amazon.com" | ||
self._headers = { | ||
"content-type": "application/json", | ||
"host": self._base_url.replace("https://", ""), | ||
"user-agent": "python-requests", | ||
"x-amz-access-token": ACCESS_TOKEN, | ||
"x-amz-date": NOW.strftime("%Y%m%dT%H%M%SZ"), | ||
} | ||
self._query_params = ANY_QUERY_PARAMS | ||
self._body = None | ||
|
||
def with_base_url(self, base_url: str) -> RequestBuilder: | ||
self._base_url = base_url | ||
return self | ||
|
||
def with_headers(self, headers: Optional[Union[str, Mapping[str, str]]]) -> RequestBuilder: | ||
self._headers = headers | ||
return self | ||
|
||
def with_query_params(self, query_params: Union[str, Mapping[str, Union[str, List[str]]]]) -> RequestBuilder: | ||
self._query_params = query_params | ||
return self | ||
|
||
def with_body(self, body: Union[str, bytes, Mapping[str, Any]]) -> RequestBuilder: | ||
self._body = body | ||
return self | ||
|
||
def _url(self) -> str: | ||
return f"{self._base_url}/{self._resource}" if self._resource else self._base_url | ||
|
||
def build(self) -> HttpRequest: | ||
return HttpRequest(url=self._url(), query_params=self._query_params, headers=self._headers, body=self._body) |
19 changes: 19 additions & 0 deletions
19
...ations/connectors/source-amazon-seller-partner/unit_tests/integration/response_builder.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
# | ||
# Copyright (c) 2024 Airbyte, Inc., all rights reserved. | ||
# | ||
|
||
|
||
import json | ||
from http import HTTPStatus | ||
from typing import Any, Mapping, Optional | ||
|
||
from airbyte_cdk.test.mock_http import HttpResponse | ||
|
||
|
||
def response_with_status(status_code: HTTPStatus, body: Optional[Mapping[str, Any]] = None) -> HttpResponse: | ||
body = body or {} | ||
return HttpResponse(body=json.dumps(body), status_code=status_code) | ||
|
||
|
||
def build_response(body: Mapping[str, Any], status_code: HTTPStatus) -> HttpResponse: | ||
return HttpResponse(body=json.dumps(body), status_code=status_code) |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we need this? I think we don't have this for stripe. The argument should be added like this
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a sort of workaround. Since I use
pytest
to run the tests, when passing thehttp_mocker
param to a test,pytest
tries to find a fixture with such name instead of just accepting the param from@HttpMocker
. I had to add this "empty" fixture because didn't find a better solution to make it work.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pytest is so invasive :(