Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

source-delighted: Unix changed to normal timestamp #13439

Merged
merged 11 commits into from
Jun 10, 2022
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,7 @@
- name: Delighted
sourceDefinitionId: cc88c43f-6f53-4e8a-8c4d-b284baaf9635
dockerRepository: airbyte/source-delighted
dockerImageTag: 0.1.3
dockerImageTag: 0.1.4
documentationUrl: https://docs.airbyte.io/integrations/sources/delighted
icon: delighted.svg
sourceType: api
Expand Down
14 changes: 9 additions & 5 deletions airbyte-config/init/src/main/resources/seed/source_specs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1546,7 +1546,7 @@
supportsNormalization: false
supportsDBT: false
supported_destination_sync_modes: []
- dockerImage: "airbyte/source-delighted:0.1.3"
- dockerImage: "airbyte/source-delighted:0.1.4"
spec:
documentationUrl: "https://docsurl.com"
connectionSpecification:
Expand All @@ -1559,15 +1559,19 @@
additionalProperties: false
properties:
since:
type: "integer"
description: "An Unix timestamp to retrieve records created on or after\
\ this time."
title: "Since"
type: "string"
description: "The date from which you'd like to replicate the data"
examples:
- 1625328167
- "2022-05-30 04:50:23"
pattern: "^[0-9]{4}-[0-9]{2}-[0-9]{2} ([0-9]{2}:[0-9]{2}:[0-9]{2})?$"
order: 0
api_key:
title: "Delighted API Key"
type: "string"
description: "A Delighted API key."
airbyte_secret: true
order: 1
supportsNormalization: false
supportsDBT: false
supported_destination_sync_modes: []
Expand Down
1 change: 1 addition & 0 deletions airbyte-integrations/builds.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
| Chartmogul | [![source-chartmogul](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-chartmogul%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-chartmogul/) |
| Cart.com | [![source-cart](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-cart%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-cart/) |
| Close.com | [![source-close-com](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-close-com%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-close-com/) |
| Delighted | [![source-delighted](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-delighted%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-delighted) |
| Dixa | [![source-dixa](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-dixa%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-dixa) |
| Drift | [![source-drift](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-drift%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-drift) |
| End-to-End Testing | [![source-e2e-test](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-e2e-test%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-e2e-test) |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,5 @@ RUN pip install .
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=0.1.3
LABEL io.airbyte.version=0.1.4
LABEL io.airbyte.name=airbyte/source-delighted
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{
"api_key": "wrong api key",
"since": 1625328197
"since": "2022-01-01 00:00:00"
}
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
from typing import Any, Iterable, List, Mapping, MutableMapping, Optional, Tuple
from urllib.parse import parse_qsl, urlparse

import pendulum
import requests
from airbyte_cdk.models import SyncMode
from airbyte_cdk.sources import AbstractSource
Expand All @@ -27,10 +28,14 @@ class DelightedStream(HttpStream, ABC):
# Define primary key to all streams as primary key
primary_key = "id"

def __init__(self, since: int, **kwargs):
def __init__(self, since: pendulum.datetime, **kwargs):
super().__init__(**kwargs)
self.since = since

@property
def since_ts(self) -> int:
return int(self.since.timestamp())

def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]:
response_data = response.json()
if len(response_data) == self.limit:
Expand All @@ -40,7 +45,7 @@ def next_page_token(self, response: requests.Response) -> Optional[Mapping[str,
def request_params(
self, stream_state: Mapping[str, Any], stream_slice: Mapping[str, any] = None, next_page_token: Mapping[str, Any] = None
) -> MutableMapping[str, Any]:
params = {"per_page": self.limit, "since": self.since}
params = {"per_page": self.limit, "since": self.since_ts}
if next_page_token:
params.update(**next_page_token)
return params
Expand Down Expand Up @@ -157,8 +162,7 @@ def check_connection(self, logger, config) -> Tuple[bool, any]:

try:
auth = self._get_authenticator(config)
args = {"authenticator": auth, "since": config["since"]}
stream = SurveyResponses(**args)
stream = SurveyResponses(authenticator=auth, since=pendulum.parse(config["since"]))
records = stream.read_records(sync_mode=SyncMode.full_refresh)
next(records)
return True, None
Expand All @@ -167,10 +171,10 @@ def check_connection(self, logger, config) -> Tuple[bool, any]:

def streams(self, config: Mapping[str, Any]) -> List[Stream]:
auth = self._get_authenticator(config)
args = {"authenticator": auth, "since": config["since"]}
stream_kwargs = {"authenticator": auth, "since": pendulum.parse(config["since"])}
return [
Bounces(**args),
People(**args),
SurveyResponses(**args),
Unsubscribes(**args),
Bounces(**stream_kwargs),
People(**stream_kwargs),
SurveyResponses(**stream_kwargs),
Unsubscribes(**stream_kwargs),
]
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,19 @@
"additionalProperties": false,
"properties": {
"since": {
"type": "integer",
"description": "An Unix timestamp to retrieve records created on or after this time.",
"examples": [1625328167]
"title": "Since",
"type": "string",
"description": "The date from which you'd like to replicate the data",
"examples": ["2022-05-30 04:50:23"],
"pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2} ([0-9]{2}:[0-9]{2}:[0-9]{2})?$",
"order": 0
},
"api_key": {
"title": "Delighted API Key",
"type": "string",
"description": "A Delighted API key.",
"airbyte_secret": true
"airbyte_secret": true,
"order": 1
}
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
#

import pendulum
import pytest
import responses
from airbyte_cdk.models import SyncMode
Expand All @@ -12,7 +13,7 @@
def test_config():
return {
"api_key": "test_api_key",
"since": "1641289584",
"since": "2022-01-01 00:00:00",
}


Expand Down Expand Up @@ -74,7 +75,7 @@ def test_not_output_records_where_cursor_field_equals_state(state, test_config,
status=200,
)

stream = stream_class(test_config["since"], authenticator=SourceDelighted()._get_authenticator(config=test_config))
stream = stream_class(pendulum.parse(test_config["since"]), authenticator=SourceDelighted()._get_authenticator(config=test_config))
records = [r for r in stream.read_records(SyncMode.incremental, stream_state=state[stream.name])]
assert not records

Expand Down
1 change: 1 addition & 0 deletions docs/integrations/sources/delighted.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ This connector supports `API PASSWORD` as the authentication method.

| Version | Date | Pull Request | Subject |
| :--- | :--- | :--- | :--- |
| 0.1.4 | 2022-06-10 | [13439](https://github.com/airbytehq/airbyte/pull/13439) | Change since parameter input to iso date |
| 0.1.3 | 2022-01-31 | [9550](https://github.com/airbytehq/airbyte/pull/9550) | Output only records in which cursor field is greater than the value in state for incremental streams |
| 0.1.2 | 2022-01-06 | [9333](https://github.com/airbytehq/airbyte/pull/9333) | Add incremental sync mode to streams in `integration_tests/configured_catalog.json` |
| 0.1.1 | 2022-01-04 | [9275](https://github.com/airbytehq/airbyte/pull/9275) | Fix pagination handling for `survey_responses`, `bounces` and `unsubscribes` streams |
Expand Down