Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[source-tiktok] Remove audience reports hourly streams #20598

Merged
merged 6 commits into from Dec 19, 2022
Merged

[source-tiktok] Remove audience reports hourly streams #20598

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
27c85d1
get rid of audience report streams using hourly dimension
brianjlai Dec 16, 2022
473d0b7
bump version and docs changelog
brianjlai Dec 16, 2022
b8a9262
fix tests
brianjlai Dec 16, 2022
f9ed877
update backwards compatibility tests since some streams are no longer…
brianjlai Dec 17, 2022
9dd5afe
auto-bump connector version
octavia-squidington-iii Dec 18, 2022
e699cd1
Merge branch 'master' into brian/tiktok_remove_hourly_audience_reports
girarda Dec 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-tiktok-marketing/Dockerfile
View file
Open in desktop
Expand Up @@ -32,5 +32,5 @@ COPY source_tiktok_marketing ./source_tiktok_marketing
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=1.0.0
LABEL io.airbyte.version=1.0.1
LABEL io.airbyte.name=airbyte/source-tiktok-marketing
12 changes: 6 additions & 6 deletions airbyte-integrations/connectors/source-tiktok-marketing/acceptance-test-config.yml
View file
Open in desktop
Expand Up @@ -28,22 +28,22 @@ tests:
discovery:
- config_path: "secrets/prod_config.json"
backward_compatibility_tests_config:
disable_for_version: "0.1.17"
disable_for_version: "1.0.0"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing streams are considered a breaking change, however we need to do this because the streams aren't usable any more. We will communicate this to the customers attempting to use these non-working streams to reset or disable them.

- config_path: "secrets/prod_config_with_day_granularity.json"
backward_compatibility_tests_config:
disable_for_version: "0.1.17"
disable_for_version: "1.0.0"
- config_path: "secrets/prod_config_with_lifetime_granularity.json"
backward_compatibility_tests_config:
disable_for_version: "0.1.17"
disable_for_version: "1.0.0"
- config_path: "secrets/config.json"
backward_compatibility_tests_config:
disable_for_version: "0.1.17"
disable_for_version: "1.0.0"
- config_path: "secrets/new_config_prod.json"
backward_compatibility_tests_config:
disable_for_version: "0.1.17"
disable_for_version: "1.0.0"
- config_path: "secrets/config_oauth.json"
backward_compatibility_tests_config:
disable_for_version: "0.1.17"
disable_for_version: "1.0.0"

basic_read:
# New style streams (for >= 0.1.13):
Expand Down
28 changes: 0 additions & 28 deletions airbyte-integrations/connectors/source-tiktok-marketing/integration_tests/streams.json
View file
Open in desktop
Expand Up @@ -108,13 +108,6 @@
"json_schema": {},
"supported_sync_modes": ["full_refresh"]
},
{
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This streams.json I don't think is actually used in our acceptance test YAML but just keeping this up to date regardless

"name": "advertisers_audience_reports_hourly",
"json_schema": {},
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_cursor": true,
"default_cursor_field": ["stat_time_hour"]
},
{
"name": "advertisers_audience_reports_daily",
"json_schema": {},
Expand All @@ -127,41 +120,20 @@
"json_schema": {},
"supported_sync_modes": ["full_refresh"]
},
{
"name": "ads_audience_reports_hourly",
"json_schema": {},
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_cursor": true,
"default_cursor_field": ["stat_time_hour"]
},
{
"name": "ads_audience_reports_daily",
"json_schema": {},
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_cursor": true,
"default_cursor_field": ["stat_time_day"]
},
{
"name": "ad_group_audience_reports_hourly",
"json_schema": {},
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_cursor": true,
"default_cursor_field": ["stat_time_hour"]
},
{
"name": "ad_group_audience_reports_daily",
"json_schema": {},
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_cursor": true,
"default_cursor_field": ["stat_time_day"]
},
{
"name": "campaigns_audience_reports_by_country_hourly",
"json_schema": {},
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_cursor": true,
"default_cursor_field": ["stat_time_hour"]
},
{
"name": "campaigns_audience_reports_by_country_daily",
"json_schema": {},
Expand Down
15 changes: 10 additions & 5 deletions airbyte-integrations/connectors/source-tiktok-marketing/source_tiktok_marketing/source.py
View file
Open in desktop
Expand Up @@ -158,18 +158,23 @@ def streams(self, config: Mapping[str, Any]) -> List[Stream]:

# 2. Basic report streams:
reports = [AdsReports, AdGroupsReports, CampaignsReports]
audience_reports = [AdsAudienceReports, AdGroupAudienceReports, CampaignsAudienceReportsByCountry]
if is_production:
# 2.1 streams work only in prod env
reports.extend([AdvertisersReports, AdvertisersAudienceReports])
reports.append(AdvertisersReports)
audience_reports.append(AdvertisersAudienceReports)

for Report in reports:
for Granularity in [Hourly, Daily, Lifetime]:
streams.append(get_report_stream(Report, Granularity)(**args))

# 3. Audience report streams:
# Audience report supports lifetime metrics only at the ADVERTISER level (see 2.1).
for Report in [AdsAudienceReports, AdGroupAudienceReports, CampaignsAudienceReportsByCountry]:
for Granularity in [Hourly, Daily]:
streams.append(get_report_stream(Report, Granularity)(**args))
for Report in audience_reports:
# As per TikTok's documentation, audience reports only support daily (not hourly) time dimension for metrics
streams.append(get_report_stream(Report, Daily)(**args))

# Audience report supports lifetime metrics only at the ADVERTISER level (see 2.1).
if Report == AdvertisersAudienceReports:
streams.append(get_report_stream(Report, Lifetime)(**args))

return streams
4 changes: 2 additions & 2 deletions airbyte-integrations/connectors/source-tiktok-marketing/unit_tests/unit_test.py
View file
Open in desktop
Expand Up @@ -134,8 +134,8 @@ def test_random_items(prepared_prod_args):
@pytest.mark.parametrize(
"config, stream_len",
[
(PROD_CONFIG_FILE, 26),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have 4 less hourly streams (1 for each audience report) in prod accounts
We have 3 less hourly streams (advertiser audience report is prod only) for sandbox accounts.

(SANDBOX_CONFIG_FILE, 19),
(PROD_CONFIG_FILE, 22),
(SANDBOX_CONFIG_FILE, 16),
],
)
def test_source_streams(config, stream_len):
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/sources/tiktok-marketing.md
View file
Open in desktop
Expand Up @@ -550,6 +550,7 @@ The connector is restricted by [requests limitation](https://ads.tiktok.com/mark

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:---------------------------------------------------------|:----------------------------------------------------------------------------------------------|
| 1.0.1 | 2022-12-16 | [20598](https://github.com/airbytehq/airbyte/pull/20598) | Remove Audience Reports with Hourly granularity due to deprecated dimension. |
| 1.0.0 | 2022-12-05 | [19758](https://github.com/airbytehq/airbyte/pull/19758) | Convert `mobile_app_id` from integer to string in AudienceReport streams. |
| 0.1.17 | 2022-10-04 | [17557](https://github.com/airbytehq/airbyte/pull/17557) | Retry error 50002 |
| 0.1.16 | 2022-09-28 | [17326](https://github.com/airbytehq/airbyte/pull/17326) | Migrate to per-stream state |
Expand Down