✨ Source Google-Ads: add incremental streams based on events #28970

tolik0 · 2023-08-02T12:09:01Z

What

Resolve:

Add new streams: Change Status and Campaign Criterion
Implement Incremental sync and delete for Ad Group and Campaign Criterion streams

How

Incremental sync for Ad Group and Campaign Criterion streams is implemented using the Change Status stream. If state is empty sync is full refresh, otherwise updates are synced from change_status resource then updated records are synced using id and deleted records returned with empty fields.

🚨 User Impact 🚨

No breaking changes

github-actions · 2023-08-02T12:09:25Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

sherifnada

Hi @tolik0 👋🏼 I left inline comments about the PKs and FKs. But my main outstanding question is how IncrementalEvents are supposed to work

airbyte-integrations/connectors/source-google-ads/acceptance-test-config.yml

airbyte-integrations/connectors/source-google-ads/source_google_ads/schemas/ad_group_ads.json

airbyte-integrations/connectors/source-google-ads/source_google_ads/streams.py

airbyte-integrations/connectors/source-google-ads/source_google_ads/source.py

airbyte-integrations/connectors/source-google-ads/source_google_ads/streams.py

airbyte-integrations/connectors/source-google-ads/unit_tests/test_source.py

github-actions · 2023-09-07T16:41:12Z

source-google-ads test report (commit `9f29c96d17`) - ❌

⏲️ Total pipeline duration: 01mn51s

Step	Result
Connector package install	✅
Build source-google-ads docker image for platform linux/x86_64	✅
Unit tests	❌
Code format checks	❌
Validate airbyte-integrations/connectors/source-google-ads/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-google-ads test

github-actions · 2023-09-08T10:33:00Z

source-google-ads test report (commit `10dc3ce887`) - ❌

⏲️ Total pipeline duration: 07mn21s

Step	Result
Connector package install	✅
Build source-google-ads docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	❌
Code format checks	❌
Validate airbyte-integrations/connectors/source-google-ads/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-google-ads test

github-actions · 2023-09-08T16:26:53Z

source-google-ads test report (commit `cc82662b7d`) - ✅

⏲️ Total pipeline duration: 07mn58s

Step	Result
Connector package install	✅
Build source-google-ads docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-google-ads/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-google-ads test

github-actions · 2023-09-08T16:54:17Z

source-google-ads test report (commit `02654726a6`) - ✅

⏲️ Total pipeline duration: 02mn59s

Step	Result
Connector package install	✅
Build source-google-ads docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-google-ads/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-google-ads test

airbyte-integrations/connectors/source-google-ads/acceptance-test-config.yml

airbyte-integrations/connectors/source-google-ads/source_google_ads/google_ads.py

airbyte-integrations/connectors/source-google-ads/source_google_ads/schemas/accounts.json

...integrations/connectors/source-google-ads/source_google_ads/schemas/ad_group_criterions.json

airbyte-integrations/connectors/source-google-ads/source_google_ads/source.py

airbyte-integrations/connectors/source-google-ads/source_google_ads/streams.py

github-actions · 2023-09-14T21:21:29Z

source-google-ads test report (commit `277bbbfd4a`) - ✅

⏲️ Total pipeline duration: 11mn00s

Step	Result
Connector package install	✅
Build source-google-ads docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-google-ads/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-google-ads test

sherifnada · 2023-09-15T05:18:07Z

@tolik0 could you address the point about the 10k limit above?

From a user point of view, it's not an acceptable outcome to say that there will be data loss. If 10k is the maximum then we'll need to find a way around that e.g: make a request for data after date t and if it's 10k records then we make another request with date = max(records_returned_in_first_request) and so on and so forth

tolik0 · 2023-09-15T08:12:06Z

@sherifnada I fixed 10k records for a day by decreasing the granularity of the cursor to microseconds. I also updated the corresponding documentation.

@tolik0 could you address the point about the 10k limit above?

From a user point of view, it's not an acceptable outcome to say that there will be data loss. If 10k is the maximum then we'll need to find a way around that e.g: make a request for data after date t and if it's 10k records then we make another request with date = max(records_returned_in_first_request) and so on and so forth

katmarkham · 2023-09-16T02:04:06Z

Does this support : #11539 ?

airbyte-integrations/connectors/source-google-ads/source_google_ads/google_ads.py

...integrations/connectors/source-google-ads/source_google_ads/schemas/ad_group_criterions.json

...-integrations/connectors/source-google-ads/source_google_ads/schemas/campaign_criterion.json

airbyte-integrations/connectors/source-google-ads/source_google_ads/streams.py

airbyte-integrations/connectors/source-google-ads/unit_tests/test_incremental_events_streams.py

github-actions · 2023-09-19T02:18:39Z

source-google-ads test report (commit `2b915ff792`) - ❌

⏲️ Total pipeline duration: 10mn41s

Step	Result
Connector package install	✅
Build source-google-ads docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	❌
Code format checks	❌
Validate airbyte-integrations/connectors/source-google-ads/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-google-ads test

airbyte-integrations/connectors/source-google-ads/source_google_ads/streams.py

github-actions · 2023-09-19T09:11:23Z

source-google-ads test report (commit `aca1798d41`) - ❌

⏲️ Total pipeline duration: 07mn13s

Step	Result
Connector package install	✅
Build source-google-ads docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅
Code format checks	❌
Validate airbyte-integrations/connectors/source-google-ads/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-google-ads test

github-actions · 2023-09-19T09:29:43Z

source-google-ads test report (commit `5b34cfca13`) - ✅

⏲️ Total pipeline duration: 08mn06s

Step	Result
Connector package install	✅
Build source-google-ads docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-google-ads/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-google-ads test

github-actions · 2023-09-19T11:57:08Z

source-google-ads test report (commit `f53a0506ed`) - ✅

⏲️ Total pipeline duration: 07mn17s

Step	Result
Connector package install	✅
Build source-google-ads docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-google-ads/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-google-ads test

lazebnyi

LGTM

tolik0 linked an issue Aug 2, 2023 that may be closed by this pull request

[EPIC] Source Google Ads to Silver #28204

Closed

octavia-squidington-iii added area/connectors Connector related issues connectors/source/google-ads labels Aug 2, 2023

tolik0 force-pushed the tolik0/source-google-ads/28204-epic-source-google-ads-to-ga branch from 6ffea1b to 86d55b6 Compare August 2, 2023 12:27

tolik0 force-pushed the tolik0/source-google-ads/28204-epic-source-google-ads-to-ga branch from 255c337 to d255428 Compare August 15, 2023 12:20

tolik0 force-pushed the tolik0/source-google-ads/28204-epic-source-google-ads-to-ga branch from 3afaa57 to 71de5c2 Compare August 23, 2023 15:23

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Aug 23, 2023

tolik0 force-pushed the tolik0/source-google-ads/28204-epic-source-google-ads-to-ga branch from 45c9a0b to 26315fd Compare August 25, 2023 12:13

sherifnada suggested changes Aug 28, 2023

View reviewed changes

tolik0 changed the title ~~Source Google-Ads: move to next releaseStage~~ ✨ Source Google-Ads: add incremental streams based on events Sep 1, 2023

tolik0 force-pushed the tolik0/source-google-ads/28204-epic-source-google-ads-to-ga branch 3 times, most recently from a91e2a6 to 67ec032 Compare September 4, 2023 18:15

tolik0 requested a review from lazebnyi September 4, 2023 18:15

tolik0 force-pushed the tolik0/source-google-ads/28204-epic-source-google-ads-to-ga branch from b56e6ca to d3e264b Compare September 7, 2023 15:00

tolik0 requested a review from sherifnada September 7, 2023 16:36

tolik0 marked this pull request as ready for review September 7, 2023 16:36

tolik0 requested review from a team as code owners September 8, 2023 16:16

octavia-squidington-iii added CDK Connector Development Kit connectors/source/postgres labels Sep 8, 2023

tolik0 removed CDK Connector Development Kit connectors/source/postgres labels Sep 8, 2023

lazebnyi requested changes Sep 8, 2023

View reviewed changes

sherifnada removed the request for review from a team September 8, 2023 21:01

airbytehq deleted a comment from github-actions bot Sep 14, 2023

lazebnyi approved these changes Sep 15, 2023

View reviewed changes

lazebnyi requested a review from brianjlai September 15, 2023 19:53

sherifnada suggested changes Sep 17, 2023

View reviewed changes

Fix comments

8113642

tolik0 requested a review from lazebnyi September 19, 2023 00:22

Delete customer.id from criterions

2b915ff

tolik0 requested a review from sherifnada September 19, 2023 07:06

sherifnada reviewed Sep 19, 2023

View reviewed changes

airbyte-integrations/connectors/source-google-ads/source_google_ads/streams.py Outdated Show resolved Hide resolved

Fix handling query limit hit

aca1798

Fix formatting

5b34cfc

tolik0 requested a review from sherifnada September 19, 2023 09:19

airbytehq deleted a comment from github-actions bot Sep 19, 2023

Update docstring

f53a050

tolik0 requested review from sherifnada and removed request for sherifnada September 19, 2023 12:01

sherifnada approved these changes Sep 19, 2023

View reviewed changes

lazebnyi approved these changes Sep 19, 2023

View reviewed changes

tolik0 merged commit b2542d6 into master Sep 19, 2023
29 of 30 checks passed

tolik0 deleted the tolik0/source-google-ads/28204-epic-source-google-ads-to-ga branch September 19, 2023 12:13

dalebradman mentioned this pull request Sep 20, 2023

✨ Adding images and videos streams to TikTok Marketing connector tryarcane/airbyte#1

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ Source Google-Ads: add incremental streams based on events #28970

✨ Source Google-Ads: add incremental streams based on events #28970

tolik0 commented Aug 2, 2023 •

edited

github-actions bot commented Aug 2, 2023 •

edited by tolik0

sherifnada left a comment

github-actions bot commented Sep 7, 2023

github-actions bot commented Sep 8, 2023

github-actions bot commented Sep 8, 2023

github-actions bot commented Sep 8, 2023

github-actions bot commented Sep 14, 2023

sherifnada commented Sep 15, 2023

tolik0 commented Sep 15, 2023

katmarkham commented Sep 16, 2023

github-actions bot commented Sep 19, 2023

github-actions bot commented Sep 19, 2023

github-actions bot commented Sep 19, 2023

github-actions bot commented Sep 19, 2023

lazebnyi left a comment

✨ Source Google-Ads: add incremental streams based on events #28970

✨ Source Google-Ads: add incremental streams based on events #28970

Conversation

tolik0 commented Aug 2, 2023 • edited

What

How

Recommended reading order

🚨 User Impact 🚨

github-actions bot commented Aug 2, 2023 • edited by tolik0

Before Merging a Connector Pull Request

sherifnada left a comment

Choose a reason for hiding this comment

github-actions bot commented Sep 7, 2023

source-google-ads test report (commit 9f29c96d17) - ❌

github-actions bot commented Sep 8, 2023

source-google-ads test report (commit 10dc3ce887) - ❌

github-actions bot commented Sep 8, 2023

source-google-ads test report (commit cc82662b7d) - ✅

github-actions bot commented Sep 8, 2023

source-google-ads test report (commit 02654726a6) - ✅

github-actions bot commented Sep 14, 2023

source-google-ads test report (commit 277bbbfd4a) - ✅

sherifnada commented Sep 15, 2023

tolik0 commented Sep 15, 2023

katmarkham commented Sep 16, 2023

github-actions bot commented Sep 19, 2023

source-google-ads test report (commit 2b915ff792) - ❌

github-actions bot commented Sep 19, 2023

source-google-ads test report (commit aca1798d41) - ❌

github-actions bot commented Sep 19, 2023

source-google-ads test report (commit 5b34cfca13) - ✅

github-actions bot commented Sep 19, 2023

source-google-ads test report (commit f53a0506ed) - ✅

lazebnyi left a comment

Choose a reason for hiding this comment

tolik0 commented Aug 2, 2023 •

edited

github-actions bot commented Aug 2, 2023 •

edited by tolik0

source-google-ads test report (commit `9f29c96d17`) - ❌

source-google-ads test report (commit `10dc3ce887`) - ❌

source-google-ads test report (commit `cc82662b7d`) - ✅

source-google-ads test report (commit `02654726a6`) - ✅

source-google-ads test report (commit `277bbbfd4a`) - ✅

source-google-ads test report (commit `2b915ff792`) - ❌

source-google-ads test report (commit `aca1798d41`) - ❌

source-google-ads test report (commit `5b34cfca13`) - ✅

source-google-ads test report (commit `f53a0506ed`) - ✅