-
Notifications
You must be signed in to change notification settings - Fork 4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
✨ Source Google-Ads: add incremental streams based on events #28970
✨ Source Google-Ads: add incremental streams based on events #28970
Conversation
Before Merging a Connector Pull RequestWow! What a great pull request you have here! 🎉 To merge this PR, ensure the following has been done/considered for each connector added or updated:
If the checklist is complete, but the CI check is failing,
|
6ffea1b
to
86d55b6
Compare
255c337
to
d255428
Compare
3afaa57
to
71de5c2
Compare
45c9a0b
to
26315fd
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @tolik0 👋🏼 I left inline comments about the PKs and FKs. But my main outstanding question is how IncrementalEvents
are supposed to work
airbyte-integrations/connectors/source-google-ads/acceptance-test-config.yml
Outdated
Show resolved
Hide resolved
airbyte-integrations/connectors/source-google-ads/acceptance-test-config.yml
Outdated
Show resolved
Hide resolved
airbyte-integrations/connectors/source-google-ads/acceptance-test-config.yml
Outdated
Show resolved
Hide resolved
airbyte-integrations/connectors/source-google-ads/source_google_ads/schemas/ad_group_ads.json
Outdated
Show resolved
Hide resolved
airbyte-integrations/connectors/source-google-ads/source_google_ads/streams.py
Outdated
Show resolved
Hide resolved
airbyte-integrations/connectors/source-google-ads/source_google_ads/source.py
Outdated
Show resolved
Hide resolved
airbyte-integrations/connectors/source-google-ads/source_google_ads/streams.py
Outdated
Show resolved
Hide resolved
airbyte-integrations/connectors/source-google-ads/source_google_ads/streams.py
Show resolved
Hide resolved
airbyte-integrations/connectors/source-google-ads/source_google_ads/streams.py
Outdated
Show resolved
Hide resolved
airbyte-integrations/connectors/source-google-ads/unit_tests/test_source.py
Show resolved
Hide resolved
a91e2a6
to
67ec032
Compare
b56e6ca
to
d3e264b
Compare
source-google-ads test report (commit
|
Step | Result |
---|---|
Connector package install | ✅ |
Build source-google-ads docker image for platform linux/x86_64 | ✅ |
Unit tests | ❌ |
Code format checks | ❌ |
Validate airbyte-integrations/connectors/source-google-ads/metadata.yaml | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-google-ads test
source-google-ads test report (commit
|
Step | Result |
---|---|
Connector package install | ✅ |
Build source-google-ads docker image for platform linux/x86_64 | ✅ |
Unit tests | ✅ |
Acceptance tests | ❌ |
Code format checks | ❌ |
Validate airbyte-integrations/connectors/source-google-ads/metadata.yaml | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-google-ads test
source-google-ads test report (commit
|
Step | Result |
---|---|
Connector package install | ✅ |
Build source-google-ads docker image for platform linux/x86_64 | ✅ |
Unit tests | ✅ |
Acceptance tests | ✅ |
Code format checks | ✅ |
Validate airbyte-integrations/connectors/source-google-ads/metadata.yaml | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-google-ads test
source-google-ads test report (commit
|
Step | Result |
---|---|
Connector package install | ✅ |
Build source-google-ads docker image for platform linux/x86_64 | ✅ |
Unit tests | ✅ |
Acceptance tests | ✅ |
Code format checks | ✅ |
Validate airbyte-integrations/connectors/source-google-ads/metadata.yaml | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-google-ads test
airbyte-integrations/connectors/source-google-ads/acceptance-test-config.yml
Outdated
Show resolved
Hide resolved
airbyte-integrations/connectors/source-google-ads/source_google_ads/google_ads.py
Outdated
Show resolved
Hide resolved
airbyte-integrations/connectors/source-google-ads/source_google_ads/google_ads.py
Outdated
Show resolved
Hide resolved
airbyte-integrations/connectors/source-google-ads/source_google_ads/schemas/accounts.json
Outdated
Show resolved
Hide resolved
...integrations/connectors/source-google-ads/source_google_ads/schemas/ad_group_criterions.json
Outdated
Show resolved
Hide resolved
airbyte-integrations/connectors/source-google-ads/source_google_ads/source.py
Outdated
Show resolved
Hide resolved
airbyte-integrations/connectors/source-google-ads/source_google_ads/streams.py
Show resolved
Hide resolved
airbyte-integrations/connectors/source-google-ads/source_google_ads/streams.py
Outdated
Show resolved
Hide resolved
source-google-ads test report (commit
|
Step | Result |
---|---|
Connector package install | ✅ |
Build source-google-ads docker image for platform linux/x86_64 | ✅ |
Unit tests | ✅ |
Acceptance tests | ✅ |
Code format checks | ✅ |
Validate airbyte-integrations/connectors/source-google-ads/metadata.yaml | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-google-ads test
@tolik0 could you address the point about the 10k limit above?
|
@sherifnada I fixed 10k records for a day by decreasing the granularity of the cursor to microseconds. I also updated the corresponding documentation.
|
Does this support : #11539 ? |
airbyte-integrations/connectors/source-google-ads/source_google_ads/google_ads.py
Outdated
Show resolved
Hide resolved
airbyte-integrations/connectors/source-google-ads/source_google_ads/google_ads.py
Outdated
Show resolved
Hide resolved
...integrations/connectors/source-google-ads/source_google_ads/schemas/ad_group_criterions.json
Outdated
Show resolved
Hide resolved
...-integrations/connectors/source-google-ads/source_google_ads/schemas/campaign_criterion.json
Outdated
Show resolved
Hide resolved
airbyte-integrations/connectors/source-google-ads/source_google_ads/streams.py
Show resolved
Hide resolved
airbyte-integrations/connectors/source-google-ads/source_google_ads/streams.py
Show resolved
Hide resolved
airbyte-integrations/connectors/source-google-ads/source_google_ads/streams.py
Show resolved
Hide resolved
airbyte-integrations/connectors/source-google-ads/source_google_ads/streams.py
Show resolved
Hide resolved
airbyte-integrations/connectors/source-google-ads/source_google_ads/streams.py
Show resolved
Hide resolved
airbyte-integrations/connectors/source-google-ads/unit_tests/test_incremental_events_streams.py
Show resolved
Hide resolved
source-google-ads test report (commit
|
Step | Result |
---|---|
Connector package install | ✅ |
Build source-google-ads docker image for platform linux/x86_64 | ✅ |
Unit tests | ✅ |
Acceptance tests | ❌ |
Code format checks | ❌ |
Validate airbyte-integrations/connectors/source-google-ads/metadata.yaml | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-google-ads test
airbyte-integrations/connectors/source-google-ads/source_google_ads/streams.py
Outdated
Show resolved
Hide resolved
source-google-ads test report (commit
|
Step | Result |
---|---|
Connector package install | ✅ |
Build source-google-ads docker image for platform linux/x86_64 | ✅ |
Unit tests | ✅ |
Acceptance tests | ✅ |
Code format checks | ❌ |
Validate airbyte-integrations/connectors/source-google-ads/metadata.yaml | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-google-ads test
source-google-ads test report (commit
|
Step | Result |
---|---|
Connector package install | ✅ |
Build source-google-ads docker image for platform linux/x86_64 | ✅ |
Unit tests | ✅ |
Acceptance tests | ✅ |
Code format checks | ✅ |
Validate airbyte-integrations/connectors/source-google-ads/metadata.yaml | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-google-ads test
source-google-ads test report (commit
|
Step | Result |
---|---|
Connector package install | ✅ |
Build source-google-ads docker image for platform linux/x86_64 | ✅ |
Unit tests | ✅ |
Acceptance tests | ✅ |
Code format checks | ✅ |
Validate airbyte-integrations/connectors/source-google-ads/metadata.yaml | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-google-ads test
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
What
Resolve:
How
Incremental sync for Ad Group and Campaign Criterion streams is implemented using the Change Status stream. If state is empty sync is full refresh, otherwise updates are synced from change_status resource then updated records are synced using id and deleted records returned with empty fields.
Recommended reading order
source-google-ads/source.py
source-google-ads/streams.py
🚨 User Impact 🚨
No breaking changes