🐛 Source Google Ads: Fix custom queries #33603

tolik0 · 2023-12-18T21:31:11Z

What

This PR addresses two key issues in the custom queries: automatic addition of segments.date in the query, which hindered some custom queries, and incorrect field type for DATE fields. The changes resolve the following issues:

How

This update introduces a config migration for existing custom queries to align with the new format. It enables the creation of custom queries without the automatic breakdown by time, maintaining compatibility with all previous query formats. Additionally, it corrects the field type for DATE fields by identifying and listing fields that are truly of type date. This correction addresses an issue caused by Google Ads not differentiating between date and datetime fields.

vercel · 2023-12-18T21:31:18Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 9, 2024 4:42pm

github-actions · 2023-12-18T21:31:40Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

github-actions · 2023-12-21T10:36:10Z

Warning

🚨 Connector code freeze is in effect until 2024-01-02. This PR is changing connector code. Please contact the current OC engineers if you want to merge this change to master.

artem1205 · 2023-12-22T11:20:33Z

airbyte-integrations/connectors/source-google-ads/setup.py

@@ -7,7 +7,7 @@

 # pin protobuf==3.20.0 as other versions may cause problems on different architectures
 # (see https://github.com/airbytehq/airbyte/issues/13580)
-MAIN_REQUIREMENTS = ["airbyte-cdk>=0.51.3", "google-ads==22.1.0", "protobuf", "pendulum"]
+MAIN_REQUIREMENTS = ["airbyte-cdk>=0.51.3", "google-ads==22.1.0", "protobuf", "pendulum>=3.0.0"]


would not this cause any problem?

pendulum was pinned to <3.0 in CDK
#33572

Pinned <3.0.0 to align with the CDK

alafanechere

@tolik0 I'd prefer that we merge this change in after code freeze.
Did you perform any sort of manual testing to make sure the migration work?
I'd suggest pre-releasing a version and manually test in on an internal cloud workspace to make sure no breaking change is introduced.

alafanechere · 2023-12-27T13:19:03Z

airbyte-integrations/connectors/source-google-ads/source_google_ads/config_migrations.py

+class MigrateCustomQuery:
+    """
+    This class stands for migrating the config at runtime,
+    while providing the backward compatibility when falling back to the previous source version.


Could you please share how backward compatibility is guaranteed?

I updated the docs. Backward compatibility is achieved by creating a new property instead of editing an existing one.

alafanechere · 2023-12-27T13:21:26Z

airbyte-integrations/connectors/source-google-ads/source_google_ads/spec.json

@@ -84,7 +84,7 @@
        "order": 6,
        "format": "date"
      },
-      "custom_queries": {
+      "custom_queries_array": {


Why are we renaming the field?

This property was renamed for backwards compatibility. When falling back to the previous version old property will be used.

alafanechere · 2023-12-27T13:26:33Z

airbyte-integrations/connectors/source-google-ads/source_google_ads/config_migrations.py

+]
+
+
+class MigrateCustomQuery:


@airbytehq/connector-extensibility would you consider adding a migration abstract class to the CDK to expose a common migration interface?

alafanechere · 2023-12-27T13:30:55Z

docs/integrations/sources/google-ads.md

-| `0.1.1`  | 2021-06-23 | [4288](https://github.com/airbytehq/airbyte/pull/4288)   | Fix `Bugfix: Correctly declare required parameters`                                                                                  |
+| Version  | Date       | Pull Request                                             | Subject                                                                                                                                            |
+|:---------|:-----------|:---------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------|
+| `3.1.0`  | 2023-12-18 | [33603](https://github.com/airbytehq/airbyte/pull/33603) | Fix custom queries                                                                                                                                 |


Could you please be more descriptive about the changes you made? If it's a minor bump it means that you introduced a new feature and not only a fix.

octavia-squidington-iii added area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/google-ads labels Dec 18, 2023

tolik0 marked this pull request as ready for review December 18, 2023 21:32

tolik0 requested review from artem1205, askarpets and a team December 18, 2023 21:33

tolik0 self-assigned this Dec 18, 2023

vercel bot deployed to Preview December 18, 2023 21:34 View deployment

vercel bot deployed to Preview December 19, 2023 12:34 View deployment

octavia-squidington-iv requested a review from a team December 19, 2023 12:34

askarpets approved these changes Dec 19, 2023

View reviewed changes

tolik0 force-pushed the tolik0/source-google-ads/fix-custom-queries-segments branch from acee414 to 62ff722 Compare December 20, 2023 17:58

vercel bot deployed to Preview December 20, 2023 18:09 View deployment

vercel bot deployed to Preview December 20, 2023 18:42 View deployment

vercel bot deployed to Preview December 21, 2023 10:37 View deployment

tolik0 force-pushed the tolik0/source-google-ads/fix-custom-queries-segments branch from f9933ed to d56a474 Compare December 21, 2023 17:58

vercel bot deployed to Preview December 21, 2023 18:00 View deployment

tolik0 requested a review from erohmensing December 21, 2023 18:28

bazarnov approved these changes Dec 21, 2023

View reviewed changes

artem1205 reviewed Dec 22, 2023

View reviewed changes

tolik0 force-pushed the tolik0/source-google-ads/fix-custom-queries-segments branch from 3321b2b to fa07d34 Compare December 22, 2023 14:40

tolik0 requested a review from alafanechere December 27, 2023 13:13

alafanechere reviewed Dec 27, 2023

View reviewed changes

vercel bot deployed to Preview December 28, 2023 13:22 View deployment

tolik0 requested a review from alafanechere December 28, 2023 23:15

tolik0 force-pushed the tolik0/source-google-ads/fix-custom-queries-segments branch from 6a4332d to 80f9396 Compare January 9, 2024 12:12

vercel bot deployed to Preview January 9, 2024 12:17 View deployment

vercel bot deployed to Preview January 9, 2024 13:18 View deployment

tolik0 force-pushed the tolik0/source-google-ads/fix-custom-queries-segments branch from d88368c to f7f04af Compare January 9, 2024 15:40

octavia-squidington-iv requested a review from a team January 9, 2024 15:41

tolik0 added 7 commits January 9, 2024 18:38

Add migration to new format of custom queries

6eeb173

Fix date type for custom queries

8a50a0a

Update version

b562f14

Minor refactoring

703937c

Add files for migration tests

5f1aa65

Fix pendulum version

9f5d844

Minor docs updates

be9cb5e

tolik0 force-pushed the tolik0/source-google-ads/fix-custom-queries-segments branch from f7f04af to be9cb5e Compare January 9, 2024 16:40

vercel bot deployed to Preview January 9, 2024 16:42 View deployment

tolik0 merged commit c09d5d3 into master Jan 9, 2024
26 checks passed

tolik0 deleted the tolik0/source-google-ads/fix-custom-queries-segments branch January 9, 2024 17:27

tolik0 mentioned this pull request Feb 7, 2024

✨Source Google Ads: Add possibility to sync all connected accounts #33707

Merged

jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024

🐛 Source Google Ads: Fix custom queries (airbytehq#33603)

a7ceb5a

jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024

🐛 Source Google Ads: Fix custom queries (airbytehq#33603)

a444c21

jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024

🐛 Source Google Ads: Fix custom queries (airbytehq#33603)

11ec92b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Source Google Ads: Fix custom queries #33603

🐛 Source Google Ads: Fix custom queries #33603

tolik0 commented Dec 18, 2023 •

edited

vercel bot commented Dec 18, 2023 •

edited

github-actions bot commented Dec 18, 2023 •

edited by tolik0

github-actions bot commented Dec 21, 2023 •

edited

artem1205 Dec 22, 2023

tolik0 Dec 22, 2023

alafanechere left a comment

alafanechere Dec 27, 2023

tolik0 Dec 28, 2023

alafanechere Dec 27, 2023

tolik0 Dec 28, 2023

alafanechere Dec 27, 2023

alafanechere Dec 27, 2023

tolik0 Dec 28, 2023

🐛 Source Google Ads: Fix custom queries #33603

🐛 Source Google Ads: Fix custom queries #33603

Conversation

tolik0 commented Dec 18, 2023 • edited

What

How

vercel bot commented Dec 18, 2023 • edited

github-actions bot commented Dec 18, 2023 • edited by tolik0

Before Merging a Connector Pull Request

github-actions bot commented Dec 21, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alafanechere left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tolik0 commented Dec 18, 2023 •

edited

vercel bot commented Dec 18, 2023 •

edited

github-actions bot commented Dec 18, 2023 •

edited by tolik0

github-actions bot commented Dec 21, 2023 •

edited