🐛 Source Google Ads: Delete required start date #30071

tolik0 · 2023-09-01T01:12:29Z

What

Delete start_date from required parameters
Fix pagination

How

Set the default value for start_date as two years ago because incremental queries should include WHERE date clause.
Fix pagination to iterate over the pages of response.

source-google-ads test report (commit `00cf0172c2`) - ❌

⏲️ Total pipeline duration: 48.22s

Step	Result
Connector package install	✅
Build source-google-ads docker image for platform linux/x86_64	✅
Unit tests	❌
Code format checks	✅
Validate airbyte-integrations/connectors/source-google-ads/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-google-ads test

github-actions · 2023-09-01T09:16:40Z

source-google-ads test report (commit `feacc8bd99`) - ✅

⏲️ Total pipeline duration: 16mn23s

Step	Result
Connector package install	✅
Build source-google-ads docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-google-ads/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-google-ads test

lazebnyi · 2023-09-04T11:47:15Z

airbyte-integrations/connectors/source-google-ads/acceptance-test-config.yml

+        ignored_fields:
+          accounts:
+            - name: customer.optimization_score_weight
+              bypass_reason: "can be updated, also it is sometimes integer, sometimes float"


can manually validate typing and update it if needed?

lazebnyi · 2023-09-04T11:51:34Z

airbyte-integrations/connectors/source-google-ads/source_google_ads/streams.py

@@ -59,7 +59,7 @@ def chunk_date_range(
    """
    today = pendulum.today(tz=time_zone)
    end_date = min(pendulum.parse(end_date, tz=time_zone), today) if end_date else today
-    start_date = pendulum.parse(start_date, tz=time_zone)
+    start_date = pendulum.parse(start_date, tz=time_zone) if start_date else pendulum.now().subtract(years=2)


If the start date is not specified, we ought to attempt to retrieve all available data. Alternatively, we can include a comment indicating that the date range query parameters are mandatory, prompting us to assign a default start date if it is absent.

Date range in mandatory parameter for the query.

github-actions · 2023-09-04T12:45:33Z

source-google-ads test report (commit `9f0a88a939`) - ✅

⏲️ Total pipeline duration: 17mn30s

Step	Result
Connector package install	✅
Build source-google-ads docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-google-ads/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-google-ads test

github-actions · 2023-09-04T16:23:41Z

source-google-ads test report (commit `8d93f365b5`) - ❌

⏲️ Total pipeline duration: 01mn07s

Step	Result
Connector package install	✅
Build source-google-ads docker image for platform linux/x86_64	✅
Unit tests	❌
Code format checks	✅
Validate airbyte-integrations/connectors/source-google-ads/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-google-ads test

github-actions · 2023-09-04T16:45:33Z

source-google-ads test report (commit `8003eb3e71`) - ✅

⏲️ Total pipeline duration: 14mn28s

Step	Result
Connector package install	✅
Build source-google-ads docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-google-ads/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-google-ads test

lazebnyi · 2023-09-07T08:26:54Z

airbyte-integrations/connectors/source-google-ads/source_google_ads/schemas/accounts.json

@@ -42,7 +42,7 @@
      "type": ["null", "number"]
    },
    "customer.optimization_score_weight": {
-      "type": ["null", "number"]
+      "type": ["null", "number", "integer"]


I’m not sure that we support two types for one field

But we also have this type of matching in a few different connectors:
source-clockify
source-amplitude
And other
It also doesn't raise errors.

I believe we can update typing during parse response.

github-actions · 2023-09-07T13:14:33Z

source-google-ads test report (commit `65c171b776`) - ❌

⏲️ Total pipeline duration: 29mn48s

Step	Result
Connector package install	✅
Build source-google-ads docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	❌
Code format checks	✅
Validate airbyte-integrations/connectors/source-google-ads/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-google-ads test

github-actions · 2023-09-07T13:46:19Z

source-google-ads test report (commit `468fbb3b78`) - ❌

⏲️ Total pipeline duration: 29mn25s

Step	Result
Connector package install	✅
Build source-google-ads docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	❌
Code format checks	✅
Validate airbyte-integrations/connectors/source-google-ads/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-google-ads test

github-actions · 2023-09-07T14:08:49Z

source-google-ads test report (commit `62aa09e496`) - ✅

⏲️ Total pipeline duration: 20mn14s

Step	Result
Connector package install	✅
Build source-google-ads docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-google-ads/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-google-ads test

github-actions · 2023-09-07T14:35:38Z

source-google-ads test report (commit `ff9eed9879`) - ✅

⏲️ Total pipeline duration: 07mn03s

Step	Result
Connector package install	✅
Build source-google-ads docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-google-ads/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-google-ads test

sentry-io · 2023-09-07T16:48:22Z

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

‼️ google.ads.googleads.errors.GoogleAdsException: (<_InactiveRpcError of RPC that terminated with: /airbyte/integration_code/source_google_ads/goo... View Issue

_{Did you find this useful? React with a 👍 or 👎}

oswaldobiolo · 2023-09-12T16:51:09Z

Hey guys.
After updating my connector version to 0.80, it started showing the following error.
Also, it became very slow after upgrading to this version. Yesterday took more than 4 hours
I rolled back to version 0.7.4 and it works fine again, running in less than 40 minutes

tolik0 · 2023-09-12T17:52:28Z

@oswaldobiolo Apologies for the 0.80 issues. A pagination fix is in #28970, expected to merge today.

tolik0 added 3 commits September 1, 2023 03:32

Delete required start_date from spec

40bb281

Fix pagination over results

987e7ee

Fix expected records

e8fed3c

octavia-squidington-iii added area/connectors Connector related issues connectors/source/google-ads labels Sep 1, 2023

tolik0 changed the title ~~ource google ads:delete required start datetime~~ Source Google Ads: Delete required start date Sep 1, 2023

Update version

00cf017

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Sep 1, 2023

tolik0 changed the title ~~Source Google Ads: Delete required start date~~ 🐛 Source Google Ads: Delete required start date Sep 1, 2023

tolik0 marked this pull request as ready for review September 1, 2023 08:43

Fix unit tests

feacc8b

tolik0 self-assigned this Sep 1, 2023

tolik0 requested review from lazebnyi and darynaishchenko September 1, 2023 12:18

lazebnyi requested changes Sep 4, 2023

View reviewed changes

Update field customer.optimization_score_weight

9f0a88a

Moved setting default start and end dates

8d93f36

tolik0 requested a review from lazebnyi September 4, 2023 16:20

Fix unit test

8003eb3

tolik0 mentioned this pull request Sep 4, 2023

✨ Source Google-Ads: add incremental streams based on events #28970

Merged

lazebnyi requested changes Sep 7, 2023

View reviewed changes

Add changing type of field from int to float

65c171b

lazebnyi approved these changes Sep 7, 2023

View reviewed changes

Minor fix

468fbb3

Increase timeout for incremental tests

62aa09e

Empty commit

ff9eed9

tolik0 merged commit 8fe6989 into master Sep 7, 2023
23 of 24 checks passed

tolik0 deleted the tolik0/source-google-ads/delete-required-start-datetime branch September 7, 2023 14:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Source Google Ads: Delete required start date #30071

🐛 Source Google Ads: Delete required start date #30071

tolik0 commented Sep 1, 2023 •

edited

github-actions bot commented Sep 1, 2023 •

edited by tolik0

github-actions bot commented Sep 1, 2023

github-actions bot commented Sep 1, 2023

lazebnyi Sep 4, 2023

tolik0 Sep 4, 2023

lazebnyi Sep 4, 2023

tolik0 Sep 4, 2023

github-actions bot commented Sep 4, 2023

github-actions bot commented Sep 4, 2023

github-actions bot commented Sep 4, 2023

lazebnyi Sep 7, 2023

tolik0 Sep 7, 2023

lazebnyi Sep 7, 2023

tolik0 Sep 7, 2023

github-actions bot commented Sep 7, 2023

github-actions bot commented Sep 7, 2023

github-actions bot commented Sep 7, 2023

github-actions bot commented Sep 7, 2023

sentry-io bot commented Sep 7, 2023

oswaldobiolo commented Sep 12, 2023

tolik0 commented Sep 12, 2023

🐛 Source Google Ads: Delete required start date #30071

🐛 Source Google Ads: Delete required start date #30071

Conversation

tolik0 commented Sep 1, 2023 • edited

What

How

Recommended reading order

github-actions bot commented Sep 1, 2023 • edited by tolik0

Before Merging a Connector Pull Request

github-actions bot commented Sep 1, 2023

source-google-ads test report (commit 00cf0172c2) - ❌

github-actions bot commented Sep 1, 2023

source-google-ads test report (commit feacc8bd99) - ✅

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Sep 4, 2023

source-google-ads test report (commit 9f0a88a939) - ✅

github-actions bot commented Sep 4, 2023

source-google-ads test report (commit 8d93f365b5) - ❌

github-actions bot commented Sep 4, 2023

source-google-ads test report (commit 8003eb3e71) - ✅

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Sep 7, 2023

source-google-ads test report (commit 65c171b776) - ❌

github-actions bot commented Sep 7, 2023

source-google-ads test report (commit 468fbb3b78) - ❌

github-actions bot commented Sep 7, 2023

source-google-ads test report (commit 62aa09e496) - ✅

github-actions bot commented Sep 7, 2023

source-google-ads test report (commit ff9eed9879) - ✅

sentry-io bot commented Sep 7, 2023

Suspect Issues

oswaldobiolo commented Sep 12, 2023

tolik0 commented Sep 12, 2023

tolik0 commented Sep 1, 2023 •

edited

github-actions bot commented Sep 1, 2023 •

edited by tolik0

source-google-ads test report (commit `00cf0172c2`) - ❌

source-google-ads test report (commit `feacc8bd99`) - ✅

source-google-ads test report (commit `9f0a88a939`) - ✅

source-google-ads test report (commit `8d93f365b5`) - ❌

source-google-ads test report (commit `8003eb3e71`) - ✅

source-google-ads test report (commit `65c171b776`) - ❌

source-google-ads test report (commit `468fbb3b78`) - ❌

source-google-ads test report (commit `62aa09e496`) - ✅

source-google-ads test report (commit `ff9eed9879`) - ✅