🐛Source Google ads: connector doesn't pull data for today #8225
Conversation
|
/test connector=connectors/source-google-ads
|
|
/test connector=connectors/source-google-ads
|
airbyte-integrations/connectors/source-google-ads/source_google_ads/streams.py
Outdated
Show resolved
Hide resolved
| @@ -83,9 +92,8 @@ def stream_slices(self, stream_state: Mapping[str, Any] = None, **kwargs) -> Ite | |||
| days_of_data_storage=self.days_of_data_storage, | |||
| ) | |||
|
|
|||
| @staticmethod | |||
| def get_date_params(stream_slice: Mapping[str, Any], cursor_field: str, end_date: pendulum.datetime = None, time_unit: str = "months"): | |||
| end_date = end_date or pendulum.yesterday() | |||
There was a problem hiding this comment.
I think there were some reasons to do it.
I suspect that Google Ads do not generate reports in real time.
As far as I remember google need some time (a day) to have full report for given day.
If we set end_day to today then report can be empty or incomplete. So please do additional research for this question
There was a problem hiding this comment.
I didn't find any information about impossibility sync reports in real-time. Google Ads returns data for today
There was a problem hiding this comment.
@sherifnada We need your opinion here.
-
As for me if we include 'today' into the current sync means that next sync starts from tomorrow so client will lose data between now and the end of today's day.
-
as far as I remember from my own experience with Google Ads - it does not provide 'real-time' reports and full data is available only in a few hours .
If we want to include todays report into sync then an alternative solution can be - we should have 1 day overlaps between scans. It guarantees that customers have full reports.
There was a problem hiding this comment.
Actually, I have found Google docs about it
https://support.google.com/google-ads/answer/2544985?hl=en
- such as clicks, conversions, and impressions) are delayed by less than 3 hours
- conversions attributed using attribution models other than “Last click” are typically delayed up to 15 hours
etc
There was a problem hiding this comment.
I think it's acceptable to sync data only up to yesterday if there is no other way to guarantee data loss. Ideally, we can pick up a sync exactly where we left off. But if it's not possible, that's acceptable imo.
What is the technical possibility of doing it that way?
There was a problem hiding this comment.
@sherifnada we cannot sync data up to today. We use segments.date field for filtering data in incremental sync. Google Ads accepts only date without time for filtering. I also update google ads doc.
There was a problem hiding this comment.
Ok, lets sync only until yesterday then. we can update that ticket with this information, and mention we can't do it due to limit from api side
github-actions
bot
added
the
area/documentation
…ads-fix-end-date # Conflicts: # airbyte-integrations/connectors/source-google-ads/source_google_ads/spec.json
|
/test connector=connectors/source-google-ads
|
| @@ -83,9 +92,8 @@ def stream_slices(self, stream_state: Mapping[str, Any] = None, **kwargs) -> Ite | |||
| days_of_data_storage=self.days_of_data_storage, | |||
| ) | |||
|
|
|||
| @staticmethod | |||
| def get_date_params(stream_slice: Mapping[str, Any], cursor_field: str, end_date: pendulum.datetime = None, time_unit: str = "months"): | |||
| end_date = end_date or pendulum.yesterday() | |||
There was a problem hiding this comment.
Ok, lets sync only until yesterday then. we can update that ticket with this information, and mention we can't do it due to limit from api side
airbyte-integrations/connectors/source-google-ads/source_google_ads/streams.py
Outdated
Show resolved
Hide resolved
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
|
/test connector=connectors/source-google-ads
|
|
/test connector=connectors/source-google-ads
|
|
/publish connector=connectors/source-google-ads
|
|
/publish connector=connectors/source-google-ads
|
…8225) * fix end_date for incremental streams * upd imports * upd unit test * upd after review * fix unit tests * change type * remove streams for manager account * bump version * add unit_tests * upd imports * update is_manager_account method * add unit tests * Update docs/integrations/sources/google-ads.md Co-authored-by: Sherif A. Nada <snadalive@gmail.com> * check metrics in custom query * change end date and upd doc * Update docs/integrations/sources/google-ads.md Co-authored-by: Sherif A. Nada <snadalive@gmail.com> * optimize imports * bump version * bump version Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
What
For incremental streams
end_dateparam is yesterday's date now. Data is not pulled for today. Could reproduce it locally.Closes #7047 and OnCall#37.
How
Update date to today and take timezone into account.
Recommended reading order
streams.py🚨 User Impact 🚨
Are there any breaking changes? If yes, please make sure to include it here and in any changelogs with the 🚨🚨 emoji
What is the end result perceived by the user?
Pre-merge Checklist
Expand the relevant checklist and delete the others.
Updating a connector
Community member or Airbyter
airbyte_secret./gradlew :airbyte-integrations:connectors:<name>:integrationTest.README.mdbootstrap.md. See description and examplesdocs/integrations/<source or destination>/<name>.mdincluding changelog. See changelog exampleAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>command is passing./publishcommand described here