fix(Google Search Console): Wait an extra day before expecting Google Search Console export data to exist #8220

sean-rose · 2025-10-08T16:38:19Z

Description

The table partition sensors in the bqetl_google_search_console DAG have timed out waiting for Google Search Console export data in 18 of the last 30 days, so it seems Google Search Console has been struggling to keep up with its typical schedule for BigQuery exports.

Related Tickets & Documents

N/A

Reviewer, please follow this checklist

… Search Console export data to exist.

dataops-ci-bot · 2025-10-08T16:43:15Z

Integration report for "fix(Google Search Console): Wait an extra day before expecting Google Search Console export data to exist."

`sql.diff`

Click to expand!

diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_google_search_console.py /tmp/workspace/generated-sql/dags/bqetl_google_search_console.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_google_search_console.py	2025-10-08 16:39:19.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_google_search_console.py	2025-10-08 16:41:42.000000000 +0000
@@ -66,7 +66,7 @@
             project_id="moz-fx-data-marketing-prod",
             dataset_id="searchconsole_addons",
             table_id="searchdata_url_impression",
-            partition_id="{{ data_interval_start.subtract(days=1) | ds_nodash }}",
+            partition_id="{{ data_interval_start.subtract(days=2) | ds_nodash }}",
             gcp_conn_id="google_cloud_shared_prod",
             deferrable=False,
             mode="reschedule",
@@ -81,7 +81,7 @@
             project_id="moz-fx-data-marketing-prod",
             dataset_id="searchconsole_blog",
             table_id="searchdata_url_impression",
-            partition_id="{{ data_interval_start.subtract(days=1) | ds_nodash }}",
+            partition_id="{{ data_interval_start.subtract(days=2) | ds_nodash }}",
             gcp_conn_id="google_cloud_shared_prod",
             deferrable=False,
             mode="reschedule",
@@ -96,7 +96,7 @@
             project_id="moz-fx-data-marketing-prod",
             dataset_id="searchconsole_getpocket",
             table_id="searchdata_url_impression",
-            partition_id="{{ data_interval_start.subtract(days=1) | ds_nodash }}",
+            partition_id="{{ data_interval_start.subtract(days=2) | ds_nodash }}",
             gcp_conn_id="google_cloud_shared_prod",
             deferrable=False,
             mode="reschedule",
@@ -111,7 +111,7 @@
             project_id="moz-fx-data-marketing-prod",
             dataset_id="searchconsole_mdn",
             table_id="searchdata_url_impression",
-            partition_id="{{ data_interval_start.subtract(days=1) | ds_nodash }}",
+            partition_id="{{ data_interval_start.subtract(days=2) | ds_nodash }}",
             gcp_conn_id="google_cloud_shared_prod",
             deferrable=False,
             mode="reschedule",
@@ -126,7 +126,7 @@
             project_id="moz-fx-data-marketing-prod",
             dataset_id="searchconsole_support",
             table_id="searchdata_url_impression",
-            partition_id="{{ data_interval_start.subtract(days=1) | ds_nodash }}",
+            partition_id="{{ data_interval_start.subtract(days=2) | ds_nodash }}",
             gcp_conn_id="google_cloud_shared_prod",
             deferrable=False,
             mode="reschedule",
@@ -141,7 +141,7 @@
             project_id="moz-fx-data-marketing-prod",
             dataset_id="searchconsole_www",
             table_id="searchdata_url_impression",
-            partition_id="{{ data_interval_start.subtract(days=1) | ds_nodash }}",
+            partition_id="{{ data_interval_start.subtract(days=2) | ds_nodash }}",
             gcp_conn_id="google_cloud_shared_prod",
             deferrable=False,
             mode="reschedule",
@@ -156,7 +156,7 @@
             project_id="moz-fx-data-marketing-prod",
             dataset_id="searchconsole_addons",
             table_id="searchdata_site_impression",
-            partition_id="{{ data_interval_start.subtract(days=1) | ds_nodash }}",
+            partition_id="{{ data_interval_start.subtract(days=2) | ds_nodash }}",
             gcp_conn_id="google_cloud_shared_prod",
             deferrable=False,
             mode="reschedule",
@@ -171,7 +171,7 @@
             project_id="moz-fx-data-marketing-prod",
             dataset_id="searchconsole_blog",
             table_id="searchdata_site_impression",
-            partition_id="{{ data_interval_start.subtract(days=1) | ds_nodash }}",
+            partition_id="{{ data_interval_start.subtract(days=2) | ds_nodash }}",
             gcp_conn_id="google_cloud_shared_prod",
             deferrable=False,
             mode="reschedule",
@@ -186,7 +186,7 @@
             project_id="moz-fx-data-marketing-prod",
             dataset_id="searchconsole_getpocket",
             table_id="searchdata_site_impression",
-            partition_id="{{ data_interval_start.subtract(days=1) | ds_nodash }}",
+            partition_id="{{ data_interval_start.subtract(days=2) | ds_nodash }}",
             gcp_conn_id="google_cloud_shared_prod",
             deferrable=False,
             mode="reschedule",
@@ -201,7 +201,7 @@
             project_id="moz-fx-data-marketing-prod",
             dataset_id="searchconsole_mdn",
             table_id="searchdata_site_impression",
-            partition_id="{{ data_interval_start.subtract(days=1) | ds_nodash }}",
+            partition_id="{{ data_interval_start.subtract(days=2) | ds_nodash }}",
             gcp_conn_id="google_cloud_shared_prod",
             deferrable=False,
             mode="reschedule",
@@ -216,7 +216,7 @@
             project_id="moz-fx-data-marketing-prod",
             dataset_id="searchconsole_support",
             table_id="searchdata_site_impression",
-            partition_id="{{ data_interval_start.subtract(days=1) | ds_nodash }}",
+            partition_id="{{ data_interval_start.subtract(days=2) | ds_nodash }}",
             gcp_conn_id="google_cloud_shared_prod",
             deferrable=False,
             mode="reschedule",
@@ -231,7 +231,7 @@
             project_id="moz-fx-data-marketing-prod",
             dataset_id="searchconsole_www",
             table_id="searchdata_site_impression",
-            partition_id="{{ data_interval_start.subtract(days=1) | ds_nodash }}",
+            partition_id="{{ data_interval_start.subtract(days=2) | ds_nodash }}",
             gcp_conn_id="google_cloud_shared_prod",
             deferrable=False,
             mode="reschedule",
@@ -242,26 +242,26 @@
 
     google_search_console_derived__search_impressions_by_page__v2 = bigquery_etl_query(
         task_id="google_search_console_derived__search_impressions_by_page__v2",
-        destination_table='search_impressions_by_page_v2${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
+        destination_table='search_impressions_by_page_v2${{ macros.ds_format(macros.ds_add(ds, -2), "%Y-%m-%d", "%Y%m%d") }}',
         dataset_id="google_search_console_derived",
         project_id="moz-fx-data-shared-prod",
         owner="srose@mozilla.com",
         email=["srose@mozilla.com", "telemetry-alerts@mozilla.com"],
         date_partition_parameter=None,
         depends_on_past=False,
-        parameters=["date:DATE:{{macros.ds_add(ds, -1)}}"],
+        parameters=["date:DATE:{{macros.ds_add(ds, -2)}}"],
     )
 
     google_search_console_derived__search_impressions_by_site__v2 = bigquery_etl_query(
         task_id="google_search_console_derived__search_impressions_by_site__v2",
-        destination_table='search_impressions_by_site_v2${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
+        destination_table='search_impressions_by_site_v2${{ macros.ds_format(macros.ds_add(ds, -2), "%Y-%m-%d", "%Y%m%d") }}',
         dataset_id="google_search_console_derived",
         project_id="moz-fx-data-shared-prod",
         owner="srose@mozilla.com",
         email=["srose@mozilla.com", "telemetry-alerts@mozilla.com"],
         date_partition_parameter=None,
         depends_on_past=False,
-        parameters=["date:DATE:{{macros.ds_add(ds, -1)}}"],
+        parameters=["date:DATE:{{macros.ds_add(ds, -2)}}"],
     )
 
     google_search_console_derived__search_impressions_by_page__v2.set_upstream(
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/google_search_console_derived/search_impressions_by_page_v2/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/google_search_console_derived/search_impressions_by_page_v2/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/google_search_console_derived/search_impressions_by_page_v2/metadata.yaml	2025-10-08 16:34:46.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/google_search_console_derived/search_impressions_by_page_v2/metadata.yaml	2025-10-08 16:34:42.000000000 +0000
@@ -24,26 +24,26 @@
 scheduling:
   dag_name: bqetl_google_search_console
   date_partition_parameter: date
-  date_partition_offset: -1
+  date_partition_offset: -2
   depends_on_table_partitions_existing:
   - task_id: wait_for_google_search_console_addons_url_impressions
     table_id: moz-fx-data-marketing-prod.searchconsole_addons.searchdata_url_impression
-    partition_id: '{{ data_interval_start.subtract(days=1) | ds_nodash }}'
+    partition_id: '{{ data_interval_start.subtract(days=2) | ds_nodash }}'
   - task_id: wait_for_google_search_console_blog_url_impressions
     table_id: moz-fx-data-marketing-prod.searchconsole_blog.searchdata_url_impression
-    partition_id: '{{ data_interval_start.subtract(days=1) | ds_nodash }}'
+    partition_id: '{{ data_interval_start.subtract(days=2) | ds_nodash }}'
   - task_id: wait_for_google_search_console_getpocket_url_impressions
     table_id: moz-fx-data-marketing-prod.searchconsole_getpocket.searchdata_url_impression
-    partition_id: '{{ data_interval_start.subtract(days=1) | ds_nodash }}'
+    partition_id: '{{ data_interval_start.subtract(days=2) | ds_nodash }}'
   - task_id: wait_for_google_search_console_mdn_url_impressions
     table_id: moz-fx-data-marketing-prod.searchconsole_mdn.searchdata_url_impression
-    partition_id: '{{ data_interval_start.subtract(days=1) | ds_nodash }}'
+    partition_id: '{{ data_interval_start.subtract(days=2) | ds_nodash }}'
   - task_id: wait_for_google_search_console_support_url_impressions
     table_id: moz-fx-data-marketing-prod.searchconsole_support.searchdata_url_impression
-    partition_id: '{{ data_interval_start.subtract(days=1) | ds_nodash }}'
+    partition_id: '{{ data_interval_start.subtract(days=2) | ds_nodash }}'
   - task_id: wait_for_google_search_console_www_url_impressions
     table_id: moz-fx-data-marketing-prod.searchconsole_www.searchdata_url_impression
-    partition_id: '{{ data_interval_start.subtract(days=1) | ds_nodash }}'
+    partition_id: '{{ data_interval_start.subtract(days=2) | ds_nodash }}'
 bigquery:
   time_partitioning:
     type: day
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/google_search_console_derived/search_impressions_by_site_v2/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/google_search_console_derived/search_impressions_by_site_v2/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/google_search_console_derived/search_impressions_by_site_v2/metadata.yaml	2025-10-08 16:34:46.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/google_search_console_derived/search_impressions_by_site_v2/metadata.yaml	2025-10-08 16:34:42.000000000 +0000
@@ -24,26 +24,26 @@
 scheduling:
   dag_name: bqetl_google_search_console
   date_partition_parameter: date
-  date_partition_offset: -1
+  date_partition_offset: -2
   depends_on_table_partitions_existing:
   - task_id: wait_for_google_search_console_addons_site_impressions
     table_id: moz-fx-data-marketing-prod.searchconsole_addons.searchdata_site_impression
-    partition_id: '{{ data_interval_start.subtract(days=1) | ds_nodash }}'
+    partition_id: '{{ data_interval_start.subtract(days=2) | ds_nodash }}'
   - task_id: wait_for_google_search_console_blog_site_impressions
     table_id: moz-fx-data-marketing-prod.searchconsole_blog.searchdata_site_impression
-    partition_id: '{{ data_interval_start.subtract(days=1) | ds_nodash }}'
+    partition_id: '{{ data_interval_start.subtract(days=2) | ds_nodash }}'
   - task_id: wait_for_google_search_console_getpocket_site_impressions
     table_id: moz-fx-data-marketing-prod.searchconsole_getpocket.searchdata_site_impression
-    partition_id: '{{ data_interval_start.subtract(days=1) | ds_nodash }}'
+    partition_id: '{{ data_interval_start.subtract(days=2) | ds_nodash }}'
   - task_id: wait_for_google_search_console_mdn_site_impressions
     table_id: moz-fx-data-marketing-prod.searchconsole_mdn.searchdata_site_impression
-    partition_id: '{{ data_interval_start.subtract(days=1) | ds_nodash }}'
+    partition_id: '{{ data_interval_start.subtract(days=2) | ds_nodash }}'
   - task_id: wait_for_google_search_console_support_site_impressions
     table_id: moz-fx-data-marketing-prod.searchconsole_support.searchdata_site_impression
-    partition_id: '{{ data_interval_start.subtract(days=1) | ds_nodash }}'
+    partition_id: '{{ data_interval_start.subtract(days=2) | ds_nodash }}'
   - task_id: wait_for_google_search_console_www_site_impressions
     table_id: moz-fx-data-marketing-prod.searchconsole_www.searchdata_site_impression
-    partition_id: '{{ data_interval_start.subtract(days=1) | ds_nodash }}'
+    partition_id: '{{ data_interval_start.subtract(days=2) | ds_nodash }}'
 bigquery:
   time_partitioning:
     type: day

Link to full diff

fix(Google Search Console): Wait an extra day before expecting Google…

62d8578

… Search Console export data to exist.

akkomar approved these changes Oct 8, 2025

View reviewed changes

sean-rose added this pull request to the merge queue Oct 8, 2025

Merged via the queue into main with commit d5de056 Oct 8, 2025
22 checks passed

sean-rose deleted the delay-Google-Search-Console branch October 8, 2025 17:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(Google Search Console): Wait an extra day before expecting Google Search Console export data to exist #8220

fix(Google Search Console): Wait an extra day before expecting Google Search Console export data to exist #8220

Uh oh!

sean-rose commented Oct 8, 2025

Uh oh!

dataops-ci-bot commented Oct 8, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fix(Google Search Console): Wait an extra day before expecting Google Search Console export data to exist #8220

fix(Google Search Console): Wait an extra day before expecting Google Search Console export data to exist #8220

Uh oh!

Conversation

sean-rose commented Oct 8, 2025

Description

Related Tickets & Documents

Uh oh!

dataops-ci-bot commented Oct 8, 2025

Integration report for "fix(Google Search Console): Wait an extra day before expecting Google Search Console export data to exist."

sql.diff

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

`sql.diff`