Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DENG-3462 - Update telemetry.clients_last_seen to use V2 instead of V1 #5447

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

kwindau
Copy link
Contributor

@kwindau kwindau commented Apr 26, 2024

…ead of v1

Checklist for reviewer:

  • Commits should reference a bug or github issue, if relevant (if a bug is referenced, the pull request should include the bug number in the title).
  • If the PR comes from a fork, trigger integration CI tests by running the Push to upstream workflow and provide the <username>:<branch> of the fork as parameter. The parameter will also show up
    in the logs of the manual-trigger-required-for-fork CI task together with more detailed instructions.
  • If adding a new field to a query, ensure that the schema and dependent downstream schemas have been updated.
  • When adding a new derived dataset, ensure that data is not available already (fully or partially) and recommend extending an existing dataset in favor of creating new ones. Data can be available in the bigquery-etl repository, looker-hub or in looker-spoke-default.

For modifications to schemas in restricted namespaces (see CODEOWNERS):

┆Issue is synchronized with this Jira Task

@kwindau kwindau marked this pull request as ready for review April 26, 2024 19:10
@dataops-ci-bot

This comment has been minimized.

@dataops-ci-bot

This comment has been minimized.

@kwindau kwindau enabled auto-merge (squash) April 30, 2024 20:15
@kwindau kwindau disabled auto-merge April 30, 2024 20:17
@kwindau
Copy link
Contributor Author

kwindau commented Apr 30, 2024

Note to self - waiting to push until the backfill of the source tables is done

@dataops-ci-bot

This comment has been minimized.

@dataops-ci-bot
Copy link

Integration report for "Merge branch 'main' into DENG-3462-clients-last-seen"

sql.diff

Click to expand!
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_analytics_aggregations.py /tmp/workspace/generated-sql/dags/bqetl_analytics_aggregations.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_analytics_aggregations.py	2024-05-08 15:25:18.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_analytics_aggregations.py	2024-05-08 15:27:02.000000000 +0000
@@ -140,10 +140,10 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_telemetry_derived__clients_last_seen__v1 = ExternalTaskSensor(
-        task_id="wait_for_telemetry_derived__clients_last_seen__v1",
+    wait_for_checks__fail_telemetry_derived__clients_last_seen__v2 = ExternalTaskSensor(
+        task_id="wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
         external_dag_id="bqetl_main_summary",
-        external_task_id="telemetry_derived__clients_last_seen__v1",
+        external_task_id="checks__fail_telemetry_derived__clients_last_seen__v2",
         execution_delta=datetime.timedelta(seconds=8100),
         check_existence=True,
         mode="reschedule",
@@ -288,6 +288,18 @@
         )
     )
 
+    wait_for_telemetry_derived__clients_last_seen__v1 = ExternalTaskSensor(
+        task_id="wait_for_telemetry_derived__clients_last_seen__v1",
+        external_dag_id="bqetl_main_summary",
+        external_task_id="telemetry_derived__clients_last_seen__v1",
+        execution_delta=datetime.timedelta(seconds=8100),
+        check_existence=True,
+        mode="reschedule",
+        allowed_states=ALLOWED_STATES,
+        failed_states=FAILED_STATES,
+        pool="DATA_ENG_EXTERNALTASKSENSOR",
+    )
+
     active_users_aggregates_attribution_v1 = bigquery_etl_query(
         task_id="active_users_aggregates_attribution_v1",
         destination_table="active_users_aggregates_attribution_v1",
@@ -614,11 +626,11 @@
     )
 
     checks__warn_firefox_desktop_derived__active_users_aggregates__v3.set_upstream(
-        firefox_desktop_active_users_aggregates
+        wait_for_checks__fail_telemetry_derived__clients_last_seen__v2
     )
 
     checks__warn_firefox_desktop_derived__active_users_aggregates__v3.set_upstream(
-        wait_for_telemetry_derived__clients_last_seen__v1
+        firefox_desktop_active_users_aggregates
     )
 
     checks__warn_firefox_ios_derived__active_users_aggregates__v3.set_upstream(
@@ -658,7 +670,7 @@
     )
 
     firefox_desktop_active_users_aggregates.set_upstream(
-        wait_for_telemetry_derived__clients_last_seen__v1
+        wait_for_checks__fail_telemetry_derived__clients_last_seen__v2
     )
 
     firefox_ios_active_users_aggregates.set_upstream(
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_analytics_tables.py /tmp/workspace/generated-sql/dags/bqetl_analytics_tables.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_analytics_tables.py	2024-05-08 15:25:18.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_analytics_tables.py	2024-05-08 15:27:03.000000000 +0000
@@ -211,10 +211,10 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_telemetry_derived__clients_last_seen__v1 = ExternalTaskSensor(
-        task_id="wait_for_telemetry_derived__clients_last_seen__v1",
+    wait_for_checks__fail_telemetry_derived__clients_last_seen__v2 = ExternalTaskSensor(
+        task_id="wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
         external_dag_id="bqetl_main_summary",
-        external_task_id="telemetry_derived__clients_last_seen__v1",
+        external_task_id="checks__fail_telemetry_derived__clients_last_seen__v2",
         check_existence=True,
         mode="reschedule",
         allowed_states=ALLOWED_STATES,
@@ -672,5 +672,5 @@
     )
 
     telemetry_derived__clients_first_seen_28_days_later__v1.set_upstream(
-        wait_for_telemetry_derived__clients_last_seen__v1
+        wait_for_checks__fail_telemetry_derived__clients_last_seen__v2
     )
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_desktop_funnel.py /tmp/workspace/generated-sql/dags/bqetl_desktop_funnel.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_desktop_funnel.py	2024-05-08 15:25:18.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_desktop_funnel.py	2024-05-08 15:27:00.000000000 +0000
@@ -52,11 +52,11 @@
     tags=tags,
 ) as dag:
 
-    wait_for_copy_deduplicate_all = ExternalTaskSensor(
-        task_id="wait_for_copy_deduplicate_all",
-        external_dag_id="copy_deduplicate",
-        external_task_id="copy_deduplicate_all",
-        execution_delta=datetime.timedelta(seconds=10800),
+    wait_for_checks__fail_telemetry_derived__clients_last_seen__v2 = ExternalTaskSensor(
+        task_id="wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
+        external_dag_id="bqetl_main_summary",
+        external_task_id="checks__fail_telemetry_derived__clients_last_seen__v2",
+        execution_delta=datetime.timedelta(seconds=7200),
         check_existence=True,
         mode="reschedule",
         allowed_states=ALLOWED_STATES,
@@ -64,11 +64,11 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_telemetry_derived__clients_last_seen__v1 = ExternalTaskSensor(
-        task_id="wait_for_telemetry_derived__clients_last_seen__v1",
-        external_dag_id="bqetl_main_summary",
-        external_task_id="telemetry_derived__clients_last_seen__v1",
-        execution_delta=datetime.timedelta(seconds=7200),
+    wait_for_copy_deduplicate_all = ExternalTaskSensor(
+        task_id="wait_for_copy_deduplicate_all",
+        external_dag_id="copy_deduplicate",
+        external_task_id="copy_deduplicate_all",
+        execution_delta=datetime.timedelta(seconds=10800),
         check_existence=True,
         mode="reschedule",
         allowed_states=ALLOWED_STATES,
@@ -110,11 +110,11 @@
     )
 
     telemetry_derived__desktop_funnel_activation_day_6__v1.set_upstream(
-        wait_for_copy_deduplicate_all
+        wait_for_checks__fail_telemetry_derived__clients_last_seen__v2
     )
 
     telemetry_derived__desktop_funnel_activation_day_6__v1.set_upstream(
-        wait_for_telemetry_derived__clients_last_seen__v1
+        wait_for_copy_deduplicate_all
     )
 
     telemetry_derived__desktop_funnel_installs__v1.set_upstream(
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_feature_usage.py /tmp/workspace/generated-sql/dags/bqetl_feature_usage.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_feature_usage.py	2024-05-08 15:25:18.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_feature_usage.py	2024-05-08 15:27:00.000000000 +0000
@@ -70,6 +70,18 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
+    wait_for_checks__fail_telemetry_derived__clients_last_seen__v2 = ExternalTaskSensor(
+        task_id="wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
+        external_dag_id="bqetl_main_summary",
+        external_task_id="checks__fail_telemetry_derived__clients_last_seen__v2",
+        execution_delta=datetime.timedelta(seconds=10800),
+        check_existence=True,
+        mode="reschedule",
+        allowed_states=ALLOWED_STATES,
+        failed_states=FAILED_STATES,
+        pool="DATA_ENG_EXTERNALTASKSENSOR",
+    )
+
     wait_for_copy_deduplicate_all = ExternalTaskSensor(
         task_id="wait_for_copy_deduplicate_all",
         external_dag_id="copy_deduplicate",
@@ -106,18 +118,6 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_telemetry_derived__clients_last_seen__v1 = ExternalTaskSensor(
-        task_id="wait_for_telemetry_derived__clients_last_seen__v1",
-        external_dag_id="bqetl_main_summary",
-        external_task_id="telemetry_derived__clients_last_seen__v1",
-        execution_delta=datetime.timedelta(seconds=10800),
-        check_existence=True,
-        mode="reschedule",
-        allowed_states=ALLOWED_STATES,
-        failed_states=FAILED_STATES,
-        pool="DATA_ENG_EXTERNALTASKSENSOR",
-    )
-
     wait_for_telemetry_derived__main_remainder_1pct__v1 = ExternalTaskSensor(
         task_id="wait_for_telemetry_derived__main_remainder_1pct__v1",
         external_dag_id="bqetl_main_summary",
@@ -149,6 +149,10 @@
 
     telemetry_derived__feature_usage__v2.set_upstream(wait_for_bq_main_events)
 
+    telemetry_derived__feature_usage__v2.set_upstream(
+        wait_for_checks__fail_telemetry_derived__clients_last_seen__v2
+    )
+
     telemetry_derived__feature_usage__v2.set_upstream(wait_for_copy_deduplicate_all)
 
     telemetry_derived__feature_usage__v2.set_upstream(wait_for_event_events)
@@ -158,9 +162,5 @@
     )
 
     telemetry_derived__feature_usage__v2.set_upstream(
-        wait_for_telemetry_derived__clients_last_seen__v1
-    )
-
-    telemetry_derived__feature_usage__v2.set_upstream(
         wait_for_telemetry_derived__main_remainder_1pct__v1
     )
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_gud.py /tmp/workspace/generated-sql/dags/bqetl_gud.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_gud.py	2024-05-08 15:25:18.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_gud.py	2024-05-08 15:27:01.000000000 +0000
@@ -50,10 +50,10 @@
     tags=tags,
 ) as dag:
 
-    wait_for_telemetry_derived__clients_last_seen__v1 = ExternalTaskSensor(
-        task_id="wait_for_telemetry_derived__clients_last_seen__v1",
+    wait_for_checks__fail_telemetry_derived__clients_last_seen__v2 = ExternalTaskSensor(
+        task_id="wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
         external_dag_id="bqetl_main_summary",
-        external_task_id="telemetry_derived__clients_last_seen__v1",
+        external_task_id="checks__fail_telemetry_derived__clients_last_seen__v2",
         execution_delta=datetime.timedelta(seconds=3600),
         check_existence=True,
         mode="reschedule",
@@ -355,7 +355,7 @@
     )
 
     telemetry_derived__smoot_usage_desktop__v2.set_upstream(
-        wait_for_telemetry_derived__clients_last_seen__v1
+        wait_for_checks__fail_telemetry_derived__clients_last_seen__v2
     )
 
     telemetry_derived__smoot_usage_desktop_compressed__v2.set_upstream(
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_kpis_shredder.py /tmp/workspace/generated-sql/dags/bqetl_kpis_shredder.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_kpis_shredder.py	2024-05-08 15:25:18.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_kpis_shredder.py	2024-05-08 15:27:02.000000000 +0000
@@ -86,10 +86,10 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_telemetry_derived__clients_last_seen__v1 = ExternalTaskSensor(
-        task_id="wait_for_telemetry_derived__clients_last_seen__v1",
+    wait_for_checks__fail_telemetry_derived__clients_last_seen__v2 = ExternalTaskSensor(
+        task_id="wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
         external_dag_id="bqetl_main_summary",
-        external_task_id="telemetry_derived__clients_last_seen__v1",
+        external_task_id="checks__fail_telemetry_derived__clients_last_seen__v2",
         check_existence=True,
         mode="reschedule",
         allowed_states=ALLOWED_STATES,
@@ -218,15 +218,15 @@
     )
 
     firefox_desktop_active_users_aggregates_for_deletion_requests.set_upstream(
-        wait_for_copy_deduplicate_all
+        wait_for_checks__fail_telemetry_derived__clients_last_seen__v2
     )
 
     firefox_desktop_active_users_aggregates_for_deletion_requests.set_upstream(
-        wait_for_search_derived__mobile_search_clients_daily__v1
+        wait_for_copy_deduplicate_all
     )
 
     firefox_desktop_active_users_aggregates_for_deletion_requests.set_upstream(
-        wait_for_telemetry_derived__clients_last_seen__v1
+        wait_for_search_derived__mobile_search_clients_daily__v1
     )
 
     firefox_ios_active_users_aggregates_for_deletion_requests.set_upstream(
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_main_summary.py /tmp/workspace/generated-sql/dags/bqetl_main_summary.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_main_summary.py	2024-05-08 15:25:18.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_main_summary.py	2024-05-08 15:27:00.000000000 +0000
@@ -131,6 +131,32 @@
         "checks__fail_telemetry_derived__clients_last_seen__v2_external",
     ) as checks__fail_telemetry_derived__clients_last_seen__v2_external:
         ExternalTaskMarker(
+            task_id="bqetl_analytics_aggregations__wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
+            external_dag_id="bqetl_analytics_aggregations",
+            external_task_id="wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
+            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=78300)).isoformat() }}",
+        )
+
+        ExternalTaskMarker(
+            task_id="bqetl_kpis_shredder__wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
+            external_dag_id="bqetl_kpis_shredder",
+            external_task_id="wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
+        )
+
+        ExternalTaskMarker(
+            task_id="bqetl_search_dashboard__wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
+            external_dag_id="bqetl_search_dashboard",
+            external_task_id="wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
+            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=77400)).isoformat() }}",
+        )
+
+        ExternalTaskMarker(
+            task_id="bqetl_analytics_tables__wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
+            external_dag_id="bqetl_analytics_tables",
+            external_task_id="wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
+        )
+
+        ExternalTaskMarker(
             task_id="bqetl_desktop_engagement_model__wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
             external_dag_id="bqetl_desktop_engagement_model",
             external_task_id="wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
@@ -138,6 +164,34 @@
         )
 
         ExternalTaskMarker(
+            task_id="bqetl_desktop_funnel__wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
+            external_dag_id="bqetl_desktop_funnel",
+            external_task_id="wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
+            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=79200)).isoformat() }}",
+        )
+
+        ExternalTaskMarker(
+            task_id="bqetl_feature_usage__wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
+            external_dag_id="bqetl_feature_usage",
+            external_task_id="wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
+            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=75600)).isoformat() }}",
+        )
+
+        ExternalTaskMarker(
+            task_id="bqetl_gud__wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
+            external_dag_id="bqetl_gud",
+            external_task_id="wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
+            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=82800)).isoformat() }}",
+        )
+
+        ExternalTaskMarker(
+            task_id="bqetl_unified__wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
+            external_dag_id="bqetl_unified",
+            external_task_id="wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
+            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=82800)).isoformat() }}",
+        )
+
+        ExternalTaskMarker(
             task_id="taar_daily__wait_for_clients_last_seen",
             external_dag_id="taar_daily",
             external_task_id="wait_for_clients_last_seen",
@@ -440,26 +494,6 @@
         "telemetry_derived__clients_last_seen__v1_external",
     ) as telemetry_derived__clients_last_seen__v1_external:
         ExternalTaskMarker(
-            task_id="bqetl_analytics_aggregations__wait_for_telemetry_derived__clients_last_seen__v1",
-            external_dag_id="bqetl_analytics_aggregations",
-            external_task_id="wait_for_telemetry_derived__clients_last_seen__v1",
-            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=78300)).isoformat() }}",
-        )
-
-        ExternalTaskMarker(
-            task_id="bqetl_kpis_shredder__wait_for_telemetry_derived__clients_last_seen__v1",
-            external_dag_id="bqetl_kpis_shredder",
-            external_task_id="wait_for_telemetry_derived__clients_last_seen__v1",
-        )
-
-        ExternalTaskMarker(
-            task_id="bqetl_search_dashboard__wait_for_telemetry_derived__clients_last_seen__v1",
-            external_dag_id="bqetl_search_dashboard",
-            external_task_id="wait_for_telemetry_derived__clients_last_seen__v1",
-            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=77400)).isoformat() }}",
-        )
-
-        ExternalTaskMarker(
             task_id="bqetl_addons__wait_for_telemetry_derived__clients_last_seen__v1",
             external_dag_id="bqetl_addons",
             external_task_id="wait_for_telemetry_derived__clients_last_seen__v1",
@@ -467,37 +501,10 @@
         )
 
         ExternalTaskMarker(
-            task_id="bqetl_analytics_tables__wait_for_telemetry_derived__clients_last_seen__v1",
-            external_dag_id="bqetl_analytics_tables",
-            external_task_id="wait_for_telemetry_derived__clients_last_seen__v1",
-        )
-
-        ExternalTaskMarker(
-            task_id="bqetl_desktop_funnel__wait_for_telemetry_derived__clients_last_seen__v1",
-            external_dag_id="bqetl_desktop_funnel",
-            external_task_id="wait_for_telemetry_derived__clients_last_seen__v1",
-            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=79200)).isoformat() }}",
-        )
-
-        ExternalTaskMarker(
-            task_id="bqetl_feature_usage__wait_for_telemetry_derived__clients_last_seen__v1",
-            external_dag_id="bqetl_feature_usage",
-            external_task_id="wait_for_telemetry_derived__clients_last_seen__v1",
-            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=75600)).isoformat() }}",
-        )
-
-        ExternalTaskMarker(
-            task_id="bqetl_gud__wait_for_telemetry_derived__clients_last_seen__v1",
-            external_dag_id="bqetl_gud",
-            external_task_id="wait_for_telemetry_derived__clients_last_seen__v1",
-            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=82800)).isoformat() }}",
-        )
-
-        ExternalTaskMarker(
-            task_id="bqetl_unified__wait_for_telemetry_derived__clients_last_seen__v1",
-            external_dag_id="bqetl_unified",
+            task_id="bqetl_analytics_aggregations__wait_for_telemetry_derived__clients_last_seen__v1",
+            external_dag_id="bqetl_analytics_aggregations",
             external_task_id="wait_for_telemetry_derived__clients_last_seen__v1",
-            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=82800)).isoformat() }}",
+            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=78300)).isoformat() }}",
         )
 
         ExternalTaskMarker(
@@ -727,7 +734,7 @@
     crashes_daily_v1.set_upstream(wait_for_copy_deduplicate_all)
 
     firefox_desktop_exact_mau28_by_client_count_dimensions.set_upstream(
-        telemetry_derived__clients_last_seen__v1
+        checks__fail_telemetry_derived__clients_last_seen__v2
     )
 
     firefox_desktop_exact_mau28_by_dimensions.set_upstream(
@@ -735,7 +742,7 @@
     )
 
     firefox_desktop_exact_mau28_by_dimensions_v2.set_upstream(
-        telemetry_derived__clients_last_seen__v1
+        checks__fail_telemetry_derived__clients_last_seen__v2
     )
 
     telemetry_derived__clients_daily__v6.set_upstream(
@@ -746,6 +753,10 @@
 
     telemetry_derived__clients_daily_event__v1.set_upstream(wait_for_event_events)
 
+    telemetry_derived__clients_daily_joined__v1.set_upstream(
+        checks__fail_telemetry_derived__clients_last_seen__v2
+    )
+
     telemetry_derived__clients_daily_joined__v1.set_upstream(crashes_daily_v1)
 
     telemetry_derived__clients_daily_joined__v1.set_upstream(
@@ -756,10 +767,6 @@
         telemetry_derived__clients_daily_event__v1
     )
 
-    telemetry_derived__clients_daily_joined__v1.set_upstream(
-        telemetry_derived__clients_last_seen__v1
-    )
-
     telemetry_derived__clients_first_seen__v1.set_upstream(
         telemetry_derived__clients_daily__v6
     )
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_search_dashboard.py /tmp/workspace/generated-sql/dags/bqetl_search_dashboard.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_search_dashboard.py	2024-05-08 15:25:18.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_search_dashboard.py	2024-05-08 15:26:59.000000000 +0000
@@ -47,10 +47,10 @@
     tags=tags,
 ) as dag:
 
-    wait_for_telemetry_derived__clients_last_seen__v1 = ExternalTaskSensor(
-        task_id="wait_for_telemetry_derived__clients_last_seen__v1",
+    wait_for_checks__fail_telemetry_derived__clients_last_seen__v2 = ExternalTaskSensor(
+        task_id="wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
         external_dag_id="bqetl_main_summary",
-        external_task_id="telemetry_derived__clients_last_seen__v1",
+        external_task_id="checks__fail_telemetry_derived__clients_last_seen__v2",
         execution_delta=datetime.timedelta(seconds=9000),
         check_existence=True,
         mode="reschedule",
@@ -172,7 +172,7 @@
     )
 
     search_derived__desktop_search_aggregates_by_userstate__v1.set_upstream(
-        wait_for_telemetry_derived__clients_last_seen__v1
+        wait_for_checks__fail_telemetry_derived__clients_last_seen__v2
     )
 
     search_derived__desktop_search_aggregates_for_searchreport__v1.set_upstream(
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_unified.py /tmp/workspace/generated-sql/dags/bqetl_unified.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_unified.py	2024-05-08 15:25:18.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_unified.py	2024-05-08 15:27:01.000000000 +0000
@@ -56,6 +56,18 @@
     tags=tags,
 ) as dag:
 
+    wait_for_checks__fail_telemetry_derived__clients_last_seen__v2 = ExternalTaskSensor(
+        task_id="wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
+        external_dag_id="bqetl_main_summary",
+        external_task_id="checks__fail_telemetry_derived__clients_last_seen__v2",
+        execution_delta=datetime.timedelta(seconds=3600),
+        check_existence=True,
+        mode="reschedule",
+        allowed_states=ALLOWED_STATES,
+        failed_states=FAILED_STATES,
+        pool="DATA_ENG_EXTERNALTASKSENSOR",
+    )
+
     wait_for_fenix_derived__clients_last_seen_joined__v1 = ExternalTaskSensor(
         task_id="wait_for_fenix_derived__clients_last_seen_joined__v1",
         external_dag_id="bqetl_glean_usage",
@@ -128,18 +140,6 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_telemetry_derived__clients_last_seen__v1 = ExternalTaskSensor(
-        task_id="wait_for_telemetry_derived__clients_last_seen__v1",
-        external_dag_id="bqetl_main_summary",
-        external_task_id="telemetry_derived__clients_last_seen__v1",
-        execution_delta=datetime.timedelta(seconds=3600),
-        check_existence=True,
-        mode="reschedule",
-        allowed_states=ALLOWED_STATES,
-        failed_states=FAILED_STATES,
-        pool="DATA_ENG_EXTERNALTASKSENSOR",
-    )
-
     wait_for_telemetry_derived__core_clients_last_seen__v1 = ExternalTaskSensor(
         task_id="wait_for_telemetry_derived__core_clients_last_seen__v1",
         external_dag_id="bqetl_core",
@@ -317,6 +317,10 @@
     )
 
     telemetry_derived__unified_metrics__v1.set_upstream(
+        wait_for_checks__fail_telemetry_derived__clients_last_seen__v2
+    )
+
+    telemetry_derived__unified_metrics__v1.set_upstream(
         wait_for_fenix_derived__clients_last_seen_joined__v1
     )
 
@@ -341,9 +345,5 @@
     )
 
     telemetry_derived__unified_metrics__v1.set_upstream(
-        wait_for_telemetry_derived__clients_last_seen__v1
-    )
-
-    telemetry_derived__unified_metrics__v1.set_upstream(
         wait_for_telemetry_derived__core_clients_last_seen__v1
     )
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry/clients_last_seen/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/clients_last_seen/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry/clients_last_seen/metadata.yaml	2024-05-08 15:23:23.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/clients_last_seen/metadata.yaml	2024-05-08 15:24:05.000000000 +0000
@@ -33,4 +33,4 @@
   - workgroup:mozilla-confidential
 references:
   view.sql:
-  - moz-fx-data-shared-prod.telemetry.clients_last_seen_v1
+  - moz-fx-data-shared-prod.telemetry.clients_last_seen_v2
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry/clients_last_seen/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/clients_last_seen/view.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry/clients_last_seen/view.sql	2024-05-08 15:21:51.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/clients_last_seen/view.sql	2024-05-08 15:21:56.000000000 +0000
@@ -4,4 +4,4 @@
 SELECT
   *
 FROM
-  `moz-fx-data-shared-prod.telemetry.clients_last_seen_v1`
+  `moz-fx-data-shared-prod.telemetry.clients_last_seen_v2`

Link to full diff

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants