Changes to active_users_aggregates query for Mobile #5396

lucia-vargas-a · 2024-04-19T00:18:11Z

This PR implements DENG-2989 & DENG-2529 to update the query for active_users_aggregates:

Use baseline_clients_last_seen tables for Mobile in order to retrieve growth metrics and combine with search for is_default_browser and uri_count.
Change language_name which is currently not used in queries for locale which has been requested.

Checklist for reviewer:

Commits should reference a bug or github issue, if relevant (if a bug is referenced, the pull request should include the bug number in the title).
If the PR comes from a fork, trigger integration CI tests by running the Push to upstream workflow and provide the <username>:<branch> of the fork as parameter. The parameter will also show up
in the logs of the manual-trigger-required-for-fork CI task together with more detailed instructions.
If adding a new field to a query, ensure that the schema and dependent downstream schemas have been updated.
When adding a new derived dataset, ensure that data is not available already (fully or partially) and recommend extending an existing dataset in favor of creating new ones. Data can be available in the bigquery-etl repository, looker-hub or in looker-spoke-default.

For modifications to schemas in restricted namespaces (see CODEOWNERS):

Follow the change control procedure

┆Issue is synchronized with this Jira Task

sql_generators/active_users_aggregates_v3/templates/mobile_query.sql

sql_generators/active_users_aggregates_v3/templates/desktop_query.sql

bochocki

The code looks good. I just have a few comments to address before we can merge.

sql_generators/active_users_aggregates_v3/templates/focus_android_view.sql

sql_generators/active_users_aggregates_v3/templates/desktop_query.sql

sql_generators/active_users_aggregates_v3/templates/mobile_query.sql

sql_generators/active_users_aggregates_v3/templates/desktop_query.sql

… attribution only when neccesary to improve performance. Delay update by 1 day to get metrics ping's data.

sql_generators/active_users_aggregates_v3/templates/focus_android_query.sql

sql_generators/active_users_aggregates_v3/templates/metadata.yaml

…growth metrics from clients_last_seen_v2, replace language_name with locale and remove search metrics based on sprint decision (see DENG-1989).

…DENG-2989_update_active_users_aggregates # Conflicts: # sql_generators/active_users_aggregates_v3/templates/desktop_query.sql # sql_generators/active_users_aggregates_v3/templates/desktop_schema.yaml

…ics based on sprint decision (see DENG-1989) and calculate the min metrics ping received between the current and next date, given that these pings can arrive in the same or next date as the baseline ping.

…is separated to [PR-5607](#5607).

lucia-vargas-a · 2024-05-17T14:48:09Z

@bochocki this PR is updated to remove search metrics and get the correct data from the metrics ping. Please give it a another review when you have time.

bochocki

Just one non-blocking question.

bochocki · 2024-05-17T17:30:43Z

sql_generators/active_users_aggregates_v3/templates/mobile_query.sql

-    attribution_data.install_source,
-    attribution_data.adjust_network
+    unioned.*,
+    {% if app_name == "fenix" or  app_name == "firefox_ios" %}


Will this account for, e.g. MozillaOnline? I guess we don't care about attribution in that case?

dataops-ci-bot · 2024-05-20T15:09:52Z

Integration report for "Update app_anme in checks for Focus"

`sql.diff`

Click to expand!

diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_analytics_aggregations.py /tmp/workspace/generated-sql/dags/bqetl_analytics_aggregations.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_analytics_aggregations.py	2024-05-20 14:55:51.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_analytics_aggregations.py	2024-05-20 15:08:59.000000000 +0000
@@ -152,6 +152,18 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
+    wait_for_fenix_derived__metrics_clients_last_seen__v1 = ExternalTaskSensor(
+        task_id="wait_for_fenix_derived__metrics_clients_last_seen__v1",
+        external_dag_id="bqetl_glean_usage",
+        external_task_id="fenix.fenix_derived__metrics_clients_last_seen__v1",
+        execution_delta=datetime.timedelta(seconds=8100),
+        check_existence=True,
+        mode="reschedule",
+        allowed_states=ALLOWED_STATES,
+        failed_states=FAILED_STATES,
+        pool="DATA_ENG_EXTERNALTASKSENSOR",
+    )
+
     wait_for_checks__fail_firefox_ios_derived__clients_activation__v1 = (
         ExternalTaskSensor(
             task_id="wait_for_checks__fail_firefox_ios_derived__clients_activation__v1",
@@ -178,10 +190,10 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_fenix_derived__clients_last_seen_joined__v1 = ExternalTaskSensor(
-        task_id="wait_for_fenix_derived__clients_last_seen_joined__v1",
+    wait_for_checks__fail_org_mozilla_ios_fennec_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
+        task_id="wait_for_checks__fail_org_mozilla_ios_fennec_derived__baseline_clients_last_seen__v1",
         external_dag_id="bqetl_glean_usage",
-        external_task_id="fenix.fenix_derived__clients_last_seen_joined__v1",
+        external_task_id="firefox_ios.checks__fail_org_mozilla_ios_fennec_derived__baseline_clients_last_seen__v1",
         execution_delta=datetime.timedelta(seconds=8100),
         check_existence=True,
         mode="reschedule",
@@ -190,10 +202,10 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_search_derived__mobile_search_clients_daily__v1 = ExternalTaskSensor(
-        task_id="wait_for_search_derived__mobile_search_clients_daily__v1",
-        external_dag_id="bqetl_mobile_search",
-        external_task_id="search_derived__mobile_search_clients_daily__v1",
+    wait_for_checks__fail_org_mozilla_ios_firefox_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
+        task_id="wait_for_checks__fail_org_mozilla_ios_firefox_derived__baseline_clients_last_seen__v1",
+        external_dag_id="bqetl_glean_usage",
+        external_task_id="firefox_ios.checks__fail_org_mozilla_ios_firefox_derived__baseline_clients_last_seen__v1",
         execution_delta=datetime.timedelta(seconds=8100),
         check_existence=True,
         mode="reschedule",
@@ -202,10 +214,10 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_firefox_ios_derived__clients_last_seen_joined__v1 = ExternalTaskSensor(
-        task_id="wait_for_firefox_ios_derived__clients_last_seen_joined__v1",
+    wait_for_checks__fail_org_mozilla_ios_firefoxbeta_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
+        task_id="wait_for_checks__fail_org_mozilla_ios_firefoxbeta_derived__baseline_clients_last_seen__v1",
         external_dag_id="bqetl_glean_usage",
-        external_task_id="firefox_ios.firefox_ios_derived__clients_last_seen_joined__v1",
+        external_task_id="firefox_ios.checks__fail_org_mozilla_ios_firefoxbeta_derived__baseline_clients_last_seen__v1",
         execution_delta=datetime.timedelta(seconds=8100),
         check_existence=True,
         mode="reschedule",
@@ -214,10 +226,58 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_focus_android_derived__clients_last_seen_joined__v1 = ExternalTaskSensor(
-        task_id="wait_for_focus_android_derived__clients_last_seen_joined__v1",
+    wait_for_firefox_ios_derived__metrics_clients_last_seen__v1 = ExternalTaskSensor(
+        task_id="wait_for_firefox_ios_derived__metrics_clients_last_seen__v1",
         external_dag_id="bqetl_glean_usage",
-        external_task_id="focus_android.focus_android_derived__clients_last_seen_joined__v1",
+        external_task_id="firefox_ios.firefox_ios_derived__metrics_clients_last_seen__v1",
+        execution_delta=datetime.timedelta(seconds=8100),
+        check_existence=True,
+        mode="reschedule",
+        allowed_states=ALLOWED_STATES,
+        failed_states=FAILED_STATES,
+        pool="DATA_ENG_EXTERNALTASKSENSOR",
+    )
+
+    wait_for_checks__fail_org_mozilla_focus_beta_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
+        task_id="wait_for_checks__fail_org_mozilla_focus_beta_derived__baseline_clients_last_seen__v1",
+        external_dag_id="bqetl_glean_usage",
+        external_task_id="focus_android.checks__fail_org_mozilla_focus_beta_derived__baseline_clients_last_seen__v1",
+        execution_delta=datetime.timedelta(seconds=8100),
+        check_existence=True,
+        mode="reschedule",
+        allowed_states=ALLOWED_STATES,
+        failed_states=FAILED_STATES,
+        pool="DATA_ENG_EXTERNALTASKSENSOR",
+    )
+
+    wait_for_checks__fail_org_mozilla_focus_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
+        task_id="wait_for_checks__fail_org_mozilla_focus_derived__baseline_clients_last_seen__v1",
+        external_dag_id="bqetl_glean_usage",
+        external_task_id="focus_android.checks__fail_org_mozilla_focus_derived__baseline_clients_last_seen__v1",
+        execution_delta=datetime.timedelta(seconds=8100),
+        check_existence=True,
+        mode="reschedule",
+        allowed_states=ALLOWED_STATES,
+        failed_states=FAILED_STATES,
+        pool="DATA_ENG_EXTERNALTASKSENSOR",
+    )
+
+    wait_for_checks__fail_org_mozilla_focus_nightly_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
+        task_id="wait_for_checks__fail_org_mozilla_focus_nightly_derived__baseline_clients_last_seen__v1",
+        external_dag_id="bqetl_glean_usage",
+        external_task_id="focus_android.checks__fail_org_mozilla_focus_nightly_derived__baseline_clients_last_seen__v1",
+        execution_delta=datetime.timedelta(seconds=8100),
+        check_existence=True,
+        mode="reschedule",
+        allowed_states=ALLOWED_STATES,
+        failed_states=FAILED_STATES,
+        pool="DATA_ENG_EXTERNALTASKSENSOR",
+    )
+
+    wait_for_focus_android_derived__metrics_clients_last_seen__v1 = ExternalTaskSensor(
+        task_id="wait_for_focus_android_derived__metrics_clients_last_seen__v1",
+        external_dag_id="bqetl_glean_usage",
+        external_task_id="focus_android.focus_android_derived__metrics_clients_last_seen__v1",
         execution_delta=datetime.timedelta(seconds=8100),
         check_existence=True,
         mode="reschedule",
@@ -238,10 +298,34 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_focus_ios_derived__clients_last_seen_joined__v1 = ExternalTaskSensor(
-        task_id="wait_for_focus_ios_derived__clients_last_seen_joined__v1",
+    wait_for_checks__fail_org_mozilla_ios_focus_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
+        task_id="wait_for_checks__fail_org_mozilla_ios_focus_derived__baseline_clients_last_seen__v1",
+        external_dag_id="bqetl_glean_usage",
+        external_task_id="focus_ios.checks__fail_org_mozilla_ios_focus_derived__baseline_clients_last_seen__v1",
+        execution_delta=datetime.timedelta(seconds=8100),
+        check_existence=True,
+        mode="reschedule",
+        allowed_states=ALLOWED_STATES,
+        failed_states=FAILED_STATES,
+        pool="DATA_ENG_EXTERNALTASKSENSOR",
+    )
+
+    wait_for_focus_ios_derived__metrics_clients_last_seen__v1 = ExternalTaskSensor(
+        task_id="wait_for_focus_ios_derived__metrics_clients_last_seen__v1",
+        external_dag_id="bqetl_glean_usage",
+        external_task_id="focus_ios.focus_ios_derived__metrics_clients_last_seen__v1",
+        execution_delta=datetime.timedelta(seconds=8100),
+        check_existence=True,
+        mode="reschedule",
+        allowed_states=ALLOWED_STATES,
+        failed_states=FAILED_STATES,
+        pool="DATA_ENG_EXTERNALTASKSENSOR",
+    )
+
+    wait_for_checks__fail_org_mozilla_ios_klar_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
+        task_id="wait_for_checks__fail_org_mozilla_ios_klar_derived__baseline_clients_last_seen__v1",
         external_dag_id="bqetl_glean_usage",
-        external_task_id="focus_ios.focus_ios_derived__clients_last_seen_joined__v1",
+        external_task_id="klar_ios.checks__fail_org_mozilla_ios_klar_derived__baseline_clients_last_seen__v1",
         execution_delta=datetime.timedelta(seconds=8100),
         check_existence=True,
         mode="reschedule",
@@ -250,10 +334,10 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_klar_ios_derived__clients_last_seen_joined__v1 = ExternalTaskSensor(
-        task_id="wait_for_klar_ios_derived__clients_last_seen_joined__v1",
+    wait_for_klar_ios_derived__metrics_clients_last_seen__v1 = ExternalTaskSensor(
+        task_id="wait_for_klar_ios_derived__metrics_clients_last_seen__v1",
         external_dag_id="bqetl_glean_usage",
-        external_task_id="klar_ios.klar_ios_derived__clients_last_seen_joined__v1",
+        external_task_id="klar_ios.klar_ios_derived__metrics_clients_last_seen__v1",
         execution_delta=datetime.timedelta(seconds=8100),
         check_existence=True,
         mode="reschedule",
@@ -349,7 +433,7 @@
 
     checks__fail_fenix_derived__active_users_aggregates__v3 = bigquery_dq_check(
         task_id="checks__fail_fenix_derived__active_users_aggregates__v3",
-        source_table="active_users_aggregates_v3",
+        source_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
         dataset_id="fenix_derived",
         project_id="moz-fx-data-shared-prod",
         is_dq_check_fail=True,
@@ -360,13 +444,13 @@
             "telemetry-alerts@mozilla.com",
         ],
         depends_on_past=False,
-        parameters=["submission_date:DATE:{{ds}}"],
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
         retries=0,
     )
 
     checks__fail_firefox_ios_derived__active_users_aggregates__v3 = bigquery_dq_check(
         task_id="checks__fail_firefox_ios_derived__active_users_aggregates__v3",
-        source_table="active_users_aggregates_v3",
+        source_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
         dataset_id="firefox_ios_derived",
         project_id="moz-fx-data-shared-prod",
         is_dq_check_fail=True,
@@ -377,13 +461,13 @@
             "telemetry-alerts@mozilla.com",
         ],
         depends_on_past=False,
-        parameters=["submission_date:DATE:{{ds}}"],
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
         retries=0,
     )
 
     checks__fail_focus_android_derived__active_users_aggregates__v3 = bigquery_dq_check(
         task_id="checks__fail_focus_android_derived__active_users_aggregates__v3",
-        source_table="active_users_aggregates_v3",
+        source_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
         dataset_id="focus_android_derived",
         project_id="moz-fx-data-shared-prod",
         is_dq_check_fail=True,
@@ -394,13 +478,13 @@
             "telemetry-alerts@mozilla.com",
         ],
         depends_on_past=False,
-        parameters=["submission_date:DATE:{{ds}}"],
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
         retries=0,
     )
 
     checks__fail_focus_ios_derived__active_users_aggregates__v3 = bigquery_dq_check(
         task_id="checks__fail_focus_ios_derived__active_users_aggregates__v3",
-        source_table="active_users_aggregates_v3",
+        source_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
         dataset_id="focus_ios_derived",
         project_id="moz-fx-data-shared-prod",
         is_dq_check_fail=True,
@@ -411,13 +495,13 @@
             "telemetry-alerts@mozilla.com",
         ],
         depends_on_past=False,
-        parameters=["submission_date:DATE:{{ds}}"],
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
         retries=0,
     )
 
     checks__fail_klar_ios_derived__active_users_aggregates__v3 = bigquery_dq_check(
         task_id="checks__fail_klar_ios_derived__active_users_aggregates__v3",
-        source_table="active_users_aggregates_v3",
+        source_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
         dataset_id="klar_ios_derived",
         project_id="moz-fx-data-shared-prod",
         is_dq_check_fail=True,
@@ -428,13 +512,13 @@
             "telemetry-alerts@mozilla.com",
         ],
         depends_on_past=False,
-        parameters=["submission_date:DATE:{{ds}}"],
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
         retries=0,
     )
 
     checks__warn_fenix_derived__active_users_aggregates__v3 = bigquery_dq_check(
         task_id="checks__warn_fenix_derived__active_users_aggregates__v3",
-        source_table="active_users_aggregates_v3",
+        source_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
         dataset_id="fenix_derived",
         project_id="moz-fx-data-shared-prod",
         is_dq_check_fail=False,
@@ -445,14 +529,13 @@
             "telemetry-alerts@mozilla.com",
         ],
         depends_on_past=False,
-        parameters=["submission_date:DATE:{{ds}}"],
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
         retries=0,
     )
 
-    checks__warn_firefox_desktop_derived__active_users_aggregates__v3 = (
-        bigquery_dq_check(
+    checks__warn_firefox_desktop_derived__active_users_aggregates__v3 = bigquery_dq_check(
             task_id="checks__warn_firefox_desktop_derived__active_users_aggregates__v3",
-            source_table="active_users_aggregates_v3",
+        source_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
             dataset_id="firefox_desktop_derived",
             project_id="moz-fx-data-shared-prod",
             is_dq_check_fail=False,
@@ -463,14 +546,13 @@
                 "telemetry-alerts@mozilla.com",
             ],
             depends_on_past=False,
-            parameters=["submission_date:DATE:{{ds}}"],
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
             retries=0,
         )
-    )
 
     checks__warn_firefox_ios_derived__active_users_aggregates__v3 = bigquery_dq_check(
         task_id="checks__warn_firefox_ios_derived__active_users_aggregates__v3",
-        source_table="active_users_aggregates_v3",
+        source_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
         dataset_id="firefox_ios_derived",
         project_id="moz-fx-data-shared-prod",
         is_dq_check_fail=False,
@@ -481,13 +563,13 @@
             "telemetry-alerts@mozilla.com",
         ],
         depends_on_past=False,
-        parameters=["submission_date:DATE:{{ds}}"],
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
         retries=0,
     )
 
     checks__warn_focus_android_derived__active_users_aggregates__v3 = bigquery_dq_check(
         task_id="checks__warn_focus_android_derived__active_users_aggregates__v3",
-        source_table="active_users_aggregates_v3",
+        source_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
         dataset_id="focus_android_derived",
         project_id="moz-fx-data-shared-prod",
         is_dq_check_fail=False,
@@ -498,13 +580,13 @@
             "telemetry-alerts@mozilla.com",
         ],
         depends_on_past=False,
-        parameters=["submission_date:DATE:{{ds}}"],
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
         retries=0,
     )
 
     checks__warn_focus_ios_derived__active_users_aggregates__v3 = bigquery_dq_check(
         task_id="checks__warn_focus_ios_derived__active_users_aggregates__v3",
-        source_table="active_users_aggregates_v3",
+        source_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
         dataset_id="focus_ios_derived",
         project_id="moz-fx-data-shared-prod",
         is_dq_check_fail=False,
@@ -515,13 +597,13 @@
             "telemetry-alerts@mozilla.com",
         ],
         depends_on_past=False,
-        parameters=["submission_date:DATE:{{ds}}"],
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
         retries=0,
     )
 
     checks__warn_klar_ios_derived__active_users_aggregates__v3 = bigquery_dq_check(
         task_id="checks__warn_klar_ios_derived__active_users_aggregates__v3",
-        source_table="active_users_aggregates_v3",
+        source_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
         dataset_id="klar_ios_derived",
         project_id="moz-fx-data-shared-prod",
         is_dq_check_fail=False,
@@ -532,13 +614,13 @@
             "telemetry-alerts@mozilla.com",
         ],
         depends_on_past=False,
-        parameters=["submission_date:DATE:{{ds}}"],
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
         retries=0,
     )
 
     fenix_active_users_aggregates = bigquery_etl_query(
         task_id="fenix_active_users_aggregates",
-        destination_table="active_users_aggregates_v3",
+        destination_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
         dataset_id="fenix_derived",
         project_id="moz-fx-data-shared-prod",
         owner="lvargas@mozilla.com",
@@ -547,13 +629,14 @@
             "lvargas@mozilla.com",
             "telemetry-alerts@mozilla.com",
         ],
-        date_partition_parameter="submission_date",
+        date_partition_parameter=None,
         depends_on_past=False,
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
     )
 
     firefox_desktop_active_users_aggregates = bigquery_etl_query(
         task_id="firefox_desktop_active_users_aggregates",
-        destination_table="active_users_aggregates_v3",
+        destination_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
         dataset_id="firefox_desktop_derived",
         project_id="moz-fx-data-shared-prod",
         owner="lvargas@mozilla.com",
@@ -562,13 +645,14 @@
             "lvargas@mozilla.com",
             "telemetry-alerts@mozilla.com",
         ],
-        date_partition_parameter="submission_date",
+        date_partition_parameter=None,
         depends_on_past=False,
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
     )
 
     firefox_ios_active_users_aggregates = bigquery_etl_query(
         task_id="firefox_ios_active_users_aggregates",
-        destination_table="active_users_aggregates_v3",
+        destination_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
         dataset_id="firefox_ios_derived",
         project_id="moz-fx-data-shared-prod",
         owner="lvargas@mozilla.com",
@@ -577,13 +661,14 @@
             "lvargas@mozilla.com",
             "telemetry-alerts@mozilla.com",
         ],
-        date_partition_parameter="submission_date",
+        date_partition_parameter=None,
         depends_on_past=False,
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
     )
 
     focus_android_active_users_aggregates = bigquery_etl_query(
         task_id="focus_android_active_users_aggregates",
-        destination_table="active_users_aggregates_v3",
+        destination_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
         dataset_id="focus_android_derived",
         project_id="moz-fx-data-shared-prod",
         owner="lvargas@mozilla.com",
@@ -592,13 +677,14 @@
             "lvargas@mozilla.com",
             "telemetry-alerts@mozilla.com",
         ],
-        date_partition_parameter="submission_date",
+        date_partition_parameter=None,
         depends_on_past=False,
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
     )
 
     focus_ios_active_users_aggregates = bigquery_etl_query(
         task_id="focus_ios_active_users_aggregates",
-        destination_table="active_users_aggregates_v3",
+        destination_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
         dataset_id="focus_ios_derived",
         project_id="moz-fx-data-shared-prod",
         owner="lvargas@mozilla.com",
@@ -607,13 +693,14 @@
             "lvargas@mozilla.com",
             "telemetry-alerts@mozilla.com",
         ],
-        date_partition_parameter="submission_date",
+        date_partition_parameter=None,
         depends_on_past=False,
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
     )
 
     klar_ios_active_users_aggregates = bigquery_etl_query(
         task_id="klar_ios_active_users_aggregates",
-        destination_table="active_users_aggregates_v3",
+        destination_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
         dataset_id="klar_ios_derived",
         project_id="moz-fx-data-shared-prod",
         owner="lvargas@mozilla.com",
@@ -622,8 +709,9 @@
             "lvargas@mozilla.com",
             "telemetry-alerts@mozilla.com",
         ],
-        date_partition_parameter="submission_date",
+        date_partition_parameter=None,
         depends_on_past=False,
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
     )
 
     telemetry_derived__cohort_daily_statistics__v1 = bigquery_etl_query(
@@ -767,19 +855,27 @@
     )
 
     fenix_active_users_aggregates.set_upstream(
-        wait_for_checks__fail_firefox_ios_derived__clients_activation__v1
+        wait_for_checks__fail_org_mozilla_fenix_derived__baseline_clients_last_seen__v1
     )
 
     fenix_active_users_aggregates.set_upstream(
-        wait_for_checks__fail_firefox_ios_derived__firefox_ios_clients__v1
+        wait_for_checks__fail_org_mozilla_fenix_nightly_derived__baseline_clients_last_seen__v1
+    )
+
+    fenix_active_users_aggregates.set_upstream(
+        wait_for_checks__fail_org_mozilla_fennec_aurora_derived__baseline_clients_last_seen__v1
     )
 
     fenix_active_users_aggregates.set_upstream(
-        wait_for_fenix_derived__clients_last_seen_joined__v1
+        wait_for_checks__fail_org_mozilla_firefox_beta_derived__baseline_clients_last_seen__v1
     )
 
     fenix_active_users_aggregates.set_upstream(
-        wait_for_search_derived__mobile_search_clients_daily__v1
+        wait_for_checks__fail_org_mozilla_firefox_derived__baseline_clients_last_seen__v1
+    )
+
+    fenix_active_users_aggregates.set_upstream(
+        wait_for_fenix_derived__metrics_clients_last_seen__v1
     )
 
     firefox_desktop_active_users_aggregates.set_upstream(
@@ -787,75 +883,63 @@
     )
 
     firefox_ios_active_users_aggregates.set_upstream(
-        wait_for_checks__fail_fenix_derived__firefox_android_clients__v1
+        wait_for_checks__fail_firefox_ios_derived__clients_activation__v1
     )
 
     firefox_ios_active_users_aggregates.set_upstream(
-        wait_for_checks__fail_firefox_ios_derived__clients_activation__v1
+        wait_for_checks__fail_firefox_ios_derived__firefox_ios_clients__v1
     )
 
     firefox_ios_active_users_aggregates.set_upstream(
-        wait_for_checks__fail_firefox_ios_derived__firefox_ios_clients__v1
+        wait_for_checks__fail_org_mozilla_ios_fennec_derived__baseline_clients_last_seen__v1
     )
 
     firefox_ios_active_users_aggregates.set_upstream(
-        wait_for_firefox_ios_derived__clients_last_seen_joined__v1
+        wait_for_checks__fail_org_mozilla_ios_firefox_derived__baseline_clients_last_seen__v1
     )
 
     firefox_ios_active_users_aggregates.set_upstream(
-        wait_for_search_derived__mobile_search_clients_daily__v1
+        wait_for_checks__fail_org_mozilla_ios_firefoxbeta_derived__baseline_clients_last_seen__v1
     )
 
-    focus_android_active_users_aggregates.set_upstream(
-        wait_for_focus_android_derived__clients_last_seen_joined__v1
+    firefox_ios_active_users_aggregates.set_upstream(
+        wait_for_firefox_ios_derived__metrics_clients_last_seen__v1
     )
 
     focus_android_active_users_aggregates.set_upstream(
-        wait_for_search_derived__mobile_search_clients_daily__v1
+        wait_for_checks__fail_org_mozilla_focus_beta_derived__baseline_clients_last_seen__v1
     )
 
     focus_android_active_users_aggregates.set_upstream(
-        wait_for_telemetry_derived__core_clients_last_seen__v1
+        wait_for_checks__fail_org_mozilla_focus_derived__baseline_clients_last_seen__v1
     )
 
-    focus_ios_active_users_aggregates.set_upstream(
-        wait_for_checks__fail_fenix_derived__firefox_android_clients__v1
+    focus_android_active_users_aggregates.set_upstream(
+        wait_for_checks__fail_org_mozilla_focus_nightly_derived__baseline_clients_last_seen__v1
     )
 
-    focus_ios_active_users_aggregates.set_upstream(
-        wait_for_checks__fail_firefox_ios_derived__clients_activation__v1
+    focus_android_active_users_aggregates.set_upstream(
+        wait_for_focus_android_derived__metrics_clients_last_seen__v1
     )
 
-    focus_ios_active_users_aggregates.set_upstream(
-        wait_for_checks__fail_firefox_ios_derived__firefox_ios_clients__v1
+    focus_android_active_users_aggregates.set_upstream(
+        wait_for_telemetry_derived__core_clients_last_seen__v1
     )
 
     focus_ios_active_users_aggregates.set_upstream(
-        wait_for_focus_ios_derived__clients_last_seen_joined__v1
+        wait_for_checks__fail_org_mozilla_ios_focus_derived__baseline_clients_last_seen__v1
     )
 
     focus_ios_active_users_aggregates.set_upstream(
-        wait_for_search_derived__mobile_search_clients_daily__v1
-    )
-
-    klar_ios_active_users_aggregates.set_upstream(
-        wait_for_checks__fail_fenix_derived__firefox_android_clients__v1
-    )
-
-    klar_ios_active_users_aggregates.set_upstream(
-        wait_for_checks__fail_firefox_ios_derived__clients_activation__v1
-    )
-
-    klar_ios_active_users_aggregates.set_upstream(
-        wait_for_checks__fail_firefox_ios_derived__firefox_ios_clients__v1
+        wait_for_focus_ios_derived__metrics_clients_last_seen__v1
     )
 
     klar_ios_active_users_aggregates.set_upstream(
-        wait_for_klar_ios_derived__clients_last_seen_joined__v1
+        wait_for_checks__fail_org_mozilla_ios_klar_derived__baseline_clients_last_seen__v1
     )
 
     klar_ios_active_users_aggregates.set_upstream(
-        wait_for_search_derived__mobile_search_clients_daily__v1
+        wait_for_klar_ios_derived__metrics_clients_last_seen__v1
     )
 
     telemetry_derived__cohort_daily_statistics__v1.set_upstream(
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_glam_export.py /tmp/workspace/generated-sql/dags/bqetl_glam_export.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_glam_export.py	2024-05-20 14:55:51.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_glam_export.py	2024-05-20 15:09:03.000000000 +0000
@@ -45,7 +45,7 @@
 with DAG(
     "bqetl_glam_export",
     default_args=default_args,
-    schedule_interval="0 19 * * *",
+    schedule_interval="0 8 * * *",
     doc_md=docs,
     tags=tags,
 ) as dag:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_glam_refresh_aggregates.py /tmp/workspace/generated-sql/dags/bqetl_glam_refresh_aggregates.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_glam_refresh_aggregates.py	2024-05-20 14:55:51.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_glam_refresh_aggregates.py	2024-05-20 15:09:00.000000000 +0000
@@ -54,7 +54,7 @@
         task_id="wait_for_extracts",
         external_dag_id="glam",
         external_task_id="extracts",
-        execution_delta=datetime.timedelta(days=-1, seconds=68400),
+        execution_delta=datetime.timedelta(seconds=21600),
         check_existence=True,
         mode="reschedule",
         allowed_states=ALLOWED_STATES,
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_glean_usage.py /tmp/workspace/generated-sql/dags/bqetl_glean_usage.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_glean_usage.py	2024-05-20 14:55:51.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_glean_usage.py	2024-05-20 15:09:03.000000000 +0000
@@ -778,6 +778,21 @@
         task_group=task_group_focus_android,
     )
 
+    with TaskGroup(
+        "checks__fail_org_mozilla_focus_beta_derived__baseline_clients_last_seen__v1_external",
+        parent_group=task_group_focus_android,
+    ) as checks__fail_org_mozilla_focus_beta_derived__baseline_clients_last_seen__v1_external:
+        ExternalTaskMarker(
+            task_id="bqetl_analytics_aggregations__wait_for_checks__fail_org_mozilla_focus_beta_derived__baseline_clients_last_seen__v1",
+            external_dag_id="bqetl_analytics_aggregations",
+            external_task_id="wait_for_checks__fail_org_mozilla_focus_beta_derived__baseline_clients_last_seen__v1",
+            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=78300)).isoformat() }}",
+        )
+
+        checks__fail_org_mozilla_focus_beta_derived__baseline_clients_last_seen__v1_external.set_upstream(
+            checks__fail_org_mozilla_focus_beta_derived__baseline_clients_last_seen__v1
+        )
+
     checks__fail_org_mozilla_focus_derived__baseline_clients_last_seen__v1 = bigquery_dq_check(
         task_id="checks__fail_org_mozilla_focus_derived__baseline_clients_last_seen__v1",
         source_table="baseline_clients_last_seen_v1",
@@ -797,6 +812,13 @@
         parent_group=task_group_focus_android,
     ) as checks__fail_org_mozilla_focus_derived__baseline_clients_last_seen__v1_external:
         ExternalTaskMarker(
+            task_id="bqetl_analytics_aggregations__wait_for_checks__fail_org_mozilla_focus_derived__baseline_clients_last_seen__v1",
+            external_dag_id="bqetl_analytics_aggregations",
+            external_task_id="wait_for_checks__fail_org_mozilla_focus_derived__baseline_clients_last_seen__v1",
+            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=78300)).isoformat() }}",
+        )
+
+        ExternalTaskMarker(
             task_id="bqetl_nondesktop__wait_for_checks__fail_org_mozilla_focus_derived__baseline_clients_last_seen__v1",
             external_dag_id="bqetl_nondesktop",
             external_task_id="wait_for_checks__fail_org_mozilla_focus_derived__baseline_clients_last_seen__v1",
@@ -828,6 +850,21 @@
         task_group=task_group_focus_android,
     )
 
+    with TaskGroup(
+        "checks__fail_org_mozilla_focus_nightly_derived__baseline_clients_last_seen__v1_external",
+        parent_group=task_group_focus_android,
+    ) as checks__fail_org_mozilla_focus_nightly_derived__baseline_clients_last_seen__v1_external:
+        ExternalTaskMarker(
+            task_id="bqetl_analytics_aggregations__wait_for_checks__fail_org_mozilla_focus_nightly_derived__baseline_clients_last_seen__v1",
+            external_dag_id="bqetl_analytics_aggregations",
+            external_task_id="wait_for_checks__fail_org_mozilla_focus_nightly_derived__baseline_clients_last_seen__v1",
+            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=78300)).isoformat() }}",
+        )
+
+        checks__fail_org_mozilla_focus_nightly_derived__baseline_clients_last_seen__v1_external.set_upstream(
+            checks__fail_org_mozilla_focus_nightly_derived__baseline_clients_last_seen__v1
+        )
+
     checks__fail_org_mozilla_ios_fennec_derived__baseline_clients_last_seen__v1 = bigquery_dq_check(
         task_id="checks__fail_org_mozilla_ios_fennec_derived__baseline_clients_last_seen__v1",
         source_table="baseline_clients_last_seen_v1",
@@ -847,6 +884,13 @@
         parent_group=task_group_firefox_ios,
     ) as checks__fail_org_mozilla_ios_fennec_derived__baseline_clients_last_seen__v1_external:
         ExternalTaskMarker(
+            task_id="bqetl_analytics_aggregations__wait_for_checks__fail_org_mozilla_ios_fennec_derived__baseline_clients_last_seen__v1",
+            external_dag_id="bqetl_analytics_aggregations",
+            external_task_id="wait_for_checks__fail_org_mozilla_ios_fennec_derived__baseline_clients_last_seen__v1",
+            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=78300)).isoformat() }}",
+        )
+
+        ExternalTaskMarker(
             task_id="bqetl_firefox_ios__wait_for_checks__fail_org_mozilla_ios_fennec_derived__baseline_clients_last_seen__v1",
             external_dag_id="bqetl_firefox_ios",
             external_task_id="wait_for_checks__fail_org_mozilla_ios_fennec_derived__baseline_clients_last_seen__v1",
@@ -904,6 +948,13 @@
         parent_group=task_group_firefox_ios,
     ) as checks__fail_org_mozilla_ios_firefox_derived__baseline_clients_last_seen__v1_external:
         ExternalTaskMarker(
+            task_id="bqetl_analytics_aggregations__wait_for_checks__fail_org_mozilla_ios_firefox_derived__baseline_clients_last_seen__v1",
+            external_dag_id="bqetl_analytics_aggregations",
+            external_task_id="wait_for_checks__fail_org_mozilla_ios_firefox_derived__baseline_clients_last_seen__v1",
+            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=78300)).isoformat() }}",
+        )
+
+        ExternalTaskMarker(
             task_id="bqetl_firefox_ios__wait_for_checks__fail_org_mozilla_ios_firefox_derived__baseline_clients_last_seen__v1",
             external_dag_id="bqetl_firefox_ios",
             external_task_id="wait_for_checks__fail_org_mozilla_ios_firefox_derived__baseline_clients_last_seen__v1",
@@ -947,6 +998,13 @@
         parent_group=task_group_firefox_ios,
     ) as checks__fail_org_mozilla_ios_firefoxbeta_derived__baseline_clients_last_seen__v1_external:
         ExternalTaskMarker(
+            task_id="bqetl_analytics_aggregations__wait_for_checks__fail_org_mozilla_ios_firefoxbeta_derived__baseline_clients_last_seen__v1",
+            external_dag_id="bqetl_analytics_aggregations",
+            external_task_id="wait_for_checks__fail_org_mozilla_ios_firefoxbeta_derived__baseline_clients_last_seen__v1",
+            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=78300)).isoformat() }}",
+        )
+
+        ExternalTaskMarker(
             task_id="bqetl_firefox_ios__wait_for_checks__fail_org_mozilla_ios_firefoxbeta_derived__baseline_clients_last_seen__v1",
             external_dag_id="bqetl_firefox_ios",
             external_task_id="wait_for_checks__fail_org_mozilla_ios_firefoxbeta_derived__baseline_clients_last_seen__v1",
@@ -990,6 +1048,13 @@
         parent_group=task_group_focus_ios,
     ) as checks__fail_org_mozilla_ios_focus_derived__baseline_clients_last_seen__v1_external:
         ExternalTaskMarker(
+            task_id="bqetl_analytics_aggregations__wait_for_checks__fail_org_mozilla_ios_focus_derived__baseline_clients_last_seen__v1",
+            external_dag_id="bqetl_analytics_aggregations",
+            external_task_id="wait_for_checks__fail_org_mozilla_ios_focus_derived__baseline_clients_last_seen__v1",
+            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=78300)).isoformat() }}",
+        )
+
+        ExternalTaskMarker(
             task_id="bqetl_nondesktop__wait_for_checks__fail_org_mozilla_ios_focus_derived__baseline_clients_last_seen__v1",
             external_dag_id="bqetl_nondesktop",
             external_task_id="wait_for_checks__fail_org_mozilla_ios_focus_derived__baseline_clients_last_seen__v1",
@@ -1026,6 +1091,13 @@
         parent_group=task_group_klar_ios,
     ) as checks__fail_org_mozilla_ios_klar_derived__baseline_clients_last_seen__v1_external:
         ExternalTaskMarker(
+            task_id="bqetl_analytics_aggregations__wait_for_checks__fail_org_mozilla_ios_klar_derived__baseline_clients_last_seen__v1",
+            external_dag_id="bqetl_analytics_aggregations",
+            external_task_id="wait_for_checks__fail_org_mozilla_ios_klar_derived__baseline_clients_last_seen__v1",
+            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=78300)).isoformat() }}",
+        )
+
+        ExternalTaskMarker(
             task_id="bqetl_nondesktop__wait_for_checks__fail_org_mozilla_ios_klar_derived__baseline_clients_last_seen__v1",
             external_dag_id="bqetl_nondesktop",
             external_task_id="wait_for_checks__fail_org_mozilla_ios_klar_derived__baseline_clients_last_seen__v1",
@@ -1600,13 +1672,6 @@
         parent_group=task_group_fenix,
     ) as fenix_derived__clients_last_seen_joined__v1_external:
         ExternalTaskMarker(
-            task_id="bqetl_analytics_aggregations__wait_for_fenix_derived__clients_last_seen_joined__v1",
-            external_dag_id="bqetl_analytics_aggregations",
-            external_task_id="wait_for_fenix_derived__clients_last_seen_joined__v1",
-            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=78300)).isoformat() }}",
-        )
-
-        ExternalTaskMarker(
             task_id="bqetl_kpis_shredder__wait_for_fenix_derived__clients_last_seen_joined__v1",
             external_dag_id="bqetl_kpis_shredder",
             external_task_id="wait_for_fenix_derived__clients_last_seen_joined__v1",
@@ -1659,6 +1724,13 @@
         parent_group=task_group_fenix,
     ) as fenix_derived__metrics_clients_last_seen__v1_external:
         ExternalTaskMarker(
+            task_id="bqetl_analytics_aggregations__wait_for_fenix_derived__metrics_clients_last_seen__v1",
+            external_dag_id="bqetl_analytics_aggregations",
+            external_task_id="wait_for_fenix_derived__metrics_clients_last_seen__v1",
+            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=78300)).isoformat() }}",
+        )
+
+        ExternalTaskMarker(
             task_id="bqetl_data_observability_test_data_copy__wait_for_fenix_derived__metrics_clients_last_seen__v1",
             external_dag_id="bqetl_data_observability_test_data_copy",
             external_task_id="wait_for_fenix_derived__metrics_clients_last_seen__v1",
@@ -2074,13 +2146,6 @@
         parent_group=task_group_firefox_ios,
     ) as firefox_ios_derived__clients_last_seen_joined__v1_external:
         ExternalTaskMarker(
-            task_id="bqetl_analytics_aggregations__wait_for_firefox_ios_derived__clients_last_seen_joined__v1",
-            external_dag_id="bqetl_analytics_aggregations",
-            external_task_id="wait_for_firefox_ios_derived__clients_last_seen_joined__v1",
-            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=78300)).isoformat() }}",
-        )
-
-        ExternalTaskMarker(
             task_id="bqetl_kpis_shredder__wait_for_firefox_ios_derived__clients_last_seen_joined__v1",
             external_dag_id="bqetl_kpis_shredder",
             external_task_id="wait_for_firefox_ios_derived__clients_last_seen_joined__v1",
@@ -2121,6 +2186,21 @@
         task_group=task_group_firefox_ios,
     )
 
+    with TaskGroup(
+        "firefox_ios_derived__metrics_clients_last_seen__v1_external",
+        parent_group=task_group_firefox_ios,
+    ) as firefox_ios_derived__metrics_clients_last_seen__v1_external:
+        ExternalTaskMarker(
+            task_id="bqetl_analytics_aggregations__wait_for_firefox_ios_derived__metrics_clients_last_seen__v1",
+            external_dag_id="bqetl_analytics_aggregations",
+            external_task_id="wait_for_firefox_ios_derived__metrics_clients_last_seen__v1",
+            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=78300)).isoformat() }}",
+        )
+
+        firefox_ios_derived__metrics_clients_last_seen__v1_external.set_upstream(
+            firefox_ios_derived__metrics_clients_last_seen__v1
+        )
+
     firefox_reality_derived__clients_last_seen_joined__v1 = bigquery_etl_query(
         task_id="firefox_reality_derived__clients_last_seen_joined__v1",
         destination_table="clients_last_seen_joined_v1",
@@ -2228,13 +2308,6 @@
         parent_group=task_group_focus_android,
     ) as focus_android_derived__clients_last_seen_joined__v1_external:
         ExternalTaskMarker(
-            task_id="bqetl_analytics_aggregations__wait_for_focus_android_derived__clients_last_seen_joined__v1",
-            external_dag_id="bqetl_analytics_aggregations",
-            external_task_id="wait_for_focus_android_derived__clients_last_seen_joined__v1",
-            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=78300)).isoformat() }}",
-        )
-
-        ExternalTaskMarker(
             task_id="bqetl_unified__wait_for_focus_android_derived__clients_last_seen_joined__v1",
             external_dag_id="bqetl_unified",
             external_task_id="wait_for_focus_android_derived__clients_last_seen_joined__v1",
@@ -2269,6 +2342,21 @@
         task_group=task_group_focus_android,
     )
 
+    with TaskGroup(
+        "focus_android_derived__metrics_clients_last_seen__v1_external",
+        parent_group=task_group_focus_android,
+    ) as focus_android_derived__metrics_clients_last_seen__v1_external:
+        ExternalTaskMarker(
+            task_id="bqetl_analytics_aggregations__wait_for_focus_android_derived__metrics_clients_last_seen__v1",
+            external_dag_id="bqetl_analytics_aggregations",
+            external_task_id="wait_for_focus_android_derived__metrics_clients_last_seen__v1",
+            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=78300)).isoformat() }}",
+        )
+
+        focus_android_derived__metrics_clients_last_seen__v1_external.set_upstream(
+            focus_android_derived__metrics_clients_last_seen__v1
+        )
+
     focus_ios_derived__clients_last_seen_joined__v1 = bigquery_etl_query(
         task_id="focus_ios_derived__clients_last_seen_joined__v1",
         destination_table="clients_last_seen_joined_v1",
@@ -2286,13 +2374,6 @@
         parent_group=task_group_focus_ios,
     ) as focus_ios_derived__clients_last_seen_joined__v1_external:
         ExternalTaskMarker(
-            task_id="bqetl_analytics_aggregations__wait_for_focus_ios_derived__clients_last_seen_joined__v1",
-            external_dag_id="bqetl_analytics_aggregations",
-            external_task_id="wait_for_focus_ios_derived__clients_last_seen_joined__v1",
-            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=78300)).isoformat() }}",
-        )
-
-        ExternalTaskMarker(
             task_id="bqetl_kpis_shredder__wait_for_focus_ios_derived__clients_last_seen_joined__v1",
             external_dag_id="bqetl_kpis_shredder",
             external_task_id="wait_for_focus_ios_derived__clients_last_seen_joined__v1",
@@ -2333,6 +2414,21 @@
         task_group=task_group_focus_ios,
     )
 
+    with TaskGroup(
+        "focus_ios_derived__metrics_clients_last_seen__v1_external",
+        parent_group=task_group_focus_ios,
+    ) as focus_ios_derived__metrics_clients_last_seen__v1_external:
+        ExternalTaskMarker(
+            task_id="bqetl_analytics_aggregations__wait_for_focus_ios_derived__metrics_clients_last_seen__v1",
+            external_dag_id="bqetl_analytics_aggregations",
+            external_task_id="wait_for_focus_ios_derived__metrics_clients_last_seen__v1",
+            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=78300)).isoformat() }}",
+        )
+
+        focus_ios_derived__metrics_clients_last_seen__v1_external.set_upstream(
+            focus_ios_derived__metrics_clients_last_seen__v1
+        )
+
     glean_dictionary_derived__events_stream__v1 = bigquery_etl_query(
         task_id="glean_dictionary_derived__events_stream__v1",
         destination_table="events_stream_v1",
@@ -2404,13 +2500,6 @@
         parent_group=task_group_klar_ios,
     ) as klar_ios_derived__clients_last_seen_joined__v1_external:
         ExternalTaskMarker(
-            task_id="bqetl_analytics_aggregations__wait_for_klar_ios_derived__clients_last_seen_joined__v1",
-            external_dag_id="bqetl_analytics_aggregations",
-            external_task_id="wait_for_klar_ios_derived__clients_last_seen_joined__v1",
-            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=78300)).isoformat() }}",
-        )
-
-        ExternalTaskMarker(
             task_id="bqetl_kpis_shredder__wait_for_klar_ios_derived__clients_last_seen_joined__v1",
             external_dag_id="bqetl_kpis_shredder",
             external_task_id="wait_for_klar_ios_derived__clients_last_seen_joined__v1",
@@ -2451,6 +2540,21 @@
         task_group=task_group_klar_ios,
     )
 
+    with TaskGroup(
+        "klar_ios_derived__metrics_clients_last_seen__v1_external",
+        parent_group=task_group_klar_ios,
+    ) as klar_ios_derived__metrics_clients_last_seen__v1_external:
+        ExternalTaskMarker(
+            task_id="bqetl_analytics_aggregations__wait_for_klar_ios_derived__metrics_clients_last_seen__v1",
+            external_dag_id="bqetl_analytics_aggregations",
+            external_task_id="wait_for_klar_ios_derived__metrics_clients_last_seen__v1",
+            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=78300)).isoformat() }}",
+        )
+
+        klar_ios_derived__metrics_clients_last_seen__v1_external.set_upstream(
+            klar_ios_derived__metrics_clients_last_seen__v1
+        )
+
     lockwise_android_derived__clients_last_seen_joined__v1 = bigquery_etl_query(
         task_id="lockwise_android_derived__clients_last_seen_joined__v1",
         destination_table="clients_last_seen_joined_v1",
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_mobile_search.py /tmp/workspace/generated-sql/dags/bqetl_mobile_search.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_mobile_search.py	2024-05-20 14:55:51.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_mobile_search.py	2024-05-20 15:08:56.000000000 +0000
@@ -100,13 +100,6 @@
         "search_derived__mobile_search_clients_daily__v1_external",
     ) as search_derived__mobile_search_clients_daily__v1_external:
         ExternalTaskMarker(
-            task_id="bqetl_analytics_aggregations__wait_for_search_derived__mobile_search_clients_daily__v1",
-            external_dag_id="bqetl_analytics_aggregations",
-            external_task_id="wait_for_search_derived__mobile_search_clients_daily__v1",
-            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=78300)).isoformat() }}",
-        )
-
-        ExternalTaskMarker(
             task_id="bqetl_kpis_shredder__wait_for_search_derived__mobile_search_clients_daily__v1",
             external_dag_id="bqetl_kpis_shredder",
             external_task_id="wait_for_search_derived__mobile_search_clients_daily__v1",
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/data-observability-dev/fenix_derived/firefox_android_anonymised_v1/query.sql /tmp/workspace/generated-sql/sql/data-observability-dev/fenix_derived/firefox_android_anonymised_v1/query.sql
--- /tmp/workspace/main-generated-sql/sql/data-observability-dev/fenix_derived/firefox_android_anonymised_v1/query.sql	2024-05-20 14:51:31.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/data-observability-dev/fenix_derived/firefox_android_anonymised_v1/query.sql	2024-05-20 14:52:22.000000000 +0000
@@ -2,7 +2,7 @@
   SHA256(client_id) AS client_id_hashed,
   first_seen_date,
   channel,
-  CAST(ASCII(SHA256(client_id)) AS STRING) AS funky_column,
+  ASCII(SHA256(client_id)) AS funky_column,
 FROM
   `data-observability-dev.fenix.firefox_android_clients`
 WHERE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-glam-prod-fca7/glam_etl/glam_desktop_beta_aggregates_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-glam-prod-fca7/glam_etl/glam_desktop_beta_aggregates_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-glam-prod-fca7/glam_etl/glam_desktop_beta_aggregates_v1/metadata.yaml	2024-05-20 14:52:14.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-glam-prod-fca7/glam_etl/glam_desktop_beta_aggregates_v1/metadata.yaml	2024-05-20 15:04:55.000000000 +0000
@@ -12,7 +12,7 @@
   depends_on:
   - task_id: extracts
     dag_name: glam
-    execution_delta: -5h
+    execution_delta: 6h
 
 # Generated by bigquery_etl.dependency
 references:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-glam-prod-fca7/glam_etl/glam_desktop_nightly_aggregates_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-glam-prod-fca7/glam_etl/glam_desktop_nightly_aggregates_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-glam-prod-fca7/glam_etl/glam_desktop_nightly_aggregates_v1/metadata.yaml	2024-05-20 14:52:14.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-glam-prod-fca7/glam_etl/glam_desktop_nightly_aggregates_v1/metadata.yaml	2024-05-20 15:04:55.000000000 +0000
@@ -12,7 +12,7 @@
   depends_on:
   - task_id: extracts
     dag_name: glam
-    execution_delta: -5h
+    execution_delta: 6h
 
 # Generated by bigquery_etl.dependency
 references:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-glam-prod-fca7/glam_etl/glam_desktop_release_aggregates_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-glam-prod-fca7/glam_etl/glam_desktop_release_aggregates_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-glam-prod-fca7/glam_etl/glam_desktop_release_aggregates_v1/metadata.yaml	2024-05-20 14:52:14.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-glam-prod-fca7/glam_etl/glam_desktop_release_aggregates_v1/metadata.yaml	2024-05-20 15:04:55.000000000 +0000
@@ -12,7 +12,7 @@
   depends_on:
   - task_id: extracts
     dag_name: glam
-    execution_delta: -5h
+    execution_delta: 6h
 
 # Generated by bigquery_etl.dependency
 references:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_cirrus_derived/baseline_clients_daily_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_cirrus_derived/baseline_clients_daily_v1/query.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_cirrus_derived/baseline_clients_daily_v1/query.sql	2024-05-20 14:52:07.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_cirrus_derived/baseline_clients_daily_v1/query.sql	2024-05-20 14:54:06.000000000 +0000
@@ -23,8 +23,6 @@
     normalized_channel,
     normalized_os,
     normalized_os_version,
-    CAST(NULL AS STRING) AS distribution_id,
-    metadata.geo.subdivision1 AS geo_subdivision,
   FROM
     `accounts_cirrus_stable.baseline_v1`
   -- Baseline pings with 'foreground' reason were first introduced in early April 2020;
@@ -91,8 +89,6 @@
     udf.mode_last(ARRAY_AGG(device_manufacturer) OVER w1) AS device_manufacturer,
     udf.mode_last(ARRAY_AGG(device_model) OVER w1) AS device_model,
     udf.mode_last(ARRAY_AGG(telemetry_sdk_build) OVER w1) AS telemetry_sdk_build,
-    udf.mode_last(ARRAY_AGG(distribution_id) OVER w1) AS distribution_id,
-    udf.mode_last(ARRAY_AGG(geo_subdivision) OVER w1) AS geo_subdivision,
   FROM
     with_date_offsets
   WHERE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/bedrock_derived/event_monitoring_live_v1/materialized_view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/bedrock_derived/event_monitoring_live_v1/materialized_view.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/bedrock_derived/event_monitoring_live_v1/materialized_view.sql	2024-05-20 14:52:08.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/bedrock_derived/event_monitoring_live_v1/materialized_view.sql	2024-05-20 14:54:06.000000000 +0000
@@ -50,7 +50,7 @@
         client_info.app_display_version AS version,
         ping_info
       FROM
-        `moz-fx-data-shared-prod.bedrock_live.events_v1`
+        `moz-fx-data-shared-prod.bedrock_live.interaction_v1`
       UNION ALL
       SELECT
         submission_timestamp,
@@ -60,7 +60,7 @@
         client_info.app_display_version AS version,
         ping_info
       FROM
-        `moz-fx-data-shared-prod.bedrock_live.interaction_v1`
+        `moz-fx-data-shared-prod.bedrock_live.non_interaction_v1`
       UNION ALL
       SELECT
         submission_timestamp,
@@ -70,7 +70,7 @@
         client_info.app_display_version AS version,
         ping_info
       FROM
-        `moz-fx-data-shared-prod.bedrock_live.non_interaction_v1`
+        `moz-fx-data-shared-prod.bedrock_live.events_v1`
     )
   CROSS JOIN
     UNNEST(events) AS event,
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/burnham_derived/baseline_clients_daily_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham_derived/baseline_clients_daily_v1/query.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/burnham_derived/baseline_clients_daily_v1/query.sql	2024-05-20 14:52:07.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham_derived/baseline_clients_daily_v1/query.sql	2024-05-20 14:54:07.000000000 +0000
@@ -23,8 +23,6 @@
     normalized_channel,
     normalized_os,
     normalized_os_version,
-    CAST(NULL AS STRING) AS distribution_id,
-    metadata.geo.subdivision1 AS geo_subdivision,
   FROM
     `burnham_stable.baseline_v1`
   -- Baseline pings with 'foreground' reason were first introduced in early April 2020;
@@ -91,8 +89,6 @@
     udf.mode_last(ARRAY_AGG(device_manufacturer) OVER w1) AS device_manufacturer,
     udf.mode_last(ARRAY_AGG(device_model) OVER w1) AS device_model,
     udf.mode_last(ARRAY_AGG(telemetry_sdk_build) OVER w1) AS telemetry_sdk_build,
-    udf.mode_last(ARRAY_AGG(distribution_id) OVER w1) AS distribution_id,
-    udf.mode_last(ARRAY_AGG(geo_subdivision) OVER w1) AS geo_subdivision,
   FROM
     with_date_offsets
   WHERE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml	2024-05-20 14:51:30.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml	2024-05-20 15:02:40.000000000 +0000
@@ -1,49 +1,49 @@
 fields:
-- mode: NULLABLE
-  name: submission_date
+- name: submission_date
   type: DATE
-- mode: NULLABLE
-  name: source
+  mode: NULLABLE
+- name: source
   type: STRING
-- mode: NULLABLE
-  name: event_type
+  mode: NULLABLE
+- name: event_type
   type: STRING
-- mode: NULLABLE
-  name: form_factor
+  mode: NULLABLE
+- name: form_factor
   type: STRING
-- mode: NULLABLE
-  name: country
+  mode: NULLABLE
+- name: country
   type: STRING
-- mode: NULLABLE
-  name: subdivision1
+  mode: NULLABLE
+- name: subdivision1
   type: STRING
-- mode: NULLABLE
-  name: advertiser
+  mode: NULLABLE
+- name: advertiser
   type: STRING
-- mode: NULLABLE
-  name: release_channel
+  mode: NULLABLE
+- name: release_channel
   type: STRING
-- mode: NULLABLE
-  name: position
+  mode: NULLABLE
+- name: position
   type: INTEGER
-- mode: NULLABLE
-  name: provider
+  mode: NULLABLE
+- name: provider
   type: STRING
-- mode: NULLABLE
-  name: match_type
+  mode: NULLABLE
+- name: match_type
   type: STRING
-- mode: NULLABLE
-  name: normalized_os
+  mode: NULLABLE
+- name: normalized_os
   type: STRING
-- mode: NULLABLE
-  name: suggest_data_sharing_enabled
+  mode: NULLABLE
+- name: suggest_data_sharing_enabled
   type: BOOLEAN
-- mode: NULLABLE
-  name: event_count
+  mode: NULLABLE
+- name: event_count
   type: INTEGER
-- mode: NULLABLE
-  name: user_count
+  mode: NULLABLE
+- name: user_count
   type: INTEGER
-- mode: NULLABLE
-  name: query_type
+  mode: NULLABLE
+- name: query_type
   type: STRING
+  mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml	2024-05-20 14:51:30.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml	2024-05-20 15:02:40.000000000 +0000
@@ -1,40 +1,40 @@
 fields:
-- mode: NULLABLE
-  name: submission_date
+- name: submission_date
   type: DATE
-- mode: NULLABLE
-  name: form_factor
+  mode: NULLABLE
+- name: form_factor
   type: STRING
-- mode: NULLABLE
-  name: country
+  mode: NULLABLE
+- name: country
   type: STRING
-- mode: NULLABLE
-  name: advertiser
+  mode: NULLABLE
+- name: advertiser
   type: STRING
-- mode: NULLABLE
-  name: normalized_os
+  mode: NULLABLE
+- name: normalized_os
   type: STRING
-- mode: NULLABLE
-  name: release_channel
+  mode: NULLABLE
+- name: release_channel
   type: STRING
-- mode: NULLABLE
-  name: position
+  mode: NULLABLE
+- name: position
   type: INTEGER
-- mode: NULLABLE
-  name: provider
+  mode: NULLABLE
+- name: provider
   type: STRING
-- mode: NULLABLE
-  name: match_type
+  mode: NULLABLE
+- name: match_type
   type: STRING
-- mode: NULLABLE
-  name: suggest_data_sharing_enabled
+  mode: NULLABLE
+- name: suggest_data_sharing_enabled
   type: BOOLEAN
-- mode: NULLABLE
-  name: impression_count
+  mode: NULLABLE
+- name: impression_count
   type: INTEGER
-- mode: NULLABLE
-  name: click_count
+  mode: NULLABLE
+- name: click_count
   type: INTEGER
-- mode: NULLABLE
-  name: query_type
+  mode: NULLABLE
+- name: query_type
   type: STRING
+  mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml	2024-05-20 14:51:30.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml	2024-05-20 15:00:41.000000000 +0000
@@ -26,6 +26,9 @@
 - name: adjust_network
   type: STRING
   mode: NULLABLE
+- name: install_source
+  type: STRING
+  mode: NULLABLE
 - name: retained_week_2
   type: BOOLEAN
   mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml	2024-05-20 14:51:30.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml	2024-05-20 15:00:40.000000000 +0000
@@ -48,6 +48,10 @@
   description: 'The type of source of a client installation.
 
     '
+- name: install_source
+  type: STRING
+  mode: NULLABLE
+  description: null
 - name: new_profiles
   type: INTEGER
   mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v3/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v3/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v3/metadata.yaml	2024-05-20 14:52:07.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v3/metadata.yaml	2024-05-20 15:05:45.000000000 +0000
@@ -27,6 +27,7 @@
 scheduling:
   dag_name: bqetl_analytics_aggregations
   task_name: fenix_active_users_aggregates
+  date_partition_offset: -1
 bigquery:
   time_partitioning:
     type: day
@@ -55,7 +56,5 @@
   - moz-fx-data-shared-prod.org_mozilla_firefox_live.baseline_v1
   query.sql:
   - fenix.firefox_android_clients
-  - firefox_ios.firefox_ios_clients
-  - moz-fx-data-shared-prod.fenix.clients_last_seen_joined
-  - moz-fx-data-shared-prod.search_derived.mobile_search_clients_daily_v1
-  - mozdata.static.csa_gblmkt_languages
+  - moz-fx-data-shared-prod.fenix.baseline_clients_last_seen
+  - moz-fx-data-shared-prod.fenix.metrics_clients_last_seen
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v3/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v3/query.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v3/query.sql	2024-05-20 14:52:07.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v3/query.sql	2024-05-20 15:04:49.000000000 +0000
@@ -6,23 +6,14 @@
     install_source
   FROM
     fenix.firefox_android_clients
-  UNION ALL
-  SELECT
-    client_id,
-    adjust_network,
-    CAST(NULL AS STRING) install_source
-  FROM
-    firefox_ios.firefox_ios_clients
 ),
 baseline AS (
   SELECT
     submission_date,
     normalized_channel,
     client_id,
-    days_since_seen,
-    days_seen_bits,
+    days_active_bits,
     days_created_profile_bits,
-    durations,
     normalized_os,
     normalized_os_version,
     locale,
@@ -32,81 +23,67 @@
     device_model,
     first_seen_date,
     submission_date = first_seen_date AS is_new_profile,
-    uri_count,
-    is_default_browser,
     CAST(NULL AS string) AS distribution_id,
     isp,
-    IF(isp = 'BrowserStack', CONCAT('Fenix', ' BrowserStack'), 'Fenix') AS app_name
+    app_name,
+    segment,
+    is_daily_user,
+    is_weekly_user,
+    is_monthly_user,
+    is_dau,
+    is_wau,
+    is_mau
   FROM
-    `moz-fx-data-shared-prod.fenix.clients_last_seen_joined`
+    `moz-fx-data-shared-prod.fenix.baseline_clients_last_seen`
   WHERE
     submission_date = @submission_date
 ),
-search_clients AS (
+metrics AS (
+    -- Metrics ping can arrive either in the same or next day as the baseline ping.
+  WITH min_metrics_ping AS (
   SELECT
     client_id,
-    submission_date,
-    ad_click,
-    organic,
-    search_count,
-    search_with_ads
+      MIN(submission_date) AS submission_date
   FROM
-    `moz-fx-data-shared-prod.search_derived.mobile_search_clients_daily_v1`
+      `moz-fx-data-shared-prod.fenix.metrics_clients_last_seen`
   WHERE
-    submission_date = @submission_date
-),
-search_metrics AS (
-  SELECT
-    baseline.client_id,
-    baseline.submission_date,
-    SUM(ad_click) AS ad_clicks,
-    SUM(organic) AS organic_search_count,
-    SUM(search_count) AS search_count,
-    SUM(search_with_ads) AS search_with_ads
-  FROM
-    baseline
-  LEFT JOIN
-    search_clients s
-    ON baseline.client_id = s.client_id
-    AND baseline.submission_date = s.submission_date
+      submission_date
+      BETWEEN @submission_date
+      AND DATE_ADD(@submission_date, INTERVAL 1 DAY)
   GROUP BY
+      client_id
+  )
+  SELECT
     client_id,
+    submission_date,
+    metrics.normalized_channel,
+    metrics.uri_count,
+    metrics.is_default_browser
+  FROM
+    `moz-fx-data-shared-prod.fenix.metrics_clients_last_seen` AS metrics
+  INNER JOIN
+    min_metrics_ping
+    USING (client_id, submission_date)
+  WHERE
     submission_date
+    BETWEEN @submission_date
+    AND DATE_A

⚠️ Only part of the diff is displayed.

Link to full diff

Changes to active_users_aggregates.

bb07c4e

lucia-vargas-a marked this pull request as draft April 19, 2024 00:18

lucia-vargas-a changed the title ~~Changes to active_users_aggregates.~~ Changes to active_users_aggregates tables Apr 22, 2024

lucia-vargas-a requested a review from bochocki April 22, 2024 14:32

lucia-vargas-a changed the title ~~Changes to active_users_aggregates tables~~ Changes to active_users_aggregates query Apr 22, 2024

lucia-vargas-a mentioned this pull request Apr 22, 2024

Changes to active_users_aggregates view #5395

Open

5 tasks

lucia-vargas-a added 3 commits April 22, 2024 17:04

Remove days_since_seen and calculate activity segment.

aa0c756

Remove is_core_active not required.

9e5e8b9

Remove days_seen_bits from mobile queries not required.

3e65076

This comment has been minimized.

Sign in to view

lucia-vargas-a commented Apr 22, 2024

View reviewed changes

sql_generators/active_users_aggregates_v3/templates/mobile_query.sql Show resolved Hide resolved

This comment has been minimized.

Sign in to view

Adjust mobile query to use the baseline_clients_last_seen views.

4cc2993

This comment has been minimized.

Sign in to view

Update Focus Android query to use the baseline_clients_last_seen view.

a9a02c2

bochocki reviewed Apr 22, 2024

View reviewed changes

sql_generators/active_users_aggregates_v3/templates/desktop_query.sql Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

bochocki reviewed Apr 23, 2024

View reviewed changes

Merge branch 'main' into DENG-2989_update_active_users_aggregates

4f6df29

kik-kik mentioned this pull request Apr 25, 2024

feat(DENG-3462): add activity fields to support KPI metrics to baseline_last_seen views #5434

Closed

5 tasks

lucia-vargas-a added 4 commits April 25, 2024 20:25

Merge branch 'main' into DENG-2989_update_active_users_aggregates

2d93ba7

Get activity segments from the views instead of hard-coding it. Query…

dac6369

… attribution only when neccesary to improve performance. Delay update by 1 day to get metrics ping's data.

Merge branch 'main' into DENG-2989_update_active_users_aggregates

f91d78d

Get app_name from clients_last_seen_v2 view.

5ecae5e

lucia-vargas-a commented Apr 25, 2024

View reviewed changes

sql_generators/active_users_aggregates_v3/templates/focus_android_query.sql Outdated Show resolved Hide resolved

lucia-vargas-a commented Apr 25, 2024

View reviewed changes

sql_generators/active_users_aggregates_v3/templates/metadata.yaml Show resolved Hide resolved

This comment has been minimized.

Sign in to view

Merge branch 'main' into DENG-2989_update_active_users_aggregates

68f434e

This comment has been minimized.

Sign in to view

bochocki approved these changes May 13, 2024

View reviewed changes

lucia-vargas-a marked this pull request as ready for review May 14, 2024 12:38

Remove first_seen_date and correct column name in desktop_query.sql

c100f7f

This comment has been minimized.

Sign in to view

lucia-vargas-a added 5 commits May 17, 2024 14:18

Merge branch 'main' into DENG-2989_update_active_users_aggregates

5bfeaf7

Update active_users_aggregates query for Desktop to get app_name and …

dfe7434

…growth metrics from clients_last_seen_v2, replace language_name with locale and remove search metrics based on sprint decision (see DENG-1989).

Merge branch 'DENG_2989_update_active_users_aggregates_desktop' into …

e494d2e

…DENG-2989_update_active_users_aggregates # Conflicts: # sql_generators/active_users_aggregates_v3/templates/desktop_query.sql # sql_generators/active_users_aggregates_v3/templates/desktop_schema.yaml

Merge branch 'main' into DENG-2989_update_active_users_aggregates

38dcf35

Update active_users_aggregates query for Mobile to remove search metr…

c2a05a3

…ics based on sprint decision (see DENG-1989) and calculate the min metrics ping received between the current and next date, given that these pings can arrive in the same or next date as the baseline ping.

lucia-vargas-a changed the title ~~Changes to active_users_aggregates query~~ Changes to active_users_aggregates query for Mobile May 17, 2024

lucia-vargas-a added 2 commits May 17, 2024 16:39

Revert changes to Desktop queries in this PR, as this implementation …

e85a7e6

…is separated to [PR-5607](#5607).

Space

d066954

lucia-vargas-a requested a review from bochocki May 17, 2024 14:44

Formatting.

6ab0f4f

Change name of activity_segment to segment.

7cf3c56

This comment has been minimized.

Sign in to view

Fix reference to app name.

44ba283

This comment has been minimized.

Sign in to view

bochocki approved these changes May 17, 2024

View reviewed changes

lucia-vargas-a added 2 commits May 20, 2024 16:49

Update app_anme in checks for Focus

32f4c71

Merge branch 'main' into DENG-2989_update_active_users_aggregates

d69958a

This comment has been minimized.

Sign in to view

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Changes to active_users_aggregates query for Mobile #5396

Changes to active_users_aggregates query for Mobile #5396

lucia-vargas-a commented Apr 19, 2024 •

edited by jira bot

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

bochocki left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

lucia-vargas-a commented May 17, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

bochocki left a comment

bochocki May 17, 2024 •

edited

This comment has been minimized.

dataops-ci-bot commented May 20, 2024

Changes to active_users_aggregates query for Mobile #5396

Are you sure you want to change the base?

Changes to active_users_aggregates query for Mobile #5396

Conversation

lucia-vargas-a commented Apr 19, 2024 • edited by jira bot

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

bochocki left a comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

lucia-vargas-a commented May 17, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

bochocki left a comment

Choose a reason for hiding this comment

bochocki May 17, 2024 • edited

Choose a reason for hiding this comment

This comment has been minimized.

dataops-ci-bot commented May 20, 2024

Integration report for "Update app_anme in checks for Focus"

sql.diff

lucia-vargas-a commented Apr 19, 2024 •

edited by jira bot

bochocki May 17, 2024 •

edited

`sql.diff`