Changes to active_users_aggregates view #5395
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lucia-vargas-a
changed the title
This comment has been minimized.
This comment has been minimized.
bochocki
approved these changes
Apr 23, 2024
* Adding a new table (moz-fx-data-shared-prod.firefox_ios_derived.ltv_states_v1) & corresponding view (moz-fx-data-shared-prod.firefox_ios.ltv_states)
* Backfill new table * update end_date to incorporate new fields
* Add UDFs for extracting campaign info * Add campaigns table * Add ad_groups table * Add schemas * Uncomment UDFs * Format sql files * Reformat UDFs * Use common most_recent_rec terminology * Replace 3 UDFs with one * reformat & fix tests * Reformat again
* Extract campaign_id and ad_group_id in fac * Add Android App Campaign Stats table * Dont rely on fac view * Reformat
* Added nightly, experiments and 1pct tables
…5386) * Added deletion date field --------- Co-authored-by: Chelsey Beck <64881557+chelseybeck@users.noreply.github.com>
* DENG-3471 suppression list from campaign monitor * Update sql/moz-fx-data-shared-prod/marketing_suppression_list_external/campaign_monitor_suppression_list_v1/metadata.yaml Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com> --------- Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com>
* updating create statement * joining on users table to filter for active and ensuring there is at least one subscription * joining on users to filter for active * adding dev subscription group * removing fxa_id in favor of has_fxa * bringing in update timestamp for downstream use * updating formatting and adding filter for active users * adding filter for one active newsletter * updating tests * adding fxa id back to users table to join to products * updating query * updating values * updating tests * fix test for subscriptions * changing schema to array * updating format * updating to pull in all subscriptions with statuses * removing create statement * updating subscriptions query to make it an array and updating associated tests * updating formatting and comment --------- Co-authored-by: Leli Schiestl <lschiestl@mozilla.com>
* DENG-3471 suppression list from campaign monitor * Update sql/moz-fx-data-shared-prod/marketing_suppression_list_external/campaign_monitor_suppression_list_v1/metadata.yaml Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com> * after seans coder review --------- Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com>
…dated private bqetl (#5419)
* DENG-3186 new desktop engagement model * DENG-3186 add distribution ID to eng client V1 query * DENG-3186 - add distribution ID to the schema.yaml for eng client * DENG-3186 add distribution ID to the agg table * DENG-3186 adding locale to both client & agg tables * DENG-3186 add app_version to client & agg tables * DENG-3168 move lifecycle stage from engagement agg table to engagement agg view * DENG-3186 - add new desktop_engagement_client_v1 table to shredder * DENG-3186 update formatting of config.py * DENG-3186 switch to get country, os, os version, and build # from clients last seen and remove join to clients daily
* Update Glean generators checks. * Remove new_profiles check, not required. * warn or fail depending on range * Set check to fail if clients are missing, which is an ETL issue. * Checks fail when difference is higher than 1%. * Checks fail when difference is higher than 1%. * Make checks fail on a smaller difference.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
…as no baseline pings (bug 1888718). (#5726) Cirrus apps generally don't have baseline pings. This should resolve bug 1888718 "Airflow task bqetl_glean_usage.checks__warn_accounts_cirrus_derived__baseline_clients_last_seen__v1 failing since exec_date 2024-03-23".
#5734) The `kpi_forecasting` DAG doesn't currently have a `wait_for_unified_metrics` task.
* Backfill geckoview_version_v1 * Add schema.yaml to geckoview_version_v1
…rch_aggregates (#5714) * RS_1233_Add payload.processes.parent.scalars.policies_is_enterprise to search aggregates and search clients daily table Add ingpayload.processes.parent.scalars.policies_is_enterprise to - search_derived/search_aggregates - search_derived/searc_ clients_daily_v8 * Update query.sql * update to schema.yaml * Fix CI issues * Fix the tests Fix tests issue --------- Co-authored-by: Alekhya Kommasani <akommasani@mozilla.com> Co-authored-by: Alekhya <88394696+alekhyamoz@users.noreply.github.com>
* Fix `format_timedelta` function's parsing of negative timedeltas. The entire timedelta can be negative. * Refactor to use a single timedelta regular expression. * Fix typo in `format_timedelta` function argument.
* The app_name is Firefox Desktop in all cases. * Formatting * Formatting
* Add BigConfig example * Add monte carlo config * Allow multiple monte carlo configs
* Add deprecation label to sql generators. * Clarification to not delete the table.
* Update active_users_aggregates query for Desktop to get app_name and growth metrics from clients_last_seen_v2, replace language_name with locale and remove search metrics based on sprint decision (see DENG-1989). * Use active_users view. * Naming adjustment --------- Co-authored-by: Brad Ochocki Szasz <brad.ochocki@gmail.com>
* Changes to active_users_aggregates. * Remove days_since_seen and calculate activity segment. * Remove is_core_active not required. * Remove days_seen_bits from mobile queries not required. * Adjust mobile query to use the baseline_clients_last_seen views. * Update Focus Android query to use the baseline_clients_last_seen view. * Get activity segments from the views instead of hard-coding it. Query attribution only when neccesary to improve performance. Delay update by 1 day to get metrics ping's data. * Get app_name from clients_last_seen_v2 view. * Set activity_segment as NULL for legacy Focus Android. * Formatting * Remove first_seen_date and correct column name in desktop_query.sql * Update active_users_aggregates query for Desktop to get app_name and growth metrics from clients_last_seen_v2, replace language_name with locale and remove search metrics based on sprint decision (see DENG-1989). * Update active_users_aggregates query for Mobile to remove search metrics based on sprint decision (see DENG-1989) and calculate the min metrics ping received between the current and next date, given that these pings can arrive in the same or next date as the baseline ping. * Revert changes to Desktop queries in this PR, as this implementation is separated to [PR-5607](#5607). * Space * Formatting. * Change name of activity_segment to segment. * Fix reference to app name. * Update app_anme in checks for Focus * Use active_users instead of baseline_clients_last_seen and improve retrieving metrics ping's data. * Formatting. * Get distribution_id for Fenix. * Get distribution_id for Fenix. --------- Co-authored-by: Brad Ochocki Szasz <brad.ochocki@gmail.com>
* Changes to active_users_aggregates. * Remove days_since_seen and calculate activity segment. * Remove is_core_active not required. * Remove days_seen_bits from mobile queries not required. * Adjust mobile query to use the baseline_clients_last_seen views. * Update Focus Android query to use the baseline_clients_last_seen view. * Get activity segments from the views instead of hard-coding it. Query attribution only when neccesary to improve performance. Delay update by 1 day to get metrics ping's data. * Get app_name from clients_last_seen_v2 view. * Set activity_segment as NULL for legacy Focus Android. * Formatting * Remove first_seen_date and correct column name in desktop_query.sql * Update active_users_aggregates query for Desktop to get app_name and growth metrics from clients_last_seen_v2, replace language_name with locale and remove search metrics based on sprint decision (see DENG-1989). * Update active_users_aggregates query for Mobile to remove search metrics based on sprint decision (see DENG-1989) and calculate the min metrics ping received between the current and next date, given that these pings can arrive in the same or next date as the baseline ping. * Revert changes to Desktop queries in this PR, as this implementation is separated to [PR-5607](#5607). * Space * Formatting. * Change name of activity_segment to segment. * Add . Klar Android query and join in the Mobile view. * Fix reference to app name. * Update app_anme in checks for Focus * Add klar android to the checks and update app_name in checks for Focus. * Formatting. * Use active_users instead of baseline_clients_last_seen and improve retrieving metrics ping's data. * Formatting. * --- (#5623) updated-dependencies: - dependency-name: requests dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Anna Scholtz <anna@scholtzan.net> * Update backfill.yaml (#5629) * DENG-3719: Allow setting billing project for managed backfills (#5605) * Added default billing project and param * DENG-3851 Updated managed backfill docs with slack channel for notifcations (#5630) * Updated docs * Update docs/cookbooks/creating_a_derived_dataset.md Co-authored-by: Alexander <anicholson@mozilla.com> --------- Co-authored-by: Alexander <anicholson@mozilla.com> * Inject some errors to fenix metrics_clients_last_seen_v1 to test observability tooling (#5631) * feat: add kpi_support_metrics SQL generator (#5594) * feat: add kpi_support_metrics SQL generator * feat: replace baseline_clients_last_seen_activity_extended with active_users * feat: add template to union all views inside telemetry namespace * feat: remove retention queries for firefox_ios and fenix as now they are generated via kpi_support_metrics generator * bug: correct app_name based logic used by templating * bug: fix formatting inside kpi_support_metrics templates * feat: implement feedback from PR by removing mozilla online logic as no longer needed and correcting app_name setting * feat: fix active_users union mismatch * feat: remove scheduling for now, will be added in the next PR. * feat: make sure we user product_specific_attribution_fields additional field list instead of app_name logic in templates (#5633) * delete new pageload experiments table (#5634) * Use information_schema to find experiment tables for shredder (#5635) * Update retention of search terms sanitized datasets (#5577) * Be explicit about columns in clients_yearly (#5636) * Add distribution_id and geo_subdivision to clients_yearly schema (#5637) * Added two new fields to col list (#5639) * feat: add backfill.yaml to org_mozilla_fennec_aurora_derived.baseline_clients_daily_v1 (#5632) * Add flipkart and home_depot views (#5638) * Add flipkart and home_depot views * remove manually created views. use the user_facing view syndication instead * fix yaml lint * DENG-3288 add Focus iOS and Focus Android and switch to use new views instead of extended views (#5641) * DENG-3288 Add "is_mobile" to mobile_engagement_clients_v1 and mobile_engagement_v1 (#5648) * Update `bqetl_subplat` DAG description with new expected Stripe report error message. (#5642) * feat: add schema files to baseline_clients_daily_v1 and baseline_clients_last_seen_v1 (#5646) * fix: add fields: map to baseline_daily and baseline_last_seen schemas (#5651) * feat: update kpi_support_metrics generation to append platform to the unioned view name (#5644) Co-authored-by: Katie Windau <153020235+kwindau@users.noreply.github.com> * fix(backfills): switch to concurrent.futures to improve debuggability (#5653) * fix(backfills): followup to concurrent.futures, raise Error if any failed dates (#5654) * Reprocess past two days in jobs_by_organization_v1 (#5652) * feat: rename baseline_last_seen references to active_users and add "platform" to only include platform specific fields when it matches (#5645) * fix(backfills): switch from process to thread to avoid pickling bigquery object (#5656) * feat: rename kpi_support_metrics to mobile_kpi_support_metrics (#5657) * [RS-1246] Fix DDG DAU in search_revenue_levers_daily (#5655) * Fix search_revenue_levers_daily * Query formatting --------- Co-authored-by: m-d-bowerman <mbowerman@mozilla.com> * DENG-2975 active_users (#5478) * Add is_desktop to engagement and retention tables (#5658) * add is_desktop to engagement and retention tables * replace clients_last_seen by active_users * format * remove unused normalized_channel * [DENG-3889] Rewrite events_stream_v1 for better performance (#5659) * Add more null channels for metrics_clients_last_seen test (#5662) * chore(glam): accommodate for dag reschedule (#5664) * [DENG-3889] Replace events_stream temp udfs with inline usages (#5666) * Add telemetry_derived.latest_version to data-observability-dev (#5663) * [RS-834] Add country to search_revenue_levers_daily (#4739) * Add country * change channel spellings so match revenue table * update country list * Reformat * Update desktop DAU definition Can't update mobile DAU by engine breakouts w/o changes to mobile_search_clients_daily. But we also don't need that option since default engine is currently buggy and searchers presumably have > 0 active seconds * Fix sql format --------- Co-authored-by: Alekhya <88394696+alekhyamoz@users.noreply.github.com> Co-authored-by: Alekhya Kommasani <akommasani@mozilla.com> * Fix PyPI package publishing in CI (#5669) * Fix PyPI package publishing in CI * Update .circleci/workflows.yml Co-authored-by: Mike Williams <102263964+mikewilli@users.noreply.github.com> --------- Co-authored-by: Mike Williams <102263964+mikewilli@users.noreply.github.com> * Use ad group ID to get campaign info (#5670) * feat: move mobile engagement logic into mobile kpi support metrics generator (#5661) * feat: move mobile_engagement SQL logic into mobile_kpi_support_metrics generator * feat: remove mobile_engagement queries from telemetry and telemetry_erived * feat: apply formatting to engagement_clients view and add checks template * feat: add engagement view template and add engagement templates to the generation logic * bug: fix engagement query errors * feat: sneak in removing of retention views inside telemetry (mobile kpi generator will overwrite these anyways) * feat: remove bqetl_mobile_engagement_model DAG from DAG list as no longer needed. * feat: fix an engagement_v1 query template and removed unecessary comment from the metadata template * fix: update engagement templates to ensure valid query is rendered * [DENG-3905] Support queries with temp udfs when billing project is set (#5668) * DENG-3237: Added deletion dates to deprecated tables (#5660) * Added deletion dates * fix: metadata.yaml generation change to allow backfill command execution (#5671) * RS-788 Add support for organic searches with ads to the mobile search counts tables (#5598) * adding organic searches with ads to this table * updating mobile_search_aggregates table with search_with_ads_organic column * updating the search revenue lever table - include search_with_ads_organic columns for Bing, Google and DDG * Fix CI issues * Fix tests CI failure * fix tests * Fix test sql failure * Update query.sql reverting back to original code for search_revenue_levers table --------- Co-authored-by: Alekhya Kommasani <akommasani@mozilla.com> Co-authored-by: Alekhya <88394696+alekhyamoz@users.noreply.github.com> * Update desktop active_users view to make dimensions available for KPIs. (#5675) * Use desktop active_users view to make dimensions available for KPIs. * Naming adjustment * Fix reference to app name. * Use active_users instead of baseline_clients_last_seen and improve retrieving metrics ping's data. * Formatting. * Get distribution_id for Fenix. * Get distribution_id for Fenix. * Format correction * Skip unavailable data, becuase it's not KPI relevant. * Formatting --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Anna Scholtz <anna@scholtzan.net> Co-authored-by: Teon L Brooks <teon.brooks@gmail.com> Co-authored-by: Winnie Chan <10429026+wwyc@users.noreply.github.com> Co-authored-by: Alexander <anicholson@mozilla.com> Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com> Co-authored-by: whd <whd@users.noreply.github.com> Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com> Co-authored-by: wil stuckey <wstuckey@mozilla.com> Co-authored-by: Katie Windau <153020235+kwindau@users.noreply.github.com> Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com> Co-authored-by: skahmann3 <16420065+skahmann3@users.noreply.github.com> Co-authored-by: m-d-bowerman <mbowerman@mozilla.com> Co-authored-by: Vanessa Sabino <vsabino@mozilla.com> Co-authored-by: Eduardo Filho <edugomfilho@gmail.com> Co-authored-by: Alekhya <88394696+alekhyamoz@users.noreply.github.com> Co-authored-by: Alekhya Kommasani <akommasani@mozilla.com> Co-authored-by: Mike Williams <102263964+mikewilli@users.noreply.github.com> Co-authored-by: Preethi Issac <102384487+pissac17@users.noreply.github.com>
Integration report for "Merge branch 'main' into DENG-2989_update_active_users_view"
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Update the column names in the view active_users_aggregates. Merge after PR-5396.
Checklist for reviewer:
<username>:<branch>of the fork as parameter. The parameter will also show upin the logs of the
manual-trigger-required-for-forkCI task together with more detailed instructions.For modifications to schemas in restricted namespaces (see
CODEOWNERS):┆Issue is synchronized with this Jira Task