Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DENG-2621: Remove pageload experiments #5482

Closed
wants to merge 3 commits into from
Closed

DENG-2621: Remove pageload experiments #5482

wants to merge 3 commits into from

Conversation

wwyc
Copy link
Contributor

@wwyc wwyc commented May 2, 2024 •
edited by data-sync-user

https://mozilla-hub.atlassian.net/browse/DENG-2621

Stakedholder has agreed that this table is no longer needed as the size of the resulting table would be too big and would not improve query efficiencies after backfill.

Checklist for reviewer:

  • Commits should reference a bug or github issue, if relevant (if a bug is referenced, the pull request should include the bug number in the title).
  • If the PR comes from a fork, trigger integration CI tests by running the Push to upstream workflow and provide the <username>:<branch> of the fork as parameter. The parameter will also show up
    in the logs of the manual-trigger-required-for-fork CI task together with more detailed instructions.
  • If adding a new field to a query, ensure that the schema and dependent downstream schemas have been updated.
  • When adding a new derived dataset, ensure that data is not available already (fully or partially) and recommend extending an existing dataset in favor of creating new ones. Data can be available in the bigquery-etl repository, looker-hub or in looker-spoke-default.

For modifications to schemas in restricted namespaces (see CODEOWNERS):

┆Issue is synchronized with this Jira Task

@wwyc wwyc changed the title DENG-2621: Delete new pageload experiments table DENG-2621: Remove pageload experiments May 2, 2024
@wwyc wwyc marked this pull request as draft May 2, 2024 21:22
@dataops-ci-bot

This comment has been minimized.

Sign in to view
@dataops-ci-bot
Copy link

Integration report for "Removed pageload exp view"

sql.diff

Click to expand! 
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_pageload_v1.py /tmp/workspace/generated-sql/dags/bqetl_pageload_v1.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_pageload_v1.py	2024-05-02 21:25:46.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_pageload_v1.py	2024-05-02 21:37:40.000000000 +0000
@@ -79,23 +79,6 @@
         arguments=["--schema_update_option=ALLOW_FIELD_ADDITION"],
     )
 
-    firefox_desktop_pageload_experiments__v1 = bigquery_etl_query(
-        task_id="firefox_desktop_pageload_experiments__v1",
-        destination_table="pageload_experiments_v1",
-        dataset_id="firefox_desktop_derived",
-        project_id="moz-fx-data-shared-prod",
-        owner="wichan@mozilla.com",
-        email=[
-            "acreskey@mozilla.com",
-            "dpalmeiro@mozilla.com",
-            "telemetry-alerts@mozilla.com",
-            "wichan@mozilla.com",
-        ],
-        date_partition_parameter="submission_date",
-        depends_on_past=False,
-        arguments=["--schema_update_option=ALLOW_FIELD_ADDITION"],
-    )
-
     firefox_desktop_pageload_nightly__v1 = bigquery_etl_query(
         task_id="firefox_desktop_pageload_nightly__v1",
         destination_table="pageload_nightly_v1",
@@ -115,6 +98,4 @@
 
     firefox_desktop_pageload_1pct__v1.set_upstream(wait_for_copy_deduplicate_all)
 
-    firefox_desktop_pageload_experiments__v1.set_upstream(wait_for_copy_deduplicate_all)
-
     firefox_desktop_pageload_nightly__v1.set_upstream(wait_for_copy_deduplicate_all)
Only in /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop: pageload_experiments
Only in /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_derived: pageload_experiments_v1
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml	2024-05-02 21:22:20.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml	2024-05-02 21:30:59.000000000 +0000
@@ -1,49 +1,49 @@
 fields:
-- mode: NULLABLE
-  name: submission_date
+- name: submission_date
   type: DATE
-- mode: NULLABLE
-  name: source
+  mode: NULLABLE
+- name: source
   type: STRING
-- mode: NULLABLE
-  name: event_type
+  mode: NULLABLE
+- name: event_type
   type: STRING
-- mode: NULLABLE
-  name: form_factor
+  mode: NULLABLE
+- name: form_factor
   type: STRING
-- mode: NULLABLE
-  name: country
+  mode: NULLABLE
+- name: country
   type: STRING
-- mode: NULLABLE
-  name: subdivision1
+  mode: NULLABLE
+- name: subdivision1
   type: STRING
-- mode: NULLABLE
-  name: advertiser
+  mode: NULLABLE
+- name: advertiser
   type: STRING
-- mode: NULLABLE
-  name: release_channel
+  mode: NULLABLE
+- name: release_channel
   type: STRING
-- mode: NULLABLE
-  name: position
+  mode: NULLABLE
+- name: position
   type: INTEGER
-- mode: NULLABLE
-  name: provider
+  mode: NULLABLE
+- name: provider
   type: STRING
-- mode: NULLABLE
-  name: match_type
+  mode: NULLABLE
+- name: match_type
   type: STRING
-- mode: NULLABLE
-  name: normalized_os
+  mode: NULLABLE
+- name: normalized_os
   type: STRING
-- mode: NULLABLE
-  name: suggest_data_sharing_enabled
+  mode: NULLABLE
+- name: suggest_data_sharing_enabled
   type: BOOLEAN
-- mode: NULLABLE
-  name: event_count
+  mode: NULLABLE
+- name: event_count
   type: INTEGER
-- mode: NULLABLE
-  name: user_count
+  mode: NULLABLE
+- name: user_count
   type: INTEGER
-- mode: NULLABLE
-  name: query_type
+  mode: NULLABLE
+- name: query_type
   type: STRING
+  mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml	2024-05-02 21:22:20.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml	2024-05-02 21:30:58.000000000 +0000
@@ -1,40 +1,40 @@
 fields:
-- mode: NULLABLE
-  name: submission_date
+- name: submission_date
   type: DATE
-- mode: NULLABLE
-  name: form_factor
+  mode: NULLABLE
+- name: form_factor
   type: STRING
-- mode: NULLABLE
-  name: country
+  mode: NULLABLE
+- name: country
   type: STRING
-- mode: NULLABLE
-  name: advertiser
+  mode: NULLABLE
+- name: advertiser
   type: STRING
-- mode: NULLABLE
-  name: normalized_os
+  mode: NULLABLE
+- name: normalized_os
   type: STRING
-- mode: NULLABLE
-  name: release_channel
+  mode: NULLABLE
+- name: release_channel
   type: STRING
-- mode: NULLABLE
-  name: position
+  mode: NULLABLE
+- name: position
   type: INTEGER
-- mode: NULLABLE
-  name: provider
+  mode: NULLABLE
+- name: provider
   type: STRING
-- mode: NULLABLE
-  name: match_type
+  mode: NULLABLE
+- name: match_type
   type: STRING
-- mode: NULLABLE
-  name: suggest_data_sharing_enabled
+  mode: NULLABLE
+- name: suggest_data_sharing_enabled
   type: BOOLEAN
-- mode: NULLABLE
-  name: impression_count
+  mode: NULLABLE
+- name: impression_count
   type: INTEGER
-- mode: NULLABLE
-  name: click_count
+  mode: NULLABLE
+- name: click_count
   type: INTEGER
-- mode: NULLABLE
-  name: query_type
+  mode: NULLABLE
+- name: query_type
   type: STRING
+  mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml	2024-05-02 21:22:20.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml	2024-05-02 21:30:07.000000000 +0000
@@ -26,6 +26,9 @@
 - name: adjust_network
   type: STRING
   mode: NULLABLE
+- name: install_source
+  type: STRING
+  mode: NULLABLE
 - name: retained_week_2
   type: BOOLEAN
   mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml	2024-05-02 21:22:20.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml	2024-05-02 21:30:07.000000000 +0000
@@ -48,6 +48,10 @@
   description: 'The type of source of a client installation.
 
     '
+- name: install_source
+  type: STRING
+  mode: NULLABLE
+  description: null
 - name: new_profiles
   type: INTEGER
   mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/pageload_experiments/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/pageload_experiments/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/pageload_experiments/metadata.yaml	2024-05-02 21:22:54.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/pageload_experiments/metadata.yaml	1970-01-01 00:00:00.000000000 +0000
@@ -1,13 +0,0 @@
-friendly_name: Pageload Experiments
-description: |-
-  Please provide a description for the query
-owners: []
-labels: {}
-bigquery: null
-workgroup_access:
-- role: roles/bigquery.dataViewer
-  members:
-  - workgroup:mozilla-confidential
-references:
-  view.sql:
-  - moz-fx-data-shared-prod.firefox_desktop_derived.pageload_experiments_v1
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/pageload_experiments/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/pageload_experiments/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/pageload_experiments/schema.yaml	2024-05-02 21:22:54.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/pageload_experiments/schema.yaml	1970-01-01 00:00:00.000000000 +0000
@@ -1,303 +0,0 @@
-fields:
-- name: additional_properties
-  type: STRING
-  mode: NULLABLE
-- name: client_info
-  type: RECORD
-  mode: NULLABLE
-  fields:
-  - name: android_sdk_version
-    type: STRING
-    mode: NULLABLE
-  - name: app_build
-    type: STRING
-    mode: NULLABLE
-  - name: app_channel
-    type: STRING
-    mode: NULLABLE
-  - name: app_display_version
-    type: STRING
-    mode: NULLABLE
-  - name: architecture
-    type: STRING
-    mode: NULLABLE
-  - name: build_date
-    type: STRING
-    mode: NULLABLE
-  - name: client_id
-    type: STRING
-    mode: NULLABLE
-  - name: device_manufacturer
-    type: STRING
-    mode: NULLABLE
-  - name: device_model
-    type: STRING
-    mode: NULLABLE
-  - name: first_run_date
-    type: STRING
-    mode: NULLABLE
-  - name: locale
-    type: STRING
-    mode: NULLABLE
-  - name: os
-    type: STRING
-    mode: NULLABLE
-  - name: os_version
-    type: STRING
-    mode: NULLABLE
-  - name: telemetry_sdk_build
-    type: STRING
-    mode: NULLABLE
-  - name: windows_build_number
-    type: INTEGER
-    mode: NULLABLE
-  - name: session_count
-    type: INTEGER
-    mode: NULLABLE
-  - name: session_id
-    type: STRING
-    mode: NULLABLE
-- name: document_id
-  type: STRING
-  mode: NULLABLE
-- name: events
-  type: RECORD
-  mode: REPEATED
-  fields:
-  - name: category
-    type: STRING
-    mode: NULLABLE
-  - name: extra
-    type: RECORD
-    mode: REPEATED
-    fields:
-    - name: key
-      type: STRING
-      mode: NULLABLE
-    - name: value
-      type: STRING
-      mode: NULLABLE
-  - name: name
-    type: STRING
-    mode: NULLABLE
-  - name: timestamp
-    type: INTEGER
-    mode: NULLABLE
-- name: metadata
-  type: RECORD
-  mode: NULLABLE
-  fields:
-  - name: geo
-    type: RECORD
-    mode: NULLABLE
-    fields:
-    - name: city
-      type: STRING
-      mode: NULLABLE
-    - name: country
-      type: STRING
-      mode: NULLABLE
-    - name: db_version
-      type: STRING
-      mode: NULLABLE
-    - name: subdivision1
-      type: STRING
-      mode: NULLABLE
-    - name: subdivision2
-      type: STRING
-      mode: NULLABLE
-  - name: header
-    type: RECORD
-    mode: NULLABLE
-    fields:
-    - name: date
-      type: STRING
-      mode: NULLABLE
-    - name: dnt
-      type: STRING
-      mode: NULLABLE
-    - name: x_debug_id
-      type: STRING
-      mode: NULLABLE
-    - name: x_foxsec_ip_reputation
-      type: STRING
-      mode: NULLABLE
-    - name: x_lb_tags
-      type: STRING
-      mode: NULLABLE
-    - name: x_pingsender_version
-      type: STRING
-      mode: NULLABLE
-    - name: x_source_tags
-      type: STRING
-      mode: NULLABLE
-    - name: x_telemetry_agent
-      type: STRING
-      mode: NULLABLE
-    - name: parsed_date
-      type: TIMESTAMP
-      mode: NULLABLE
-    - name: parsed_x_source_tags
-      type: STRING
-      mode: REPEATED
-    - name: parsed_x_lb_tags
-      type: RECORD
-      mode: NULLABLE
-      fields:
-      - name: tls_version
-        type: STRING
-        mode: NULLABLE
-      - name: tls_cipher_hex
-        type: STRING
-        mode: NULLABLE
-  - name: isp
-    type: RECORD
-    mode: NULLABLE
-    fields:
-    - name: db_version
-      type: STRING
-      mode: NULLABLE
-    - name: name
-      type: STRING
-      mode: NULLABLE
-    - name: organization
-      type: STRING
-      mode: NULLABLE
-  - name: user_agent
-    type: RECORD
-    mode: NULLABLE
-    fields:
-    - name: browser
-      type: STRING
-      mode: NULLABLE
-    - name: os
-      type: STRING
-      mode: NULLABLE
-    - name: version
-      type: STRING
-      mode: NULLABLE
-- name: metrics
-  type: RECORD
-  mode: NULLABLE
-  fields:
-  - name: labeled_counter
-    type: RECORD
-    mode: NULLABLE
-    fields:
-    - name: glean_error_invalid_label
-      type: RECORD
-      mode: REPEATED
-      fields:
-      - name: key
-        type: STRING
-        mode: NULLABLE
-      - name: value
-        type: INTEGER
-        mode: NULLABLE
-    - name: glean_error_invalid_overflow
-      type: RECORD
-      mode: REPEATED
-      fields:
-      - name: key
-        type: STRING
-        mode: NULLABLE
-      - name: value
-        type: INTEGER
-        mode: NULLABLE
-    - name: glean_error_invalid_state
-      type: RECORD
-      mode: REPEATED
-      fields:
-      - name: key
-        type: STRING
-        mode: NULLABLE
-      - name: value
-        type: INTEGER
-        mode: NULLABLE
-    - name: glean_error_invalid_value
-      type: RECORD
-      mode: REPEATED
-      fields:
-      - name: key
-        type: STRING
-        mode: NULLABLE
-      - name: value
-        type: INTEGER
-        mode: NULLABLE
-  - name: string
-    type: RECORD
-    mode: NULLABLE
-    fields:
-    - name: glean_client_annotation_experimentation_id
-      type: STRING
-      mode: NULLABLE
-- name: normalized_app_name
-  type: STRING
-  mode: NULLABLE
-- name: normalized_channel
-  type: STRING
-  mode: NULLABLE
-- name: normalized_country_code
-  type: STRING
-  mode: NULLABLE
-- name: normalized_os
-  type: STRING
-  mode: NULLABLE
-- name: normalized_os_version
-  type: STRING
-  mode: NULLABLE
-- name: ping_info
-  type: RECORD
-  mode: NULLABLE
-  fields:
-  - name: end_time
-    type: STRING
-    mode: NULLABLE
-  - name: experiments
-    type: RECORD
-    mode: REPEATED
-    fields:
-    - name: key
-      type: STRING
-      mode: NULLABLE
-    - name: value
-      type: RECORD
-      mode: NULLABLE
-      fields:
-      - name: branch
-        type: STRING
-        mode: NULLABLE
-      - name: extra
-        type: RECORD
-        mode: NULLABLE
-        fields:
-        - name: type
-          type: STRING
-          mode: NULLABLE
-        - name: enrollment_id
-          type: STRING
-          mode: NULLABLE
-  - name: ping_type
-    type: STRING
-    mode: NULLABLE
-  - name: reason
-    type: STRING
-    mode: NULLABLE
-  - name: seq
-    type: INTEGER
-    mode: NULLABLE
-  - name: start_time
-    type: STRING
-    mode: NULLABLE
-  - name: parsed_start_time
-    type: TIMESTAMP
-    mode: NULLABLE
-  - name: parsed_end_time
-    type: TIMESTAMP
-    mode: NULLABLE
-- name: sample_id
-  type: INTEGER
-  mode: NULLABLE
-- name: submission_timestamp
-  type: TIMESTAMP
-  mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/pageload_experiments/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/pageload_experiments/view.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/pageload_experiments/view.sql	2024-05-02 21:22:21.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/pageload_experiments/view.sql	1970-01-01 00:00:00.000000000 +0000
@@ -1,7 +0,0 @@
-CREATE OR REPLACE VIEW
-  `moz-fx-data-shared-prod.firefox_desktop.pageload_experiments`
-AS
-SELECT
-  *
-FROM
-  `moz-fx-data-shared-prod.firefox_desktop_derived.pageload_experiments_v1`
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_derived/pageload_experiments_v1/backfill.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_derived/pageload_experiments_v1/backfill.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_derived/pageload_experiments_v1/backfill.yaml	2024-05-02 21:22:21.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_derived/pageload_experiments_v1/backfill.yaml	1970-01-01 00:00:00.000000000 +0000
@@ -1,7 +0,0 @@
-2024-04-24:
-  start_date: 2024-01-01
-  end_date: 2024-04-23
-  reason: Backfill a smaller sample for a new table to ensure data is correct.
-  watchers:
-  - wichan@mozilla.com
-  status: Complete
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_derived/pageload_experiments_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_derived/pageload_experiments_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_derived/pageload_experiments_v1/metadata.yaml	2024-05-02 21:23:58.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_derived/pageload_experiments_v1/metadata.yaml	1970-01-01 00:00:00.000000000 +0000
@@ -1,33 +0,0 @@
-friendly_name: Pageload Experiments
-description: |-
-  Contains pings with experiments only from pageload_v1
-owners:
-- wichan@mozilla.com
-- acreskey@mozilla.com
-- dpalmeiro@mozilla.com
-labels:
-  incremental: true
-  dag: bqetl_pageload_v1
-  owner1: wichan
-  owner2: acreskey
-  owner3: dpalmeiro
-scheduling:
-  dag_name: bqetl_pageload_v1
-  task_name: firefox_desktop_pageload_experiments__v1
-  arguments:
-  - --schema_update_option=ALLOW_FIELD_ADDITION
-bigquery:
-  time_partitioning:
-    type: day
-    field: submission_timestamp
-    require_partition_filter: true
-    expiration_days: null
-  range_partitioning: null
-  clustering:
-    fields:
-    - normalized_channel
-workgroup_access:
-- role: roles/bigquery.dataViewer
-  members:
-  - workgroup:mozilla-confidential
-references: {}
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_derived/pageload_experiments_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_derived/pageload_experiments_v1/query.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_derived/pageload_experiments_v1/query.sql	2024-05-02 21:22:21.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_derived/pageload_experiments_v1/query.sql	1970-01-01 00:00:00.000000000 +0000
@@ -1,7 +0,0 @@
-SELECT
-  *
-FROM
-  `moz-fx-data-shared-prod.firefox_desktop.pageload`
-WHERE
-  DATE(submission_timestamp) = @submission_date
-  AND ARRAY_LENGTH(ping_info.experiments) > 0
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_derived/pageload_experiments_v1/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_derived/pageload_experiments_v1/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_derived/pageload_experiments_v1/schema.yaml	2024-05-02 21:22:21.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_derived/pageload_experiments_v1/schema.yaml	1970-01-01 00:00:00.000000000 +0000
@@ -1,303 +0,0 @@
-fields:
-- name: additional_properties
-  type: STRING
-  mode: NULLABLE
-- name: client_info
-  type: RECORD
-  mode: NULLABLE
-  fields:
-  - name: android_sdk_version
-    type: STRING
-    mode: NULLABLE
-  - name: app_build
-    type: STRING
-    mode: NULLABLE
-  - name: app_channel
-    type: STRING
-    mode: NULLABLE
-  - name: app_display_version
-    type: STRING
-    mode: NULLABLE
-  - name: architecture
-    type: STRING
-    mode: NULLABLE
-  - name: build_date
-    type: STRING
-    mode: NULLABLE
-  - name: client_id
-    type: STRING
-    mode: NULLABLE
-  - name: device_manufacturer
-    type: STRING
-    mode: NULLABLE
-  - name: device_model
-    type: STRING
-    mode: NULLABLE
-  - name: first_run_date
-    type: STRING
-    mode: NULLABLE
-  - name: locale
-    type: STRING
-    mode: NULLABLE
-  - name: os
-    type: STRING
-    mode: NULLABLE
-  - name: os_version
-    type: STRING
-    mode: NULLABLE
-  - name: telemetry_sdk_build
-    type: STRING
-    mode: NULLABLE
-  - name: windows_build_number
-    type: INTEGER
-    mode: NULLABLE
-  - name: session_count
-    type: INTEGER
-    mode: NULLABLE
-  - name: session_id
-    type: STRING
-    mode: NULLABLE
-- name: document_id
-  type: STRING
-  mode: NULLABLE
-- name: events
-  type: RECORD
-  mode: REPEATED
-  fields:
-  - name: category
-    type: STRING
-    mode: NULLABLE
-  - name: extra
-    type: RECORD
-    mode: REPEATED
-    fields:
-    - name: key
-      type: STRING
-      mode: NULLABLE
-    - name: value
-      type: STRING
-      mode: NULLABLE
-  - name: name
-    type: STRING
-    mode: NULLABLE
-  - name: timestamp
-    type: INTEGER
-    mode: NULLABLE
-- name: metadata
-  type: RECORD
-  mode: NULLABLE
-  fields:
-  - name: geo
-    type: RECORD
-    mode: NULLABLE
-    fields:
-    - name: city
-      type: STRING
-      mode: NULLABLE
-    - name: country
-      type: STRING
-      mode: NULLABLE
-    - name: db_version
-      type: STRING
-      mode: NULLABLE
-    - name: subdivision1
-      type: STRING
-      mode: NULLABLE
-    - name: subdivision2
-      type: STRING
-      mode: NULLABLE
-  - name: header
-    type: RECORD
-    mode: NULLABLE
-    fields:
-    - name: date
-      type: STRING
-      mode: NULLABLE
-    - name: dnt
-      type: STRING
-      mode: NULLABLE
-    - name: x_debug_id
-      type: STRING
-      mode: NULLABLE
-    - name: x_foxsec_ip_reputation
-      type: STRING
-      mode: NULLABLE
-    - name: x_lb_tags
-      type: STRING
-      mode: NULLABLE
-    - name: x_pingsender_version
-      type: STRING
-      mode: NULLABLE
-    - name: x_source_tags
-      type: STRING
-      mode: NULLABLE
-    - name: x_telemetry_agent
-      type: STRING
-      mode: NULLABLE
-    - name: parsed_date
-      type: TIMESTAMP
-      mode: NULLABLE
-    - name: parsed_x_source_tags
-      type: STRING
-      mode: REPEATED
-    - name: parsed_x_lb_tags
-      type: RECORD
-      mode: NULLABLE
-      fields:
-      - name: tls_version
-        type: STRING
-        mode: NULLABLE
-      - name: tls_cipher_hex
-        type: STRING
-        mode: NULLABLE
-  - name: isp
-    type: RECORD
-    mode: NULLABLE
-    fields:
-    - name: db_version
-      type: STRING
-      mode: NULLABLE
-    - name: name
-      type: STRING
-      mode: NULLABLE
-    - name: organization
-      type: STRING
-      mode: NULLABLE
-  - name: user_agent
-    type: RECORD
-    mode: NULLABLE
-    fields:
-    - name: browser
-      type: STRING
-      mode: NULLABLE
-    - name: os
-      type: STRING
-      mode: NULLABLE
-    - name: version
-      type: STRING
-      mode: NULLABLE
-- name: metrics
-  type: RECORD
-  mode: NULLABLE
-  fields:
-  - name: labeled_counter
-    type: RECORD
-    mode: NULLABLE
-    fields:
-    - name: glean_error_invalid_label
-      type: RECORD
-      mode: REPEATED
-      fields:
-      - name: key
-        type: STRING
-        mode: NULLABLE
-      - name: value
-        type: INTEGER
-        mode: NULLABLE
-    - name: glean_error_invalid_overflow
-      type: RECORD
-      mode: REPEATED
-      fields:
-      - name: key
-        type: STRING
-        mode: NULLABLE
-      - name: value
-        type: INTEGER
-        mode: NULLABLE
-    - name: glean_error_invalid_state
-      type: RECORD
-      mode: REPEATED
-      fields:
-      - name: key
-        type: STRING
-        mode: NULLABLE
-      - name: value
-        type: INTEGER
-        mode: NULLABLE
-    - name: glean_error_invalid_value
-      type: RECORD
-      mode: REPEATED
-      fields:
-      - name: key
-        type: STRING
-        mode: NULLABLE
-      - name: value
-        type: INTEGER
-        mode: NULLABLE
-  - name: string
-    type: RECORD
-    mode: NULLABLE
-    fields:
-    - name: glean_client_annotation_experimentation_id
-      type: STRING
-      mode: NULLABLE
-- name: normalized_app_name
-  type: STRING
-  mode: NULLABLE
-- name: normalized_channel
-  type: STRING
-  mode: NULLABLE
-- name: normalized_country_code
-  type: STRING
-  mode: NULLABLE
-- name: normalized_os
-  type: STRING
-  mode: NULLABLE
-- name: normalized_os_version
-  type: STRING
-  mode: NULLABLE
-- name: ping_info
-  type: RECORD
-  mode: NULLABLE
-  fields:
-  - name: end_time
-    type: STRING
-    mode: NULLABLE
-  - name: experiments
-    type: RECORD
-    mode: REPEATED
-    fields:
-    - name: key
-      type: STRING
-      mode: NULLABLE
-    - name: value
-      type: RECORD
-      mode: NULLABLE
-      fields:
-      - name: branch
-        type: STRING
-        mode: NULLABLE
-      - name: extra
-        type: RECORD
-        mode: NULLABLE
-        fields:
-        - name: type
-          type: STRING
-          mode: NULLABLE
-        - name: enrollment_id
-          type: STRING
-          mode: NULLABLE
-  - name: ping_type
-    type: STRING
-    mode: NULLABLE
-  - name: reason
-    type: STRING
-    mode: NULLABLE
-  - name: seq
-    type: INTEGER
-    mode: NULLABLE
-  - name: start_time
-    type: STRING
-    mode: NULLABLE
-  - name: parsed_start_time
-    type: TIMESTAMP
-    mode: NULLABLE
-  - name: parsed_end_time
-    type: TIMESTAMP
-    mode: NULLABLE
-- name: sample_id
-  type: INTEGER
-  mode: NULLABLE
-- name: submission_timestamp
-  type: TIMESTAMP
-  mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/internet_outages/global_outages_v1/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/internet_outages/global_outages_v1/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/internet_outages/global_outages_v1/schema.yaml	2024-05-02 21:22:20.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/internet_outages/global_outages_v1/schema.yaml	2024-05-02 21:31:05.000000000 +0000
@@ -1,49 +1,49 @@
 fields:
-- mode: NULLABLE
-  name: country
+- name: country
   type: STRING
-- mode: NULLABLE
-  name: city
+  mode: NULLABLE
+- name: city
   type: STRING
-- mode: NULLABLE
-  name: datetime
+  mode: NULLABLE
+- name: datetime
   type: TIMESTAMP
-- mode: NULLABLE
-  name: proportion_undefined
+  mode: NULLABLE
+- name: proportion_undefined
   type: FLOAT
-- mode: NULLABLE
-  name: proportion_timeout
+  mode: NULLABLE
+- name: proportion_timeout
   type: FLOAT
-- mode: NULLABLE
-  name: proportion_abort
+  mode: NULLABLE
+- name: proportion_abort
   type: FLOAT
-- mode: NULLABLE
-  name: proportion_unreachable
+  mode: NULLABLE
+- name: proportion_unreachable
   type: FLOAT
-- mode: NULLABLE
-  name: proportion_terminated
+  mode: NULLABLE
+- name: proportion_terminated
   type: FLOAT
-- mode: NULLABLE
-  name: proportion_channel_open
+  mode: NULLABLE
+- name: proportion_channel_open
   type: FLOAT
-- mode: NULLABLE
-  name: avg_dns_success_time
+  mode: NULLABLE
+- name: avg_dns_success_time
   type: FLOAT
-- mode: NULLABLE
-  name: missing_dns_success
+  mode: NULLABLE
+- name: missing_dns_success
   type: FLOAT
-- mode: NULLABLE
-  name: avg_dns_failure_time
+  mode: NULLABLE
+- name: avg_dns_failure_time
   type: FLOAT
-- mode: NULLABLE
-  name: missing_dns_failure
+  mode: NULLABLE
+- name: missing_dns_failure
   type: FLOAT
-- mode: NULLABLE
-  name: count_dns_failure
+  mode: NULLABLE
+- name: count_dns_failure
   type: FLOAT
-- mode: NULLABLE
-  name: ssl_error_prop
+  mode: NULLABLE
+- name: ssl_error_prop
   type: FLOAT
-- mode: NULLABLE
-  name: avg_tls_handshake_time
+  mode: NULLABLE
+- name: avg_tls_handshake_time
   type: FLOAT
+  mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_daily_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_daily_v1/query.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_daily_v1/query.sql	2024-05-02 21:22:20.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_daily_v1/query.sql	2024-05-02 21:24:56.000000000 +0000
@@ -1,6 +1,4 @@
--- Query generated by ./bqetl generate search
--- This file doesn't get overwritten by the generator. The generator output needs
--- to be written to this file manually.
+-- Query generated by bigquery-etl/search/mobile_search_clients_daily.py
 --
 -- Older versions separate source and engine with an underscore instead of period
 -- Return array of form [source, engine] if key is valid, empty array otherwise
@@ -498,7 +496,7 @@
     metrics.counter.browser_total_uri_count,
     client_info.locale,
   FROM
-    org_mozilla_ios_klar.metrics AS org_mozilla_klar_metrics
+    org_mozilla_ios_klar.metrics AS org_mozilla_ios_klar_metrics
 ),
 fenix_baseline AS (
   SELECT
@@ -874,7 +872,6 @@
     ANY_VALUE(sample_id) AS sample_id,
     udf.map_mode_last(ARRAY_CONCAT_AGG(experiments)) AS experiments,
     SUM(total_uri_count) AS total_uri_count,
-    CAST(NULL AS STRING) AS normalized_engine
   FROM
     combined_search_clients
   WHERE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/static/data_incidents_v1/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/static/data_incidents_v1/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/static/data_incidents_v1/schema.yaml	2024-05-02 21:22:20.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/static/data_incidents_v1/schema.yaml	2024-05-02 21:31:50.000000000 +0000
@@ -1,22 +1,22 @@
 fields:
-- mode: NULLABLE
-  name: start_date
+- name: start_date
   type: DATE
-- mode: NULLABLE
-  name: end_date
+  mode: NULLABLE
+- name: end_date
   type: DATE
-- mode: NULLABLE
-  name: incident
+  mode: NULLABLE
+- name: incident
   type: STRING
-- mode: NULLABLE
-  name: description
+  mode: NULLABLE
+- name: description
   type: STRING
-- mode: NULLABLE
-  name: bug
+  mode: NULLABLE
+- name: bug
   type: STRING
-- mode: NULLABLE
-  name: product
+  mode: NULLABLE
+- name: product
   type: STRING
-- mode: NULLABLE
-  name: version
+  mode: NULLABLE
+- name: version
   type: STRING
+  mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/static/monitoring_distinct_docids_notes_v1/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/static/monitoring_distinct_docids_notes_v1/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/static/monitoring_distinct_docids_notes_v1/schema.yaml	2024-05-02 21:22:20.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/static/monitoring_distinct_docids_notes_v1/schema.yaml	2024-05-02 21:31:50.000000000 +0000
@@ -1,19 +1,19 @@
 fields:
-- mode: NULLABLE
-  name: start_date
+- name: start_date
   type: DATE
-- mode: NULLABLE
-  name: end_date
+  mode: NULLABLE
+- name: end_date
   type: DATE
-- mode: NULLABLE
-  name: document_namespace
+  mode: NULLABLE
+- name: document_namespace
   type: STRING
-- mode: NULLABLE
-  name: document_type
+  mode: NULLABLE
+- name: document_type
   type: STRING
-- mode: NULLABLE
-  name: notes
+  mode: NULLABLE
+- name: notes
   type: STRING
-- mode: NULLABLE
-  name: bug
+  mode: NULLABLE
+- name: bug
   type: STRING
+  mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/static/monitoring_missing_columns_notes_v1/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/static/monitoring_missing_columns_notes_v1/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/static/monitoring_missing_columns_notes_v1/schema.yaml	2024-05-02 21:22:20.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/static/monitoring_missing_columns_notes_v1/schema.yaml	2024-05-02 21:31:50.000000000 +0000
@@ -1,25 +1,25 @@
 fields:
-- mode: NULLABLE
-  name: start_date
+- name: start_date
   type: DATE
-- mode: NULLABLE
-  name: end_date
+  mode: NULLABLE
+- name: end_date
   type: DATE
-- mode: NULLABLE
-  name: document_namespace
+  mode: NULLABLE
+- name: document_namespace
   type: STRING
-- mode: NULLABLE
-  name: document_type
+  mode: NULLABLE
+- name: document_type
   type: STRING
-- mode: NULLABLE
-  name: document_version
+  mode: NULLABLE
+- name: document_version
   type: STRING
-- mode: NULLABLE
-  name: path
+  mode: NULLABLE
+- name: path
   type: STRING
-- mode: NULLABLE
-  name: notes
+  mode: NULLABLE
+- name: notes
   type: STRING
-- mode: NULLABLE
-  name: bug
+  mode: NULLABLE
+- name: bug
   type: STRING
+  mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/static/monitoring_missing_document_namespaces_notes_v1/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/static/monitoring_missing_document_namespaces_notes_v1/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/static/monitoring_missing_document_namespaces_notes_v1/schema.yaml	2024-05-02 21:22:20.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/static/monitoring_missing_document_namespaces_notes_v1/schema.yaml	2024-05-02 21:31:50.000000000 +0000
@@ -1,22 +1,22 @@
 fields:
-- mode: NULLABLE
-  name: start_date
+- name: start_date
   type: DATE
-- mode: NULLABLE
-  name: end_date
+  mode: NULLABLE
+- name: end_date
   type: DATE
-- mode: NULLABLE
-  name: document_namespace
+  mode: NULLABLE
+- name: document_namespace
   type: STRING
-- mode: NULLABLE
-  name: document_type
+  mode: NULLABLE
+- name: document_type
   type: STRING
-- mode: NULLABLE
-  name: document_version
+  mode: NULLABLE
+- name: document_version
   type: STRING
-- mode: NULLABLE
-  name: notes
+  mode: NULLABLE
+- name: notes
   type: STRING
-- mode: NULLABLE
-  name: bug
+  mode: NULLABLE
+- name: bug
   type: STRING
+  mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/static/monitoring_schema_errors_notes_v1/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/static/monitoring_schema_errors_notes_v1/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/static/monitoring_schema_errors_notes_v1/schema.yaml	2024-05-02 21:22:20.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/static/monitoring_schema_errors_notes_v1/schema.yaml	2024-05-02 21:31:50.000000000 +0000
@@ -1,22 +1,22 @@
 fields:
-- mode: NULLABLE
-  name: start_date
+- name: start_date
   type: DATE
-- mode: NULLABLE
-  name: end_date
+  mode: NULLABLE
+- name: end_date
   type: DATE
-- mode: NULLABLE
-  name: document_namespace
+  mode: NULLABLE
+- name: document_namespace
   type: STRING
-- mode: NULLABLE
-  name: document_type
+  mode: NULLABLE
+- name: document_type
   type: STRING
-- mode: NULLABLE
-  name: path
+  mode: NULLABLE
+- name: path
   type: STRING
-- mode: NULLABLE
-  name: notes
+  mode: NULLABLE
+- name: notes
   type: STRING
-- mode: NULLABLE
-  name: bug
+  mode: NULLABLE
+- name: bug
   type: STRING
+  mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry/releases_latest/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/releases_latest/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry/releases_latest/schema.yaml	2024-05-02 21:22:54.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/releases_latest/schema.yaml	2024-05-02 21:31:52.000000000 +0000
@@ -2,21 +2,18 @@
 - name: date
   type: DATE
   mode: NULLABLE
-  description: null
 - name: product
   type: STRING
   mode: NULLABLE
 - name: category
   type: STRING
   mode: NULLABLE
-  description: null
 - name: channel
   type: STRING
   mode: NULLABLE
 - name: build_number
   type: INTEGER
   mode: NULLABLE
-  description: null
 - name: release_date
   type: DATE
   mode: NULLABLE

Link to full diff

@wwyc wwyc marked this pull request as ready for review May 23, 2024 21:52
@wwyc
Copy link
Contributor Author

wwyc commented May 23, 2024

duplicate of #5634

@wwyc wwyc closed this May 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@wwyc @dataops-ci-bot