Bug 1574490 - Add views for events. #510

mreid-moz · 2019-11-18T14:50:35Z

These views are generated based on the json configuration files from telemetry-streaming.

I used this script to generate them.

fbertsch

cc @akkomar who will be working on this. A rough overview of this and the associated script look good, but we really need to compare these views to the data in Amplitude directly - I commented in the bug about that.

sql/telemetry/fennec_ios_events_v1/view.sql

sql/telemetry/desktop_savant_events_v1/view.sql

relud · 2019-11-21T18:58:08Z

I used this script to generate them.

we should include that script in this PR, probably as templates/telemetry/make_events_view.py

Based on https://gist.github.com/mreid-moz/33837c4c511b4fb7923f67a91232beb7

akkomar · 2019-11-22T13:20:24Z

I used this script to generate them.

we should include that script in this PR, probably as templates/telemetry/make_events_view.py

I have added it under templates/telemetry/make_events_view.py and adapted for this repo.

See https://github.com/mozilla/telemetry-streaming/blob/b85cdffc72a6f9ab224f9eececc38dfa09d98b8c/src/main/scala/com/mozilla/telemetry/pings/Ping.scala#L335-L338 for current implementation

It has been unscheduled in 2018: mozilla/telemetry-airflow@ae0f236#diff-99344063164ff6dff208054f7a8ed655

Don't use event group name in `insert_id`

See https://github.com/mozilla/telemetry-streaming/blob/master/src/main/scala/com/mozilla/telemetry/pings/Ping.scala#L343

Some properties were hardcoded per ping type and not included in the schemas

See https://github.com/mozilla/telemetry-streaming/blob/b85cdffc72a6f9ab224f9eececc38dfa09d98b8c/src/main/scala/com/mozilla/telemetry/pings/Ping.scala#L351

See https://github.com/mozilla/telemetry-streaming/blob/b85cdffc72a6f9ab224f9eececc38dfa09d98b8c/src/main/scala/com/mozilla/telemetry/pings/Ping.scala#L341

`false`

akkomar · 2019-11-27T10:56:18Z

I have exported Amplitude events from a single day corresponding to each mobile view and created notebooks to validate their correctness:

fennec_ios_events_v1: https://colab.research.google.com/drive/16JQ2ExixIUCitTydtPMp-XSCPFC_a4i2
focus_android_events_v1: https://colab.research.google.com/drive/1RjGrn9M8PEUqMYHG6ohsmbYptxVkRtpT
fire_tv: https://colab.research.google.com/drive/1SsA_iKLMHrphHQs034Y8n_ishGq1_j76
rocket_android: https://colab.research.google.com/drive/1wTKE7ZU7f9tbBDMqOIg2Z3LPgw4E9NIp

Mobile views look good - individual properties, total event count and grouped by name counts all match with the exports. We also have matching and stable insert_ids which makes backfills feasible.

Depending on next steps this is safe to merge at this point as devtools schema can be handled in a different PR.

It hasn't been used in legacy ETL and was a duplicate of `os`.

For consistency with Fenix view and new process

It duplicates event_type

…Amplitude job This adds support to read from views introduced in mozilla/bigquery-etl#510. This can be used as an alternative solution if S3 ingestion takes too long to set up.

Mobile views were merged as aprt of #580

akkomar · 2019-12-17T08:14:49Z

Devtools schema is handled in #599

akkomar · 2019-12-30T09:31:50Z

This has been completed in #580 and #599.

Bug 1574490 - Add views for events.

1faffe7

mreid-moz requested a review from fbertsch November 18, 2019 14:50

Merge branch 'master' into bug1574490

2e391b6

fbertsch reviewed Nov 19, 2019

View reviewed changes

Merge branch 'master' into bug1574490

212e2b1

akkomar reviewed Nov 21, 2019

View reviewed changes

sql/telemetry/fennec_ios_events_v1/view.sql Outdated Show resolved Hide resolved

akkomar reviewed Nov 21, 2019

View reviewed changes

sql/telemetry/desktop_savant_events_v1/view.sql Outdated Show resolved Hide resolved

Add script for generating Amplitude-compatible event views

aefe0a8

Based on https://gist.github.com/mreid-moz/33837c4c511b4fb7923f67a91232beb7

akkomar and others added 3 commits November 22, 2019 14:21

Merge branch 'master' into bug1574490

9bc18f5

Make insert_id compatible with current implementation

24ec549

See https://github.com/mozilla/telemetry-streaming/blob/b85cdffc72a6f9ab224f9eececc38dfa09d98b8c/src/main/scala/com/mozilla/telemetry/pings/Ping.scala#L335-L338 for current implementation

Remove savant view

340cf51

It has been unscheduled in 2018: mozilla/telemetry-airflow@ae0f236#diff-99344063164ff6dff208054f7a8ed655

akkomar force-pushed the bug1574490 branch from 6e70a8b to 340cf51 Compare November 22, 2019 20:20

akkomar and others added 3 commits November 24, 2019 16:22

Make insert_id compatible with current implementation

627036f

Don't use event group name in `insert_id`

Use full event name as event_type

cc22074

See https://github.com/mozilla/telemetry-streaming/blob/master/src/main/scala/com/mozilla/telemetry/pings/Ping.scala#L343

Merge branch 'master' into bug1574490

6f8042e

akkomar force-pushed the bug1574490 branch from 6c15cc0 to 9c3cdc0 Compare November 25, 2019 12:14

Fennec_ios: add missing user properties

d332c44

Some properties were hardcoded per ping type and not included in the schemas

akkomar force-pushed the bug1574490 branch from 9c3cdc0 to d332c44 Compare November 25, 2019 12:37

Add city to views

56ff0ec

See https://github.com/mozilla/telemetry-streaming/blob/b85cdffc72a6f9ab224f9eececc38dfa09d98b8c/src/main/scala/com/mozilla/telemetry/pings/Ping.scala#L351

akkomar mentioned this pull request Nov 26, 2019

Bug 1574490 - Send iOS events to Amplitude from BigQuery view mozilla/telemetry-airflow#759

Closed

akkomar and others added 8 commits November 26, 2019 17:46

Switch session_id_offset to session_id in views

13207ed

See https://github.com/mozilla/telemetry-streaming/blob/b85cdffc72a6f9ab224f9eececc38dfa09d98b8c/src/main/scala/com/mozilla/telemetry/pings/Ping.scala#L341

Merge branch 'master' into bug1574490

81c4ab2

Add focus user_properties, don't coalesce null boolean preferences to

b88f8c0

`false`

Fix Fire_TV insert_id

50bd6cc

Add Fire_TV user_properties

837ad73

Add Rocket_android user_properties

c10fcf7

Merge branch 'master' into bug1574490

0937864

Add os and app name filtering

55d3bb2

akkomar added 3 commits November 27, 2019 20:40

Add time field to views, use integer for session_id

95ca296

Remove platform field from views

1d62d0b

It hasn't been used in legacy ETL and was a duplicate of `os`.

Remove submission_date from views

518489b

For consistency with Fenix view and new process

akkomar force-pushed the bug1574490 branch from 6bce761 to 518489b Compare November 28, 2019 17:20

Don't include event_name in views

5fd4d1f

It duplicates event_type

akkomar mentioned this pull request Dec 2, 2019

Bug 1574490 - Add support for reading from BigQuery views to EventsTo… mozilla/telemetry-streaming#255

Draft

akkomar mentioned this pull request Dec 9, 2019

Bug 1574490 - Add Amplitude-compatible views for mobile events #580

Merged

akkomar and others added 6 commits December 11, 2019 18:54

Merge branch 'master' into bug1574490

f3a5cfb

Don't generate mobile views

e613edd

Mobile views were merged as aprt of #580

Fix session_start_time format

a68b3d1

generate sql

63d622a

Clone script for devtools views

47c051b

Use user_id instead of device-id in devtools views

70ec6a7

akkomar closed this Dec 30, 2019

acmiyaguchi deleted the bug1574490 branch January 6, 2021 23:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug 1574490 - Add views for events. #510

Bug 1574490 - Add views for events. #510

mreid-moz commented Nov 18, 2019

fbertsch left a comment

relud commented Nov 21, 2019

akkomar commented Nov 22, 2019

akkomar commented Nov 27, 2019

akkomar commented Dec 17, 2019

akkomar commented Dec 30, 2019

Bug 1574490 - Add views for events. #510

Bug 1574490 - Add views for events. #510

Conversation

mreid-moz commented Nov 18, 2019

fbertsch left a comment

Choose a reason for hiding this comment

relud commented Nov 21, 2019

akkomar commented Nov 22, 2019

akkomar commented Nov 27, 2019

akkomar commented Dec 17, 2019

akkomar commented Dec 30, 2019