From 18b467834668855e3b10ca77765f3a33ff05cb82 Mon Sep 17 00:00:00 2001 From: SuYoungHong Date: Mon, 22 Oct 2018 20:38:44 -0700 Subject: [PATCH 1/5] adding best practices for telemetry events design --- src/SUMMARY.md | 1 + .../designing_events_best_practices.md | 36 +++++++++++++++++++ 2 files changed, 37 insertions(+) create mode 100644 src/cookbooks/designing_events_best_practices.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index d348b8a68..1ced2e850 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -34,6 +34,7 @@ * [Metrics](cookbooks/metrics.md) * [Active DAU Definition](cookbooks/active_dau.md) * [Retention Analysis](cookbooks/retention.md) + * [Designing Telemetry Events Best Practices](cookbooks/designing_events_best_practices.md) --- diff --git a/src/cookbooks/designing_events_best_practices.md b/src/cookbooks/designing_events_best_practices.md new file mode 100644 index 000000000..364d56670 --- /dev/null +++ b/src/cookbooks/designing_events_best_practices.md @@ -0,0 +1,36 @@ +# Designing Telemetry Events Best Practices + +## Naming Events/Probes: + +One of the limitations of the Telemetry Events API is that it is difficult to distinguish between events produced by different probes. The API allows for a lot of flexibility. Any probe written into Firefox can create an event with any values the following fields: + + | Timestamp | Category (string) | Method (string) | Object (string) | Value (string) | Extra (map) | + +With the following restrictions: +* The Category, Method, and Object of any event produced by the probe must have a value. +* All combinations of Category, Method, and Object values must be unqiue to that probe (no other probes can produce the same combination). + +However, there is no identifier or name field for the probe itself. Furthermore, although the combinations of Category, Method, and Object for a probe must be unique, individually, those fields can share values with other probes. + +This can make it confusing and difficult to isolate events from a particular probe, or find what probe produced a specific event. + +#### Suggested Convention: + +To deal with this API limitation, we suggest recording the probe name in the Category field of the event in this format: + +``` +"category.probe_name" +``` + +For example: +* ```"navigation.search"``` +* ```"frame.tab"``` +* ```"encounter.popup"``` + + +This provides 3 advantages: +1. Events produced by this probe will be easily identifiable. Conversely, the probe which produced the event will be easier to locate in the code. +2. Probes can be controlled easier. The category field is what we use to "turn on" and "turn off" events. By creating a 1 to 1 mapping between categories and probes, we can control events on an individual probe level. +3. By having the category field act as the probe identifier, it makes it easier to pass on events to Amplitude and other platforms. + + From a0a53fa28a462e8bb7858ad1241a38f8e9f913b5 Mon Sep 17 00:00:00 2001 From: SuYoungHong Date: Wed, 24 Oct 2018 17:42:39 -0700 Subject: [PATCH 2/5] updated events best pratice with more background and examples --- src/SUMMARY.md | 2 +- .../designing_events_best_practices.md | 36 ------- src/cookbooks/events_best_practices.md | 101 ++++++++++++++++++ 3 files changed, 102 insertions(+), 37 deletions(-) delete mode 100644 src/cookbooks/designing_events_best_practices.md create mode 100644 src/cookbooks/events_best_practices.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 1ced2e850..96294cbe5 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -34,7 +34,7 @@ * [Metrics](cookbooks/metrics.md) * [Active DAU Definition](cookbooks/active_dau.md) * [Retention Analysis](cookbooks/retention.md) - * [Designing Telemetry Events Best Practices](cookbooks/designing_events_best_practices.md) + * [Telemetry Events Best Practices](cookbooks/events_best_practices.md) --- diff --git a/src/cookbooks/designing_events_best_practices.md b/src/cookbooks/designing_events_best_practices.md deleted file mode 100644 index 364d56670..000000000 --- a/src/cookbooks/designing_events_best_practices.md +++ /dev/null @@ -1,36 +0,0 @@ -# Designing Telemetry Events Best Practices - -## Naming Events/Probes: - -One of the limitations of the Telemetry Events API is that it is difficult to distinguish between events produced by different probes. The API allows for a lot of flexibility. Any probe written into Firefox can create an event with any values the following fields: - - | Timestamp | Category (string) | Method (string) | Object (string) | Value (string) | Extra (map) | - -With the following restrictions: -* The Category, Method, and Object of any event produced by the probe must have a value. -* All combinations of Category, Method, and Object values must be unqiue to that probe (no other probes can produce the same combination). - -However, there is no identifier or name field for the probe itself. Furthermore, although the combinations of Category, Method, and Object for a probe must be unique, individually, those fields can share values with other probes. - -This can make it confusing and difficult to isolate events from a particular probe, or find what probe produced a specific event. - -#### Suggested Convention: - -To deal with this API limitation, we suggest recording the probe name in the Category field of the event in this format: - -``` -"category.probe_name" -``` - -For example: -* ```"navigation.search"``` -* ```"frame.tab"``` -* ```"encounter.popup"``` - - -This provides 3 advantages: -1. Events produced by this probe will be easily identifiable. Conversely, the probe which produced the event will be easier to locate in the code. -2. Probes can be controlled easier. The category field is what we use to "turn on" and "turn off" events. By creating a 1 to 1 mapping between categories and probes, we can control events on an individual probe level. -3. By having the category field act as the probe identifier, it makes it easier to pass on events to Amplitude and other platforms. - - diff --git a/src/cookbooks/events_best_practices.md b/src/cookbooks/events_best_practices.md new file mode 100644 index 000000000..f5ec03d83 --- /dev/null +++ b/src/cookbooks/events_best_practices.md @@ -0,0 +1,101 @@ +# Telemetry Events Best Practices + +## Overview: + +[The Telemetry Events API](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/events.html) allows users to define and record events in the browser. + +Events are defined in [Events.yaml](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/events.html#the-yaml-definition-file) and each events creates records with the following properties: +* timestamp +* category +* method +* object +* value +* extra + +With the following restrictions and features: +* The category, method, and object properties of any record produced by an event must have a value. +* All combinations values from the category, method, and object properties must be unique to that particular event (no other event can produce events with the same combination). +* Events can be 'turned on' or 'turned off' by it's category value. i.e. we can instruct the browser to "stop sending us events from 'devtools' category." + +These records are then stored in event pings and available in the events dataset. + +## Identifying Events + +One challenge with this data is it can be difficult to identify all the records from a particular event. Unlike Scalars and Histograms, which keep data in individual locations (like `scalar_parent_browser_engagement_total_uri_count` for [total_uri_count](https://dxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/Scalars.yaml#99)), all event records are stored together, regardless of which event generated them. The records themselves don't have a field identifying which event produced it[1]. + +Take, for example, the ["manage"](https://dxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/Events.yaml#105) + event in the "addonsManager" category. + +``` +addonsManager: # category + manage: # event name + description: > + ... + objects: ["extension", "theme", "locale", "dictionary", "other"] # object values + methods: ["disable", "enable", "sideload_prompt", "uninstall"] # method values + extra_keys: # extra values + ... + notification_emails: ... + expiry_version: ... + record_in_processes: ... + bug_numbers: ... + release_channel_collection: ... +``` +This event will produce records that look like: + +|timestamp| category | method | object | value | extra | +|-| -------- |---------| --------|-------|-------| +|...|addonsManager|install|extension| |...| +|...|addonsManager|update|locale| |...| +|...|addonsManager|sideload_prompt|other| |...| + +But none of these records will indicate that it was produced by the "manage" event. To find all records produced by "manage", one would have to query all records where +``` +category = ... +AND method in [...,] +AND object in [...,] +``` +which is not ideal. + +Furthermore, if one encounters this data without knowledge of how the "manage" event works, they need to look up the event based on the category, method, and object values in order to find the event, and then requery the data to find all the related events. It's not immediately clear from the data if this record: + +|timestamp| category | method | object | value | extra | +|-| -------- |---------| --------|-------|-------| +|...|addonsManager|update|locale| |...| + +and and this record: + +|timestamp| category | method | object | value | extra | +|-| -------- |---------| --------|-------|-------| +|...|addonsManager|install|extension| |...| + +are related or not. + +Another factor that can add to confusion is the fact that other events can share similar values for methods or objects (or even the combination of method and object). For example: + +|timestamp| category | method | object | value | extra | +|-| -------- |---------| --------|-------|-------| +|...|normandy|update|preference_rollout| |...| + +which can further confuser users. + +[1]: If a user defines an event in Events.yaml without specifying a list of acceptable methods, the method will default to the name of the event for records created by that event. + +#### Suggested Convention: + +To simplify things in the future, we suggest recording the event name in the category field in this format when designing new events: + +``` +"category.event_name" +``` + +For example: +* ```"navigation.search"``` +* ```"addonsManager.manage"``` +* ```"frame.tab"``` + + +This provides 3 advantages: +1. Records produced by this event will be easily identifiable. Conversely, the event which produced the record will be easier to locate in the code. +2. Events can be controlled easier. The category field is what we use to "turn on" and "turn off" events. By creating a 1 to 1 mapping between categories and events, we can control events on an individual level. +3. By having the category field act as the event identifier, it makes it easier to pass on events to Amplitude and other platforms. From 9e22b3e69e06f69d38d8ac2d1e07eccc6680373c Mon Sep 17 00:00:00 2001 From: Su-Young Hong Date: Wed, 13 Feb 2019 15:20:21 -0200 Subject: [PATCH 3/5] Update events_best_practices.md updating doc r.e. suggestions from chutten --- src/cookbooks/events_best_practices.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/cookbooks/events_best_practices.md b/src/cookbooks/events_best_practices.md index f5ec03d83..c27a56dd4 100644 --- a/src/cookbooks/events_best_practices.md +++ b/src/cookbooks/events_best_practices.md @@ -14,10 +14,10 @@ Events are defined in [Events.yaml](https://firefox-source-docs.mozilla.org/tool With the following restrictions and features: * The category, method, and object properties of any record produced by an event must have a value. -* All combinations values from the category, method, and object properties must be unique to that particular event (no other event can produce events with the same combination). +* All combinations of values from the category, method, and object properties must be unique to that particular event (no other event can produce events with the same combination). * Events can be 'turned on' or 'turned off' by it's category value. i.e. we can instruct the browser to "stop sending us events from 'devtools' category." -These records are then stored in event pings and available in the events dataset. +These records are then stored in [event pings](http://gecko-docs.mozilla.org.s3.amazonaws.com/toolkit/components/telemetry/telemetry/data/event-ping.html) and available in the [events dataset](https://docs.telemetry.mozilla.org/datasets/batch_view/events/reference.html). ## Identifying Events @@ -79,11 +79,11 @@ Another factor that can add to confusion is the fact that other events can share which can further confuser users. -[1]: If a user defines an event in Events.yaml without specifying a list of acceptable methods, the method will default to the name of the event for records created by that event. +[1]: Events do have `name` fields, but they aren't included in the event records and thus are not present in the resulting dataset. Also, If a user defines an event in Events.yaml without specifying a list of acceptable methods, the method will default to the name of the event for records created by that event. #### Suggested Convention: -To simplify things in the future, we suggest recording the event name in the category field in this format when designing new events: +To simplify things in the future, we suggest adding the event name to the category field using dot notation when designing new events: ``` "category.event_name" @@ -96,6 +96,6 @@ For example: This provides 3 advantages: -1. Records produced by this event will be easily identifiable. Conversely, the event which produced the record will be easier to locate in the code. -2. Events can be controlled easier. The category field is what we use to "turn on" and "turn off" events. By creating a 1 to 1 mapping between categories and events, we can control events on an individual level. +1. Records produced by this event will be easily identifiable. Also, the event which produced the record will be easier to locate in the code. +2. Events can be controlled more easily. The category field is what we use to "turn on" and "turn off" events. By creating a 1 to 1 mapping between categories and events, we can control events on an individual level. 3. By having the category field act as the event identifier, it makes it easier to pass on events to Amplitude and other platforms. From 3dfcafa0412f0e35e9b46dd6f5964cb2df278bae Mon Sep 17 00:00:00 2001 From: Su-Young Hong Date: Wed, 13 Feb 2019 18:34:39 -0200 Subject: [PATCH 4/5] Update events_best_practices.md fixed spelling nits (into code) for travis tests --- src/cookbooks/events_best_practices.md | 28 +++++++++++++------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/src/cookbooks/events_best_practices.md b/src/cookbooks/events_best_practices.md index c27a56dd4..f28f110e2 100644 --- a/src/cookbooks/events_best_practices.md +++ b/src/cookbooks/events_best_practices.md @@ -4,7 +4,7 @@ [The Telemetry Events API](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/events.html) allows users to define and record events in the browser. -Events are defined in [Events.yaml](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/events.html#the-yaml-definition-file) and each events creates records with the following properties: +Events are defined in [`Events.yaml`](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/events.html#the-yaml-definition-file) and each events creates records with the following properties: * timestamp * category * method @@ -15,16 +15,16 @@ Events are defined in [Events.yaml](https://firefox-source-docs.mozilla.org/tool With the following restrictions and features: * The category, method, and object properties of any record produced by an event must have a value. * All combinations of values from the category, method, and object properties must be unique to that particular event (no other event can produce events with the same combination). -* Events can be 'turned on' or 'turned off' by it's category value. i.e. we can instruct the browser to "stop sending us events from 'devtools' category." +* Events can be 'turned on' or 'turned off' by it's category value. i.e. we can instruct the browser to "stop sending us events from the `devtools` category." These records are then stored in [event pings](http://gecko-docs.mozilla.org.s3.amazonaws.com/toolkit/components/telemetry/telemetry/data/event-ping.html) and available in the [events dataset](https://docs.telemetry.mozilla.org/datasets/batch_view/events/reference.html). ## Identifying Events -One challenge with this data is it can be difficult to identify all the records from a particular event. Unlike Scalars and Histograms, which keep data in individual locations (like `scalar_parent_browser_engagement_total_uri_count` for [total_uri_count](https://dxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/Scalars.yaml#99)), all event records are stored together, regardless of which event generated them. The records themselves don't have a field identifying which event produced it[1]. +One challenge with this data is it can be difficult to identify all the records from a particular event. Unlike Scalars and Histograms, which keep data in individual locations (like `scalar_parent_browser_engagement_total_uri_count` for [`total_uri_count`](https://dxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/Scalars.yaml#99)), all event records are stored together, regardless of which event generated them. The records themselves don't have a field identifying which event produced it[1]. -Take, for example, the ["manage"](https://dxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/Events.yaml#105) - event in the "addonsManager" category. +Take, for example, the [`manage`](https://dxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/Events.yaml#105) + event in the `addonsManager` category. ``` addonsManager: # category @@ -45,11 +45,11 @@ This event will produce records that look like: |timestamp| category | method | object | value | extra | |-| -------- |---------| --------|-------|-------| -|...|addonsManager|install|extension| |...| -|...|addonsManager|update|locale| |...| -|...|addonsManager|sideload_prompt|other| |...| +|...|`addonsManager`|`install`|`extension`| |...| +|...|`addonsManager`|`update`|`locale`| |...| +|...|`addonsManager`|`sideload_prompt`|`other`| |...| -But none of these records will indicate that it was produced by the "manage" event. To find all records produced by "manage", one would have to query all records where +But none of these records will indicate that it was produced by the `manage` event. To find all records produced by `manage`, one would have to query all records where ``` category = ... AND method in [...,] @@ -57,17 +57,17 @@ AND object in [...,] ``` which is not ideal. -Furthermore, if one encounters this data without knowledge of how the "manage" event works, they need to look up the event based on the category, method, and object values in order to find the event, and then requery the data to find all the related events. It's not immediately clear from the data if this record: +Furthermore, if one encounters this data without knowledge of how the `manage` event works, they need to look up the event based on the category, method, and object values in order to find the event, and then requery the data to find all the related events. It's not immediately clear from the data if this record: |timestamp| category | method | object | value | extra | |-| -------- |---------| --------|-------|-------| -|...|addonsManager|update|locale| |...| +|...|`addonsManager`|`update`|`locale`| |...| and and this record: |timestamp| category | method | object | value | extra | |-| -------- |---------| --------|-------|-------| -|...|addonsManager|install|extension| |...| +|...|`addonsManager`|`install`|`extension`| |...| are related or not. @@ -75,11 +75,11 @@ Another factor that can add to confusion is the fact that other events can share |timestamp| category | method | object | value | extra | |-| -------- |---------| --------|-------|-------| -|...|normandy|update|preference_rollout| |...| +|...|`normandy`|`update`|`preference_rollout`| |...| which can further confuser users. -[1]: Events do have `name` fields, but they aren't included in the event records and thus are not present in the resulting dataset. Also, If a user defines an event in Events.yaml without specifying a list of acceptable methods, the method will default to the name of the event for records created by that event. +[1]: Events do have name fields, but they aren't included in the event records and thus are not present in the resulting dataset. Also, If a user defines an event in `Events.yaml` without specifying a list of acceptable methods, the method will default to the name of the event for records created by that event. #### Suggested Convention: From fb699c7c14ad03b88537be8be2dc67f39db44793 Mon Sep 17 00:00:00 2001 From: Su-Young Hong Date: Wed, 13 Feb 2019 18:37:41 -0200 Subject: [PATCH 5/5] Update events_best_practices.md nit --- src/cookbooks/events_best_practices.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/cookbooks/events_best_practices.md b/src/cookbooks/events_best_practices.md index f28f110e2..3111c46f5 100644 --- a/src/cookbooks/events_best_practices.md +++ b/src/cookbooks/events_best_practices.md @@ -57,7 +57,7 @@ AND object in [...,] ``` which is not ideal. -Furthermore, if one encounters this data without knowledge of how the `manage` event works, they need to look up the event based on the category, method, and object values in order to find the event, and then requery the data to find all the related events. It's not immediately clear from the data if this record: +Furthermore, if one encounters this data without knowledge of how the `manage` event works, they need to look up the event based on the category, method, and object values in order to find the event, and then query the data again to find all the related events. It's not immediately clear from the data if this record: |timestamp| category | method | object | value | extra | |-| -------- |---------| --------|-------|-------|