Backfill metrics pt. 1 #1067

boek · 2019-03-18T23:11:30Z

No description provided.

app/src/main/java/org/mozilla/fenix/HomeActivity.kt

boek · 2019-03-19T21:50:50Z

Request for data collection review form

All questions are mandatory. You must receive review from a data steward peer on your responses to these questions before shipping new data collection.

What questions will you answer with this data?

We expand on how users are opening the app
Do users use search to get to their destination?
Do users use the autocomplete feature?
Do users use search suggestions?
How many people use Fenix as their default Browser?

Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements?

An increase/decrease will indicate happiness, stickiness and usefulness of the browser

What alternative methods did you consider to answer these questions? Why were they not sufficient?

N/A (These are baseline metrics)

Can current instrumentation answer these questions?

Currently no, as these are some of the first metrics we're recording

List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data c ollection categories on the found on the Mozilla wiki.

All data is Category 2.

How long will this data be collected?

@mmccorks will permanently monitor this data.

What populations will you measure?

All release, beta, and nightly users with telemetry enabled.

Please provide a general description of how you will analyze this data.

Leanplum
Glean

Where do you intend to share the results of your analysis?

Only on leanplum, glean and with mobile teams.

boek · 2019-03-19T22:06:35Z

Data review for @liuche

travis79

From a glean perspective, this looks good. I left what I hope is a helpful comment on the GleanMetricsService.kt file.

travis79 · 2019-03-20T18:22:50Z

app/src/main/java/org/mozilla/fenix/components/metrics/GleanMetricsService.kt


-    override fun shouldTrack(event: Event): Boolean = Settings.getInstance(context).isTelemetryEnabled
+    override fun shouldTrack(event: Event): Boolean {


Glean does this internally based on a persistent internal flag that can be toggled using the Glean.setUploadEnabled() function. The idea was to toggle the flag when the user opted out of telemetry and let the glean library handle discarding recorded data and preventing upload.

@travis79 Good to know, the secondary case for this method is we need a way to allow/block some metrics from being sent to some providers to support Leanplum

Then that's a good case for doing this in the client app. Glean currently doesn't have a way to disable a single specific metric at run-time, only through the metrics.yaml file.

It probably wouldn't be too hard to make that work -- but maybe it doesn't buy much...

@travis79 @mdboom If you look at the code in metrics.kt you can see how I have it setup. Right now I have a generic Event type for things that I want to track inside Fenix. When we track an event it dispatches each event to a MetricsService which decides whether or not it wants to track that event and then transforms is appropriately into a format that service expects.

mdboom · 2019-03-20T18:29:34Z

app/src/main/java/org/mozilla/fenix/components/metrics/Metrics.kt

+    // Interaction Events
+    data class SearchBarTapped(val source: Source) : Event() {
+        enum class Source { HOME, BROWSER }
+        override val extras: Map<String, String>?


Just a heads up -- the extras map will become Map<Enum, String> in the next android-components release. This lets us check that the keys are valid at compile time rather than run time. Unfortunately, it's a breaking change to the API -- and you're our first external user (lucky you!)

mozilla-mobile/android-components#2403

I actually really like this change 👍

liuche

Need some changes, mostly de-duplicating documentation (which is good! writing docs once is better)

Thanks for being the guinea pig for Glean SDK usage and documentation. I think parts are working, but would be happy to see some clearer documentation on the Glean SDK side so future consumers of the library have a clearer understanding. @mdboom is that something we could work on for future consumers of Glean? I think this is a valuable first pass!

liuche · 2019-03-21T22:42:39Z

app/metrics.yaml

    data_reviews:
-    - N/A
+    - https://github.com/mozilla-mobile/fenix/pull/1067#issuecomment-474598673
    notification_emails:
    - telemetry-client-dev@mozilla.com


Will it be useful to have a Product/Team email here as well, to be aware of expiry?

liuche · 2019-03-21T22:44:33Z

app/metrics.yaml

    data_reviews:
-    - N/A
+    - https://github.com/mozilla-mobile/fenix/pull/1067#issuecomment-474598673
    notification_emails:
    - telemetry-client-dev@mozilla.com
    expires: never


In terms of expiry, I'm not a fan of "never" expiry - is setting this to 6mo-1yr acceptable? It's also a regular check for whoever the product manager is to be aware of what data we're collecting.

cc @mdboom on seeing this "never" pop up from the sample code

Actually I'm not even sure what expiry would mean here, @mdboom is this a date field? Would be really helpful to have this in the sample yaml file.

@liuche https://mozilla.github.io/glean_parser/metrics-yaml.html#expires It looks like this field isn't for when the data expires, but when we should stop collecting this data.

Definitely an interesting idea! But I don't think we'll want this to expire?

Ah, after re-reading I understand now! I will set this to a year. I'm not sure if there is a good mailing list to alert yet though.

Yes, that's the correct documentation for that field. And as you point out, data retention is a separate issue (which we hope to address at some point...)

@mdboom yeah, my point here was more to say, can you set the sample yaml file to actually use an expiration date, rather than never? I think that's a huge problem when the "default" in the sample is "never expire this" 😓

@liuche I changed it to a date, specifically 1 year from now. I think we can talk with Product about this more later as well.

@liuche: I'm not sure how to do that without breaking the sample, though. :(

Talked to @mdboom and he said he'll file a bug next week to set a date for the sample, and add a comment for any consumers of it to update the expiry date.

liuche · 2019-03-21T22:53:27Z

app/metrics.yaml

@@ -0,0 +1,20 @@
+# This file defines the metrics that are recorded by glean telemetry. They are


Nit do you need a license on this?

Also I assume this is just copied from the Glean sample file? I wonder if we should include the license there too @mdboom

It's easy enough to add. I don't feel too strongly either way. I filed a bug for glean itself here, and @boek and the fenix team can decide about this file.

liuche · 2019-03-21T22:54:32Z

app/metrics.yaml

+    description: >
+      A User opened the app
+    extra_keys:
+      source: "The source from which the app was opened"


Can you be more specific about this, including all possible values this can be?

Ah I see the higher detail later in the docs! I think the goal of Glean is to avoid having to write those docs anymore, so put all that detail and values and explanation right here in the yaml file itself.

👍 whenever possible

app/metrics.yaml

liuche · 2019-03-21T23:01:27Z

app/src/main/java/org/mozilla/fenix/components/metrics/Metrics.kt

@@ -52,6 +52,16 @@ sealed class Event {
            get() = mapOf("source" to source.name)
    }

+    data class EnteredUrl(val autoCompleted: Boolean) : Event() {
+        override val extras: Map<String, String>?
+            get() = mapOf("autocomplete" to autoCompleted.toString())


Honestly I don't love all these booleans-to-strings, because it's less clear what is happening. @mdboom can Glean support non-string, string maps? I thought maps could be printed out as strings without every item being a string.

That's a good suggestion, @liuche. I've filed a this bug to follow up on this. @boek: I'd suggest you merge this as-is in the PR for now, and we can revisit once the glean team has decided how to move forward with that. It should be easy enough to find these instances and change them...

docs/telemetry.md

liuche

data-review+

One tiny nit about LP + typo, and @boek please remember to add a Product email to the expiry notifications (or file a bug if necessary to track).
Also @mdboom to add a bug to change expiry: never to an actual date + comment in the sample.yaml code (please link here).

Data Review Form

Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?
Yes, documented by Glean SDK in metrics.yaml, and can be scraped into docs page in Automate Fenix telemetry documentation to probe-scraper #1156
Is there a control mechanism that allows the user to turn the data collection on and off?
Yes, Fenix has data collection toggle in-app
If the request is for permanent data collection, is there someone who will monitor the data over time?**
Expires in 1 year, open for renewal
Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under? **
Type 2, interaction: app opened, search bar tapped, entered url, performed search, default browser, with either booleans or a set of pre-defined event strings.
Is the data collection request for default-on or default-off?
Default on
Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?
No
Is the data collection covered by the existing Firefox privacy notice?
Yes
Does there need to be a check-in in the future to determine whether to renew the data? (Yes/No) (If yes, set a todo reminder or file a bug if appropriate)**
No, will expire - but @boek needs to add another Product email to these probes so they get notified when the probe collection is about to expire.
Does the data collection use a third-party collection tool?
No, uses Glean

liuche · 2019-03-22T23:43:03Z

docs/telemetry.md

Nit: typo in this sentence

Also, you should keep the app_opened ping because the Leanplum key is not documented in Glean I assume, and #981 is still open.

…ult browser

…pped

…o telemetry

boek · 2019-03-22T23:58:01Z

Tracking notification reminder here: #1157

ghost assigned boek Mar 18, 2019

ghost added the review label Mar 18, 2019

colintheshots reviewed Mar 19, 2019

View reviewed changes

app/src/main/java/org/mozilla/fenix/HomeActivity.kt Outdated Show resolved Hide resolved

boek force-pushed the mi-metrics branch from 27a8990 to 9c25ad4 Compare March 19, 2019 21:42

boek marked this pull request as ready for review March 19, 2019 22:06

boek requested a review from a team as a code owner March 19, 2019 22:06

boek requested a review from liuche March 19, 2019 22:06

boek mentioned this pull request Mar 20, 2019

[Meta] Integrate Glean SDK (Event Ping) #923

Closed

travis79 reviewed Mar 20, 2019

View reviewed changes

mdboom reviewed Mar 20, 2019

View reviewed changes

colintheshots approved these changes Mar 20, 2019

View reviewed changes

boek changed the title ~~Backfill metrics~~ Backfill metrics pt. 1 Mar 20, 2019

boek assigned liuche Mar 20, 2019

boek added the needs:data-review PR is awaiting a data review label Mar 20, 2019

liuche suggested changes Mar 21, 2019

View reviewed changes

boek force-pushed the mi-metrics branch from 9027838 to 59ef543 Compare March 22, 2019 06:20

sblatz mentioned this pull request Mar 22, 2019

As a developer I want to receive crash reports #906

Closed

boek force-pushed the mi-metrics branch 2 times, most recently from 5b34874 to 1feea32 Compare March 22, 2019 17:32

boek requested a review from liuche March 22, 2019 17:44

liuche mentioned this pull request Mar 22, 2019

Automate Fenix telemetry documentation to probe-scraper #1156

Closed

liuche approved these changes Mar 22, 2019

View reviewed changes

boek added 6 commits March 22, 2019 16:53

For mozilla-mobile#960 - Adds metrics to measure if Fenix is the defa…

d42a15c

…ult browser

For mozilla-mobile#968 - Adds telemetry for app entry point

836e214

For mozilla-mobile#959 - Adds telemetry for when the search bar is ta…

75dd80f

…pped

For mozilla-mobile#959 - Adds search related telemetry

7787507

For mozilla-mobile#986 - Adds support for opening Fenix with a link t…

5190e80

…o telemetry

Documents new telemetry events

6cf4e72

Addresses metrics nits

2de49bd

boek force-pushed the mi-metrics branch from 1feea32 to 2de49bd Compare March 22, 2019 23:57

boek merged commit ea01ae4 into mozilla-mobile:master Mar 23, 2019

boek deleted the mi-metrics branch March 23, 2019 00:14

ghost removed the review label Mar 23, 2019

mcomella mentioned this pull request May 12, 2020

[Bug] app_opened telemetry event unexpectedly does not run on every app launch #10616

Closed

liuche mentioned this pull request Aug 21, 2020

For #12046 - Renew telemetry probes before migration completion #13958

Merged

4 tasks

liuche mentioned this pull request Oct 6, 2020

Issue #14142 - Telemetry renewal to 08-01-2021 #15713

Merged

3 tasks

mcarare mentioned this pull request Feb 24, 2021

For #18076: Renew telemetry expiring in April. #18143

Merged

3 tasks

eliserichards mentioned this pull request Jun 14, 2021

For #19923: renew telemetry probes #19924

Merged

3 tasks

eliserichards mentioned this pull request Jul 26, 2021

For #20516: Renew product telemetry probes expiring in August #20517

Merged

3 tasks

mdboom mentioned this pull request Aug 26, 2021

For #21025: Set long term business-critical metrics as non-expiring #21038

Merged

3 tasks

eliserichards mentioned this pull request Oct 22, 2021

For #21313: Renew metrics for December - never expire updates #21788

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Backfill metrics pt. 1 #1067

Backfill metrics pt. 1 #1067

boek commented Mar 18, 2019

boek commented Mar 19, 2019 •

edited

boek commented Mar 19, 2019

travis79 left a comment

travis79 Mar 20, 2019

boek Mar 20, 2019

travis79 Mar 20, 2019

mdboom Mar 20, 2019

boek Mar 20, 2019

mdboom Mar 20, 2019

boek Mar 20, 2019

liuche left a comment

liuche Mar 21, 2019

liuche Mar 21, 2019

liuche Mar 21, 2019

boek Mar 22, 2019

boek Mar 22, 2019

mdboom Mar 22, 2019

liuche Mar 22, 2019

boek Mar 22, 2019 •

edited

mdboom Mar 22, 2019

liuche Mar 22, 2019

liuche Mar 21, 2019

liuche Mar 21, 2019

mdboom Mar 22, 2019

liuche Mar 21, 2019

liuche Mar 21, 2019

mdboom Mar 22, 2019

liuche Mar 21, 2019

mdboom Mar 22, 2019

liuche left a comment

liuche Mar 22, 2019

boek commented Mar 22, 2019


		override fun shouldTrack(event: Event): Boolean = Settings.getInstance(context).isTelemetryEnabled
		override fun shouldTrack(event: Event): Boolean {

		@@ -0,0 +1,20 @@
		# This file defines the metrics that are recorded by glean telemetry. They are

Backfill metrics pt. 1 #1067

Backfill metrics pt. 1 #1067

Conversation

boek commented Mar 18, 2019

boek commented Mar 19, 2019 • edited

Request for data collection review form

boek commented Mar 19, 2019

travis79 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liuche left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

boek Mar 22, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liuche left a comment

Choose a reason for hiding this comment

Data Review Form

Choose a reason for hiding this comment

boek commented Mar 22, 2019

boek commented Mar 19, 2019 •

edited

boek Mar 22, 2019 •

edited