add metrics overview #1662

clouserw · 2016-10-21T21:39:36Z

PR for discussion. All comments/edits are welcome.

There is a TODO in it looking for sample code. I saw several experiments doing it in different ways. What is the right way?
There are several files in docs/metrics/ which I think can be cleaned up after this lands
My hope was to be able to have the Test Pilot add-on be a broker for metrics and be able to choose what to send to Telemetry, but I couldn't figure out a good way to do that and keep engineering costs low. I think making sure Telemetry is focused on the big questions should keep the reporting code relatively simple (vs instrumenting every event and button which we'll do more of in GA).

ckprice · 2016-10-22T04:18:14Z

docs/metrics/index.md

+Running included JavaScript from the net is asking for trouble (and is actually
+not allowed at all on addons.mozilla.org).
+
+Google [provides documentation for logging


You may want to use an example from the add-on code to highlight the Measurement Protocol instead of from the website (which actually does use the GA library).

chuckharmston · 2016-10-24T21:31:15Z

There is a TODO in it looking for sample code. I saw several experiments doing it in different ways. What is the right way?

Different experiments use different frameworks, which have their own interfaces (or none at all) to nsIObserverService:

Universal Search uses the XPCOM API.
Test Pilot itself uses the SDK system/events module.
Tracking Protection uses the BroadcastChannel API, which is a special thing that @lmorchard did for WebExtensions, since they do not have access to nsIObserverService.

chuckharmston · 2016-10-26T15:39:54Z

docs/metrics/index.md

+# Overview
+
+Test Pilot loves metrics.  Our whole purpose is to ask questions, measure and
+analyaze responses, and make recommendations.  To that end, we have a few tools


chuckharmston · 2016-10-26T15:41:02Z

docs/metrics/index.md

+is just gravy.  We use GA for all basic measuring, funnel tracking,
+click-through rates, and simple A/B testing.
+
+## Telemetry


We should probably add some context about what capital-T Telemetry means.

chuckharmston · 2016-10-26T15:44:06Z

docs/metrics/index.md

+
+Google Analytics does a lot of great things generically, but comes up short
+when trying to do deep analysis on data, which is usually where those big
+defining questions lay.


Two other thing Telemetry does well:

Correcting things we measure with other behaviors or attributes that we don't or can't measure

Correcting for bias in our population. Test Pilot users are demonstrably power users, and Telemetry gives us data that can help us understand and correct for that.

chuckharmston · 2016-10-26T15:44:55Z

docs/metrics/index.md

+to [clone these two bugs](https://bugzilla.mozilla.org/buglist.cgi?quicksearch=1297199%2C+1297200).
+
+The Measurement Team will be looking at the Metrics Plan you wrote to
+understand what is needed so it's important it includes the schema you'll be


needed, so it's important that it includes

chuckharmston · 2016-10-26T15:46:33Z

docs/metrics/index.md

+```javascript
+app.sendToGA('event', {
+    eventCategory: 'Menu Interactions',
+      eventAction: 'drop-down menu',


The increasing tab size here is probably not what you intended.

chuckharmston · 2016-10-26T15:47:08Z

docs/metrics/index.md

+## Google Analytics
+To use Google Analytics we recommend you use your own Google Analytics account
+to record data.  This makes the transition easy when your experiment graduates
+from the Test Pilot program.


Because this is the case, should we mention permissions? I'd think we want to make sure that the Test Pilot team has at least read access to each of these accounts.

chuckharmston · 2016-10-26T15:47:35Z

docs/metrics/index.md

+
+We're all here to help.  This document exists as documentation for us as much
+as anyone -- don't feel like you have to go at this alone, it just needs to be
+written down so we don't forget steps. :)


An emoticon and no emoji? What is this, 2013?

ghost · 2016-10-27T23:18:05Z

I pushed a new version of the proposal. The "Sending Data" section is new so please check that out.

ghost · 2016-10-27T23:20:43Z

docs/metrics/index.md

+TODO: code here
+```
+
+The testpilot.sendEvent() function accepts these parameters:


These fields are shared across the three systems with slightly different names so I think it will be easy to remap and send to all three as a base. We'll still need to add specific fields for additional experiment meta-data.

This doesn't get us every measurement we could want, but it gets us a lot of data quickly

chuckharmston · 2016-10-28T20:43:09Z

docs/metrics/index.md

+## Sending Data
+
+The actual function to report event based data is called `testpilot.sendEvent()` and is
+provided by the `test-pilot-metrics` node module. To use it:


testpilot-metrics? As I recall, some engineering manager somewhere insisted that we don't hyphenate.

chuckharmston · 2016-11-01T21:02:22Z

docs/metrics/index.md

+
+We're all here to help.  This document exists as documentation for us as much
+as anyone -- don't feel like you have to go at this alone, it just needs to be
+written down so we don't forget steps. 😺


jaredhirsch · 2016-11-07T17:50:36Z

docs/metrics/index.md

+
+## Google Analytics
+To use Google Analytics you'll need to have your Google Analytics credentials
+to record data.  Using your own account here instead of a generic Test Pilot


I think that Mozilla has a special GA account that is extra privacy-preserving.

I think we either need to ask experiment authors to use the special non-tracking Mozilla instance of GA, or, if authors will use their own, regular GA accounts, we may need to talk to legal about revising our terms of service.

lmorchard · 2016-11-15T18:39:35Z

docs/metrics/index.md

+the appropriate endpoints.
+
+> TODO:
+> * need to specify what "system information" is, if for no other reason than we


We get a lot of system information "for free" on Telemetry thanks to the environment data added to the ping.

I think a lot of that will be inaccessible to library code running in webextension-based experiments - assuming we wanted to gather & send some of it as context to GA or the Tiles Pipeline. Need to inventory what we can actually gather in that scenario. (Could be more than I think off the top of my head)

jaredhirsch · 2016-11-28T17:47:23Z

docs/metrics/index.md

+* `variant` (optional): An identifying string if you're running different
+  variants. eg. `cohort-A`
+
+These paramters will be combined with other data (like a timestamp and system


typo: parameters

jaredhirsch · 2016-11-28T17:47:37Z

docs/metrics/index.md

+  variants. eg. `cohort-A`
+
+These paramters will be combined with other data (like a timestamp and system
+information) in the sentEvent() function and the broker will send the data to


typo: sendEvent

jaredhirsch · 2016-11-28T17:53:33Z

@clouserw Based on our discussion from a few weeks back, looks like this doc to be updated to remove references to the testpilot npm module. I think we were just going to provide good docs on how to send messages from different addon types. For reference, discussion notes are here.

ckprice · 2017-01-25T20:25:48Z

Are these docs deprecated by mozilla/testpilot-metrics#4?

jaredhirsch · 2017-01-26T17:54:52Z

@ckprice I made some edits to this PR, I'll push them up in a bit

jaredhirsch · 2017-01-26T18:31:22Z

@ckprice @clouserw updated the PR with another commit's worth of testpilot-metrics stuff. Mind taking a look?

jaredhirsch · 2017-01-27T00:15:45Z

Looks like 'event' needs to be replaced with 'method', per the Event Telemetry doc. I'm also changing the testpilot-metrics API to not have any required keys in the sendEvent method, for better backwards compatibility with existing experiments.

Once I've made those changes, I'll plan to land this PR.

jaredhirsch · 2017-02-09T21:02:05Z

planning to land this later today

jaredhirsch · 2017-02-10T23:59:36Z

OK, let's land this!

clouserw added the needs discussion label Oct 21, 2016

clouserw self-assigned this Oct 21, 2016

ckprice reviewed Oct 22, 2016

View reviewed changes

chuckharmston reviewed Oct 26, 2016

View reviewed changes

clouserw force-pushed the tga branch from 43f508f to b86fb18 Compare October 27, 2016 23:16

ghost reviewed Oct 27, 2016

View reviewed changes

chuckharmston reviewed Oct 28, 2016

View reviewed changes

chuckharmston reviewed Nov 1, 2016

View reviewed changes

clouserw force-pushed the tga branch from b86fb18 to 2699dd9 Compare November 2, 2016 02:20

ghost mentioned this pull request Nov 3, 2016

Implement metrics broker #1735

Closed

jaredhirsch reviewed Nov 7, 2016

View reviewed changes

lmorchard reviewed Nov 15, 2016

View reviewed changes

jaredhirsch reviewed Nov 28, 2016

View reviewed changes

ghost closed this Dec 21, 2016

ghost reopened this Dec 21, 2016

ghost added the status: in progress label Dec 21, 2016

jaredhirsch mentioned this pull request Feb 6, 2017

Update testpilot schema, remove testpilottest schema mozilla/ping-centre#36

Merged

clouserw and others added 2 commits February 10, 2017 15:36

add metrics overview

2cb73cc

Add testpilot-metrics details to metrics doc

895e799

jaredhirsch force-pushed the tga branch from 2d65256 to 895e799 Compare February 10, 2017 23:57

jaredhirsch merged commit df44bd0 into mozilla:master Feb 10, 2017

jaredhirsch removed the status: in progress label Feb 10, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add metrics overview #1662

add metrics overview #1662

clouserw commented Oct 21, 2016

ckprice Oct 22, 2016

chuckharmston commented Oct 24, 2016

chuckharmston Oct 26, 2016

chuckharmston Oct 26, 2016

chuckharmston Oct 26, 2016

chuckharmston Oct 26, 2016

chuckharmston Oct 26, 2016

chuckharmston Oct 26, 2016

chuckharmston Oct 26, 2016

ghost commented Oct 27, 2016

ghost Oct 27, 2016

chuckharmston Oct 28, 2016

chuckharmston Nov 1, 2016

jaredhirsch Nov 7, 2016

lmorchard Nov 15, 2016 •

edited

jaredhirsch Nov 28, 2016

jaredhirsch Nov 28, 2016

jaredhirsch commented Nov 28, 2016

ckprice commented Jan 25, 2017

jaredhirsch commented Jan 26, 2017

jaredhirsch commented Jan 26, 2017

jaredhirsch commented Jan 27, 2017

jaredhirsch commented Feb 9, 2017

jaredhirsch commented Feb 10, 2017

add metrics overview #1662

add metrics overview #1662

Conversation

clouserw commented Oct 21, 2016

Choose a reason for hiding this comment

chuckharmston commented Oct 24, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost commented Oct 27, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lmorchard Nov 15, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jaredhirsch commented Nov 28, 2016

ckprice commented Jan 25, 2017

jaredhirsch commented Jan 26, 2017

jaredhirsch commented Jan 26, 2017

jaredhirsch commented Jan 27, 2017

jaredhirsch commented Feb 9, 2017

jaredhirsch commented Feb 10, 2017

lmorchard Nov 15, 2016 •

edited