Skip to content
This repository has been archived by the owner. It is now read-only.

revise metrics.md for 54 launch #2452

Merged
merged 1 commit into from Mar 27, 2017
Merged

Conversation

@clouserw
Copy link
Collaborator

@clouserw clouserw commented Mar 21, 2017

r? @johngruen and anyone else

@ianb I noticed a bunch of checkboxes are struck through. Should I just whack those lines?


### Opt-out
0. Metrics to tell us if Firefox Screenshots is affecting Firefox. For instance, do Firefox Screenshots users engage longer with Firefox?
Copy link
Contributor

@ianb ianb Mar 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We did have a talk with Brendan Colloran about this, and he was pretty confident that we couldn't find any meaningful way to understand this, except through controlled experiment. That is, Firefox Screenshots users are going to use Firefox more, because people who use any feature are more active than the general population.

The one option – which I'd like us to consider at some point – is to actually do a controlled experiment by leaving some set of users with Screenshots disabled. And maybe we could see some differences for the brief period when we are rolling out Screenshots (but probably not).

Copy link
Contributor

@wresuolc wresuolc Mar 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've been talking about that. It sounds like you don't need to leave the set of users with the feature disabled for a super long time to get a decent result (longer is better, of course, but hey...). If our rollout lasts a couple weeks, that's already useful data. We might be able to keep a small portion out longer, but that schedule isn't finalized.

docs/METRICS.md Outdated

This information is intended to help us make Firefox Screenshots better.
No metrics data is sent to any servers until a user interacts with Page Shot.
Copy link
Contributor

@ianb ianb Mar 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, "Page Shot"


### Opt-out

The add-on reads the Telemetry opt-out preference (`toolkit.telemetry.enabled`) before sending any metrics data to the servers. If this preference is false, or if there is any issue trying to fetch the preference, then no data is sent.
Copy link
Contributor

@ianb ianb Mar 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could be more specific that the add-on checks the value of that preference each time you click the Screenshots button.

Copy link
Contributor

@wresuolc wresuolc Mar 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought there might be other entry points. Like, if a person hits the keyboard combo or if we eventually provide some other link (a my shots button?)

Copy link
Contributor

@ianb ianb Mar 24, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's slightly ad hoc, but at each entry point we try to run analytics.refreshTelemetryPref() (it ended up this way because each entry point also has its own event, so we want to refresh the pref before sending the first event). But yes, we try to do it at the beginning of each session.

docs/METRICS.md Outdated
The add-on does not communicate directly with GA, instead it POSTs an event to the Firefox Screenshots server which, in turn, sends the event to GA. This model allows us to only send the data we need for analysis instead of the more comprehensive data collection that google-analytics.js does. Besides the event information specified below we also send the User-Agent string, and the add-on provides:
* Any A/B tests the user is in, or if the user is in the control for a test
* The application (“firefox”)
* The Page Shot add-on version (“6.0.0”)
Copy link

@ckprice ckprice Mar 27, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Page Shot"

@ckprice
Copy link

@ckprice ckprice commented Mar 27, 2017

Do we need anything about Sentry in here?

@wresuolc
Copy link
Contributor

@wresuolc wresuolc commented Mar 27, 2017

None of the other metrics docs talk about sentry, so I don't think so

@wresuolc
Copy link
Contributor

@wresuolc wresuolc commented Mar 27, 2017

thanks

@wresuolc wresuolc merged commit 27608ed into mozilla-services:master Mar 27, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants