Skip to content
This repository has been archived by the owner on Apr 3, 2019. It is now read-only.

Commit

Permalink
fix(docs): overhaul the metrics events documentation
Browse files Browse the repository at this point in the history 
#1901

r=vbudhram
  • Loading branch information
@philbooth
philbooth committed May 18, 2017
1 parent 643ed85 commit 2d5943c
Showing 1 changed file with 187 additions and 97 deletions.
284 changes: 187 additions & 97 deletions docs/metrics-events.md
View file
Expand Up @@ -15,6 +15,26 @@ metrics queries in redash:
or state changes
at the account level.

* [Flow events](#flow-events)
* [`flow_metadata`](#flow_metadata)
* [`flow_events`](#flow_events)
* [`flow_experiments`](#flow_experiments)
* [Success event names](#success-event-names)
* [View names](#view-names)
* [Action names](#action-names)
* [Experiment names](#experiment-names)
* [Template names](#template-names)
* [Error event names](#error-event-names)
* [Activity events](#activity-events)
* [`activity_events`](#activity_events)
* [`daily_activity_per_device`](#daily_activity_per_device)
* [`daily_multi_device_users`](#daily_multi_device_users)
* [Event names](#event-names)
* [Sampled data sets](#sampled-data-sets)
* [Significant changes](#significant-changes)

## Flow events

Flow events are used
to plot charts for
sign-in and sign-up user funnels.
Expand All @@ -23,92 +43,27 @@ follow individual user journeys
for the length of each flow,
even when those journeys
span over multiple devices or browsers.
They also power the chart for our
"time taken for device connection" KPI.

Activity events are used
for analysing user behaviour
in a more general way.
They are behind the charts for our
"engagement ratio" and "multi-device usage" KPIs.

## Flow events

The following flow events
represent a successful step
in a sign-in or sign-up flow:

|Name|Description|
|----|-----------|
|`flow.begin`|A user has requested a page that allows them to sign in/up.|
|`flow.${viewName}.view`|A view has rendered.|
|`flow.${viewName}.engage`|A user has interacted with the the form on a page that allows them to sign in/up`.|
|`flow.${viewName}.submit`|A user has submitted the signup form on a page that allows them to sign in/up.|
|`flow.${viewName}.have-account`|A user has clicked on the 'Already have an account?' link.|
|`flow.${viewName}.create-account`|A user has clicked on the 'Create an account' link.|
|`flow.${viewName}.forgot-password`|A user has clicked on the 'Forgot password?' link.|
|`flow.${action}.attempt`|The content server has sent a sign-in/up request to the auth server.|
|`flow.experiment.${experiment}.${group}`|A user has been included in an active experiment.|
|`flow.performance`|`flow_time` for this event indicates the number of milliseconds a user waited until the first view rendered and they were able to interact with the page.|
|`flow.performance.network`|`flow_time` for this event is a number that approximates the relative speed of a user's network performance (lower is faster).|
|`flow.performance.server`|`flow_time` for this event is a number that approximates the relative speed of the server performance (lower is faster).|
|`flow.performance.client`|`flow_time` for this event is a number that approximates the relative speed of a user's client-side performance (lower is faster).|
|`account.login`|An existing account has been signed in to.|
|`account.created`|A new account has been created.|
|`email.confirmation.sent`|A sign-in confirmation email has been sent to a user.|
|`email.verification.sent`|A sign-up verification email has been sent to a user.|
|`email.confirmation.resent`|A sign-in confirmation email has been re-sent to a user.|
|`email.verification.resent`|A sign-up verification email has been re-sent to a user.|
|`email.verify_code.clicked`|A user has clicked on the link in a confirmation/verification email.|
|`email.${templateName}.delivered`|An email was delivered to a user.|
|`sms.region.${region}`|A user has tried to send SMS to `region`.|
|`sms.${templateName}.sent`|An SMS message has been sent to a user's phone.|
|`account.confirmed`|Sign-in to an existing account has been confirmed via email.|
|`account.reminder`|A new account has been verified via a reminder email.|
|`account.verified`|A new account has been verified via email.|
|`account.keyfetch`|Sync encryption keys have been fetched.|
|`account.signed`|A certificate has been signed.|
|`account.reset`|An account has been reset.|
|`account.login.confirmedUnblockCode`|A user has successfully unblocked their account.|
|`account.login.sentUnblockCode`|A sign-in unblock email has been sent to the user.|
|`password.forgot.send_code.start`|A user has initiated the password reset flow.|
|`password.forgot.send_code.completed`|A password reset email has been sent to the user.|
|`password.forgot.resend_code.start`|A user has requested that the password reset email be re-sent.|
|`password.forgot.resend_code.completed`|A password reset email has been re-sent to the user.|
|`password.forgot.verify_code.start`|A user has clicked on the link in a password reset email.|
|`password.forgot.verify_code.completed`|A password reset has been successfully completed on the server.|
|`route.${path}.200`| A route responded with a 200 status code. Example: `route./account/login.200`|
|`flow.complete`|A user has successfully completed a sign-in or sign-up flow.|

The following flow events
represent error conditions,
which may or may not be terminal
to a flow:

|Name|Description|
|----|-----------|
|`customs.blocked`|A request was blocked by the customs server.|
|`route.${path}.${statusCode}.${errno}`| A route responded with a >=400 status code. Includes `errno`. Example: `route./account/login.400.103`|
|`email.${templateName}.bounced`|An email bounced.|

In redshift,
these events are stored
in two tables:
flow event data is stored
in three tables:

* `flow_metadata`,
* [`flow_metadata`](#flow_metadata),
containing all of the data
relating to a flow
as a single entity.

* `flow_events`,
* [`flow_events`](#flow_events),
containing data
for the individual events
within each flow.

* `flow_experiments`,
* [`flow_experiments`](#flow_experiments),
containing data for flows
that are part of a feature experiment.

### flow_metadata

The `flow_metadata` table
contains the following fields:

Expand All @@ -135,6 +90,8 @@ contains the following fields:
|`utm_term`|Marketing campaign search term for the first flow in the session. Not stored if the `DNT` request header was `1`.|
|`export_date`|The date that the `flow.begin` event was exported to S3 by the metrics pipeline.|

### flow_events

The `flow_events` table
contains the following fields:

Expand All @@ -147,6 +104,8 @@ contains the following fields:
|`uid`|The user id. An opaque token, HMACed to avoid correlation back to FxA user db. Not every flow event has a `uid`.|
|`locale`|The user's locale. For cases where we aren't localised in their favoured locale(s), the value will be `en-US.default`|

### flow_experiments

The `flow_experiments` table
contains the following fields:

Expand All @@ -159,30 +118,146 @@ contains the following fields:
|`uid`|The user id. An opaque token, HMACed to avoid correlation back to FxA user db. Not every experiment has a `uid`.|
|`export_date`|The date that the experiment event was exported to S3 by the metrics pipeline.|

## Activity events
### Success event names

The following activity events
are emitted:
The following flow events
represent a successful step
in a sign-in or sign-up flow:

|Name|Description|
|----|-----------|
|`account.created`|A new account has been created.|
|`flow.begin`|A user has requested a page that allows them to sign in/up.|
|`flow.${viewName}.view`|A view has rendered.|
|`flow.${viewName}.engage`|A user has interacted with the the form on a page that allows them to sign in/up`.|
|`flow.${viewName}.submit`|A user has submitted the signup form on a page that allows them to sign in/up.|
|`flow.${viewName}.have-account`|A user has clicked on the 'Already have an account?' link.|
|`flow.${viewName}.create-account`|A user has clicked on the 'Create an account' link.|
|`flow.${viewName}.forgot-password`|A user has clicked on the 'Forgot password?' link.|
|`flow.${action}.attempt`|The content server has sent a sign-in/up request to the auth server.|
|`flow.experiment.${experiment}.${group}`|A user has been included in an active experiment.|
|`flow.performance`|`flow_time` for this event indicates the number of milliseconds a user waited until the first view rendered and they were able to interact with the page.|
|`flow.performance.network`|`flow_time` for this event is a number that approximates the relative speed of a user's network performance (lower is faster).|
|`flow.performance.server`|`flow_time` for this event is a number that approximates the relative speed of the server performance (lower is faster).|
|`flow.performance.client`|`flow_time` for this event is a number that approximates the relative speed of a user's client-side performance (lower is faster).|
|`account.login`|An existing account has been signed in to.|
|`account.verified`|A new account has been verified via email.|
|`account.created`|A new account has been created.|
|`email.confirmation.sent`|A sign-in confirmation email has been sent to a user.|
|`email.verification.sent`|A sign-up verification email has been sent to a user.|
|`email.confirmation.resent`|A sign-in confirmation email has been re-sent to a user.|
|`email.verification.resent`|A sign-up verification email has been re-sent to a user.|
|`email.verify_code.clicked`|A user has clicked on the link in a confirmation/verification email.|
|`email.${templateName}.delivered`|An email was delivered to a user.|
|`sms.region.${region}`|A user has tried to send SMS to `region`.|
|`sms.${templateName}.sent`|An SMS message has been sent to a user's phone.|
|`account.confirmed`|Sign-in to an existing account has been confirmed via email.|
|`account.reminder`|A new account has been verified via a reminder email.|
|`account.verified`|A new account has been verified via email.|
|`account.keyfetch`|Sync encryption keys have been fetched.|
|`account.signed`|A certificate has been signed.|
|`account.reset`|An account has been reset.|
|`account.deleted`|An account has been deleted.|
|`device.created`|A device record has been created for a Sync account.|
|`device.updated`|Device record is updated on a Sync account.|
|`device.deleted`|Device record has been deleted from a Sync account.|
|`sync.sentTabToDevice`|Device sent a push message for send-tab-to-device feature.|
|`account.login.confirmedUnblockCode`|A user has successfully unblocked their account.|
|`account.login.sentUnblockCode`|A sign-in unblock email has been sent to the user.|
|`password.forgot.send_code.start`|A user has initiated the password reset flow.|
|`password.forgot.send_code.completed`|A password reset email has been sent to the user.|
|`password.forgot.resend_code.start`|A user has requested that the password reset email be re-sent.|
|`password.forgot.resend_code.completed`|A password reset email has been re-sent to the user.|
|`password.forgot.verify_code.start`|A user has clicked on the link in a password reset email.|
|`password.forgot.verify_code.completed`|A password reset has been successfully completed on the server.|
|`route.${path}.200`| A route responded with a 200 status code. Example: `route./account/login.200`|
|`flow.complete`|A user has successfully completed a sign-in or sign-up flow.|

In redshift,
these events are stored
in the `activity_events` table
with the following fields:
#### View names

Possible values for `${viewName}` include,
but are not limited to:

View name|Description
---------|-----------
`signup`|The sign-up form
`confirm`|Displayed while awaiting account verification
`signup-confirmed`|The tab the user signed up from, after account verification
`signup-verified`|The tab the user verified their email in, after account verification
`signin`|The sign-in form
`confirm-signin`|Displayed while awaiting sign-in confirmation
`signin-confirmed`|The tab the user signed in from, after sign-in confirmation
`verify-email`|The tab the user confirmed their email in, after sign-in confirmation
`complete-signin`|
`reset-password`|The reset password form
`confirm-reset-password`|
`reset-password-confirmed`|
`reset-password-verified`|
`complete-reset-password`|
`signin-unblock`|The sign-in unblock screen
`choose-what-to-sync`|Choose what to Sync
`connect-another-device`|Connect another device, phase 1
`sms`|Connect another device, phase 2
`cookies-disabled`|Error page shown if local storage or cookies are disabled

#### Action names

Possible values for `${action}` are:

Action name|Description
-----------|-----------
`signup`|Create an account, i.e. `/account/create`
`signin`|Sign in to an existing account, i.e. `/account/login`

#### Experiment names

Possible values for `${experiment}` are:

Experiment name|Description
---------------|-----------
`connectAnotherDevice`|Connect another device, phase 1
`sendSms`|Connect another device, phase 2

#### Template names

Possible values for `${templateName}` are:

Template name|Description
-------------|-----------
`installFirefox`|Firefox app store link
`1`|Historically, there was [a bug] where message ids were in included in the event instead of template names. For that reason, `1` is the pre-train-86 version of `installFirefox`.

### Error event names

The following flow events
represent error conditions,
which may or may not be terminal
to a flow:

|Name|Description|
|----|-----------|
|`customs.blocked`|A request was blocked by the customs server.|
|`route.${path}.${statusCode}.${errno}`| A route responded with a >=400 status code. Example: `route./account/login.400.103`|
|`email.${templateName}.bounced`|An email bounced.|

## Activity events

Activity events are used
for analysing behaviour
at the user level.
The data is stored
in the [`activity_events` table](#activity_events).

Two further tables
summarise daily activity
for Sync-connected devices:

* [`daily_activity_per_device`](#daily_activity_per_device),
contains data about the devices
that are active on each day.

* [`daily_multi_device_users`](#daily_multi_device_users),
contains data for users
with multiple active devices
within a five-day period.

### activity_events

The `activity_events` table
contains the following fields:

|Name|Description|
|----|-----------|
Expand All @@ -195,12 +270,9 @@ with the following fields:
|`ua_version`|The user's browser version.|
|`ua_os`|The user's operating system.|

Two further tables,
summarising device usage,
are populated
based on the activity event data.
### daily_activity_per_device

The table `daily_activity_per_device`
The `daily_activity_per_device` table
contains the following fields:

|Name|Description|
Expand All @@ -213,19 +285,32 @@ contains the following fields:
|`ua_version`|The user's browser version.|
|`ua_os`|The user's operating system.|

The table `daily_multi_device_users`
### daily_multi_device_users

The `daily_multi_device_users` table
contains the following fields:

|Name|Description|
|----|-----------|
|`day`|The date of the activity.|
|`uid`|The HMACed user id.|

For this table,
a multi-device user is defined as
somebody who was also active
on a different device
in the preceding five days.
### Event names

|Name|Description|
|----|-----------|
|`account.created`|A new account has been created.|
|`account.login`|An existing account has been signed in to.|
|`account.verified`|A new account has been verified via email.|
|`account.confirmed`|Sign-in to an existing account has been confirmed via email.|
|`account.keyfetch`|Sync encryption keys have been fetched.|
|`account.signed`|A certificate has been signed.|
|`account.reset`|An account has been reset.|
|`account.deleted`|An account has been deleted.|
|`device.created`|A device record has been created for a Sync account.|
|`device.updated`|Device record is updated on a Sync account.|
|`device.deleted`|Device record has been deleted from a Sync account.|
|`sync.sentTabToDevice`|Device sent a push message for send-tab-to-device feature.|

## Sampled data sets

Expand Down Expand Up @@ -272,6 +357,11 @@ to each of the table names mentioned above:
Previously the message id
was logged in its place](https://github.com/mozilla/fxa-auth-server/pull/1843).

* [The bucketing of users
into experiments was fixed,
drastically changing the number of users
going through our `sendSms` experiment](https://github.com/mozilla/fxa-content-server/pull/4977).

### Train 84

* [The `sms.region.${region}` event
Expand Down

0 comments on commit 2d5943c

Please sign in to comment.