Skip to content

Update documentation for v2 of error_aggregates #132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Parent: Adding in documentation for newly added user states fields, in additi…
merged 1 commit into from
May 14, 2018
Merged

Update documentation for v2 of error_aggregates #132

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 16 additions & 15 deletions datasets/streaming/error_aggregates/reference.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ SELECT window_start,
sum(gpu_crashes) AS gpu,
sum(plugin_crashes) AS plugin,
sum(gmplugin_crashes) AS gmplugin
FROM telemetry.error_aggregates
FROM telemetry.error_aggregates_v2
WHERE application = 'Firefox'
AND (os_name = 'Darwin' or os_name = 'Linux' or os_name = 'Windows_NT')
AND (channel = 'beta' or channel = 'release' or channel = 'nightly' or channel = 'esr')
Expand All @@ -40,7 +40,7 @@ Get the number of `main_crashes` on Windows over a small interval

```sql
SELECT window_start as time, sum(main_crashes) AS main_crashes
FROM telemetry.error_aggregates_v1
FROM telemetry.error_aggregates_v2
WHERE application = 'Firefox'
AND os_name = 'Windows_NT'
AND channel = 'release'
Expand All @@ -54,45 +54,46 @@ GROUP BY window_start

### Data sources

The aggregates in this data source are derived from main and crash [pings](../../pings.md).
Crash pings are only used to count/gather main crash events, all other errors (including all other crashes) are gathered from main pings.
The aggregates in this data source are derived from main, crash and core [pings](../../pings.md):
* crash pings are used to count/gather main and content crash events, all other errors from desktop clients (including all other crashes) are gathered from main pings
* core pings are used to count usage hours, first subsession and unique client counts.

## Scheduling

The `error_aggregates` job is run continuously, using the Spark Streaming infrastructure

## Schema

The `error_aggregates` table has the following columns which define its dimensions:
The `error_aggregates_v2` table has the following columns which define its dimensions:

* `window_start`: Beginning of interval when this sample was taken
* `window_end`: End of interval when this sample was taken (will always be 5 minutes more
* `window_start`: beginning of interval when this sample was taken
* `window_end`: end of interval when this sample was taken (will always be 5 minutes more
than `window_start` for any given row)
* `submission_date_s3`: the date pings were submitted for a particular aggregate
* `channel`: the channel, like `release` or `beta`
* `version`: the version e.g. `57.0.1`
* `display_version`: like version, but includes beta number if applicable e.g. `57.0.1b4`
* `build_id`: the `YYYYMMDDhhmmss` timestamp the program was built, like `20160123180541`. This is also known as the `build ID` or `buildid`
* `application`: application name (e.g. `Firefox`)
* `application`: application name (e.g. `Firefox` or `Fennec`)
* `os_name`: name of the OS (e.g. `Darwin` or `Windows_NT`)
* `os_version`: version of the OS
* `architecture`: build architecture, e.g. `x86`
* `country`: country code for the user (determined using geoIP), like `US` or `UK`
* `experiment_id`: identifier of the experiment being participated in, such as `e10s-beta46-noapz@experiments.mozilla.org`, or null if no experiment
* `experiment_branch`: the branch of the experiment being participated in, such as `control` or `experiment`, or null if no experiment
* `e10s_enabled`: whether E10s is enabled
* `gfx_compositor`: the graphics backend compositor used by the program, such as `d3d11`, `opengl` and `simple`. Null values may be reported as `none` as well
* `quantum_ready`: whether the ping was submitted from an instance of Firefox that was Quantum-ready (i.e. no old-style addons)

And these are the various measures we are counting:

* `usage_hours`: number of usage hours (i.e. total number of session hours reported by the pings in this aggregate, note that this might include time where
people are not actively using the browser or their computer is asleep)
* `count`: number of pings processed in this aggregate
* `main_crashes`: number of main process crashes (or just program crashes, in the non-e10s case)
* `content_crashes`: number of content process crashes
* `startup_crashes` : number of startup crashes
* `content_crashes`: number of content process crashes (`version => 58` only)
* `gpu_crashes`: number of GPU process crashes
* `plugin_crashes`: number of plugin process crashes
* `gmplugin_crashes`: number of Gecko media plugin (often abbreviated `GMPlugin`) process crashes
* `content_shutdown_crashes`: number of content process crashes that were caused by failure to shut down in a timely manner (should always be
less than the number of `content_crashes`)
* `gpu_crashes`: number of GPU process crashes
* `content_shutdown_crashes`: number of content process crashes that were caused by failure to shut down in a timely manner (`version => 58` only)
* `browser_shim_usage_blocked`: number of times a CPOW shim was blocked from being created by browser code
* `permissions_sql_corrupted`: number of times the permissions SQL error occurred (beta/nightly only)
* `defective_permissions_sql_removed`: number of times there was a removal of defective `permissions.sqlite` (beta/nightly only)
Expand Down