Skip to content
This repository has been archived by the owner on Feb 20, 2023. It is now read-only.

For #6174 - Add telemetry for WebExtensions #8318

Merged
merged 1 commit into from Apr 2, 2020

Conversation

gabrielluong
Copy link
Member

@gabrielluong gabrielluong commented Feb 11, 2020

Fixes #6174. When an user interacts with an installed add-on in the browser menu, we emit a BrowserMenuFact with the add-on id. To cover the installed/removed/enabled/disabled add-ons, we do a metric ping using the list of enabled and installed add-ons that is emitted via WebExtensionFact.

Pull Request checklist

  • Tests: This PR includes thorough tests or an explanation of why it does not
  • Screenshots: This PR includes screenshots or GIFs of the changes made or an explanation of why it does not
  • Accessibility: The code in this PR follows accessibility best practices or does not include any user facing features. In addition, it includes a screenshot of a successful accessibility scan to ensure no new defects are added to the product.

After merge

  • Milestone: Make sure issues finished by this pull request are added to the milestone of the version currently in development.

To download an APK when reviewing a PR:

  1. click on Show All Checks,
  2. click Details next to "Taskcluster (pull_request)" after it appears and then finishes with a green checkmark,
  3. click on the "Fenix - assemble" task, then click "Run Artifacts".
  4. the APK links should be on the left side of the screen, named for each CPU architecture

@gabrielluong gabrielluong added the pr:work-in-progress PRs that are not ready to be reviewed yet and are actively being worked on label Feb 11, 2020
@gabrielluong gabrielluong marked this pull request as ready for review February 27, 2020 21:11
@gabrielluong gabrielluong added needs:data-review PR is awaiting a data review and removed pr:work-in-progress PRs that are not ready to be reviewed yet and are actively being worked on labels Feb 27, 2020
@gabrielluong
Copy link
Member Author

Request for data collection review form

All questions are mandatory. You must receive review from a data steward peer on your responses to these questions before shipping new data collection.

  1. What questions will you answer with this data?
  • How users interact with WebExtensions in Fenix in order to better prioritize related bugs or enhancements
  • How often users look for/check out add-ons in settings? Is this feature interesting to users? How often users check out the available add-ons VS actually install an add-on?
  • How many user have a specific add-on installed, and how many have the add-on installed and enabled, over a specific time period
  • How often users interact with an installed add-on (in the toolbar menu) to look up or modify settings? This will help us learn whether we should collaborate with the add-on developer to invest in improving settings/UI for individual add-ons.
  1. Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements?
  • This will help us prioritize bugs and features related to WebExtensions based on user need.
  1. What alternative methods did you consider to answer these questions? Why were they not sufficient?
  • No current alternative methods.
  1. Can current instrumentation answer these questions?
  • Currently there are no events to track this.
  1. List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories on the found on the Mozilla wiki.
  • All data is Category 2.
  1. How long will this data be collected?
  • Until 09/01/2020
  1. What populations will you measure?
  • All release, beta, and nightly users with telemetry enabled.
  1. Please provide a general description of how you will analyze this data.
  • Glean / Amplitude
  1. Where do you intend to share the results of your analysis?
  • Only on Glean, Amplitude and with mobile teams.

@codecov-io
Copy link

codecov-io commented Feb 27, 2020

Codecov Report

Merging #8318 into master will decrease coverage by 0.43%.
The diff coverage is 10.81%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master    #8318      +/-   ##
============================================
- Coverage     19.36%   18.92%   -0.44%     
+ Complexity      519      490      -29     
============================================
  Files           335      322      -13     
  Lines         13485    12925     -560     
  Branches       1793     1707      -86     
============================================
- Hits           2611     2446     -165     
+ Misses        10650    10275     -375     
+ Partials        224      204      -20
Impacted Files Coverage Δ Complexity Δ
...la/fenix/components/metrics/GleanMetricsService.kt 7.66% <0%> (-1.29%) 3 <0> (ø)
...ava/org/mozilla/fenix/settings/SettingsFragment.kt 0% <0%> (ø) 0 <0> (ø) ⬇️
...va/org/mozilla/fenix/components/metrics/Metrics.kt 18.93% <12.9%> (-1.21%) 0 <0> (ø)
...mozilla/fenix/library/history/HistoryInteractor.kt 0% <0%> (-100%) 0% <0%> (-12%)
...lla/fenix/components/toolbar/DefaultToolbarMenu.kt 0% <0%> (-45.5%) 0% <0%> (-11%)
...in/java/org/mozilla/fenix/AppRequestInterceptor.kt 55.26% <0%> (-17.16%) 7% <0%> (+1%)
...mponents/searchengine/FenixSearchEngineProvider.kt 52.17% <0%> (-11.66%) 10% <0%> (-5%)
.../main/java/org/mozilla/fenix/theme/ThemeManager.kt 0% <0%> (-10.82%) 0% <0%> (ø)
...ix/home/sessioncontrol/SessionControlController.kt 64.16% <0%> (-8.99%) 0% <0%> (ø)
...e/intent/FennecBookmarkShortcutsIntentProcessor.kt 83.33% <0%> (-8.34%) 3% <0%> (-2%)
... and 108 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 44432d2...ca3cdc6. Read the comment docs.

@gabrielluong
Copy link
Member Author

gabrielluong commented Feb 28, 2020

Added a "do not land" for the time being. I intend on switching the the FEATURE_ADDON component flag to SUPPORT_WEBEXTENSIONS in Facts.toEvent().

@gabrielluong
Copy link
Member Author

Ready to be reviewed. Prior "do-not-land" was to correct the component name that was emitting the facts. mozilla-mobile/android-components#6109

Copy link
Contributor

@boek boek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove all identifying (Names and id's) from the data collection for now and file a follow up issue for collecting that? It is going to have to be a bigger discussion

app/metrics.yaml Outdated Show resolved Hide resolved
app/metrics.yaml Outdated Show resolved Hide resolved
app/metrics.yaml Outdated Show resolved Hide resolved
@gabrielluong gabrielluong added the pr:work-in-progress PRs that are not ready to be reviewed yet and are actively being worked on label Mar 3, 2020
@liuche
Copy link
Contributor

liuche commented Mar 27, 2020

Hey @gabrielluong, is this ready for re-review? it still has a WIP label on it. Please ping if it's ready to be reviewed.

app/metrics.yaml Outdated Show resolved Hide resolved
@gabrielluong gabrielluong removed the pr:work-in-progress PRs that are not ready to be reviewed yet and are actively being worked on label Apr 1, 2020
@gabrielluong gabrielluong requested a review from boek April 1, 2020 22:07
Copy link
Contributor

@boek boek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Data Review Form (to be filled by Data Stewards)

  1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?
    Yes, metrics.yaml and metrics.md

  2. Is there a control mechanism that allows the user to turn the data collection on and off?
    Yes, under data controls

  3. If the request is for permanent data collection, is there someone who will monitor the data over time?
    Has expiry

  4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
    Type 2

  5. Is the data collection request for default-on or default-off?
    Default on

  6. Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?
    No

  7. Is the data collection covered by the existing Firefox privacy notice?
    Yes

  8. Does there need to be a check-in in the future to determine whether to renew the data?
    Has expiry

  9. Does the data collection use a third-party collection tool?
    no

@boek boek dismissed grigoryk’s stale review April 2, 2020 17:11

Changes were made

@boek boek merged commit dc6d479 into mozilla-mobile:master Apr 2, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
needs:data-review PR is awaiting a data review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Telemetry] WebExtensions
5 participants