#10824: Onboarding telemetry

[eliserichards]

For #10824

Events:

• OnboardingManualSignIn
• OnboardingAutomaticSignIn
• OnboardingThemePicker
  • dark, light, automatic (follow device settings)
  • default: automatic
• OnboardingTrackingProtection
  • strict, standard
  • default: ON, standard
• OnboardingPrivateBrowsing
• OnboardingPrivacyNotice
• OnboardingFinish
• OnboardingToolbarPositionPicker
  • top, bottom
  • default: bottom
• OnboardingWhatsNew

This PR is covering:

• User toggled any of the onboarding toggles

Already exists in preference metrics

• Track final position of these toggles through our metrics ping

Pull Request checklist

• Tests: This PR includes thorough tests or an explanation of why it does not
• Screenshots: This PR includes screenshots or GIFs of the changes made or an explanation of why it does not
• Accessibility: The code in this PR follows accessibility best practices or does not include any user facing features. In addition, it includes a screenshot of a successful accessibility scan to ensure no new defects are added to the product.

After merge

• Milestone: Make sure issues finished by this pull request are added to the milestone of the version currently in development.

To download an APK when reviewing a PR:

1. click on Show All Checks,
2. click Details next to "Taskcluster (pull_request)" after it appears and then finishes with a green checkmark,
3. click on the "Fenix - assemble" task, then click "Run Artifacts".
4. the APK links should be on the left side of the screen, named for each CPU architecture

[eliserichards]

This ordering got all messed up. I pulled preference_toggled down into it's own section (per discussion with @sblatz and Marissa) and nested all onboarding clicks within that

[codecov-commenter]

Codecov Report

Merging #11867 into master will decrease coverage by 0.07%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             master   #11867      +/-   ##
============================================
- Coverage     22.11%   22.04%   -0.08%     
+ Complexity      727      726       -1     
============================================
  Files           377      377              
  Lines         15121    15153      +32     
  Branches       1961     1965       +4     
============================================
- Hits           3344     3340       -4     
- Misses        11491    11527      +36     
  Partials        286      286              

Impacted Files	Coverage Δ	Complexity Δ
...la/fenix/components/metrics/GleanMetricsService.kt	13.77% <0.00%> (-0.68%)	4.00 <0.00> (ø)
...va/org/mozilla/fenix/components/metrics/Metrics.kt	19.77% <0.00%> (-0.71%)	0.00 <0.00> (ø)
.../onboarding/OnboardingAutomaticSignInViewHolder.kt	0.00% <0.00%> (ø)	0.00 <0.00> (ø)
...ewholders/onboarding/OnboardingFinishViewHolder.kt	0.00% <0.00%> (ø)	0.00 <0.00> (ø)
...ers/onboarding/OnboardingManualSignInViewHolder.kt	0.00% <0.00%> (ø)	0.00 <0.00> (ø)
...rs/onboarding/OnboardingPrivacyNoticeViewHolder.kt	0.00% <0.00%> (ø)	0.00 <0.00> (ø)
.../onboarding/OnboardingPrivateBrowsingViewHolder.kt	0.00% <0.00%> (ø)	0.00 <0.00> (ø)
...ders/onboarding/OnboardingThemePickerViewHolder.kt	0.00% <0.00%> (ø)	0.00 <0.00> (ø)
...rding/OnboardingToolbarPositionPickerViewHolder.kt	0.00% <0.00%> (ø)	0.00 <0.00> (ø)
...boarding/OnboardingTrackingProtectionViewHolder.kt	0.00% <0.00%> (ø)	0.00 <0.00> (ø)
... and 10 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c958cc0...6a82259. Read the comment docs.

[liuche]

@eliserichards can you include a data review form please?

[eliserichards]

Request for data collection review form

All questions are mandatory. You must receive review from a data steward peer on your responses to these questions before shipping new data collection.

1. What questions will you answer with this data?

• What percentage of users interact with onboarding cards by changing a toggle or tapping on a link (any of them)?
• What percentage of users interact with each individual card (which cards resonate more with users and make them want to explore/take action/change anything?)
• What percentage of users who change a preference (e.g. ETP) do it during onboarding vs in settings later on?
• Are users who interact with onboarding more likely to come back in the 7 days following install?

2. Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements?

• We would like to better understand the onboarding flow in order to streamline the experience for users, and to ensure that the initial interaction with the app is useful and doesn't drive people away.

3. What alternative methods did you consider to answer these questions? Why were they not sufficient?

• There is not currently an alternative to measure this.

4. Can current instrumentation answer these questions?

• No, there are no currently metrics around the onboarding actions.

5. List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories on the found on the Mozilla wiki.

• All data is Category 2.

6. How long will this data be collected?

• Until 09/01/2020

7. What populations will you measure?

• All release, beta, and nightly users with telemetry enabled.

8. Please provide a general description of how you will analyze this data.

• Glean / Amplitude

9. Where do you intend to share the results of your analysis?

• Only on Glean, Amplitude and with mobile teams.

[sblatz]

Looks good overall to me, just a couple comments!

[sblatz]

These all look great from an eng standpoint :)

[liuche]

Data Review Form (to be filled by Data Stewards)

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, interaction data with onboarding is in metrics.md

2. Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, included under Fenix data settings

3. If the request is for permanent data collection, is there someone who will monitor the data over time?

Has expiry

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Type 2, user interaction with onboarding cards

5. Is the data collection request for default-on or default-off?

Default on

6. Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No, interaction data only

7. Is the data collection covered by the existing Firefox privacy notice?

yes

8. Does there need to be a check-in in the future to determine whether to renew the data?

No, has expiry

9. Does the data collection use a third-party collection tool?

no

[liuche]

Does this automatically trigger the PB mode being saved to the metrics ping for private browsing setting? https://github.com/mozilla-mobile/fenix/blob/master/docs/metrics.md#metrics-1

[eliserichards]

Yes it does!