MR2 Telemetry: Opening screen preference

[MozillaNoah]

#22145

Pull Request checklist

• Tests: This PR includes thorough tests or an explanation of why it does not
• Screenshots: This PR includes screenshots or GIFs of the changes made or an explanation of why it does not
• Accessibility: The code in this PR follows accessibility best practices or does not include any user facing features. In addition, it includes a screenshot of a successful accessibility scan to ensure no new defects are added to the product.

To download an APK when reviewing a PR:

1. click on Show All Checks,
2. click Details next to "Taskcluster (pull_request)" after it appears and then finishes with a green checkmark,
3. click on the "Fenix - assemble" task, then click "Run Artifacts".
4. the APK links should be on the left side of the screen, named for each CPU architecture

[MozillaNoah]

Request for data collection review form

All questions are mandatory. You must receive review from a data steward peer on your responses to these questions before shipping new data collection.

1. What questions will you answer with this data?

• This data will help us understand how users interact with the homescreen.

2. Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements?

• This will help us understand what screen users prefer to be presented with during a new session.


3. What alternative methods did you consider to answer these questions? Why were they not sufficient?

• There are no other alternatives.


4. Can current instrumentation answer these questions?

• No.


5. List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories found on the Mozilla wiki.

   Note that the data steward reviewing your request will characterize your data collection based on the highest (and most sensitive) category.

Measurement Description	Data Collection Category	Tracking Bug #
What opening screen preference do users have?	Category 2 - interaction data	#22145

6. Please provide a link to the documentation for this data collection which describes the ultimate data set in a public, complete, and accurate way.

• https://dictionary.telemetry.mozilla.org/apps/fenix


7. How long will this data be collected? Choose one of the following:

• One year (ending on November 1st, 2022), with the option to renew at that point.


8. What populations will you measure?

• All channels, all locales, all countries


9. If this data collection is default on, what is the opt-out mechanism for users?

• Default Glean SDK opt-out mechanism.


10. Please provide a general description of how you will analyze this data.

• Glean and Amplitude.


11. Where do you intend to share the results of your analysis?

• 
Only on Glean, Amplitude, and with mobile teams.


12. Is there a third-party tool (i.e. not Telemetry) that you are proposing to use for this data collection?

• No third-party tools.

[MozillaNoah]

I was unsure what's going on here with the Nimbus code, so I chose to put this particular metric outside of it. Is this location alright, or should it be moved inside the experiment run block?

[jonalmeida]

This observer waits for Nimbus (our experimentation framework) to finish it's async startup before we send back telemetry results which may be based an experiment group the user is in.

The questions worth asking in that case are: do these telemetry probes you're collection dependant on an experiment, are the options visible/invisible, etc.

[jonalmeida]

If the answer is, they don't depend on an experiment. They should be fine there, although I'd leave that as a comment in the code because it would definitely look odd to the next person coming by and wondering the same thing.

[MozillaNoah]

Yeah, that's where my uncertainly is mainly coming from. I'm primarily unsure whether the experiment code here shows/hides stuff (and maybe is the experiment even still running), in which case I'd need to put my new probe with the existing ones.

[jonalmeida]

(and maybe is the experiment even still running)

It's harder to determine future state but we can check the current state - so let's do that!

I've elaborated my investigation below if that helps you in the future on how to find the answers you're looking for. 🙂

Let's look at all the references to alwaysOpenTheHomepageWhenOpeningTheApp, alwaysOpenTheLastTabWhenOpeningTheApp, openHomepageAfterFourHoursOfInactivity: they all seem to come from Settings.shouldStartOnHome and that is only used on the HomeActivity.

In that block itself we see a feature flag, FeatureFlags.showStartOnHomeSettings which is defaulted to on. Looking at other references to the HomeActivity method there, I see none that suspect they are related to feature flags or nimbus.

So let's look at the FeatureFlags.showStartOnHomeSettings: with a git blame on the file, we can see that we flipped the flag true as part of our 94 release. With 94 now in release, we typically remove the feature flag since we rarely have a need to keep it around for much longer.

This tells me that we should be fine with collecting the preference where you currently have, or even after Nimbus is done loading because the value isn't dependant on it.

[MozillaNoah]

This makes sense. Thank you so much for taking the time to explain your thought process of your exploration and for the additional context about the retirement of feature flags.

[jonalmeida]

eng r+

(please see the comment below though!)

[jonalmeida]

(and maybe is the experiment even still running)

It's harder to determine future state but we can check the current state - so let's do that!

I've elaborated my investigation below if that helps you in the future on how to find the answers you're looking for. 🙂

Let's look at all the references to alwaysOpenTheHomepageWhenOpeningTheApp, alwaysOpenTheLastTabWhenOpeningTheApp, openHomepageAfterFourHoursOfInactivity: they all seem to come from Settings.shouldStartOnHome and that is only used on the HomeActivity.

In that block itself we see a feature flag, FeatureFlags.showStartOnHomeSettings which is defaulted to on. Looking at other references to the HomeActivity method there, I see none that suspect they are related to feature flags or nimbus.

So let's look at the FeatureFlags.showStartOnHomeSettings: with a git blame on the file, we can see that we flipped the flag true as part of our 94 release. With 94 now in release, we typically remove the feature flag since we rarely have a need to keep it around for much longer.

This tells me that we should be fine with collecting the preference where you currently have, or even after Nimbus is done loading because the value isn't dependant on it.

[rocketsroger]

Request for data collection review form

All questions are mandatory. You must receive review from a data steward peer on your responses to these questions before shipping new data collection.

1. What questions will you answer with this data?


* This data will help us understand how users interact with the homescreen.


1. Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements?


* This will help us understand what screen users prefer to be presented with during a new session.


1. What alternative methods did you consider to answer these questions? Why were they not sufficient?


* There are no other alternatives.


1. Can current instrumentation answer these questions?


* No.


1. List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories found on the Mozilla wiki.
   
   Note that the data steward reviewing your request will characterize your data collection based on the highest (and most sensitive) category.

Measurement Description Data Collection Category Tracking Bug #
What opening screen preference do users have? Category 2 - interaction data #22145

1. Please provide a link to the documentation for this data collection which describes the ultimate data set in a public, complete, and accurate way.


* [https://dictionary.telemetry.mozilla.org/apps/fenix
  ](https://dictionary.telemetry.mozilla.org/apps/fenix%E2%80%A8)


1. How long will this data be collected? Choose one of the following:


* One year (ending on November 1st, 2022), with the option to renew at that point.


1. What populations will you measure?


* All channels, all locales, all countries


1. If this data collection is default on, what is the opt-out mechanism for users?


* Default Glean SDK opt-out mechanism.


1. Please provide a general description of how you will analyze this data.


* Glean and Amplitude.


1. Where do you intend to share the results of your analysis?


* Only on Glean, Amplitude, and with mobile teams.


1. Is there a third-party tool (i.e. not Telemetry) that you are proposing to use for this data collection?


* No third-party tools.

Data Review

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, through the metrics.yaml file and the Glean Dictionary

2. Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, through the "Send Usage Data" preference in the application settings

3. If the request is for permanent data collection, is there someone who will monitor the data over time?

N/A, collection set to end or be renewed by 2022-11-01

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction data

5. Is the data collection request for default-on or default-off?

default-on

6. Does the instrumentation include the addition of any new identifiers?

No

7. Is the data collection covered by the existing Firefox privacy notice?

Yes

8. Does the data collection use a third-party collection tool?

No

Result

data-review+