for #11830 added new metric for collecting startup method from all startup phases #11940

sraturi · 2020-06-25T07:37:14Z

Pull Request checklist

Tests: This PR includes thorough tests or an explanation of why it does not
Screenshots: This PR includes screenshots or GIFs of the changes made or an explanation of why it does not
Accessibility: The code in this PR follows accessibility best practices or does not include any user facing features. In addition, it includes a screenshot of a successful accessibility scan to ensure no new defects are added to the product.

After merge

Milestone: Make sure issues finished by this pull request are added to the milestone of the version currently in development.

To download an APK when reviewing a PR:

click on Show All Checks,
click Details next to "Taskcluster (pull_request)" after it appears and then finishes with a green checkmark,
click on the "Fenix - assemble" task, then click "Run Artifacts".
the APK links should be on the left side of the screen, named for each CPU architecture

sraturi · 2020-06-25T07:55:49Z

Test Flows

Start app from app icon (cold startup) -> press home button ->press the app icon again(warm startup)
Start app from app icon (cold startup) -> press home button ->open Gmail link [open to browser option](warm startup)
- Recording telemetry inside onStart was using the HomeActivty's existing intent to get the startup information. For example, on cold startup, it would report APP_ICON and than on warm startup(Gmail link) it would again report APP_ICON. ~~Therefore, had to record the telemetry inside the onNewIntent() to report the correct startup method, LINK in this case.~~
open Gmail link[in app browser](cold startup)-> press home button -> open twitter link [in app browser] (cold startup)
- This always begins with a cold startup... I'm assuming because Each time, we create a new ExternalAppBrowserActivity instance in the respective in-app processes?
open Gmail link [open to browser option](cold startup) -> press home button ->open twitter link [open to browser option](warm startup)

mcomella

I'm not sure this is 100% accurate to what we're looking for.

app/metrics.yaml

app/src/main/java/org/mozilla/fenix/HomeActivity.kt

mcomella · 2020-06-25T20:16:52Z

app/src/main/java/org/mozilla/fenix/HomeActivity.kt

@@ -229,6 +234,16 @@ open class HomeActivity : LocaleAwareAppCompatActivity() {
        val intentHandled = intentProcessors.any { it.process(intent, navHost.navController, this.intent) }
        browsingModeManager.mode = getModeFromIntentOrLastKnown(intent)

+        if (settings().isTelemetryEnabled) {
+            /* If there is a warm or hot startup, onNewIntent method is always called first.


One issue I see with using onNewIntent directly is that it will trigger if the app receives an intent while it's open. I assume this can happen when clicking notifications in the notification tray.

I wonder if we can take this new intent, set it in a variable, and use it in onStart (I don't know what the lifecycle is though). This may not be worth the effort though (I tend to be too much of a perfectionist – call me out for it!). If making this work gets too complicated, we may want to land something like this sooner and document the known caveats given it probably doesn't happen that often so the data can still be useful.

I added a new class AllSourceStartupTelemetry to handle the logic of triggering recording our startup telemetry.

mcomella · 2020-06-25T20:18:03Z

app/src/main/java/org/mozilla/fenix/HomeActivity.kt

+        return when {
+            intent.isLauncherIntent -> Event.OpenedAppAllStart.Source.APP_ICON
+            intent.action == Intent.ACTION_VIEW -> Event.OpenedAppAllStart.Source.LINK
+            else -> null


I feel like we should record "UNKNOWN" in the else case.

Will update it!

@mcomella if we record unknown, should we at least store the value of the "UNKNOWN" so that we have some insight on what those different starts were in order to adapt the code afterwards?

app/src/main/java/org/mozilla/fenix/components/metrics/Metrics.kt

app/src/main/java/org/mozilla/fenix/HomeActivity.kt

mcomella · 2020-06-25T21:02:35Z

app/src/main/java/org/mozilla/fenix/HomeActivity.kt

@@ -308,6 +323,15 @@ open class HomeActivity : LocaleAwareAppCompatActivity() {
        }
    }

+    @VisibleForTesting(otherwise = PROTECTED)


nit: I don't see this being used for testing: shall we make this method protected and skip the annotation?

Ah yeah, most of the functions was copied from the existing code for "app_opened" metric. I was wondering if unit testing needs to be part of this PR?

app/src/main/java/org/mozilla/fenix/HomeActivity.kt

sraturi · 2020-06-25T22:48:15Z

Request for data collection review form

All questions are mandatory. You must receive review from a data steward peer on your responses to these questions before shipping new data collection.

1. What questions will you answer with this data?
How do users open the app (custom tab, app_icon, link, unknown) most frequently on all startups (cold, warm, hot)

2. Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:
Creating accurate understanding of real world usage to prioritize performance improvements

3. What alternative methods did you consider to answer these questions? Why were they not sufficient?
There is an existing app_opened metrics, However it is only used for cold startup.

4. Can current instrumentation answer these questions?
no

List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories found on the Mozilla wiki.
data is Interaction data.How the user opens app , options -> custom tab, app_icon, link

6. Please provide a link to the documentation for this data collection which describes the ultimate data set in a public, complete, and accurate way.
This is documented in the metrics.md file

7. How long will this data be collected?
Until 2020-12-01

8. What populations will you measure?
All release, beta, and nightly users with telemetry enabled.

9. If this data collection is default on, what is the opt-out mechanism for users?
default on, opt out by Settings ->Data collection

10. Please provide a general description of how you will analyze this data.
analyze so we can set up better tests and prioritize analyses to improve performance in the most impactful areas.

11. Where do you intend to share the results of your analysis?
Glean, Amplitude and with mobile teams.

12. Is there a third-party tool (i.e. not Telemetry) that you are proposing to use for this data collection? If so:
N/A

@liuche

liuche

data-review+ only, nit on clearer documentation

Data Review Form (to be filled by Data Stewards)

Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, in metrics.md

Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, through telemetry controls in Fenix

If the request is for permanent data collection, is there someone who will monitor the data over time?
Expires in 12/2020
Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Yes, type 2, when home activity gets opened by user

Is the data collection request for default-on or default-off?

default on

Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

no

Is the data collection covered by the existing Firefox privacy notice?

yes

Does there need to be a check-in in the future to determine whether to renew the data? (Yes/No) (If yes, set a todo reminder or file a bug if appropriate)**

has expiry

Does the data collection use a third-party collection tool? If yes, escalate to legal.

no

liuche · 2020-06-26T16:46:03Z

app/metrics.yaml

+    bugs:
+      - https://github.com/mozilla-mobile/fenix/issues/11830
+    data_reviews:
+      - NA


Link here to the PR where you filed the data review request (which will be this issue).

liuche · 2020-06-26T16:49:48Z

app/metrics.yaml

+    type: event
+    description: |
+      A user opened the app from any one of the following flows:
+      hot, warm or cold start to the HomeActivity.


Could you give an example of what each of these mean in this description?

This description is used by anyone who might be using this telemetry, which could include data science, and they may not have a clear idea of what the difference is between each item (fwiw I'm not really clear if hot startup means just switching to HomeActivity from any other activity, or if it means something else!)

mcomella

Code looks okay assuming the assumptions are correct (onStart is before onNewIntent): here are the nits while I test it out.

mcomella · 2020-06-26T21:37:15Z

app/metrics.yaml

+      source:
+        description: |
+          The method used to open Fenix. Possible values are `app_icon`,
+          `custom_tab` or `link` or `unknown`


nit: custom_tab, link, or unknown

app/metrics.yaml

mcomella · 2020-06-26T21:48:43Z

app/metrics.yaml

+    type: event
+    description: |
+      A user opened the app from any one of the following flows:
+      hot, warm or cold start to the HomeActivity.


We may want to elaborate what "HomeActivity" represents – as liuche mentions, this can be used be anyone. I'd say something like, "to the HomeActivity. The HomeActivity encompasses the home screen and browser screen but may include other screens."

mcomella · 2020-06-26T21:49:10Z

app/src/main/java/org/mozilla/fenix/HomeActivity.kt

+
+    override fun onStart() {
+        super.onStart()
+        allSourceStartupTelemetry.onStart()


nit: -> ...onStartHomeActivity or similar. We want to disambiguate what this call is onStart of.

mcomella · 2020-06-26T21:50:12Z

app/src/main/java/org/mozilla/fenix/components/metrics/AllSourceStartupTelemetry.kt

+ * For example, Start app through APP_ICON -> run adb intent command to open a link.
+ * We want to ignore these adb commands since app is already on foreground.
+ */
+open class AllSourceStartupTelemetry(private var currentIntent: Intent?, val metrics: MetricController) {


nit: don't include open, we're not overriding this anywhere

mcomella · 2020-06-26T21:50:54Z

app/src/main/java/org/mozilla/fenix/components/metrics/AllSourceStartupTelemetry.kt

+ * For example, Start app through APP_ICON -> run adb intent command to open a link.
+ * We want to ignore these adb commands since app is already on foreground.
+ */
+open class AllSourceStartupTelemetry(private var currentIntent: Intent?, val metrics: MetricController) {


nit: private val metrics

mcomella · 2020-06-26T21:51:04Z

app/src/main/java/org/mozilla/fenix/HomeActivity.kt

@@ -188,6 +190,16 @@ open class HomeActivity : LocaleAwareAppCompatActivity() {
            intent.removeExtra(START_IN_RECENTS_SCREEN)
            moveTaskToBack(true)
        }
+        allSourceStartupTelemetry =


nit: newline above

mcomella · 2020-06-26T21:51:21Z

app/src/main/java/org/mozilla/fenix/HomeActivity.kt

@@ -245,6 +257,7 @@ open class HomeActivity : LocaleAwareAppCompatActivity() {
                ?.let { it as? TabTrayDialogFragment }
                ?.also { it.dismissAllowingStateLoss() }
        }
+        allSourceStartupTelemetry.onNewIntent(intent)


nit: newline above

mcomella · 2020-06-26T21:54:54Z

app/src/main/java/org/mozilla/fenix/components/metrics/AllSourceStartupTelemetry.kt

+        currentIntent = intent
+    }
+
+    open fun getIntentAllStartSource(intent: SafeIntent): Event.OpenedAppAllStart.Source? {


nit: not open, private

app/src/main/java/org/mozilla/fenix/components/metrics/AllSourceStartupTelemetry.kt

mcomella

Testing locally on my P2, onNewIntent gets called after onStart so I don't think the current code will work

mcomella

Code looks good - submitting nits while I test.

app/src/main/java/org/mozilla/fenix/HomeActivity.kt

mcomella · 2020-06-26T23:49:08Z

app/metrics.yaml

+  app_opened_all_startup:
+    type: event
+    description: |
+      A user opened the app to the HomeActivity. The HomeActivity


"The system received an Intent for the HomeActivity. An intent is received an external entity wants to the app to display content. Intents can be received when the app is closed – at which point the app will be opened – or when the app is already opened – at which point the already open app will make changes such as loading a url. This can be used loosely as a heuristic for when the user requested to open the app. The HomeActivity encompasses..."

Or something like that. We need to make it clear this is not "user opened the app" anymore.

Honestly, I would put all those details below what was initially written. Let's get the simple version first so it's easy for someone to understand the high-level, and then move all the details afterwards.

@sraturi Can you address this in the follow-up? It should be okay to change the description

app/metrics.yaml

app/src/main/java/org/mozilla/fenix/HomeActivity.kt

mcomella · 2020-06-26T23:51:58Z

We should get another data review but let's not block the PR on that.

for mozilla-mobile#11830 collecting startup metric inaside onNewIntent in HomeActivity clean up, added comments nitpicks cleanup move all source startup telemetry into its own logic and added an UNKOWN state nit cleanup minor changes added comment metric description nit cleanup

nit changes

mcomella

The intuitive behaviors I tried seem to be working correctly:

Open app
Press home, open app
App link when app is closed (adb & gmail)
app link when app is opened (adb)
custom tab when app is closed (fastmail)
force-stop app and try some of ^

If I wasn't so tired I'd probably try harder to break this but this lgtm for now after the nits are addressed (particular the event name and the docs).

mcomella

lgtm, let's land as is and follow-up for the nits.

app/src/main/java/org/mozilla/fenix/components/metrics/Metrics.kt

sraturi requested review from mcomella and MarcLeclair June 25, 2020 07:55

sraturi linked an issue Jun 25, 2020 that may be closed by this pull request

[Telemetry] Add telemetry to record how the app was opened on cold/warm/hot startup(home screen, applink, custom tab) #11830

Closed

7 tasks

mcomella suggested changes Jun 25, 2020

View reviewed changes

sraturi requested a review from liuche June 25, 2020 23:37

sraturi added the needs:data-review PR is awaiting a data review label Jun 26, 2020

sraturi force-pushed the telemetry/#11830 branch 3 times, most recently from a75705e to 2a88a87 Compare June 26, 2020 20:14

sraturi requested a review from mcomella June 26, 2020 20:16

sraturi force-pushed the telemetry/#11830 branch 2 times, most recently from ce96afa to 557f996 Compare June 26, 2020 20:49

liuche approved these changes Jun 26, 2020

View reviewed changes

mcomella reviewed Jun 26, 2020

View reviewed changes

mcomella suggested changes Jun 26, 2020

View reviewed changes

sraturi force-pushed the telemetry/#11830 branch from 557f996 to fb00580 Compare June 26, 2020 23:32

mcomella reviewed Jun 26, 2020

View reviewed changes

sraturi added 2 commits June 26, 2020 17:05

switched back to onNewIntent solution

78f6553

nit changes

mcomella approved these changes Jun 27, 2020

View reviewed changes

renamed the metric

3ac67e9

sraturi force-pushed the telemetry/#11830 branch from fb00580 to 3ac67e9 Compare June 27, 2020 00:15

mcomella approved these changes Jun 27, 2020

View reviewed changes

app/src/main/java/org/mozilla/fenix/components/metrics/Metrics.kt Show resolved Hide resolved

sraturi merged commit 4b9cc95 into mozilla-mobile:master Jun 27, 2020

liuche mentioned this pull request Jun 27, 2020

Releng 78.0.1/79.0.0 #12048

Closed

12 tasks

sraturi mentioned this pull request Jul 2, 2020

Telemetry 11830, track how app was opened on all startup #12114

Merged

4 tasks

mdboom mentioned this pull request Aug 10, 2020

Data review on #11940 is missing data collection category #13454

Closed

liuche mentioned this pull request Aug 21, 2020

For #12046 - Renew telemetry probes before migration completion #13958

Merged

4 tasks

mcarare mentioned this pull request Feb 24, 2021

For #18076: Renew telemetry expiring in April. #18143

Merged

3 tasks

eliserichards mentioned this pull request Jun 14, 2021

For #19923: renew telemetry probes #19924

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

for #11830 added new metric for collecting startup method from all startup phases #11940

for #11830 added new metric for collecting startup method from all startup phases #11940

sraturi commented Jun 25, 2020

sraturi commented Jun 25, 2020 •

edited

Loading

mcomella left a comment

mcomella Jun 25, 2020

sraturi Jun 26, 2020

mcomella Jun 25, 2020

sraturi Jun 26, 2020

MarcLeclair Jun 26, 2020

mcomella Jun 25, 2020

sraturi Jun 25, 2020

sraturi commented Jun 25, 2020 •

edited

Loading

liuche left a comment •

edited

Loading

liuche Jun 26, 2020

sraturi Jun 26, 2020

liuche Jun 26, 2020

sraturi Jun 26, 2020

mcomella left a comment

mcomella Jun 26, 2020

sraturi Jun 26, 2020

mcomella Jun 26, 2020

sraturi Jun 26, 2020

mcomella Jun 26, 2020

mcomella Jun 26, 2020

mcomella Jun 26, 2020

mcomella Jun 26, 2020

mcomella Jun 26, 2020

mcomella Jun 26, 2020

mcomella left a comment

mcomella left a comment

mcomella Jun 26, 2020

liuche Jun 27, 2020

mcomella Jun 27, 2020

mcomella commented Jun 26, 2020

mcomella left a comment

mcomella left a comment

for #11830 added new metric for collecting startup method from all startup phases #11940

for #11830 added new metric for collecting startup method from all startup phases #11940

Conversation

sraturi commented Jun 25, 2020

Pull Request checklist

After merge

To download an APK when reviewing a PR:

sraturi commented Jun 25, 2020 • edited Loading

Test Flows

mcomella left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sraturi commented Jun 25, 2020 • edited Loading

Request for data collection review form

liuche left a comment • edited Loading

Choose a reason for hiding this comment

Data Review Form (to be filled by Data Stewards)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mcomella left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mcomella left a comment

Choose a reason for hiding this comment

mcomella left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mcomella commented Jun 26, 2020

mcomella left a comment

Choose a reason for hiding this comment

mcomella left a comment

Choose a reason for hiding this comment

sraturi commented Jun 25, 2020 •

edited

Loading

sraturi commented Jun 25, 2020 •

edited

Loading

liuche left a comment •

edited

Loading