Engagement notification for inactive users

[onemahon]

User Story

We'd like to use a local notification to encourage new users to re-engage with the app after an absence of usage as a way to try to convert them into regularly active users.

Acceptance Criteria

• only show this notification for new users
• schedule notification for 48 hours after first use
• don't show the notification if the user winds up doing any other activity in the app after first use
• don't show the notification if Firefox is set as the default browser

Telemetry

• notification triggered
• notification clicked on
• whether or not the app was granted permissions to send notifications

┆Issue is synchronized with this Jira Task

[onemahon]

Just created this ticket from some ongoing product conversations. I also wanted to ask a few questions about the possible approach here:

Questions

• schedule notification for 48 hours after first use

1. When, precisely, should the notification occur? Do we have a strict preference? E.g., should it be 48 hours after the user's first activity? Or should it be "some time on the day 2 days after first use"?

• don't show the notification if the user winds up doing any other activity in the app after first use

2. Do we have a clear understanding of when "first use" ends? E.g., does activity 10 minutes after first use count? 1 hr? Anytime within the first day of use?

• don't show the notification if Firefox is set as the default browser

3. Is there any way to cancel the notification in the edge case where a user sets Firefox to their default browser without re-opening the app? (I'd guess not, and that's probably fine from a product perspective, but worth double-checking)

[rocketsroger]

Clarification of these requirements:

1. When, precisely, should the notification occur? Do we have a strict preference? E.g., should it be 48 hours after the user's first activity? Or should it be "some time on the day 2 days after first use"?

This notification should occur 48 hours after Firefox for Android first started.

2. Do we have a clear understanding of when "first use" _ends_? E.g., does activity 10 minutes after first use count? 1 hr? Anytime within the first day of use?

First use here means user have no interaction after 24 hours counting from the time Firefox for Android first started.

3. Is there any way to cancel the notification in the edge case where a user sets Firefox to their default browser _without_ re-opening the app? (I'd guess not, and that's probably fine from a product perspective, but worth double-checking)

Default browser check will be done right before showing the notification. If Firefox is their default browser at that moment, we do not show the notification.

[jonalmeida]

Consider also that first use might happen at 1am and it might not be a great experience to send a notification at the same time on a different day.

We should also do some testing to ensure this doesn't consider install time as first use. Auto-updates happen at night again, so 48 hours after that would be distributive to send a notification.

It's not in the acceptance criteria, but shouldn't we have a Nimbus feature flag to disable this if we find a bug once this has already been released in order to limit affected users?

[rocketsroger]

Consider also that first use might happen at 1am and it might not be a great experience to send a notification at the same time on a different day.

We should also do some testing to ensure this doesn't consider install time as first use. Auto-updates happen at night again, so 48 hours after that would be distributive to send a notification.

This notification will only show up for a new user. No current user should get this notification. Auto-update should not trigger the scheduling of the re-engagement notification.

It's not in the acceptance criteria, but shouldn't we have a Nimbus feature flag to disable this if we find a bug once this has already been released in order to limit affected users?

We do have another issue that tracks the Nimbus flag. However, I'll be implementing that at the same time as this issue.

[AdinaPetridean]

Tested on the latest Nightly 109.0a1 (2022-12-07) with Motorola Moto G9 plus (Android 11).
After 48 hours of inactivity, the notification is triggered only when the app is brought to the foreground (when it is active).

@rocketsroger can you confirm this is the expected behaviour?

[delia-pop]

Upon further testing I managed to receive the notification correctly following specific STR, after 48h of inactivity on Nightly 109.0a1 from 12/08.

The following STR were used:
Make sure the Notifications are enabled;

1. With a fresh install/clear data, open the app. Note the time when the app is opened (e.g. 11:06 a.m.).
2. Close the app.
3. Go to Settings > Date & Time.
4. Set the Time to the same value in Step 1 (11:06 a.m.).
5. Set the Date with 48h ahead.
6. Wait for a minute if needed, then observe is the notification is received.

Observe the attached video for more information:

Correctly_Received_App_Closed.mp4

Please note the following inconsistencies observe during testing:

1. The notification is not received automatically if in the STR above instead of closing the app in Step 2, the app is sent in background. In this case, the user has to open the app in order for the notification to be sent (please observe the video below), as mentioned in the comment above:

Apps_Opened.mp4

2. On multiple occasions, when all products are installed on the device, the "Set to default browser" notifications for Nightly, Beta and RC would also not be automatically received (the user has to open the app in order to trigger them).

Regarding telemetry, the events for the engagement notification were correctly captured:

Devices used:

• Google Pixel 6 (Android 13)
• Xiaomi 12 Pro (Android 12)
• Lenovo Yoga Tab 11 (Android 11)
• Xiaomi Redmi Note 8T (Android 9)

Although the testing results seem to be depending on the timing and not on the devices used, the happy flow was only reproduced with Google Pixel 6 (Android 13) and the application closed. We will also test this by leaving the app inactive for 48h without changing the Time and Date from Settings.

@onemahon , @rocketsroger if you have other proposal on how can be test this differently, please let us know.

[onemahon]

Thanks @delia-pop and @AdinaPetridean!

I'm guessing the reason the notification doesn't arrive is because the app is still running in the background - even though the clock is set forward by two days. I think this edge case is likely acceptable (very few phones, if any, will maintain the app actively running in the background for 48 hours).

I'll create a follow-up ticket for this case, but I'd vote we defer that fix until the next version. @channingcarter are you OK with that plan? (Note: if so, we should be vigilant looking at telemetry for unexpected data or behavior, just in case I'm wrong about the scale of impact. Not sure if this is possible, but in an ideal world, it would be helpful to have a chart of notification pings according to the amount of time between the re_engagement_notif_shown timestamp and, if it exists, a timestamp of the first time the user opens the app. We'd want to see them all showing up around the same time - 48 hours. We'd also want to display experiment-targeted users who never actually receive a re_engagement_notif_shown ping at all).

@delia-pop as for this note:

On multiple occasions, when all products are installed on the device, the "Set to default browser" notifications for Nightly, Beta and RC would also not be automatically received (the user has to open the app in order to trigger them).

It sounds like you're describing a new issue for the "set to default" notification, separate from the "try private browsing" notification - that's a little worrying, if so (just because I'm not sure why that would be affected). When you say "on multiple occasions" - was that rare, or frequent in your testing?

[channingcarter]

I'm fine with that plan to defer the fix until 110 @onemahon .

[delia-pop]

Thank you, @onemahon for your input and for reporting the edge case.
Regarding the "Set as default browser" notification, I re-tested the notifications on the latest Nightly 110 , as well on Beta 109.0b2 and Release 108.1.0 again, and couldn't reproduce the inconsistencies initially reported. I only noticed that, similar to the engagement notification, the "Set to default browser" one is only received after opening the application foreground.

I will close this ticket as Verified Fixed, but we will further test the notifications to make sure they work correctly.

[rocketsroger]

@delia-pop to answer your question. From my testing, when modifying system time, the app (Fenix in this case) has to be forced closed for it to work. If the app is not forced closed, I have to wait the full delay time even with system time modified. I don't think this is a bug but a difference in how Android OS determines when the delay time expires when the app is running vs app is closed.

With the way we're currently using system time to test the notification, Fenix will have to be forced closed for that to work. However, I did test (with a shorter delay) that without force closing the app, the notification will still show up.

[rocketsroger]

@AdinaPetridean, It's not expected that we have to open the app for the notification to show. Workmanager takes different considerations when determining what's the best time to perform the task (In this case showing the notification). It's not very precise. In this case when you opened Fenix, it determined that it's the best time to perform the task since the app is running.

I would suggest waiting 1 to 2 hours past 48 hours to make sure the notification shows up without user interaction with Fenix. Thanks