Skip to content
This repository has been archived by the owner on Feb 20, 2023. It is now read-only.

[Telemetry] Onboarding #10824

Closed
1 of 7 tasks
vesta0 opened this issue May 21, 2020 · 6 comments
Closed
1 of 7 tasks

[Telemetry] Onboarding #10824

vesta0 opened this issue May 21, 2020 · 6 comments
Assignees
@eliserichards
Labels
E3 Estimation Point: average, 2 - 3 days Feature:Telemetry P1 Current sprint

Comments

@vesta0
Copy link
Collaborator

vesta0 commented May 21, 2020 •
edited by data-sync-user

Description & Product Manager / Data Scientist User Story

We'd like to know if users find onboarding cards useful in helping them customize their experience and knowing what preferences are available to them.

What questions will you answer with this data?

-What percentage of users interact with onboarding cards by changing a toggle or tapping on a link (any of them)?
-What percentage of users interact with each individual card (which cards resonate more with users and make them want to explore/take action/change anything?)
-What percentage of users who change a preference (e.g. ETP) do it during onboarding vs in settings later on?
-Are users who interact with onboarding more likely to come back in the 7 days following install?

Acceptance Criteria

  • ENG files a DS JIRA request outlining their methodology.
  • DS sign off on instrumentation methodology addressing product questions.
  • Event pings can be queried via re:dash
  • Event pings can be queried via amplitude
  • We are sending telemetry events for the actions listed in the requirements
  • We have documented the telemetry
  • We have asked a data steward to review the telemetry

┆Issue is synchronized with this Jira Task
@github-actions github-actions bot added the needs:triage Issue needs triage label May 21, 2020
@vesta0 vesta0 added this to Backlog in Feature engagement May 21, 2020
@vesta0 vesta0 added this to Prioritized Backlog in Fenix Sprint Kanban May 21, 2020
@vesta0 vesta0 moved this from Prioritized Backlog to Inbox in Fenix Sprint Kanban May 21, 2020
@vesta0
Copy link
Collaborator Author

vesta0 commented May 21, 2020

Please note down your methodology here and create a DS bug (see acceptance criteria) for DS review.

@liuche liuche added the size S label May 21, 2020
@liuche liuche moved this from Inbox to Sprint in Fenix Sprint Kanban May 21, 2020
@vesta0 vesta0 moved this from Backlog to Waiting for DS in Feature engagement May 21, 2020
@sblatz sblatz self-assigned this May 22, 2020
@liuche liuche added E2 Estimation Point: easy, half a day to 2 days E3 Estimation Point: average, 2 - 3 days and removed E2 Estimation Point: easy, half a day to 2 days labels May 22, 2020
@sblatz sblatz removed their assignment May 22, 2020
@ekager ekager removed the needs:triage Issue needs triage label May 22, 2020
@vesta0 vesta0 added the P1 Current sprint label May 29, 2020
@vesta0 vesta0 moved this from Waiting for DS to Backlog in Feature engagement Jun 3, 2020
@sblatz
Copy link
Contributor

sblatz commented Jun 3, 2020

JIRA request with high priority (P1) filed!

@sblatz sblatz moved this from Sprint to In Progress in Fenix Sprint Kanban Jun 3, 2020
@sblatz sblatz added the 🙅 waiting Issues that are blocked or has dependencies that are not ready label Jun 3, 2020
@liuche
Copy link
Contributor

liuche commented Jun 4, 2020

@vesta0 to bump DS priority

@sblatz sblatz moved this from Backlog to Waiting for DS in Feature engagement Jun 9, 2020
@sblatz sblatz removed 🙅 waiting Issues that are blocked or has dependencies that are not ready needs:data-science labels Jun 15, 2020
@sblatz sblatz moved this from Waiting for DS to Backlog in Feature engagement Jun 16, 2020
@sblatz sblatz moved this from Backlog to Waiting for DS in Feature engagement Jun 16, 2020
@sblatz sblatz moved this from Waiting for DS to Eng Ready in Feature engagement Jun 16, 2020
@liuche liuche moved this from In Progress to Sprint in Fenix Sprint Kanban Jun 22, 2020
@eliserichards
Copy link
Contributor

eliserichards commented Jun 23, 2020 •
edited

categories:

  • OnboardingSectionHeader (used for "Get to know {appName}")
  • OnboardingManualSignIn
  • OnboardingAutomaticSignIn
  • OnboardingThemePicker
    • dark, light, automatic (follow device settings)
  • OnboardingTrackingProtection
  • OnboardingPrivateBrowsing
  • OnboardingPrivacyNotice
  • OnboardingFinish
  • OnboardingToolbarPositionPicker
  • OnboardingWhatsNew

@eliserichards eliserichards mentioned this issue Jun 23, 2020
Merged
13 tasks
@sblatz sblatz moved this from Sprint to In Dev Review in Fenix Sprint Kanban Jun 25, 2020
eliserichards added a commit that referenced this issue Jun 25, 2020
@eliserichards
#10824: Onboarding telemetry (#11867)
665fb8d 
* Onboarding events and tracking

* Rename metrics for consistency

* Add reference to PR for the data review

* Add default values to onboarding events
@eliserichards eliserichards moved this from In Dev Review to Ready for QA in Fenix Sprint Kanban Jun 25, 2020
@eliserichards eliserichards added the eng:qa:needed QA Needed label Jun 25, 2020
@sblatz sblatz moved this from Eng Ready to In progress in Feature engagement Jun 25, 2020
@liuche liuche mentioned this issue Jun 27, 2020
Closed
12 tasks
@AndiAJ
Copy link
Collaborator

AndiAJ commented Jun 30, 2020

Hi, I've just checked this matter on both Nightly Build 200629 from 6/30 and Fenix 79.0.1-beta.1 from 6/30 using a Google Pixel 3a (Android 10)

The pings are generated for each onboarding item ✔️
On the other hand, I've noticed that the following:

  1. The automatic sign in card isn't displayed on Fenix 79.0.1-beta.1 @liuche is this expected ❓
  2. Some events seem to be duplicated ❓ @eliserichards could you please review and advise. ☺️

• Tracking protection set to Strict

"events": [
        {
          "category": "onboarding",
          "extra": {
            "setting": "STRICT"
          },
          "name": "pref_toggled_tracking_prot",
          "timestamp": 0
        },
        {
          "category": "onboarding",
          "extra": {
            "setting": "STRICT"
          },
          "name": "pref_toggled_tracking_prot",
          "timestamp": 0
        }

• Tracking protection set to Strict

"events": [
       {
         "category": "onboarding",
         "extra": {
           "setting": "STANDARD"
         },
         "name": "pref_toggled_tracking_prot",
         "timestamp": 0
       },
       {
         "category": "onboarding",
         "extra": {
           "setting": "STANDARD"
         },
         "name": "pref_toggled_tracking_prot",
         "timestamp": 0
       }

• Theme picker onboarding card set to Light

"events": [
        {
          "category": "onboarding",
          "extra": {
            "theme": "LIGHT"
          },
          "name": "pref_toggled_theme_picker",
          "timestamp": 0
        },
        {
          "category": "onboarding",
          "extra": {
            "theme": "LIGHT"
          },
          "name": "pref_toggled_theme_picker",
          "timestamp": 3
        }

• Theme picker onboarding card set to Dark

"events": [
        {
          "category": "onboarding",
          "extra": {
            "theme": "DARK"
          },
          "name": "pref_toggled_theme_picker",
          "timestamp": 0
        },
        {
          "category": "onboarding",
          "extra": {
            "theme": "DARK"
          },
          "name": "pref_toggled_theme_picker",
          "timestamp": 3
        }

• Theme picker onboarding card set to Automatic

"events": [
        {
          "category": "onboarding",
          "extra": {
            "theme": "FOLLOW_DEVICE"
          },
          "name": "pref_toggled_theme_picker",
          "timestamp": 0
        },
        {
          "category": "onboarding",
          "extra": {
            "theme": "FOLLOW_DEVICE"
          },
          "name": "pref_toggled_theme_picker",
          "timestamp": 4
        }

• Toolbar position preference set to Top

"events": [
        {
          "category": "onboarding",
          "extra": {
            "position": "TOP"
          },
          "name": "pref_toggled_toolbar_position",
          "timestamp": 0
        },
        {
          "category": "onboarding",
          "extra": {
            "position": "TOP"
          },
          "name": "pref_toggled_toolbar_position",
          "timestamp": 3
        }

• Toolbar position preference set to Bottom

  "events": [
        {
          "category": "onboarding",
          "extra": {
            "position": "BOTTOM"
          },
          "name": "pref_toggled_toolbar_position",
          "timestamp": 0
        },
        {
          "category": "onboarding",
          "extra": {
            "position": "BOTTOM"
          },
          "name": "pref_toggled_toolbar_position",
          "timestamp": 4
        }

✔️ Nightly Build 200630 from 6/30

► Normal Browsing

Action Status Ping
The onboarding manual sign in card was tapped ✔️ 7abb71be-be62-404f-9707-e2e6c10c6e19
The onboarding What's New card was tapped ✔️ 1612ad9b-4254-4967-b5bf-c77a28928b7f
The Strict tracking protection preference was chosen from the onboarding card ✔️ 4c08a1f1-c8f3-40b6-a80e-425ebcb13161
The Standard tracking protection preference was chosen from the onboarding card ✔️ ca202e58-9973-4839-8b1d-7f51f84e98fa
The Light device theme was chosen using the theme picker onboarding card ✔️ 23d05306-8b11-4e90-bb69-49da1dc4bcc1
The Dark device theme was chosen using the theme picker onboarding card ✔️ dc628b93-be63-4b14-9c5d-b2d651533d3a
The Automatic device theme was chosen using the theme picker onboarding card ✔️ 3b7ede78-e0aa-4784-8585-553aa6a07fbc
The private browsing preference was selected from the onboarding card ✔️ 5c0e87ce-39af-4551-a028-4c77ab01bc23
The Top toolbar position preference was chosen from the onboarding card ✔️ 6ee56f1f-7783-46de-a09d-99a96a0bff11
The Bottom toolbar position preference was chosen from the onboarding card ✔️ d444434a-6443-4050-a0c1-df0ad517de60
The onboarding privacy notice card was tapped ✔️ 328e5d02-2bad-47f9-97e1-ca9226495ad3
The user taps starts browsing and ends the onboarding experience ✔️ 0e16725b-0f34-401e-9d82-daacc96538b2

Logcat
Glean dashboard

► Private Browsing

Action Status Ping
The onboarding automatic sign in card was tapped. ✔️ e70d21ea-9a4c-47ae-a887-8c499ef05739
The onboarding What's New card was tapped ✔️ 1d4bdd44-6f20-44db-913c-8e34b0ccb803
The Strict tracking protection preference was chosen from the onboarding card ✔️ 6e7f079e-9bba-471b-bac7-d27a0ae5523d
The Standard tracking protection preference was chosen from the onboarding card ✔️ 083a235e-89c3-4ca0-bda9-9a8e46aa1fb5
The Light device theme was chosen using the theme picker onboarding card ✔️ bcc3cf28-9e4a-4615-9c3f-ad3773a7c52d
The Dark device theme was chosen using the theme picker onboarding card ✔️ 662e001d-c920-4b51-9a03-e02d2865b1ba
The Automatic device theme was chosen using the theme picker onboarding card ✔️ 10bad87f-f5c6-4e20-8a00-06a921dc5da1
The private browsing preference was selected from the onboarding card ✔️ d09286b5-f7e9-48cf-be54-4277c33a7e68
The Top toolbar position preference was chosen from the onboarding card ✔️ 28453b77-4efc-4736-a56b-ccc04497e25e
The Bottom toolbar position preference was chosen from the onboarding card ✔️ e0a40b6a-4353-48b3-ad56-320d798c83cc
The onboarding privacy notice card was tapped ✔️ c706daa2-2692-4681-8836-758768c9ecd5
The user taps starts browsing and ends the onboarding experience ✔️ cf9107f9-a845-4b18-ac8b-26b9879aff07

Logcat
Glean dashboard

✔️ Fenix 79.0.1-beta.1 from 6/30

► Normal Browsing

Action Status Ping
The onboarding manual sign in card was tapped ✔️ 8b3ef25f-3821-493a-9f48-14deeec19a84
The onboarding What's New card was tapped ✔️ 0c9c5097-c516-4e20-9979-72f4f1e1b5c5
The Strict tracking protection preference was chosen from the onboarding card ✔️ 516fec72-3f70-4962-9d99-f705deb96de1
The Standard tracking protection preference was chosen from the onboarding card ✔️ e1144f4e-f034-42b2-a759-69e302fa1d81
The Light device theme was chosen using the theme picker onboarding card ✔️ 7d232b9f-b41f-45d2-8ebd-2b45bddbd367
The Dark device theme was chosen using the theme picker onboarding card ✔️ 02d3799e-44c2-484d-bfcc-f57ad2dcfabe
The Automatic device theme was chosen using the theme picker onboarding card ✔️ 719adc02-a4a9-484f-b277-2d8cb9c74cb3
The private browsing preference was selected from the onboarding card ✔️ 36e40854-12ed-4746-bd1a-37d505575a72
The Top toolbar position preference was chosen from the onboarding card ✔️ 8cae41f5-a73b-4955-a333-d75483ea2929
The Bottom toolbar position preference was chosen from the onboarding card ✔️ 915806d2-0eba-44f1-8a01-9b2a81c6ab58
The onboarding privacy notice card was tapped ✔️ b5b94494-5926-4539-9b30-73a9aea1c154
The user taps starts browsing and ends the onboarding experience ✔️ 922774c1-5402-4efb-a1d8-7655ff4fa955

Logcat
Glean dashboard

► Private Browsing

Action Status Ping
The onboarding manual sign in card was tapped ✔️ d6ba0484-a2b9-4724-a800-a25ce6f9df74
The onboarding What's New card was tapped ✔️ 9ac92ffb-5e84-4516-98ee-903c4eea555d
The Strict tracking protection preference was chosen from the onboarding card ✔️ 5e2dee35-7b3f-48b2-ac64-4c19ab959021
The Standard tracking protection preference was chosen from the onboarding card ✔️ 69ee190e-a711-4384-b491-87fc534c2ea6
The Light device theme was chosen using the theme picker onboarding card ✔️ e5815904-12ac-428a-adca-ca81834e1da1
The Dark device theme was chosen using the theme picker onboarding card ✔️ 3b7346e7-35db-4641-a83f-7b8f3779b863
The Automatic device theme was chosen using the theme picker onboarding card ✔️ 606e615f-d188-4ce4-b02b-942fe4e93c15
The private browsing preference was selected from the onboarding card ✔️ ec26fb74-2504-4354-9090-b91e20fb489e
The Top toolbar position preference was chosen from the onboarding card ✔️ 2f6fea29-f241-4225-9186-b76b0478d72c
The Bottom toolbar position preference was chosen from the onboarding card ✔️ d8204e36-7af2-44a4-8b93-03b405480d1c
The onboarding privacy notice card was tapped ✔️ 8f94b69a-c71b-469f-b12c-9250e0fe7dbc
The user taps starts browsing and ends the onboarding experience ✔️ 20c68b2c-3987-493e-b5a6-6f70830b099d

Logcat
Glean dashboard

@AndiAJ AndiAJ removed the eng:qa:needed QA Needed label Jun 30, 2020
@eliserichards
Copy link
Contributor

This is ok for now. We will file followups as needed. Thanks @AndiAJ!

Fenix Sprint Kanban automation moved this from Ready for QA to Sprint 20.11 Done Jul 21, 2020
@eliserichards eliserichards removed this from Sprint 20.11 Done in Fenix Sprint Kanban Jul 21, 2020
@eliserichards eliserichards removed this from In progress in Feature engagement Jul 21, 2020
@data-sync-user data-sync-user changed the title [Telemetry] Onboarding FNX-4604 ⁃ [Telemetry] Onboarding Aug 10, 2020
@data-sync-user data-sync-user changed the title FNX-4604 ⁃ [Telemetry] Onboarding FNX3-14185 ⁃ [Telemetry] Onboarding Aug 10, 2020
@data-sync-user data-sync-user changed the title FNX3-14185 ⁃ [Telemetry] Onboarding FNX-4604 ⁃ [Telemetry] Onboarding Aug 11, 2020
@data-sync-user data-sync-user changed the title FNX-4604 ⁃ [Telemetry] Onboarding FNX2-13271 ⁃ [Telemetry] Onboarding Aug 11, 2020
@data-sync-user data-sync-user changed the title FNX2-13271 ⁃ [Telemetry] Onboarding [Telemetry] Onboarding May 18, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@eliserichards eliserichards

Labels
E3 Estimation Point: average, 2 - 3 days Feature:Telemetry P1 Current sprint
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@liuche @sblatz @ekager @vesta0 @eliserichards @AndiAJ @data-sync-user