Skip to content

Conversation

@cmonfortep
Copy link
Contributor

@cmonfortep cmonfortep commented Feb 10, 2021

Task/Issue URL: https://app.asana.com/0/488551667048375/1199920127722390/f
Tech Design URL: https://app.asana.com/0/1142021229838617/1199920405912714/f
CC:

Description:
We want to run an experiment to measure the effect of enabling Fireproof login detection by default.

Control group: no changes.

Experimental variant: we've introduced the following product changes:

  • Fireproof login detection enabled by default
  • Allow users to disable login detection if they dismiss twice in a row the Fireproof login prompt dialog
  • Introduce pixels as defined in https://app.asana.com/0/0/1199699295733011/f

Steps to test this PR:

Control group:

  1. Hardcode variant "zq"
  2. Fresh intall
  3. Go to settings and ensure login detection is disabled
  4. Enable login detection
  5. Perform login in two different websites, and dismiss the Fireproof login dialog in all of them
  6. Ensure we never asked you to disable login detection

Experiment Test 1: when Login dialog dismissed twice in a row, ask user to disable login detection

  1. Hardcode variant "zw"
  2. Fresh intall
  3. Go to settings and ensure login detection is enabled
  4. Go back to the browser
  5. Perform login in two different websites, and dismiss the Fireproof login dialog in all of them
  6. Ensure dialog asking to disable login detection shown
  7. Press disable
  8. Go to fireproof settings and ensure login detection is disabled

Experiment Test 2: we ask to disable login detection only once

  1. Hardcode variant "zw"
  2. Fresh intall
  3. Perform login in two different websites, and dismiss the Fireproof login dialog in all of them
  4. Ensure dialog asking to disable login detection shown
  5. Press "Cancel"
  6. Clear all data (this is optional, clear data so we can login on sites again)
  7. Perform login in two different websites, and dismiss the Fireproof login dialog in all of them
  8. Ensure we don't show again the dialog to disable login detection

Experiment Test 3: if user enables login detection, never ask them to disable login detection

  1. Hardcode variant "zw"
  2. Fresh intall
  3. Go to Settings
  4. Ensure login detection is enabled
  5. Disable login detection
  6. Enable it again
  7. Perform login in two different websites, and dismiss the Fireproof login dialog in all of them
  8. Ensure we don't show again the dialog to disable login detection

Experiment Test 4: Validate pixels (user did not try the fire button)

  1. Hardcode variant "zw"
  2. Fresh intall
  3. Perform login on a website
  4. Ensure Fireproof login dialog shown and pixel m_fw_ld_s with fe=false
  5. Dismiss dialog and pixel m_fw_l_d with fe=false
  6. Do login on a second site and dismiss Fireproof login dialog
  7. Ensure dialog asking to disable login detection shown and pixel m_fw_dd_s with fe=false
  8. Dismiss dialog and pixel m_fw_dd_c with fe=false
  9. Do login on a third site and when asked to fireproof website click "Fireproof"
  10. Dialog dismissed and pixel m_fw_l_a with fe=falsesent

Experiment Test 5: Validate pixels (user already tried the fire button)

  1. Hardcode variant "zw"
  2. Fresh intall
  3. Clear all data
  4. Perform login on a website
  5. Ensure Fireproof login dialog shown and pixel m_fw_ld_s with fe=true
  6. Dismiss dialog and pixel m_fw_l_d with fe=true
  7. Do login on a second site and dismiss Fireproof login dialog
  8. Ensure dialog asking to disable login detection shown and pixel m_fw_dd_s with fe=true
  9. Dismiss dialog and pixel m_fw_dd_c with fe=true
  10. Do login on another site and when asked to fireproof website click "Fireproof"
  11. Dialog dismissed and pixel m_fw_l_a with fe=true sent

Internal references:

Software Engineering Expectations
Technical Design Template

@cmonfortep cmonfortep changed the title Feature/cristian/experiment login enabled Experiment with enabling prompted Fireproofing by default Feb 10, 2021
@cmonfortep cmonfortep marked this pull request as ready for review February 12, 2021 10:16
Copy link
Member

@CDRussell CDRussell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All scenarios tested and working well on Pixel 2 running latest Android; functionality as expected and pixels as expected. I'm having problems getting a reliable emulator to test out on Lollipop, so my testing is less thorough there - just FYI.

# Conflicts:
#	app/src/main/java/com/duckduckgo/app/browser/di/BrowserModule.kt
#	app/src/main/res/values/string-untranslated.xml
@cmonfortep cmonfortep merged commit a712b09 into develop Feb 16, 2021
@cmonfortep cmonfortep deleted the feature/cristian/experiment_login_enabled branch February 16, 2021 11:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants