[Epic] Boomerang

[juliemccracken]

All issues for the Skyline boomerang work would be opened under this epic for tracking purposes.

PRD:
https://docs.google.com/document/d/1fqyXTcAfJ8HPaCsEtML_bfIItlYYtG5yk32ZRaOXorY/edit#

┆Issue is synchronized with this Jira Epic
┆Issue Number: FXA-196

[davismtl]

• Targetting train 144
• Working on server routes and email.
• We've done the planning for the A/B testing.
• Usertesting.com sessions have been done. Ready for analysis. \o/

[davismtl]

• Goes out as A/B test next week for registrations
• Will be on by default for proxy
• September 23rd for the Sign in flows.

[davismtl]

Here is a dashboard to track performance with registrations:
https://analytics.amplitude.com/mozilla-corp/dashboard/rzomsbr

Why did the experiment only start on Friday the 13th?

[vbudhram]

Why did the experiment only start on Friday the 13th?

There was a fix for amplitude metrics that went out that day 👻

[davismtl]

Numbers are still showing a 22% increase in registrations. There is a slight decrease in retention.

@vbudhram is looking to turn this up to 100% as of next week since this seems like a win.

We will continue to keep an eye on retention/

[shane-tomlinson]

Numbers are still showing a 22% increase in registrations. There is a slight decrease in retention.

WOOOOW! Fantastic work @vbudhram, @ryanfeeley, and everyone else who pushed on this!

[davismtl]

Tagged for train 146. Should go out today.
100% for registrations.

Going to 50% for sign-ins.

[davismtl]

For the registration experiment, we need to log results here:
https://docs.google.com/document/d/1fqyXTcAfJ8HPaCsEtML_bfIItlYYtG5yk32ZRaOXorY/edit#heading=h.cf325ti2rkj4

[davismtl]

Since the release, we seem to be losing a lot of multi-device users.

What could we do to recover this lost knowing that we prefer this flow for many reasons?

https://analytics.amplitude.com/mozilla-corp/dashboard/q2nfqqt

[davismtl]

Not rolled out at 100% ???
#2847

[vbudhram]

#2847 has been fixed and is in train-148

[davismtl]

This bug with analysis might be showing that the mobile drop has nothing to do with codes since it also impacted CAD on logins which had no codes in train 146.

#3059

[davismtl]

• Log results to feature doc and next steps:
  https://docs.google.com/document/d/1fqyXTcAfJ8HPaCsEtML_bfIItlYYtG5yk32ZRaOXorY/edit#
• We still need to investigate the change that happened a week later with train 147.
• Leif and Kimmy are meeting this afternoon to prepare recommendations.
• From there, we should graduate the experiment and remove the experiment logic

[davismtl]

We should have an analysis to discuss next week.

[davismtl]

Analysis from Kimmy:

https://docs.google.com/document/d/15LyHLE3e4BnQxspTKEtRw7olQPRHSgdAe3s_z9W_uvc/edit#heading=h.gjolbth6cmmx

[irrationalagent]

Logged results for registration and conversion to mobile in the feature doc. Will add results for login after I review Kimmy's analysis.

[davismtl]

Moved card to remove A/B logic.
Leif, Kimmy or Alex should walk through results in the next product meeting.