First Run Value Prop CTA

[davismtl]

Test improved first run based on this Feature Doc test recipe:
https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=64619429

[vladikoff]

from mtg: moving to measuring

cc @davismtl

[davismtl]

from mtg:

• stopped test
• re launched it yesterday
• there was an unexplained drop in traffic half way which caused statistical significance problems (might be related to new optimizely tracking)

[vladikoff]

from mtg: got results, one of the results - it might be a good idea to experiment with putting CWTS first in firstrun

[rfk]

@davismtl will wrap up conclusion, learnings and next-steps in the feature card.

[davismtl]

from mtg:

• had meeting with pdol, feeley and verdi
• Key learning and next steps:
  • The biggest drop off occurs through the first step relating to the button
  • We should run a test that adds the descriptive copy next or above the form without the button/extra step
  • Control out-performs our current production. We believe that EN-US out-performs all other languages. We will add locales to telemetry data moving forward.
  • Try different headlines instead of "Almost Done". Pick top winners and then run them through a user test.
  • Would it help users understand the service to put the preferences first? (requires more engineering)
  • Let's test the new funnelcake landing page (EN-US)

[davismtl]

from mtg:

• control is still winning overall (account and md users)
• version B "Complete your experience" is runner up and for account creation and bridges the gap even more when comparing volume of md users
• Alex will review statistical significance this afternoon and stop test if it is good to go.

[davismtl]

from mtg:

• stopped the test
• not likely going to hit statistical significance for a second best (Almost done is still best in terms of numbers)
• Will take best runner ups and UT them
• Need to complete full report in Mana

[davismtl]

from mtg:

• results are logged in the mana
• Probably going with Free as a Fox
• We will observe a slight decrease in registration and multi-device users
• Need to verify impact on rates
• Done, will move to done column