Email deliverability

[rfk]

Our sender score is hurting on Outlook. This feature card is about grouping efforts to improve deliverability.

https://github.com/mozilla/fxa/blob/master/features/FxA-56-email-deliverability/README.md

https://waffle.io/mozilla/fxa?milestone=FxA-56:%20email%20deliverability

Rendered

[vladikoff]

from mtg:

• flow events for bounces
• complaint rate
• bounce rate
• common email services, maybe a dashboard for those. example: what's the percentage of bounces for gmail vs all bounces?
• figure out how to stop resending emails if the email keeps bouncing....

@vladikoff to draft a doc to push this into defined ....

[vladikoff]

Started here: mozilla/fxa#211

[davismtl]

from mtg:

• moving to building
• deployed check spam this week on various screens
• gathering stats from amazon
• will be looking into whitelist
• if all else fails, will setup a new IP

[vbudhram]

Bounces with flowEvents: https://sql.telemetry.mozilla.org/queries/2323#4293

[davismtl]

from mtg:

• sean is blacklisting the bounced emails
  -need to create flow charts that result in bounces
• created issue for whitelisting email server as per recommended by outlook.

[rfk]

from mtg:

• @rfk to get feature doc merged this week

[davismtl]

from mtg:

• Alex (me) needed for some reviews
• We can maybe start strict, measure how many we are blocking and see if we need to loosen up. If we block 1000 ppl per day, are we loosing 1000 registrations per day?
• Travis is investigating senderscore to see if it is really valuable

[davismtl]

from mtg:

• vijay is breaking this up into issues to make a bunch of dashboards that will help measure success of this enhancement
• rfk to investigate about whitelist (certified sender)

[davismtl]

from mtg:

• travis doesn't like the whitelist service recommended by Outlook
• travis recommended that we look into a better ESP rather than SES
• train 81 shipping code to track bounces
• sean will have a demo for 82 where he will show how we don't send emails to bounced emails
• question from Ryan: how do we want to manage sending emails for password changes and resets?
  • prevent users from doing a password reset and locking themselves out completely
  • password changes could work since no confirmation is needed however we should push the user into a flow to add a second email to his account
  • don't send for new device added
• julie is looking into how we can be better at update our user flows after such changes

[davismtl]

from mtg:

• being prioritized in Q2 OKRs to increase progress
• Parts of this shipping in 83
  • Black list is going out (flag us as spam, bounced)
• metrics by domain: @vbudhram has a stradegy. A few PR in review.
• smtp relay is now in its own feature card

[davismtl]

• next step is to import data into re:dash
• as of 83, we are not resending emails to bounced emails (blacklist)
• @shane-tomlinson can improve the front end messaging of bounced emails during registration if he desires to do so ;)

[davismtl]

from mtg:

• part of this is still in flight: there is no messaging to the user to say that their email bounced so this may lead them to keep pushing the resend button on confirmation page.
• we just need to pop-up some existing messaging in a few more cases

[vbudhram]

I was able to get email event imported and created the first graph of bounces, https://sql.telemetry.mozilla.org/queries/4236. I think we can use these rates as a baseline for our email deliverability OKR.

[davismtl]

Oh, the charts are already painting a picture! Did we release @seanmonstar 's bounced email work on the 6th of April?

We seem to be having better deliverability since then!

[davismtl]

Or was it the work done for validating emails in the registration form? (the improvement seems to mostly impact registration emails sent)

[rfk]

Did we release @seanmonstar 's bounced email work on the 6th of April?

This sounds about right, it was released while we were at the work-week. The impact here is likely mostly due to us refusing to re-send to addresses we know to bounce though, not sure if we would be measure any increase in delivery to good addresses yet.

[davismtl]

from mtg:

• we have deliverability metrics now \o/
• we have a functioning black list
• We've observed that Outlook.com deliverability passed for the first time on Email on Acid for the first time. Related?

Next Steps

• @rfk to put feature doc in card.
• @vbudhram log results in the feature doc from this graph:
  https://sql.telemetry.mozilla.org/queries/4236#8440
• vj to also make the other related graphs defined in the feature doc
• UX review for users that are getting a bounced message even if their account is legit. @ryanfeeley to investigate improvements.

[seanmonstar]

So, I've been thinking about this some. Our current graphs show that keeping a bounce list means our numbers look better, which is yay! But, I'm afraid it's slightly deceitful. Of course our bounce rate will go down, since we refuse to email addresses that previously bounced. However, we want to know if keeping our bounces low means better email for everyone else.

To do that, I think we'd need this:

• The rate of people sent and email and then we receive a bounce notification for their address.
• Take those numbers, and pull those users out of how we calculate the rate of people who signin/up but don't verify their email.
• Compare that the new rate increases.

We're especially interested in the case where the email did not bounce, but was flagged as spam or junk. We want to know if, by reducing the amount of times we send to a bouncing address, if that increases our reputation with the spam/junk filters, and gets us into more inboxes.

[rfk]

@rfk to put feature doc in card.

I've linked the feature-doc from the issue description, also putting it here for reference: https://github.com/mozilla/fxa/blob/master/features/FxA-56-email-deliverability/README.md

Take those numbers, and pull those users out of how we calculate the rate of people who signin/up
but don't verify their email.

The feature-doc has a similar-but-not-quite-the-same measure called "lost mail rate":

https://github.com/mozilla/fxa/blob/master/features/FxA-56-email-deliverability/README.md#lost-mail-rate-for-email-flows

IIUC what you're proposing is something like:

• T = total number of signup flows for which we actually sent an email
• C = number of signup flows where the user successfully clicked through
• B = number of signup flows for which we received delivery failure notification
• Graph the "unclicked mail rate" = 100.0 * (T - C - B) / T

@seanmonstar does that match what you're thinking about? I expect we should have all the necessary metrics to produce such a graph already in re:dash.

[seanmonstar]

The 'lost mail rate' isn't what I was referring to. From the feature doc, it reads like that is the number of emails where we received neither a bounce nor a delivery notification.

However, that 'unclicked mail rate' equation looks right. I'll poke around.

[rfk]

@seanmonstar made a very interesting graph here: https://sql.telemetry.mozilla.org/queries/4431/source#8900

Which seems to show an improvement in click-rate for outlook in early April.

[rfk]

Next steps:

• follow up with @seanmonstar about moar graphs
• @ryanfeeley to propose improved UI for the error case when we refuse to send an email due to previous bounciness

[ryanfeeley]

https://mozilla.invisionapp.com/share/DCBJXVL8B#/231955150_Default

[davismtl]

from mtg:

• sean is going on pto so generating graphs will take a bit longer
• we are looking into if we want to do make more flexible rules for users that are randomly bouncing. Need to determine if we want to include that in this card.
  • if so do we want any messaging to say, try again in 24h?
• see @ryanfeeley 's link above

[davismtl]

FEATURE CARD MOVED TO NEW PRODUCT BOARD. ADD NEW COMMENT HERE:
https://projects.growthhackers.com/s2/IPrLmJEFrRWA0N370VeYNA