✉️ Separate Email into an Independent Service

[nelsonic]

Context

As described in dwyl/app#267, email is very important to the success of our App.

At present the Email section of the README.md https://github.com/dwyl/auth#email indicates that auth uses SES and Bamboo for sending email. It informs people that they need to have environment variables for SMTP_USERNAME, SMTP_PASSWORD, SES_SERVER and SES_PORT and links to dwyl/learn-phoenix-framework/sending-emails.md for more detail.

This is good for a barebones MVP that just needs to show a PO/PM that "email works",
but it is not a long-term solution to for reliably sending email because:

The Problem 😕

a) We are sending email in a "fire and forget" manner expecting them to automagically reach their target recipient. But we all know there is a lot more to email deliverability than simply sending the email.
How do we know it was actually delivered to the recipient?

b) We don't have any end-to-end tests for email so we have no way of knowing it's working other than to manually test sending an email to ourselves. Manual testing gets old fast.

c) Assuming the email is successfully delivered we have no way of knowing if it was opened/read.
We have no metrics for how effective our email is so we cannot be data-driven.

d) What happens when the email bounces or worse there is a complaint about our service? Our deliverability rating will plummet and along with it the fate of our App.

e) We are forced to maintain email related code in the auth App which is really not the "focus" of the auth App. Email should be a single function invocation that delegates to a dedicated service.

All of these issues lead us to believe that there is a much better way of doing email.

More detail on why email is super important to @dwyl is contained in: /app/issues/267

Proposed Solution 💡

Instead of using Bamboo and writing Elixir code to send email,
we build a separate service that has only one job email. 💌

Given that we are already using Amazon Simple Email Service (SES) for sending our email,
why not make the email service a Lambda Function that we can invoke from any App or Programming Language?

Advantages 🎉

• A separate email Lambda function/service is much easier to test and maintain.
  • It allows us to reuse and extend our existing work https://github.com/dwyl/sendemail
    that in turn allows people to create email templates in Handlebars
    which is arguably a simpler and more beginner-friendly templating language than EEx.
• auth can delegate email and focus on the one thing it needs to do; authenticate people.
• The Lambda service will handle all Notifications for SES e.g: delivery, bounce & complaint
  see: https://docs.aws.amazon.com/ses/latest/DeveloperGuide/notification-contents.html
  All events/notifications will be fed back to the Auth App and the status of the email will be recorded
  so that the team can see the delivery, bounce, open and clickthrough rate metrics! 📈
• A separate email system can more easily be used and improved upon by the community.

Downside? 😞

The downside of having a separate project/function for email is that it's another thing to think about (and hold in your head) from the developer's perspective.

I feel this is minor given that a developer is already invoking a Bamboo function,
the only difference will be we invoke a Lambda function with the same data.

---

In case you're wondering, we intend to re-use most of the work done in #36 and #35 (responsive email templates), so almost no work will be "wasted", we just think there is a more effective and long-term maintainable way of doing email.

---

Todo

• Update our knowledge of AWS Lambda: https://github.com/dwyl/learn-aws-lambda
• Update our Lambda deployment tool dwyl/aws-lambda-deploy#63, 64 & 65
• Create the Lambda Function to Send Emails
• Deploy the Lambda Function using dpl 📦
• Invoke the Lambda Function from Elixir: Invoke AWS Lambda Function from Elixir aws-ses-lambda#8
• Extend the Lambda to listen for SES Notifications delivery & bounce Create SES topic for bounced emails aws-ses-lambda#1

There is obviously a lot more we can do with email and that's part of the reason we want to split it out into it's own independent service. But this is all we need for the auth service for now.
This will allow us to get back to focussing on building our App. 📱

[nelsonic]

Invoking a JavaScript AWS Lambda Function works flawlessly and is not much code at all!
See: https://github.com/dwyl/elixir-invoke-lambda-example#7-conclusion 🚀

[nelsonic]

Email service is working exactly as we need it to!

Next step is to integrate the /api/send into the Auth app: #41