fix(email): Fix SendGrid fallback integration

[zwliew]

Problem

We currently use SendGrid as our email fallback. This causes the
following functions to fail:

1. Verifying custom from addresses (e.g. adding a new address, storing
   a campaign template with a custom address)
2. Sending emails with custom from addresses will fail
3. Receiving delivery report callbacks since the code currently parses the callbacks incorrectly

Closes #1021

Solution

Since custom from addresses are not verified with SendGrid, we shouldn't use
them at all.

In the future, if we do intend to authenticate custom from addresses with SendGrid,
we will also have to rewire verifyFromAddress() to use SendGrid APIs.

Features:

1. Send all emails using the default address during the fallback.
2. Disable verification of custom from addresses during the fallback.
3. Disable storing of templates using custom from addresses.
4. Documented instructions to activate the email fallback on the incident response doc
5. Fix parsing of SendGrid webhook events

Before & After Screenshots

Click to show images

AFTER:
Attempting to verify a custom from address:

Attempting to save an email template using a custom from address:

Creating a new email campaign:

Tests

Test 1: all campaign emails should be sent using the default from address during fallback

• Store a campaign using a custom from address
• Set EMAIL_FALLBACK_ACTIVATE to true for the worker
• Send the campaign
• The campaign emails should be sent using the default from address

Test 2: verification of custom from addresses should be disabled during fallback

• Set EMAIL_FALLBACK_ACTIVATE to true for the backend
• Try to verify a custom from address
• The UI should display an error message like in the first screenshot above

Test 3: storing of templates with custom from addresses should be disabled during fallback

• Set EMAIL_FALLBACK_ACTIVATE to true for the backend
• Try to store a campaign using a custom from address
• The UI should display an error message like in the second screenshot above

Test 4: SES callbacks should be received and parsed correctly

• Test sending via SES
  1. Deploy the branch on Travis with STAGING_BRANCH=sendgrid-email-fallback and DEPLOY_WORKER=true
  2. Send an email campaign (using these emails: https://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-simulator.html)
  3. Ensure that the status and error_code columns of theemail_messages are updated correctly]

Test 5: SendGrid callbacks should be received and parsed correctly

• Test sending via SendGrid
  1. Set EMAIL_FALLBACK_ACTIVATE=true on Secrets Manager
  2. Prefix the Secrets Manager variables listed below with BAK_
  3. Add SendGrid-variants of the variables previously prefixed with BAK_
  4. Set the SendGrid event webhook HTTP POST URL to the correct callback server endpoint, with the correct username and password
  5. Deploy the branch on Travis with STAGING_BRANCH=sendgrid-email-fallback and DEPLOY_WORKER=true
  6. Send an email campaign (using a valid email and bouncetest@tribulant.com)
  7. Ensure that the status and error_code columns of theemail_messages are updated correctly

Deploy Notes

New environment variables:

• EMAIL_FALLBACK_ACTIVATE: Switch to true to use SendGrid as fallback for all emails. Ensure that the SMTP settings are properly configured as well. (worker and backend)

Todo

• Update the incident response doc to explain about the SendGrid fallback

BACKEND_SES_HOST
BACKEND_SES_USER
BACKEND_SES_PASS
WORKER_SES_HOST
WORKER_SES_USER
WORKER_SES_PASS

[miazima]

We want to log email bounce sub types in email callbacks, here is the relevant pr #1062. Sendgrid provides bounce reasons which can be stored as error sub types too. I have not included this in the pr as we are unable to test the Sendgrid flow right now.

Here is the previous code written before I reverted the Sendgrid changes.
https://github.com/opengovsg/postmangovsg/blob/c5afd8cc36dbee6e1439f901fbaad5916d21d02f/backend/src/email/utils/callback/parsers/sendgrid.ts

[zwliew]

We want to log email bounce sub types in email callbacks, here is the relevant pr #1062. Sendgrid provides bounce reasons which can be stored as error sub types too. I have not included this in the pr as we are unable to test the Sendgrid flow right now.

Here is the previous code written before I reverted the Sendgrid changes.
https://github.com/opengovsg/postmangovsg/blob/c5afd8cc36dbee6e1439f901fbaad5916d21d02f/backend/src/email/utils/callback/parsers/sendgrid.ts

Oh I see, thanks! Since we are unable to test the Sendgrid flow in time, perhaps it should be in a separate PR?

[lamkeewei]

Update - pending SendGrid account setup before continue on review

[zwliew]

SendGrid events contain carriage return and newline characters (\r\n) and some even have unicode-encoded characters. The JSON body-parser that we currently use strips the newline characters and decodes the unicode characters, which causes the ECDSA verification to fail.

I had a less intrusive fix here: 201d0f4
But, I think it's not great for reasons stated in the revert: 5473c72

I ended up with this solution since it isn't super intrusive, but it might affect SES callbacks. I doubt it would in practice though.

One other solution which doesn't affect SES callbacks is to split the SES and SendGrid callback endpoints, and only apply the text body-parser for SendGrid, but that would be sort of a breaking change. What do you think?

[lamkeewei]

I think this is fine. I've checked and only one body-parser middleware will run each time.

[lamkeewei]

lgtm! Tested the following and works as expected:

1. Activation of email fallback should prevent verifying new custom from address
2. Activation of email fallback should prevent usage of custom from address
3. SendGrid events are parsed and handled correctly
4. SES events are parsed and handled correctly
5. Sendgrid is able to deliver to SGMail email addresses