Migrate from Sendgrid to SES

[nigeljonez]

Sendgrid is becoming problematic due to reputation issues (Spamhaus) and resulting delivery reliability.

Plan: Migrate from Sendgrid to SES

Blocking issues:

• Backscatter from alaveteli bounce setting
• Inability to tag certain e-mails with SES ConfigurationSets
• Postfix configuration accepting invalid hashes

[nigeljonez]

Although the backscatter hasn't caused us problems, implementing SES means we should also tackle it by avoiding Alaveteli softbounces, our sources appear to be:

Source	Current	Proposed
Invalid To: addresses that don't match a request	Route to holding_pen	pgsql Lookup view to validate prefix-id-idhash combinations and hard-reject if invalid
Requests marked authority_only	Default is to software bounce	Code change to set holding_pen as default, and DB migration to retire bounce
Old requests	Migrated to authority_only (old months old), Migrated to nobody (very_old months old) (src)	No action required, defaulting to holding_pen will avoid bounces

[nigeljonez]

The issue with e-mails to Ministers may be related, in case it is, this is now high priority.

[olineham]

Backscatter problem has been temporarily solved by updating all requests from bounce to holding_pen. Sendgrid stats show the huge surge of blocks/bounces and overall volume has been eliminated. This is at the expense of a big increase in the number of holding pen emails to deal with. I've opened #8 to discuss longer term options.

[olineham]

Inability to tag certain e-mails with SES ConfigurationSets: is this truly a blocker for SES migration? A single configuration and no tagging is already the situation with Sendgrid. Certainly good to have for SNS but we'd be no worse off than we were before without it.

Logging, on the other hand, is an absolute requirement. We need to keep evidence (even if found by manual search) whether a given outgoing email was delivered.

Postfix config accepting invalid hashes: also not a blocker I think since backscatter is gone (in the short term). I've opened #9 for this as an independent enhancement.