[PROD] Emails not getting sent to email queue - registration - intermittent issue

[nanometrenat]

Issue

Spin off from MSM ticket SRV-21991 (more details re specific email addresses there)

Whilst on the whole the user registration process is working correctly (users get an email when they register, then they have to click on that email to verify their email address, then either they are activated - if on whitelist - or emails go to their verifiers).

However it was identified over the weekend that in at least one case an email was not sent to one of the verifiers (although an email WAS sent to the other verifier) so effectively that user was stuck in pending users until manual action is taken from an admin to activate them.

This ticket is to track investigations as to why the email was never sent to the internal email API that sends emails.

Whilst this particular missing email is the only one that investigations have already happened for (confirming it's not a user error), it is possible that there are wider intermittent email-sending issues elsewhere. e.g. notifications. so once dev-investigations are done we will need to define what testing is required.

Related feature

IFRCGo/go-frontend#889

Impact

Unknown - intermittent

Severity

Complete blocker for those affected - they would need to contact IM for a manual update.

[teklal]

Commenting here as also related to registration: there has been at least one occurrence (potentially more) of the verification email not being sent to the user and the user hence not being able to verify and activate their account. This was observed for a user whose email address is in the whitelisted domains.

[teklal]

Also tracking here as mentioned in the ticket that may be related to other emails as well:
I did not receive notification emails about surge alerts today although there have been 4 put on GO.

So it is possible that this is a wider problem.

[nanometrenat]

xref SRV-23022

[GergiH]

Added Notification GUID "module" under the Notifications part on the Django Admin site. It saves all the emails' IDs which reach the sender API at least.
For most of the other cases more logging was added into the existing logs. If the recipients conversion fails a Cronjob log record warning will also be created with the address of one of the recipients.

All deployed to staging.

[GergiH]

@karitotp @teklal Please help me test the new functionality.

Notification GUID (Django Admin - under 'Notifications')

• API GUID - the ID coming back from the email sender API for the mail
• Email type - (has some additional info as well in most cases)
  • Field Report - new Field Report created
  • Emergency - new Emergency created
  • Validate account - new registration
  • Approve an account - to admins to approve a non-whitelisted account
  • Approved account successfully - self explanatory
  • Password recovery - forgotten password
  • Username recovery - forgotten usernam
  • Ingestion error - would be harder to test, but there is already a record on staging about it, nightly running jobs' errors
• To list - recipients basically

[nanometrenat]

There seems to be a wider problem with emails at the moment (i.e. not intermittent), users report that the usual weekly updates and field report notifications haven't been received since Friday, and looking in https://goadmin.ifrc.org/notifications/notificationguid/ I can see that the below were the last ones -yet there have been several Field Reports since then that were not notified:

API GUID	EMAIL TYPE	CREATED AT
Followed Emergency notification - 1 followed event modified	June 26, 2020, 7:40 p.m.
Followed Emergency notification - 1 followed event modified	June 26, 2020, 7:40 p.m.
Field Report notification (ifrc) - 2 new field reports	June 26, 2020, 6:50 p.m.
Field Report notification (non_ifrc) - 2 new field reports	June 26, 2020, 6:50 p.m.

[nanometrenat]

Note also that there are several users that have been set up since then - so recovery actions are required for these users to be able to proceed - thanks a lot.