Send emails to report subscribers

[graphiclunarkid]

When people submit URLs for testing we ask whether they want to subscribe to regular blocking reports about that URL. If they say yes we store their email address in the contacts table and a flag against the request record ("subscribereports") indicating they'd like to receive reports. We need a mechanism that:

1. Detects when a URL has changed state
2. Finds all the reports for that URL where (a) subscribereports is set, and; (b) we have a valid email address
3. Filters this down to one email per URL per contact (no point spamming someone with 100 emails about the same URL!)
4. Generates a suitable email depending on what has happened
5. Sends the email via whatever mechanism (Set up email server or service #51)

Acceptance criteria from call with @JimKillock and Pam:

• We want to send send an email to report subscribers every 3 months reminding them of our existence even if the URL hasn't changed state.
• Emails should be limited to a maximum of one per day.
• Reports should be sent only when ORG probes detect changes in state - not volunteer probes. We should say as much in the emails too.
• Reports should be sent for all changes of state (i.e. blocked -> unblocked state changes and vice versa)
• We should send an email as soon as we detect a state change for a single ISP.

[dantheta]

I've got a database script that can detect status changes very cheaply and easily, populating a url_status_changes table that can be used for reporting and scheduled email sends.

When the url_latest_status table is being updated, the script has got the old and new status in the same context, so it can compare them and write a "changed" row to another table really efficiently. This saves having to run queries that get the latest and latest-but-one status to compare, which would scale pretty poorly as the number of results grows.

The email alerting can also be run from a queue (that way if we get a really large number of people interested in the status of a particular URL, the status update and results submission steps in the API don't get held up preparing all the emails).

[dantheta]

There's some work-in-progress on the new-url-subscription branch which moves the URL status subscription from the requests table to a new, uniquely-indexed url_subscriptions table. This will make it easier to pull out the list of subscribers for a URL, since we won't have to deal with multiple requests by the same contact for the same URL.

[dantheta]

Just a quick question:

"Emails should be limited to a maximum of one per day."

Is this one email per url per day, or one email per url/isp combination? In the event of a URL changing status on more than one ISP on the same day, do we want to email alert on the first ISP only, or once for each ISP? (sorry if it seems like I'm obsessing over the improbable).

[JimKillock]

Yes, the idea is if the status results keep flipping for some reason (and also for some reason we detect this), then people should not be given repeated results.

[JimKillock]

So just one email per url per day.

[dantheta]

Here's the current status: The database table changes and the code for testing an email opt-in token are written and are on the feature/new-url-subscription branch of Blocking-Middleware. This is pretty much ready for merging, once it's been reviewed. It covers steps 1-4 of Richards original mini-spec.

Once it has been deployed, we'll need a URL on the web frontend that can receive the opt-in email validation tokens and pass them to the API. This URL will be the one linked to in emails.

After that, we can write in the code that will send the verification emails (we may need some new copy written for this).

Then we can start sending emails to the people who have subscribed to status updates on URLs and start sending them the updates according to Jim and Pam's rules.

Daniel.

[dantheta]

I've updated the wiki API docs with the proposed API endpoint for verifying an email token.

https://wiki.openrightsgroup.org/wiki/Censorship_Monitoring_Project_API#POST_.2Fverify.2Femail_.28proposed.29

[dantheta]

Currently blocked on #18