Queues (for communications and other)

[pwalsh]

We don't have a queue for dispatching tasks. Currently, the only area we really could use it is in dispatching email and sms messages via API to the backend providers. The initial implementation of that uses Node's event emitters to dispatch these jobs, but there is no management of such jobs like retry etc.

It would be good if we could have a simple queue framework that uses Postgres, to reduce the out of the box dependencies for the code base, but probably we'll go for a rabbit mq or redid backed queue framework.

[noamoss]

@pwalsh not sure I understand the core of this issue, can you elaborate a bit?

[pwalsh]

@noamoss in our context, queuing systems are used to manage asynchronous jobs that do not need (for various types of "need" - performance, application semantics, etc.) to be dispatched and returned in the request/response cycle of a web app.

Dispatching emails is a classic example of this. Dispatching them in a request/response cycle means the web app has more processing to do before it can respond back to the client (especially true in languages like python).

So, what is commonly done is to dispatch a "job" (a job might be a function and its arguments), put it "somewhere", and call it "later", allowing the response to return without calling the job. This design has additional benefits like the ability to retry the job if there is a failure, to set dependencies across jobs etc.

In GovFlow, because Node is an event-driven web server (it is asynchronous by design), we do the "dispatch" part using some node functionality, but, we don't get the benefit of the ability to re-run jobs, inspect job health, etc.

A concrete example is, if we try to dispatch an email and the receiving server is temporarily down, we don't know and we can't try again later.

A job queue will allow us to manage such tasks, which run run the background of the web server, more robustly.

[pwalsh]

@noamoss @amirozer I looked into it last week and it seems this is probably the best solution for this in node:

https://github.com/OptimalBits/bull

@amirozer if you have other choices that are in use, let me know.

[pwalsh]

So, SendGrid has a send_at functionality - and it turns out that Twilio does too. This basically solves our whole issue:

• we dont need to manage our own queue or schedule - they do it
• we just dispatch message tasks to those backends as we normally do
• we do need to update our communication model like in the previous comment to ensure we are tracking what got send properly (enough), and save some rendered message data for potential debugging purposes

[pwalsh]

This is now done in ba32772 by using the functionality of our service providers Twilio and SendGrid.