Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Redesigning async message queues (second attempt) #2159
Here's what I fixed or added:
This is a second attempt at redesigning the async message queues. It contains many of the changes from #2157, but has a different approach to dispatching.
In addition to the changes in #2157, I have introduced a queue management CLI command to list and dispatch events, and removed the dispatch API endpoint for the time being (because dispatching expensive events in an api call represents a DoS vector.
The queue service now makes use of the management API to retrieve a list of events pending, as well as to garbage collect, however actual dispatch is made using a system call to the event management tool.
Here's why I did it:
The queue management API calls introduced in the previous attempt are actually quite useful, since for one thing it removes the requirement to maintain an open DB connection.
Dispatching the event using the event management tool now means that we can run the dispatch as a CLI process (keeping the advantages of that), but since it now runs as a separate process, any database connections are single use.
Keeping the management api, and introducing a management tool also means that in future we can rewrite the queue service to be driven by something else - node, whatever.