Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
Move away from Storage Queues #21
This change removes the use of Azure Storage Queues from the system. Instead, we now use Azure ServiceBus for the queuing mechanism.
This has several advantages. Notably, ServiceBus has a push-style API (via AMQP) and Storage Queues only expose a REST API. This means that to integrate with Storage Queues we had to periodically poll the REST API, check if there are new messages and then act on them. With ServiceBus, we can instead just define an event listener for incoming messages and act on them as they get pushed to the receiver via AMQP. This is neater in terms of code and also more efficient in terms of processing.
Note that the push-style API for ServiceBus is not available in the Python SDK so the queue listener is implemented in C#. The listener indefinitely waits for messages from ServiceBus and on receipt makes a request to a Python service running in the same pod which then acts on the message. All the previous queue processing jobs have thus been converted to HTTP endpoints so that the queue listener can easily interact with them.
The new separation of concerns also has a few incidental added benefits. For instance, the distinction between business logic (processing of the queue message) and action triggers (receipt of the queue message) is now very neat and clear. Additionally, we now have a unified interface for all the actions in the system: everything can be triggered via a simple HTTP call whereas previously we had a mix of Python HTTP endpoints and Python daemons. This new uniformity for example simplifies local testing.
The approach implemented by this change is similar to what I described in this article: https://firstname.lastname@example.org/using-the-ambassador-pattern-for-reliable-pipes-and-filters-text-processing-9ebbea8f82c4
There are also a few other changes included in this pull request including: