Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move away from Storage Queues #21

Merged
merged 19 commits into from Jun 16, 2018

Conversation

Projects
None yet
1 participant
@c-w
Copy link
Member

commented Jun 13, 2018

This change removes the use of Azure Storage Queues from the system. Instead, we now use Azure ServiceBus for the queuing mechanism.

This has several advantages. Notably, ServiceBus has a push-style API (via AMQP) and Storage Queues only expose a REST API. This means that to integrate with Storage Queues we had to periodically poll the REST API, check if there are new messages and then act on them. With ServiceBus, we can instead just define an event listener for incoming messages and act on them as they get pushed to the receiver via AMQP. This is neater in terms of code and also more efficient in terms of processing.

Note that the push-style API for ServiceBus is not available in the Python SDK so the queue listener is implemented in C#. The listener indefinitely waits for messages from ServiceBus and on receipt makes a request to a Python service running in the same pod which then acts on the message. All the previous queue processing jobs have thus been converted to HTTP endpoints so that the queue listener can easily interact with them.

The new separation of concerns also has a few incidental added benefits. For instance, the distinction between business logic (processing of the queue message) and action triggers (receipt of the queue message) is now very neat and clear. Additionally, we now have a unified interface for all the actions in the system: everything can be triggered via a simple HTTP call whereas previously we had a mix of Python HTTP endpoints and Python daemons. This new uniformity for example simplifies local testing.

The approach implemented by this change is similar to what I described in this article: https://medium.com/@clemens.wolff/using-the-ambassador-pattern-for-reliable-pipes-and-filters-text-processing-9ebbea8f82c4

There are also a few other changes included in this pull request including:

  • Connection strings are now stored in Docker secrets where previously they were stored in environment variables. The client code stays the same since there's a docker-entrypoint script that loads the secrets into environment variables at runtime.
  • Adding some new IDE folders to git-ignore and docker-ignore.
  • I was using some of the checked-in raw MIME emails for testing and was surprised that they didn't get processed by the system. This is because we only accept emails that are sent to a subdomain of lokole.ca, so I updated our test data to reflect that.

Resolves #16

@c-w c-w merged commit b26f15d into master Jun 16, 2018

3 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
pyup.io/safety-ci No dependencies with known security vulnerabilities.
Details

@c-w c-w deleted the servicebus branch Jun 16, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.