Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add transports based on Azure PaaS #891

Merged
merged 4 commits into from Aug 30, 2018
Merged

Add transports based on Azure PaaS #891

merged 4 commits into from Aug 30, 2018

Conversation

@c-w
Copy link
Contributor

@c-w c-w commented Jun 18, 2018

This pull request adds two new transport implementations:

  • azurestoragequeues is implemented on top of Azure Storage Queues. This offers a simple but scalable and low-cost PaaS transport for Celery users in Azure. The transport is intended to be used in conjunction with the Azure Block Blob Storage backend.

  • azureservicebus is implemented on top of Azure Service Bus and offers PaaS support for more demanding Celery workloads in Azure. The transport is intended to be used in conjunction with the Azure CosmosDB backend.

This pull request was created together with @ankurokok, @dkisselev, @evandropaula, @martinpeck and @michaelperel.

@codecov
Copy link

@codecov codecov bot commented Jun 18, 2018

Codecov Report

Merging #891 into master will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master     #891   +/-   ##
=======================================
  Coverage   88.52%   88.52%           
=======================================
  Files          63       63           
  Lines        6512     6512           
  Branches      775      775           
=======================================
  Hits         5765     5765           
  Misses        665      665           
  Partials       82       82
Impacted Files Coverage Δ
kombu/transport/__init__.py 100% <ø> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 200a60a...b4f6050. Read the comment docs.

Copy link
Member

@auvipy auvipy left a comment

Looks good from a high level overview

auvipy
auvipy approved these changes Jul 3, 2018
@auvipy
Copy link
Member

@auvipy auvipy commented Jul 19, 2018

@c-w is there anything remaining from your side?

@c-w
Copy link
Contributor Author

@c-w c-w commented Jul 19, 2018

Hi @auvipy. Thanks for your review! If you're happy to merge this, I'm all for it. (I don't have write access to the repository so one of your maintainers will have to hit the merge button.)

Copy link
Member

@georgepsarakis georgepsarakis left a comment

I added some comments of minor importance. Nice work @c-w !

def sbq(self):
if self._sbq is None:
self._sbq = ServiceBusService(
service_namespace=getenv('AZURE_SERVICEBUS_ACCOUNT',
Copy link
Member

@georgepsarakis georgepsarakis Jul 20, 2018

I am not sure environment variables should work at this level and I don't see any other transport reading from the environment. The configuration should have been done at a higher level (in this case Celery).

Copy link
Contributor Author

@c-w c-w Jul 24, 2018

Done in f260341.


super(Channel, self).__init__(*args, **kwargs)

for queue in self.sbq.list_queues():
Copy link
Member

@georgepsarakis georgepsarakis Jul 20, 2018

There is an open issue with AWS SQS transport, not requiring ListQueues permissions. In the proposed fix, a predefined list of queue names is given by configuration. Would that apply here as well?

Copy link
Contributor Author

@c-w c-w Jul 24, 2018

Azure ServiceBus has three kinds of permissions: listen, send, and manage (docs). Given that we're creating new queues from the connector, we'll anyways need the manage permissions so might as well use the list queues call here. Thoughts?

return n

@property
def sbq(self):
Copy link
Member

@georgepsarakis georgepsarakis Jul 20, 2018

Not sure, but perhaps the abbreviation will be difficult to read for readers unfamiliar with the service?

Copy link
Contributor Author

@c-w c-w Jul 24, 2018

Done in a7df447.

@c-w
Copy link
Contributor Author

@c-w c-w commented Jul 24, 2018

@georgepsarakis Thanks for the comments. Addressed them. Let me know if there's anything else.

auvipy
auvipy approved these changes Jul 24, 2018
@auvipy auvipy added this to the 4.3 milestone Aug 9, 2018
@auvipy
Copy link
Member

@auvipy auvipy commented Aug 29, 2018

please rebase

c-w added 4 commits Aug 30, 2018
This pull request adds two new transport implementations:

- `azurestoragequeues` is implemented on top of Azure Storage
  Queues [1]. This offers a simple but scalable and low-cost PaaS
  transport for Celery users in Azure. The transport is intended to be
  used in conjunction with the Azure Block Blob Storage backend [2].

- `azureservicebus` is implemented on top of Azure Service Bus [3] and
  offers PaaS support for more demanding Celery workloads in Azure. The
  transport is intended to be used in conjunction with the Azure
  CosmosDB backend [4].

This pull request was created together with @ankurokok, @dkisselev,
@evandropaula, @martinpeck and @michaelperel.

[1] https://azure.microsoft.com/en-us/services/storage/queues/
[2] celery/celery#4685
[3] https://azure.microsoft.com/en-us/services/service-bus/
[4] celery/celery#4720
There is test coverage for the transports but the tests require Azure
credentials to run (passed via environment variables) so codecov doesn't
exercise them.
@c-w
Copy link
Contributor Author

@c-w c-w commented Aug 30, 2018

Done, @auvipy.

@auvipy auvipy merged commit 3263355 into celery:master Aug 30, 2018
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants