Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add transports based on Azure PaaS #891

Merged
merged 4 commits into from Aug 30, 2018

Conversation

Projects
None yet
3 participants
@c-w
Copy link
Contributor

commented Jun 18, 2018

This pull request adds two new transport implementations:

  • azurestoragequeues is implemented on top of Azure Storage Queues. This offers a simple but scalable and low-cost PaaS transport for Celery users in Azure. The transport is intended to be used in conjunction with the Azure Block Blob Storage backend.

  • azureservicebus is implemented on top of Azure Service Bus and offers PaaS support for more demanding Celery workloads in Azure. The transport is intended to be used in conjunction with the Azure CosmosDB backend.

This pull request was created together with @ankurokok, @dkisselev, @evandropaula, @martinpeck and @michaelperel.

@codecov

This comment has been minimized.

Copy link

commented Jun 18, 2018

Codecov Report

Merging #891 into master will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master     #891   +/-   ##
=======================================
  Coverage   88.52%   88.52%           
=======================================
  Files          63       63           
  Lines        6512     6512           
  Branches      775      775           
=======================================
  Hits         5765     5765           
  Misses        665      665           
  Partials       82       82
Impacted Files Coverage Δ
kombu/transport/__init__.py 100% <ø> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 200a60a...b4f6050. Read the comment docs.

@auvipy
Copy link
Member

left a comment

Looks good from a high level overview

@auvipy

auvipy approved these changes Jul 3, 2018

@auvipy

This comment has been minimized.

Copy link
Member

commented Jul 19, 2018

@c-w is there anything remaining from your side?

@c-w

This comment has been minimized.

Copy link
Contributor Author

commented Jul 19, 2018

Hi @auvipy. Thanks for your review! If you're happy to merge this, I'm all for it. (I don't have write access to the repository so one of your maintainers will have to hit the merge button.)

@georgepsarakis
Copy link
Member

left a comment

I added some comments of minor importance. Nice work @c-w !

def sbq(self):
if self._sbq is None:
self._sbq = ServiceBusService(
service_namespace=getenv('AZURE_SERVICEBUS_ACCOUNT',

This comment has been minimized.

Copy link
@georgepsarakis

georgepsarakis Jul 20, 2018

Member

I am not sure environment variables should work at this level and I don't see any other transport reading from the environment. The configuration should have been done at a higher level (in this case Celery).

This comment has been minimized.

Copy link
@c-w

c-w Jul 24, 2018

Author Contributor

Done in f260341.


super(Channel, self).__init__(*args, **kwargs)

for queue in self.sbq.list_queues():

This comment has been minimized.

Copy link
@georgepsarakis

georgepsarakis Jul 20, 2018

Member

There is an open issue with AWS SQS transport, not requiring ListQueues permissions. In the proposed fix, a predefined list of queue names is given by configuration. Would that apply here as well?

This comment has been minimized.

Copy link
@c-w

c-w Jul 24, 2018

Author Contributor

Azure ServiceBus has three kinds of permissions: listen, send, and manage (docs). Given that we're creating new queues from the connector, we'll anyways need the manage permissions so might as well use the list queues call here. Thoughts?

return n

@property
def sbq(self):

This comment has been minimized.

Copy link
@georgepsarakis

georgepsarakis Jul 20, 2018

Member

Not sure, but perhaps the abbreviation will be difficult to read for readers unfamiliar with the service?

This comment has been minimized.

Copy link
@c-w

c-w Jul 24, 2018

Author Contributor

Done in a7df447.

@c-w

This comment has been minimized.

Copy link
Contributor Author

commented Jul 24, 2018

@georgepsarakis Thanks for the comments. Addressed them. Let me know if there's anything else.

@auvipy

auvipy approved these changes Jul 24, 2018

@auvipy auvipy added this to the 4.3 milestone Aug 9, 2018

@auvipy

This comment has been minimized.

Copy link
Member

commented Aug 29, 2018

please rebase

c-w added some commits Jun 18, 2018

Add transports based on Azure PaaS
This pull request adds two new transport implementations:

- `azurestoragequeues` is implemented on top of Azure Storage
  Queues [1]. This offers a simple but scalable and low-cost PaaS
  transport for Celery users in Azure. The transport is intended to be
  used in conjunction with the Azure Block Blob Storage backend [2].

- `azureservicebus` is implemented on top of Azure Service Bus [3] and
  offers PaaS support for more demanding Celery workloads in Azure. The
  transport is intended to be used in conjunction with the Azure
  CosmosDB backend [4].

This pull request was created together with @ankurokok, @dkisselev,
@evandropaula, @martinpeck and @michaelperel.

[1] https://azure.microsoft.com/en-us/services/storage/queues/
[2] celery/celery#4685
[3] https://azure.microsoft.com/en-us/services/service-bus/
[4] celery/celery#4720
Exclude Azure transports from code coverage
There is test coverage for the transports but the tests require Azure
credentials to run (passed via environment variables) so codecov doesn't
exercise them.
@c-w

This comment has been minimized.

Copy link
Contributor Author

commented Aug 30, 2018

Done, @auvipy.

@auvipy auvipy merged commit 3263355 into celery:master Aug 30, 2018

4 checks passed

codecov/patch Coverage not affected when comparing 200a60a...b4f6050
Details
codecov/project 88.52% remains the same compared to 200a60a
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.