Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS DynamoDB result backend #3736

Merged
merged 13 commits into from Jan 23, 2017

Conversation

@georgepsarakis
Copy link
Member

commented Jan 4, 2017

Description

The changes in this pull request add the ability to define AWS DynamoDB as the result backend.

DynamoDB is a NoSQL store that has both document-oriented and key-value semantics.

Motivation

  • Adheres to the IAM permission system.
  • Scalability with provisioned read/write throughput.
  • No storage restrictions.
  • Combined with SQS, can work seamlessly in an EC2 environment.

Notes

  • The module structure and test cases was for the most part, modeled after the Riak result backend.
  • The table is automatically created if not exists. After creation, the code polls until the table is available (state ACTIVE).
  • Perhaps more configuration/customization capabilities should be added, especially regarding the table schema.
  • Currently can only be configured via URL (documentation pending).

Table Schema

Attribute Name Data Type Description
id String The task result identifier
result Bytes The base64-encoded serialized task result
timestamp Number The current timestamp as a floating-point number
@georgepsarakis

This comment has been minimized.

Copy link
Member Author

commented Jan 4, 2017

This is still a work in progress (documentation pending mainly), but any feedback will be greatly appreciated. Thanks!

@georgepsarakis

This comment has been minimized.

Copy link
Member Author

commented Jan 4, 2017

For anyone willing to test these changes, I used the following sample code:

Create the task definitions module tasks.py

from celery import Celery

# Assumes that AWS credentials will be available as environment variables
# or in ~/.aws/credentials 
app = Celery(
    'tasks',
    broker='redis://localhost',
    backend='dynamodb://@us-east-1'
)


@app.task
def add(x, y):
    return x + y

Start the worker

$ celery -A tasks worker --loglevel=info

Add task

from tasks import add
task = add.delay(4, 4)
print task.get() # 8

@georgepsarakis georgepsarakis changed the title [WIP] AWS DynamoDB result backend AWS DynamoDB result backend Jan 4, 2017

@georgepsarakis

This comment has been minimized.

Copy link
Member Author

commented Jan 4, 2017

Documentation added and CI checks passed, should be ready for review.

@thedrow
Copy link
Member

left a comment

I'm not sure I'm 100% comfortable to merge this without integration tests that are run regularly to ensure the result backend won't break.
We have enough support tickets as it is and we'd like to avoid breaking features for our users.
At the very least, we need functional tests that will pass with moto in order to merge this IMO.
@ask What do you think?

@georgepsarakis

This comment has been minimized.

Copy link
Member Author

commented Jan 5, 2017

@thedrow I don't have a problem with implementing the tests. For the time being I used mocking over the boto3 function calls. I will take a look at moto in the meanwhile. Thanks for the feedback so far!

@georgepsarakis

This comment has been minimized.

Copy link
Member Author

commented Jan 5, 2017

@thedrow I discovered that AWS offers DynamoDB as a downloadable Java application.

I have enabled the integration tests and I added the dependencies in Travis CI.

Let me know what you think!

@georgepsarakis georgepsarakis force-pushed the georgepsarakis:feature/dynamodb-result-backend branch 2 times, most recently from e8aab15 to 88963e7 Jan 5, 2017

@georgepsarakis

This comment has been minimized.

Copy link
Member Author

commented Jan 6, 2017

@thedrow I have successfully enabled the integration tests with DynamoDB as a result backend.

If you think it is necessary, I will also add functional tests with moto, but could you please share some thoughts on what the format of these tests should be?

Thanks again for taking the time to review this.

@auvipy
Copy link
Member

left a comment

rebase to

@georgepsarakis georgepsarakis force-pushed the georgepsarakis:feature/dynamodb-result-backend branch 4 times, most recently from e3e75ab to 1c09abb Jan 9, 2017

@georgepsarakis

This comment has been minimized.

Copy link
Member Author

commented Jan 9, 2017

@auvipy I have rebased but I am trying to fix Travis CI issues related to #3750 and #3749 .

@georgepsarakis

This comment has been minimized.

Copy link
Member Author

commented Jan 9, 2017

I followed this approach to specify the interpreter correctly.

@georgepsarakis georgepsarakis force-pushed the georgepsarakis:feature/dynamodb-result-backend branch 2 times, most recently from ce3a7e2 to 0394dc2 Jan 9, 2017

@auvipy

This comment has been minimized.

Copy link
Member

commented Jan 9, 2017

wait for @ask

@georgepsarakis georgepsarakis force-pushed the georgepsarakis:feature/dynamodb-result-backend branch 4 times, most recently from e33e913 to d3b6be7 Jan 9, 2017

@georgepsarakis

This comment has been minimized.

Copy link
Member Author

commented Jan 9, 2017

@auvipy I finally resolved the Travis CI issues. I added Python 3.6 to the build matrix. Perhaps this solution is of interest for #3750.

@georgepsarakis

This comment has been minimized.

Copy link
Member Author

commented Jan 9, 2017

Unit tests fail with Python 3.6, but this is out-of-scope for this pull request. All integration tests seem to pass with 3.6 though.

@ask

This comment has been minimized.

Copy link
Contributor

commented Jan 18, 2017

Looks really good to me! Very good work, I think this can be merged :)

@georgepsarakis

This comment has been minimized.

Copy link
Member Author

commented Jan 18, 2017

Thanks @ask :) !

@georgepsarakis

This comment has been minimized.

Copy link
Member Author

commented Jan 20, 2017

I think I found out what is causing the failure in unit tests with Python 3.6. Triggering the RuntimeError in test_recursive, results in an endless loop.

Changing this line to:

'Logger {0!r} parents recursive'.format(l.name)

resolves the error.

The reason for this is that in order to construct the representation of the logger l, while generating the exception message, the getEffectiveLevel function never returns, since the logger's parent is the logger itself.

@georgepsarakis georgepsarakis force-pushed the georgepsarakis:feature/dynamodb-result-backend branch from e45b7d2 to b12ac9d Jan 21, 2017

@georgepsarakis

This comment has been minimized.

Copy link
Member Author

commented Jan 21, 2017

Some fixes:

  • Resolved unit tests issue with endless loop for Python 3.6.
  • Unit tests were being skipped due to missing dependency boto3, added to requirements/test-ci-default.txt.

Unfortunately, the documentation build job is failing, but I don't think that it is related because it was passing before. I also found this relevant Sphinx issue.

@georgepsarakis georgepsarakis force-pushed the georgepsarakis:feature/dynamodb-result-backend branch from 0a3dd58 to ab55e01 Jan 21, 2017

@georgepsarakis

This comment has been minimized.

Copy link
Member Author

commented Jan 21, 2017

Issue with Sphinx documentation build resolved using explicit hash seed value in tox environment.

@georgepsarakis georgepsarakis force-pushed the georgepsarakis:feature/dynamodb-result-backend branch from ab55e01 to e38e66c Jan 21, 2017

@thedrow thedrow merged commit 9c950b4 into celery:master Jan 23, 2017

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@thedrow

This comment has been minimized.

Copy link
Member

commented Jan 23, 2017

Thank you very much!

SebastianBerchtold pushed a commit to smartlane/celery that referenced this pull request Jun 11, 2018
AWS DynamoDB result backend (celery#3736)
* Add result backend for AWS DynamoDB

* Dependencies for DynamoDB result backend

* Add DynamoDB backend in aliases

* Test cases for DynamoDB result backend

* Documentation for DynamoDB backend

* Configurable endpoint URL for DynamoDB local instance

* Enable integration tests for DynamoDB result backend

- Run before_install script only for integration environments

* Fix invalid type error for primary key in Python3

* Add Python 3.6 in Travis CI build matrix

- Instruct Travis CI to include Python 3.6 interpreter in jobs
- Optimize Travis CI build matrix

* Optimize Travis CI build matrix

* Fix endless loop in logger_isa (Python 3.6)

* Add test cases for AWS client construction

- Add/improve log messages during table initialization
- Enable skipped unit tests due to missing dependency boto3

* Use explicit hash seed value for apicheck tox environment

- Related Sphinx issue: sphinx-doc/sphinx#2324
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.