Skeleton for remote query execution using celery.

[bkyryliuk]

Current change adds ability to execute the SQL queries on the celery worker.
More info about celery can be found here:

• http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html

Worker can be started via command caravel worker
Celery is configured in the config.py and by default it is turned off. (CELERY_CONFIG = None)
Fixed some lint issues in the tests.

Tested:

1. Manually tested if the query workes fine with the CELERY_CONFIG = None and with

class CeleryConfig(object):
    BROKER_URL = 'sqla+sqlite:///celerydb.sqlite'
    CELERY_IMPORTS = ('caravel.tasks', )
    CELERY_RESULT_BACKEND = 'db+sqlite:///celery_results.sqlite'
    CELERY_ANNOTATIONS = {'tasks.add': {'rate_limit': '10/s'}}
CELERY_CONFIG = CeleryConfig

Task is passed and executed:

Query works for both cases:

Reviewers:

• @georgeke
• @mistercrunch
• @ascott

[mistercrunch]

i think you could just have a single function get_sql_results decorated with @celery_app.task, and you can call it with .async where needed (on an alternative endpoint)

[mistercrunch]

I wouldn't put this in module scope, it will print on import.

[bkyryliuk]

haven't cleaned up after debugging, sorry.

[mistercrunch]

That would assume waiting for the table which isn't what we want to do here, we can remove 1302 and 1303

[bkyryliuk]

Updated todo to be:
# TODO(bkyryliuk): add query id to the response and implement the endpoint to poll the status and results.

[mistercrunch]

+trailing comma

[coveralls]

Coverage decreased (-0.04%) to 81.435% when pulling 15a23d3 on bkyryliuk:celery into 572c6ee on airbnb:master.

[coveralls]

Coverage decreased (-0.04%) to 81.435% when pulling 15a23d3 on bkyryliuk:celery into 572c6ee on airbnb:master.

[coveralls]

Coverage decreased (-0.04%) to 81.435% when pulling 15a23d3 on bkyryliuk:celery into 572c6ee on airbnb:master.