Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhancement/Export data via Celery #432

wants to merge 5 commits into
base: master


Copy link

c-w commented Oct 31, 2019

Currently doccano's data export happens within the HTTP request/response cycle. However when the exported datasets are large, the export can take some time to run which often leads to a HTTP timeout. To fix this problem, this pull request moves the data export into a background worker implemented with Celery.

For production setups, the Celery worker can be run as a stand-alone container and the Django server can communicate to the worker via an external queue such as RabbitMQ (this is demonstrated in docker-compose.yml). However, to also keep offering a simple one click deployment model, this pull request additionally implements running both the Django server and the Celery worker inside of the same Docker container, communicating via the local SQLite database.

In the future, other long-running operations (e.g. data import) could also be moved to Celery to improve performance.

Resolves #241

@c-w c-w force-pushed the CatalystCode:enhancement/celery branch from cf7207b to f79d97e Nov 21, 2019
@Hironsan Hironsan added this to To do in v1.0.2 Nov 22, 2019
@Hironsan Hironsan moved this from To do to In progress in v1.0.2 Nov 26, 2019
@c-w c-w force-pushed the CatalystCode:enhancement/celery branch from 1f66818 to 84f6fd2 Dec 1, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
In progress
1 participant
You can’t perform that action at this time.