Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Enhancement/Export data via Celery #432
Currently doccano's data export happens within the HTTP request/response cycle. However when the exported datasets are large, the export can take some time to run which often leads to a HTTP timeout. To fix this problem, this pull request moves the data export into a background worker implemented with Celery.
For production setups, the Celery worker can be run as a stand-alone container and the Django server can communicate to the worker via an external queue such as RabbitMQ (this is demonstrated in docker-compose.yml). However, to also keep offering a simple one click deployment model, this pull request additionally implements running both the Django server and the Celery worker inside of the same Docker container, communicating via the local SQLite database.
In the future, other long-running operations (e.g. data import) could also be moved to Celery to improve performance.