Export taking very long

[Ivowings]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Mint 20.1
• Using the docker image

Describe the problem

I annotated around 5000(1080808 total samples )text classification samples in Doccano. I want to export these. This is however, taking very long. I left my computer running over night but is still loading. Is there a different method for extracting the annotations or am i doing something wrong?

[github-actions]

Would you write your environment? Thank you!

[iwmaas]

Found a workaround for anyone experiencing the same issue. In the docker container is a file called 'doccano.db'. This is an sqlite file from which I've extracted the annotations.

[mehmetilker]

Having same problem.
-Fresh install with pip on Ubuntu 18 (WSL)
-Uploaded 20.000 document consisted of short paraghraph + half annotated.
-Edited 250 doc to annotate.
-Tried to export.

I am seeing following exception:
Database is locked keeps repeation until I break with ctrl+c. (I have tried 2 times.)

[2021-06-04 09:22:56,707: INFO/ForkPoolWorker-2] Task api.tasks.export_dataset[6f28a853-9c90-4364-bc79-1a8ade42464d] succeeded in 22.551829000003636s: '/home/my-name/projects/x-docanno/.env/lib/python3.8/site-packages/backend/media/5e75d8d6-6088-49d3-8d17-7adbf8d5ebd6.zip'
[2021-06-04 09:23:02,859: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/kombu/transport/sqlalchemy/__init__.py", line 178, in _get
    raise Empty()
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 990, in _commit_impl
    self.engine.dialect.do_commit(self.connection)
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 670, in do_commit
    dbapi_connection.commit()
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/celery/worker/consumer/consumer.py", line 326, in start
    blueprint.start(self)
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/celery/worker/consumer/consumer.py", line 618, in start
    c.loop(*c.loop_args())
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/celery/worker/loops.py", line 111, in synloop
    connection.drain_events(timeout=2.0)
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/kombu/connection.py", line 318, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 960, in drain_events
    get(self._deliver, timeout=timeout)
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/kombu/utils/scheduling.py", line 52, in get
    return self.fun(resource, callback, **kwargs)
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 998, in _drain_channel
    return channel.drain_events(callback=callback, timeout=timeout)
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 742, in drain_events
    return self._poll(self.cycle, callback, timeout=timeout)
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 399, in _poll
    return cycle.get(callback)
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/kombu/utils/scheduling.py", line 52, in get
    return self.fun(resource, callback, **kwargs)
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 402, in _get_and_deliver
    message = self._get(queue)
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/kombu/transport/sqlalchemy/__init__.py", line 180, in _get
    self.session.commit()
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1428, in commit
    self._transaction.commit(_to_root=self.future)
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 836, in commit
    trans.commit()
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2330, in commit
    self._do_commit()
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2513, in _do_commit
    self._connection_commit_impl()
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2484, in _connection_commit_impl
    self.connection._commit_impl()
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 992, in _commit_impl
    self._handle_dbapi_exception(e, None, None, None, None)
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1994, in _handle_dbapi_exception
    util.raise_(
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 990, in _commit_impl
    self.engine.dialect.do_commit(self.connection)
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 670, in do_commit
    dbapi_connection.commit()
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
(Background on this error at: http://sqlalche.me/e/14/e3q8)
[2021-06-04 09:23:02,885: WARNING/MainProcess] /home/my-name/projects/my-project/.env/lib/python3.8/site-packages/celery/worker/consumer/consumer.py:361: CPendingDeprecationWarning:
In Celery 5.1 we introduced an optional breaking change which
on connection loss cancels all currently executed tasks with late acknowledgement enabled.
These tasks cannot be acknowledged as the connection is gone, and the tasks are automatically redelivered back to the queue.
You can enable this behavior using the worker_cancel_long_running_tasks_on_connection_loss setting.
In Celery 5.1 it is set to False by default. The setting will be set to True by default in Celery 6.0.

  warnings.warn(CANCEL_TASKS_BY_DEFAULT, CPendingDeprecationWarning)

[2021-06-04 09:23:02,900: INFO/MainProcess] Connected to sqla+sqlite:////home/my-name/projects/my-project/.env/lib/python3.8/site-packages/backend/db.sqlite3
[2021-06-04 09:23:07,929: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1770, in _execute_context
    self.dialect.do_execute(
  File "/home/my-name/projects/my-project/.env/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 717, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: database is locked



^C
worker: Hitting Ctrl+C again will terminate all running tasks!

worker: Warm shutdown (MainProcess)
 my-name    my-project  羽1m 51.23s 