Question: Unable to create the django_migrations table (database is locked) - When OUTPUT_DIR to SAMBA share

[stephenmetzger]

Hello,

Running ArchiveBox in docker (docker compose) am trying to output my archives to a samba share mounted on the archivebox host.

I am configuring the directories, and the docker-compose.yml file, then running the docker-compose run archivebox init --setup command. Once finished, I'm updating the ArchiveBox.conf file to contain OUTPUT_DIR = /mnt/archivebox (which is where i have my mounted SAMBA share directory mounted to the container from the host using a bind mount), and running the docker-compose up -d command.

In the logs, there is a long pause at [+] Building main SQL index and running initial migrations..., then a failure to load. Full log:

    > /mnt/archivebox
[+] Initializing a new ArchiveBox v0.6.2 collection...
----------------------------------------------------------------------
[+] Building archive folder structure...
    + ./archive, ./sources, ./logs...
    + ./ArchiveBox.conf...
[+] Building main SQL index and running initial migrations...
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/base/base.py", line 242, in _commit
    return self.connection.commit()
sqlite3.OperationalError: database is locked
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/db/migrations/recorder.py", line 68, in ensure_schema
    editor.create_model(self.Migration)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/sqlite3/schema.py", line 36, in __exit__
    super().__exit__(exc_type, exc_value, traceback)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/base/schema.py", line 117, in __exit__
    self.atomic.__exit__(exc_type, exc_value, traceback)
  File "/usr/local/lib/python3.9/site-packages/django/db/transaction.py", line 232, in __exit__
    connection.commit()
  File "/usr/local/lib/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/base/base.py", line 266, in commit
    self._commit()
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/base/base.py", line 242, in _commit
    return self.connection.commit()
  File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/base/base.py", line 242, in _commit
    return self.connection.commit()
django.db.utils.OperationalError: database is locked
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/bin/archivebox", line 33, in <module>
    sys.exit(load_entry_point('archivebox', 'console_scripts', 'archivebox')())
  File "/app/archivebox/cli/__init__.py", line 140, in main
    run_subcommand(
  File "/app/archivebox/cli/__init__.py", line 80, in run_subcommand
    module.main(args=subcommand_args, stdin=stdin, pwd=pwd)    # type: ignore
  File "/app/archivebox/cli/archivebox_server.py", line 64, in main
    server(
  File "/app/archivebox/util.py", line 114, in typechecked_function
    return func(*args, **kwargs)
  File "/app/archivebox/main.py", line 1243, in server
    run_subcommand('init', subcommand_args=['--quick'], stdin=None, pwd=out_dir)
  File "/app/archivebox/cli/__init__.py", line 80, in run_subcommand
    module.main(args=subcommand_args, stdin=stdin, pwd=pwd)    # type: ignore
  File "/app/archivebox/cli/archivebox_init.py", line 43, in main
�
(
  File "/app/archivebox/util.py", line 114, in typechecked_function
    return func(*args, **kwargs)
  File "/app/archivebox/main.py", line 328, in init
    for migration_line in apply_migrations(out_dir):
  File "/app/archivebox/util.py", line 114, in typechecked_function
    return func(*args, **kwargs)
  File "/app/archivebox/index/sql.py", line 138, in apply_migrations
    call_command("migrate", interactive=False, stdout=out)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 168, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 85, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 243, in handle
    post_migrate_state = executor.migrate(
  File "/usr/local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 91, in migrate
    self.recorder.ensure_schema()
  File "/usr/local/lib/python3.9/site-packages/django/db/migrations/recorder.py", line 70, in ensure_schema
    raise MigrationSchemaMissing("Unable to create the django_migrations table (%s)" % exc)
django.db.migrations.exceptions.MigrationSchemaMissing: Unable to create the django_migrations table (database is locked)

I have exact setup working succesfully working another machine with ArchiveBox running in docker, but cannot seem to get it to work on this machine (Ubuntu Server 21.10).

[stephenmetzger]

Not sure why I was getting the issue above, but ended up finding a workaround (probably the 'right' way to do it).

I started from scratch and ran a new --setup and docker-compose up with a typical configuration as outlined in the docs to get the package to get the directories and files set up, then edited my docker-compose.yml file to manually mount everything where I wanted it to live. My volumes section under archivebox now looks like this:

            - /mnt/BAlink/archivebox/archive:/data/archive
            - ./archivebox/data/ArchiveBox.conf:/data/ArchiveBox.conf
            - ./archivebox/data/index.sqlite3:/data/index.sqlite3
            - ./archivebox/data/logs:/data/logs
            - ./archivebox/data/sources:/data/sources

Any issues doing it this way?

[pirate]

Nope, that's a valid way to do it! But I recommend this instead so you don't have to hardcode those file paths:

            - ./archivebox/data:/data
            - /mnt/BAlink/archivebox/archive:/data/archive

[stephenmetzger]

Fantastic. Thanks. So, for docker generally, you can specify a subdirectory of a directory to be linked to a different path than the rest of the base-directory? Does the order they are listed matter?

[pirate]

Yes, and the order matters. Sub-path overrides for paths within a parent must be listed after the parent.

[pirate]

Note I've added a new DB/filesystem troubleshooting area to the wiki that may help people arriving here from Google: https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives#database-troubleshooting

Contributions/suggestions welcome there.

[p0n1]

Nope, that's a valid way to do it! But I recommend this instead so you don't have to hardcode those file paths:

            - ./archivebox/data:/data
            - /mnt/BAlink/archivebox/archive:/data/archive

This is working. When the index.sqlite3 created successfully in local, I just moved all files into /mnt/smb/archivebox and redeployed the container. The ArchiveBox could operate normally. No more database is locked issues. Not sure why.

BTW, I also tried NFS mounting and didn't encounter database locked issue.