Question: How to run AB on localhost but store data on NAS?

[iwconfig]

Hello!

I'm using docker-compose. Following the title if this issue: I first tried changing the whole data dir to a mounted path pointing to my NAS, but I got this error:

  archivebox_1  | [X] OSError: Failed to write /data/ArchiveBox.conf with fcntl.F_FULLFSYNC. ([Errno 22] Invalid argument)
  archivebox_1  |     You can store the archive/ subfolder on a hard drive or network share that doesn't support support syncronous writes,
  archivebox_1  |     but the main folder containing the index.sqlite3 and ArchiveBox.conf files must be on a filesystem that supports FSYNC.

I cannot figure out how to separate data/archive/ from the rest as I wish to store this on my NAS. I tried symlinking it but it just complains that data/archive/ already exists.

I would prefer to have as much data on my NAS so all or any of data/{logs,sonic,sources}/ as well.

How do I setup this with docker-compose?

Thank you kindly! in advance.

[pirate]

Symlink works but you have to mount the destination of the symlink in docker-compose as a volume as well (to the same location the symlink is pointing to, except inside the container).

[iwconfig]

I'm sorry, I don't understand this. Can you give me an example?

Since ArchiveBox.conf and index.sqlite3 needs to be on host and not on NAS, how do I separate them from the rest?

I've tried many different variations, for example:

volumes:
    - ./data:/data
    - /mnt/arkivet/ArchiveBox-data/archive:/mnt/arkivet/ArchiveBox-data/archive
    - /mnt/arkivet/ArchiveBox-data/sources:/mnt/arkivet/ArchiveBox-data/sources
    - /mnt/arkivet/ArchiveBox-data/logs:/mnt/arkivet/ArchiveBox-data/logs

But this doesn't work as the folders are created in host ./data. I don't know when I should create my symlinks either because AB complains if a folder (symlink) already exist or creates the folders before I have a chance to create my symlinks. It doesn't matter if I remove folder(s) and then create my symlink(s), it still complains about what I believe is FSYNC.

What am I missing?

Traceback (most recent call last):
  File "/app/archivebox/system.py", line 90, in atomic_write
    f.write(contents)
  File "/usr/local/lib/python3.9/contextlib.py", line 124, in __exit__
    next(self.gen)
  File "/app/archivebox/vendor/atomicwrites.py", line 172, in _open
    self.commit(f)
  File "/app/archivebox/vendor/atomicwrites.py", line 205, in commit
    replace_atomic(f.name, self._path)
  File "/app/archivebox/vendor/atomicwrites.py", line 99, in replace_atomic
    return _replace_atomic(src, dst)
  File "/app/archivebox/vendor/atomicwrites.py", line 56, in _replace_atomic
    _sync_directory(os.path.normpath(os.path.dirname(dst)))
  File "/app/archivebox/vendor/atomicwrites.py", line 50, in _sync_directory
    _proper_fsync(fd)
OSError: [Errno 22] Invalid argument

[cmuench]

@iwconfig I was able to share my archive directory on my Synology NAS.
I mouted the archive directory directly from NFS.

version: '2.4'

services:
    archivebox:        
        image: ${DOCKER_IMAGE:-archivebox/archivebox:master}
        command: server --quick-init 0.0.0.0:8000
        restart: unless-stopped
        environment:
            - ALLOWED_HOSTS=*
            - MEDIA_MAX_SIZE=750m
        volumes:
            - ./volumes/data:/data
            - archivebox-archive:/data/archive

volumes:
  archivebox-archive:
      driver: local
      driver_opts:
        type: "nfs"
        o: "nfsvers=4,addr=192.168.1.12,rw"
        device: ":/volume1/archivebox/archive"

Difference is that I used docker to create the mount instead of fstab. Hope that also works on your side.

[iwconfig]

Thank you so much! I got it working now 😃 I just had one issue with permissions inside the docker container, so I had to do

docker exec -it archivebox_archivebox_1 chmod -R 777 /data/archive

Awesome! Again, thank you!

@pirate Maybe this should be mentioned in the documentation? If it already is, I'm sorry, I haven't seen it.

[pirate]

Note I've added a new DB/filesystem troubleshooting area to the wiki that may help people arriving here from Google: https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives#database-troubleshooting

Contributions/suggestions welcome there.