Suggested layout when using NAS to store archive

[guidoism]

After years of playing around with ArchiveBox by creating little archives on removable disks and forgetting about them I finally have an environment capable of maintaining a growing archive: I have a TrueNAS server with lots of free space and a separate server capable of running ArchiveBox within Proxmox.

I know that it's common practice to run ArchiveBox in a VM on a NAS directly but I want storage separated from processes (I have a habit of screwing stuff up because there are so many interesting knobs to play around with so I've made an agreement with myself to not run anything on the NAS and not screw with any of the knobs in exchange for running a separate server that I can screw around with as much as I want).

At this point the NAS is working great. I have a SMB share that exposes only the archive dataset. And I have an Ubuntu LXC running inside Proxmox that mounts the dataset with CIFS.

I know that it is bad practice to store a SQLite database on a network store and indeed the archivebox init command fails during my experiments of running it directly on the share.

Therefore, what would be your suggestions for laying this out?

According to the disk layout we've got:

• The database (index.sqlite3)
• The config (ArchiveBox.conf)
• The list of all urls imported (sources/)
• The archive (archive/)

Clearly the archive should go on the network share. But what about the other stuff?

My guess is that both the config and the sources would be fine to put on the network share. Yes?

So, how about this layout:

/home/archive/archivebox/:
    archive -> /mnt/archivebox/archive
    sources -> /mnt/archivebox/sources
    ArchiveBox.config -> /mnt/archivebox/ArchiveBox.conf
    index.sqlite3 (local)

My plan is to rsync index.sqlite3 to the share after each run of archivebox add. Does this make sense to do?

[pirate]

Yup! That looks typical for an ArchiveBox NAS setup and is very similar to how I do it with NFS/SMB/ZFS on my TrueNAS box + experiments server at home.

My only suggestions are:

• also store the ArchiveBox.conf file locally and rsync it (ArchiveBox is picky about only writing to that file atomically to prevent race conditions from multiple processes editing it)
• make sure index.sqlite3-wal is also stored locally (you can rsync it but it's not required)
• configure these recommended ZFS options on the underlying ZFS dataset on the NAS
  • compression=lz4 (archived data compresses very well and lz4 uses very little extra CPU)
  • atime=off xattr=sa dnodesize=auto (common performance improvements that provide a small speed boost)
  • sync=standard (or disabled for much faster performance if you dont mind a few seconds of data loss on power outages)
  • acltype=posixacl + aclinherit=passthrough (makes sure permissions work reliably across Linux, Docker, and macOS)
  • utf8only=on + normalization=formD (prevent weird emoji/unicode in archived URLs from creating bad filesystem paths)
  • casesensitivity=insensitive (optional but recommended, ArchiveBox doesn't rely on case-sensitivity)
  • autoexpand=on (auto-resize the dataset if you add more vdev disk storage to the pool)
  • once it's released, you should also set up ZFS's brand-new Fast Dedupe feature, for now you can dedupe archive data using fdupes or rdfind

If you run into permissions problems or IO errors, the foolproof method is to store everything except ./archive locally, sometimes there are issues with sources/ and logs/ writes timing out on laggy network mounts or having their permissions messed with by all_squash/root_squash NFS options.

---

Some historical context:

In general the latest ArchiveBox versions are deliberately quite strict about their filesystem expectations. Poorly setup NASs have let to tons of subtle bugs in the past, so I've decided to enforce more and more of the FS requirements as our userbase has increased.

It has technically always been possible to store everything (including index.sqlite3) on a network mount, but you have to change some advanced NFS/SMB settings on the server to get multi-client FSYNCs working safely, and even then the risk of data loss is higher. After some people corrupted their indexes trying to run multiple instances on a shared network DB, I started steering people away from that option and adding warnings to prevent it. If you're an NFS/SMB/ZFS expert it is still possible to do however, but you have to know what options to tweak and be willing to take on the risk yourself.

---

P.S. We have a new TrueNAS chart that's in the incubator train if you ever consider running it directly on your NAS it would help us to have more users beta testing it :)

• add(archivebox): Add new ArchiveBox chart  truecharts/charts#18258
• Bug: Can't init archivebox on FreeBSD iocage jail #760 (comment)