Backup error: attempt to write a readonly database

[PAndreini]

Hi!

First of all, thank you for your work and for the useful scripts!

I have this problem: most of the times the backup fails with the error "Error: attempt to write a readonly database".
The strange thing is that sometimes the backup completes successfully.
The cron job runs under my user who has read/write permissions on the backup folder.
This is what is logged:

+ cd /home/paolo/vaultwarden_dock
+ mkdir -p /home/paolo/vaultwarden_dock/backup/vaultwarden-20211015-1129
+ max_tries=10
+ tries=0
+ /usr/bin/sqlite3 'file:vw-data/db.sqlite3?mode=ro' '.backup '\''/home/paolo/vaultwarden_dock/backup/vaultwarden-20211015-1129/db.sqlite3'\'''
Error: attempt to write a readonly database
+ ((  ++tries >= max_tries  ))
+ echo 'Backup failed. Retry #1...'
Backup failed. Retry #1...

And the error repeats for the remaining retries.

Into the backup root folder I can see the temporary backup folder ('vaultwarden-20211015-1129' in the logged case) and a zero dimension 'db.sqlite3' file into it.

Any ideas what's the problem?

Thank you!

[jjlin]

It does indeed seem strange that it would work sometimes, and not other times. Are there any network file systems (like NFS or Samba/CIFS) involved?

[PAndreini]

No, there are not network file system involved, it's a local disk.
This is a screenshot of the last backup files:

As you can see, in the last weekend there are no successful backup. The last successful backup is today at 11:00 (the cron job runs every hour). I didn't made any changes in the meantime.

I've also tried to modify the zero length db.sqlite3 file created by the failed backup job and I can save the changes.

I think it could be something related to file access permissions, because if I run the backup command with the root user, the backup completes (except of course the rclone steps, cause it's not configured), but I can't figure out what.

Any ideas?

[PAndreini]

Yeah, that looks fine. Is there anything at all unusual about this machine's hardware? Does it run any unusual software like antivirus or any kind of file watching program that might interfere with locking? When it succeeds, is it on the first try or after one or more retries?

Sorry I saw your reply after my last post.
No, there are no antivirus or other monitor software installed. It's a virtual machine dedicated to vaultwarden. It's a Debian distribution with base software installation, where I've installed only the necessary software to run vaultwarden and the backup solution. Neither desktop gui.
When it works, it works at the first of the ten retries.

[jjlin]

Try replacing

vaultwarden-backup/backup.sh

Line 34 in 05b8ef6

until ${SQLITE3} "file:${DATA_DIR}/${DB_FILE}?mode=ro" ".backup '${BACKUP_DIR_PATH}/${DB_FILE}'"; do

with

until strace ${SQLITE3} "file:${DATA_DIR}/${DB_FILE}?mode=ro" ".backup '${BACKUP_DIR_PATH}/${DB_FILE}'" 2>/tmp/strace.$(date +%s); do

and wait for a failure. Or if the issue is easy to reproduce on demand, you can just run the straced command manually.

Hopefully the strace output will provide some insight into what's happening. Not sure how large the log will end up being, but it would probably be cleaner to include it as an attachment rather than pasting.

[PAndreini]

strace.log

Bingo! I think you have found the way to uncover the cause of the problem, but I also wait for your opinion.

It seems that the backup command tries to write the file 'db.sqlite3-wal' into the data folder, where neither my user nor the vaultwarden group (that I'm member of) have write permission.
I gave the write permission to vaultwarden group, and the next two backups jobs completed successfully!

I'm still not sure why the job sometimes worked... It could be that in some cases the backup does not need to write this file?

[jjlin]

If the WAL file is already there, then it wouldn't need to create it. I'm not familiar enough with the SQLite internals to understand why it really needs to create that file in read-only mode, but presumably there's a reason, and anyway, it turns out this behavior is documented at https://sqlite.org/wal.html#read_only_databases.

In any case, it's good that there's a resolution. I've added a note to README.md mentioning this pitfall. Thanks for helping to get to the bottom of it.

[PAndreini]

Thanks to you for your help, your time and your really useful scripts!