Borg Backup does not play nice with rclone-mounted repositories

[RaitoBezarius]

Describe the bug
ReadWritePaths set up in services.borgbackup does not play really nice with fuse-mounted endpoints and create read-only filesystems.

To Reproduce
Steps to reproduce the behavior:

1. Have a fuse-mounted rclone endpoint somewhere.
2. Create a borg repository there.
3. Use services.borgbackup to set up automatic backups.
4. Observe in journalctl failure due to read-only FS.

Expected behavior
The endpoint should not be read-only during the execution of the systemd script.

Additional context
Here's some logs:

Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]: Exception ignored in: <function Repository.__del__ at 0x7f7e458935f0>
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]: Traceback (most recent call last):
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/repository.py", line 179, in __del__
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     self.close()
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/repository.py", line 446, in close
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     self.lock.release()
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/locking.py", line 383, in release
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     self._roster.modify(EXCLUSIVE, REMOVE)
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/locking.py", line 286, in modify
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     self.save(roster)
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/locking.py", line 257, in save
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     with open(self.path, "w") as f:
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]: OSError: [Errno 30] Read-only file system: '/run/rclone/vieuxtype-backup/system-wide/lock.roster'
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]: Local Exception
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]: Traceback (most recent call last):
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/archiver.py", line 166, in wrapper
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     return method(self, args, repository=repository, **kwargs)
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/archiver.py", line 574, in do_create
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     create_inner(archive, cache)
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/archiver.py", line 537, in create_inner
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     read_special=args.read_special, dry_run=dry_run, st=st)
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/archiver.py", line 651, in _process
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     read_special=read_special, dry_run=dry_run)
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/archiver.py", line 651, in _process
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     read_special=read_special, dry_run=dry_run)
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/archiver.py", line 651, in _process
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     read_special=read_special, dry_run=dry_run)
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/archiver.py", line 625, in _process
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     status = archive.process_file(path, st, cache)
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/archive.py", line 1071, in process_file
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     self.chunk_file(item, cache, self.stats, backup_io_iter(self.chunker.chunkify(fd, fh)))
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/archive.py", line 999, in chunk_file
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     item.chunks.append(chunk_processor(data))
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/archive.py", line 987, in chunk_processor
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     chunk_entry = cache.add_chunk(self.key.id_hash(data), data, stats, wait=False)
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/cache.py", line 897, in add_chunk
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     data = self.key.encrypt(chunk)
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/crypto/key.py", line 370, in encrypt
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     self.nonce_manager.ensure_reservation(num_aes_blocks(len(data)))
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/crypto/nonces.py", line 85, in ensure_reservation
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     self.commit_repo_nonce_reservation(reservation_end, repo_free_nonce)
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/crypto/nonces.py", line 48, in commit_repo_nonce_reservation
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     self.repository.commit_nonce_reservation(next_unreserved, start_nonce)
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/repository.py", line 345, in commit_nonce_reservation
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     with SaveFile(nonce_path, binary=False) as fd:
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/platform/base.py", line 167, in __enter__
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     self.fd = platform.SyncFile(self.tmppath, self.binary)
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/platform/base.py", line 106, in __init__
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     self.fd = open(path, mode)
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]: OSError: [Errno 30] Read-only file system: '/run/rclone/vieuxtype-backup/system-wide/nonce.tmp'
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]: During handling of the above exception, another exception occurred:
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]: Traceback (most recent call last):
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/archiver.py", line 4529, in main
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     exit_code = archiver.run(args)
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/archiver.py", line 4461, in run
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     return set_ec(func(args))
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/archiver.py", line 166, in wrapper
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     return method(self, args, repository=repository, **kwargs)
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/repository.py", line 204, in __exit__
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     self.close()
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/repository.py", line 446, in close
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     self.lock.release()
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/locking.py", line 383, in release
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     self._roster.modify(EXCLUSIVE, REMOVE)
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/locking.py", line 286, in modify
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     self.save(roster)
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:   File "/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/lib/python3.7/site-packages/borg/locking.py", line 257, in save
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]:     with open(self.path, "w") as f:
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]: OSError: [Errno 30] Read-only file system: '/run/rclone/vieuxtype-backup/system-wide/lock.roster'
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]: Platform: Linux vieuxtype 5.4.28 #1-NixOS SMP Wed Mar 25 07:26:00 UTC 2020 x86_64
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]: Linux:
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]: Borg: 1.1.11  Python: CPython 3.7.6 msgpack: 0.5.6
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]: PID: 16073  CWD: /
Apr 06 16:44:26 vieuxtype 7620pyksvzgcsv1hin7vhpzi6yl9a3cs-unit-script-borgbackup-job-rootBackup-start[16070]: sys.argv: ['/nix/store/qvzazarkwxd3fqvi90f3ykp5c7hc5i3c-borgbackup-1.1.11/bin/borg', 'create', '--compression', 'auto,lzma', '--exclude-from', '/nix/store/axchncibn3id6p1v7x2z2k6xrl2hlafn-excludefile', '::vieuxtype-rootBackup-2020-04-06T16:44:25.failed', '/']

Metadata

- system: `"x86_64-linux"`
 - host os: `Linux 5.4.28, NixOS, 20.09pre218613.ae6bdcc5358 (Nightingale)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.3`
 - channels(root): `"nixos-20.09pre218613.ae6bdcc5358"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Maintainer information:

module: services.borgbackup

cc @dotlambda

[dotlambda]

I think the solution is to add systemd.services.borgbackup-job-rootBackup.after = [ "mount-rclone.service" ], where mount-rclone.service is the name of the systemd service that mounts your destination using rclone.

[stale]

I marked this as stale due to inactivity. → More info