borg create throws os.fsync error on cifs mounted repo #1287

alraban · 2016-07-08T13:49:35Z

Observed both on Arch linux systems running the binary package and Debian GNU/Linux systems compiling from pip. With borg 1.05, any attempt to create a new archive in a repository that is accessible via cifs begins to run the create and then errors as soon as it reaches the first new file that needs to be written. Backups to purely local filesystems or remote backups via ssh with the same repositories appear to work perfectly.

The systems in question had no similar problem with any prior versions of borg, and a repository check on the relevant repositories comes back clean.

An example of the errors thrown:

Exception ignored in: <bound method Repository.__del__ of <Repository /mnt/backups/Saturnine.borg>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/borg/repository.py", line 72, in __del__
    self.close()
  File "/usr/local/lib/python3.4/dist-packages/borg/repository.py", line 189, in close
    self.io.close()
  File "/usr/local/lib/python3.4/dist-packages/borg/repository.py", line 532, in close
    self.close_segment()
  File "/usr/local/lib/python3.4/dist-packages/borg/repository.py", line 752, in close_segment
    self._write_fd.flush()
ValueError: flush of closed file
Local Exception.
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 81, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 247, in do_create
    create_inner(archive, cache)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 219, in create_inner
    read_special=args.read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 301, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 301, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 301, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 301, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 301, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 301, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 301, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 301, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 301, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 301, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 301, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 301, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 275, in _process
    status = archive.process_file(path, st, cache, self.ignore_inode)
  File "/usr/local/lib/python3.4/dist-packages/borg/archive.py", line 681, in process_file
    self.add_item(item)
  File "/usr/local/lib/python3.4/dist-packages/borg/archive.py", line 279, in add_item
    self.items_buffer.add(item)
  File "/usr/local/lib/python3.4/dist-packages/borg/archive.py", line 133, in add
    self.flush()
  File "/usr/local/lib/python3.4/dist-packages/borg/archive.py", line 148, in flush
    self.chunks.append(self.write_chunk(chunk))
  File "/usr/local/lib/python3.4/dist-packages/borg/archive.py", line 164, in write_chunk
    id_, _, _ = self.cache.add_chunk(self.key.id_hash(chunk), chunk, self.stats)
  File "/usr/local/lib/python3.4/dist-packages/borg/cache.py", line 365, in add_chunk
    self.repository.put(id, data, wait=False)
  File "/usr/local/lib/python3.4/dist-packages/borg/repository.py", line 481, in put
    segment, offset = self.io.write_put(id, data)
  File "/usr/local/lib/python3.4/dist-packages/borg/repository.py", line 721, in write_put
    fd = self.get_write_fd(raise_full=raise_full)
  File "/usr/local/lib/python3.4/dist-packages/borg/repository.py", line 608, in get_write_fd
    self.close_segment()
  File "/usr/local/lib/python3.4/dist-packages/borg/repository.py", line 759, in close_segment
    sync_dir(os.path.dirname(self._write_fd.name))
  File "/usr/local/lib/python3.4/dist-packages/borg/platform.py", line 9, in sync_dir
    os.fsync(fd)
OSError: [Errno 22] Invalid argument

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 1601, in main
    exit_code = archiver.run(args)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 1538, in run
    return args.func(args)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 81, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/borg/repository.py", line 94, in __exit__
    self.close()
  File "/usr/local/lib/python3.4/dist-packages/borg/repository.py", line 189, in close
    self.io.close()
  File "/usr/local/lib/python3.4/dist-packages/borg/repository.py", line 532, in close
    self.close_segment()
  File "/usr/local/lib/python3.4/dist-packages/borg/repository.py", line 752, in close_segment
    self._write_fd.flush()
ValueError: flush of closed file

Platform: Linux Saturnine 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2+deb8u2 (2016-06-25) x86_64 
Linux: debian 8.5 
Borg: 1.0.5  Python: CPython 3.4.2
PID: 19650  CWD: /home/alraban
sys.argv: ['/usr/local/bin/borg', 'create', '-sv', '-C', 'lz4', '-e', '/proc/', '-e', '/sys/', '-e', '/run/', '-e', '/tmp/', '-e', '/dev/', '-e', '/mnt/', '-e', '/home/pi/backups', '/mnt/backups/Saturnine.borg::2016-07-08-09', '/']

The error on the Arch systems is substantially similar. Let me know if I can provide any additional information or testing.

The text was updated successfully, but these errors were encountered:

ThomasWaldmann · 2016-07-08T15:11:56Z

Looks like smbfs does not like a flush on a directory fd (this is new in 1.0.4/1.0.5 to make sure the directory is really on-disk to be more robust, e.g. in case of crashes or power failures).

Workaround: go back to 1.0.3 until a fix is available.

Can you test a sourcecode patch for 1.0.5?

ThomasWaldmann · 2016-07-08T15:16:44Z

Maybe we just use the diaper pattern here.

enkore · 2016-07-08T15:17:09Z

(Previous comment deleted, since not accurate).

Ignoring EINVAL should be ok. sshfs issue isn't related, just tested. I have no traceback for that one, just an error code, so ...

ThomasWaldmann · 2016-07-08T15:18:35Z

@enkore can we be sure that EINVAL is all we get to see from this? And not some other fs throwing another error and it blows up again?

ThomasWaldmann · 2016-07-08T15:19:43Z

btw, we need more beta testers.

enkore · 2016-07-08T15:20:55Z

fsync has EBADF¹, EINTR², EINVAL, EIO³ (and those of read/write with the same meanings). Anything else means the FS/Kernel is severely broken.

¹ can't happen - open succeeded
² handled by Python.
³ don't want to shadow EIO

enkore · 2016-07-08T15:21:17Z

btw, we need more beta testers.

Yes.

Also a checklist when we touch IO code. Works on proper FS. Works on NTFS. Works on non-compliant FS (sshfs, cifs/smb, nfs, ...).

enkore · 2016-07-08T15:24:44Z

@alraban can you test #1288?

alraban · 2016-07-08T17:11:21Z

Ok, just ran a few tests with the patch; the patched version appears to work normally now on cifs repositories. Thanks much for jumping right on this! Let me know if I can do any additional testing or provide any more info.

enkore · 2016-07-08T17:18:54Z

Thanks for the report and the patch test :)

alraban changed the title ~~Borg 1.05 create errors and fails to write to repositories on cifs mounts~~ Borg 1.05 - create command throws error and fails to write to repositories on cifs mounts Jul 8, 2016

ThomasWaldmann added the bug label Jul 8, 2016

ThomasWaldmann added this to the 1.0.6 fixes for 1.0-maint milestone Jul 8, 2016

enkore self-assigned this Jul 8, 2016

enkore mentioned this issue Jul 8, 2016

sync_dir: silence fsync() failing with EINVAL #1288

Merged

ThomasWaldmann changed the title ~~Borg 1.05 - create command throws error and fails to write to repositories on cifs mounts~~ borg create throws os.fsync error in sync_dir on cifs mounts Jul 8, 2016

ThomasWaldmann changed the title ~~borg create throws os.fsync error in sync_dir on cifs mounts~~ borg create throws os.fsync error on cifs mounted repo Jul 8, 2016

enkore mentioned this issue Jul 8, 2016

Test on many file systems automatically #1289

Open

enkore closed this as completed Jul 8, 2016

enkore mentioned this issue Jul 12, 2016

flush of closed file #1316

Closed

enkore mentioned this issue Aug 26, 2016

traceback-less messaging for OSErrors #1534

Closed

zeebo mentioned this issue May 8, 2018

thousands of .tsm files created with "invalid argument" error, disk filled influxdata/influxdb#9630

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

borg create throws os.fsync error on cifs mounted repo #1287

borg create throws os.fsync error on cifs mounted repo #1287

alraban commented Jul 8, 2016 •

edited

ThomasWaldmann commented Jul 8, 2016 •

edited

ThomasWaldmann commented Jul 8, 2016

enkore commented Jul 8, 2016 •

edited

ThomasWaldmann commented Jul 8, 2016

ThomasWaldmann commented Jul 8, 2016

enkore commented Jul 8, 2016 •

edited

enkore commented Jul 8, 2016 •

edited

enkore commented Jul 8, 2016

alraban commented Jul 8, 2016

enkore commented Jul 8, 2016 •

edited

borg create throws os.fsync error on cifs mounted repo #1287

borg create throws os.fsync error on cifs mounted repo #1287

Comments

alraban commented Jul 8, 2016 • edited

ThomasWaldmann commented Jul 8, 2016 • edited

ThomasWaldmann commented Jul 8, 2016

enkore commented Jul 8, 2016 • edited

ThomasWaldmann commented Jul 8, 2016

ThomasWaldmann commented Jul 8, 2016

enkore commented Jul 8, 2016 • edited

enkore commented Jul 8, 2016 • edited

enkore commented Jul 8, 2016

alraban commented Jul 8, 2016

enkore commented Jul 8, 2016 • edited

alraban commented Jul 8, 2016 •

edited

ThomasWaldmann commented Jul 8, 2016 •

edited

enkore commented Jul 8, 2016 •

edited

enkore commented Jul 8, 2016 •

edited

enkore commented Jul 8, 2016 •

edited

enkore commented Jul 8, 2016 •

edited