New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Input/output error on sshfs #583

Closed
bpereto opened this Issue Jan 20, 2016 · 27 comments

Comments

Projects
None yet
5 participants
@bpereto
Copy link

bpereto commented Jan 20, 2016

Hi,

I have mounted an sshfs filesystem and trying to use borg on it. every time i want to init the repository, a locking error appeared.

[root@wall-e data]# borg init --debug /data/backup/test
using builtin fallback logging configuration
Using a pure-python msgpack! This will result in lower performance.
Initializing repository at "/data/backup/test"
Encryption NOT enabled.
Use the "--encryption=repokey|keyfile" to enable encryption.
Synchronizing chunks cache...
Archives: 0, w/ cached Idx: 0, w/ outdated Idx: 0, w/o cached Idx: 0.
Done.
Exception ignored in: <bound method Repository.__del__ of <Repository /data/backup/test>>
Traceback (most recent call last):
  File "/usr/lib64/python3.4/site-packages/borg/repository.py", line 66, in __del__
    self.close()
  File "/usr/lib64/python3.4/site-packages/borg/repository.py", line 158, in close
    self.lock.release()
  File "/usr/lib64/python3.4/site-packages/borg/locking.py", line 296, in release
    self._lock.release()
  File "/usr/lib64/python3.4/site-packages/borg/locking.py", line 153, in release
    os.unlink(self.unique_name)
OSError: [Errno 5] Input/output error: '/data/backup/test/lock.exclusive/wall-e.21040-0'

he repository is created and the create command returning:

using builtin fallback logging configuration
Using a pure-python msgpack! This will result in lower performance.
Failed to create/acquire the lock /data/backup/test/lock.exclusive (timeout).

my sshfs fuse mount options:
auto_cache,reconnect,allow_root,idmap=user,uid=0,gid=0,comment=sshfs,transform_symlinks,exec,auto

any idea?

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Jan 20, 2016

os.unlink is just a "delete <filename>" command. if that gives a Input/Output error, your filesystem has problems. can you try without special mount options?

also (unrelated): your msgpack is slow, because it is not compiled correctly.

@bpereto

This comment has been minimized.

Copy link

bpereto commented Jan 20, 2016

same without special mount options.

(unrelated) installed msgpack via pip install msgpack-python

@bpereto

This comment has been minimized.

Copy link

bpereto commented Jan 21, 2016

Disabling the sshfs cache with various mount options did the trick.
I will test this further, if this options have impact on the backup progress.

this sshfs mount options are working for me:
reconnect,allow_other,idmap=user,uid=0,gid=0,cache=no,noauto_cache,entry_timeout=0

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Jan 21, 2016

about uncompiled msgpack: maybe you have no compiler installed or some other dependency is missing. the pip log or screen output might show some hints.

@bpereto

This comment has been minimized.

Copy link

bpereto commented Jan 21, 2016

I runned various backups with the mount options above and it has not shown an error. also the borg check is ok.

Seems to be working for me.

pip does not show any useful output. runned setup.py manually and you were right, it was missing gcc-c++.

@bpereto bpereto closed this Jan 21, 2016

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Jan 21, 2016

can you please give your operating system (and distribution) and version?

so we can check if we need to add something to the docs.

@bpereto

This comment has been minimized.

Copy link

bpereto commented Jan 21, 2016

Platform: Linux 3.10.0-327.4.4.el7.x86_64 #1 SMP Tue Jan 5 16:07:00 UTC 2016 x86_64 x86_64
Linux: CentOS Linux 7.2.1511 Core LibC: glibc 2.3.4
Python: CPython 3.4.3

@espHorst

This comment has been minimized.

Copy link

espHorst commented Mar 19, 2016

I had the same problem with revision 1.0.0 on cygwin.
pip install msgpack-python --verbose
showed that gcc throwed an error "spawn" is missing.
Therefore I added the module gcc-g++ instead of the proposed gcc-core and
everything is fine now.

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Mar 19, 2016

@ikog so on cygwin it is enough to install gcc-g++ (and it will also install all other needed gcc stuff automatically)?

@espHorst

This comment has been minimized.

Copy link

espHorst commented Mar 20, 2016

I'm not sure as I already had installed gcc-core on this machine.
I will try today on another windows machine to install only gcc-g++.
I wil try to install related to https://borgbackup.readthedocs.org/en/stable/installation.html)

python3 python3-setuptools
binutils gcc-g++
libopenssl openssl-devel
liblz4_1 liblz4-devel # from cygwinports.org
git make openssh

replacing ggc-core with gcc-g++. I will report success/fail later on.
Should I try to drop another libary?

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Mar 20, 2016

no, the other libs are required.

@espHorst

This comment has been minimized.

Copy link

espHorst commented Mar 20, 2016

On a quite fresh installed Windows 10 machine I had success running borg using cygwin 2.4.1 64 bit. During the installation of cygwin I additionally selected the following packages:

python3 python3-setuptools
binutils gcc-g++
openssl-devel
liblz4_1 liblz4-devel # from cygwinports.org
make openssh
mc #in order to be able to edit/search the paths

(libopenssl was already selected in the cygwin installer
and I did see no need for installing git.
In line with the comment python3-cython # not needed for releases
I did not select python3-cython).

After cygwin finished I opened a cygwin and performed
easy_install-3.4 pip
pip install borgbackup --verbose

and borg worked!

However, in order to store files to a external NTFS drive I had to
start again the cygwin terminal with administrator rights.

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Mar 21, 2016

Reopening to adjust the cygwin docs.

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Mar 21, 2016

40dbb74 fixed

@georgeowell

This comment has been minimized.

Copy link

georgeowell commented May 6, 2017

Hi, I'm still having this problem same as @bpereto. Using the suggested flags above in the fstab solves this under Ubuntu 16.04. Is this actually in the documentation yet?

Would be good to know exactly what the problem is here :)

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented May 6, 2017

@georgeowell So, can you confirm the options mention in #583 (comment) are ok?

I suspect the noauto_cache makes the difference?

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented May 6, 2017

Reopening to add sshfs options to docs / FAQ.

@ThomasWaldmann ThomasWaldmann reopened this May 6, 2017

@georgeowell

This comment has been minimized.

Copy link

georgeowell commented May 7, 2017

@ThomasWaldmann yep, can confirm. The issue should in theory be easy to recreate.

My backup script mounts an sshfs "dumb" remote volume using fstab and then tries to backup to /mnt/backup. Without the above options the sshfs volume unmounts after maybe 20 seconds and the backup fails. This is also means the lock.exclusive folder is left in place.

@enkore enkore changed the title Input/output error with FUSE mounted filesystem Input/output error on sshfs May 7, 2017

@enkore

This comment has been minimized.

Copy link
Contributor

enkore commented May 7, 2017

I clarified the ambiguous title accordingly.

Perhaps also report this to the sshfs developers, since it seems like a bug in their code. Documenting this in the FAQ as a workaround works for me, too.

@georgeowell

This comment has been minimized.

Copy link

georgeowell commented May 7, 2017

Ok, good idea. I'll also find some more info out about the flags and what they specifically do and report back.

@enkore enkore modified the milestones: 1.0.11rc1, 1.0.1 - fixes May 7, 2017

@georgeowell

This comment has been minimized.

Copy link

georgeowell commented Jun 3, 2017

Sorry about that I've been afk for a while.

reconnect,allow_other,idmap=user,uid=0,gid=0,cache=no,noauto_cache,entry_timeout=0

-o reconnect
    reconnect to server 

-o allow_other
    allow access to other users 

-o idmap=TYPE
    user/group ID mapping, possible types are: 
    none

    no translation of the ID space (default)

    user

    only translate UID of connecting user

    file

    translate UIDs/GIDs based upon the contents of uidfile and gidfile 

-o uid=N
    set file owner 

-o gid=N
    set file group 

-o cache=BOOL
    enable caching {yes,no} (default: yes) 

-o noauto_cache  
     disable caching based on mtime (file modification timestamp)

-o entry_timeout=T
    cache timeout for names (1.0s) 

I'm assuming it's going to be the options to disable the caching that are making this work? I can't really find more detail than that.

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Jun 16, 2017

@georgeowell if you have the same issue, the most interesting thing would be to find out which option(s) make the difference from "not working" to "working".

These are the ones we should point out in the docs.

there is a bounty for doing this:

https://www.bountysource.com/issues/30058389-input-output-error-on-sshfs

@georgeowell

This comment has been minimized.

Copy link

georgeowell commented Jun 19, 2017

Sorry, I really don't have time to test this in my life right now. I believe it is the last three options regarding the mount cache are the ones.

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Jun 19, 2017

@georgeowell ok, thanks for feedback.

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Jun 21, 2017

Tried using our centos7_64 vagrant vm:

  • did sshfs vagrant@localhost: mnt (with default mount options)
  • borg init -e none mnt/repo + create + check - no issues at all
  • repeated same with given mount options from top post - no issues either
@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Jun 21, 2017

I'll leave this open for someone else to work on, but as I can't reproduce, I'll remove it from next milestone.

@ThomasWaldmann ThomasWaldmann removed this from the 1.0.11rc1 milestone Jun 21, 2017

@ThomasWaldmann ThomasWaldmann removed their assignment Jul 30, 2017

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented May 6, 2018

Looks like this is pretty stale. As I could not reproduce it, I'll just close it.
If you encounter such an issue, please open a new ticket and post all interesting information there.

I'll collect the bounty for this ticket temporarily to my personal account and will donate the same amount to the borgbackup organisation afterwards.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment