borg mount stopped working on RedHat 7: "Transport endpoint is not connected" when trying to get in the mount directory

[mzannoni]

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

Bug / Issue

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

[root@goliath ~]# borg -V
borg 1.1.11

Operating system (distribution) and version.

[root@goliath neofetch]# ./neofetch --off
icdev@goliath 
------------- 
OS: Red Hat Enterprise Linux Workstation release 7.8 (Maipo) x86_64 
Host: Super Server 0123456789 
Kernel: 3.10.0-1127.el7.x86_64 

Hardware / network configuration, and filesystems used.

Borg called on a RedHat server machine, repo sitting on a Synology NAS partition, mounted on the RedHat machine with cifs.

How much data is handled by borg?

~500 GB

Full borg commandline that lead to the problem (leave away excludes and passwords)

[root@goliath ~]# borg mount /opt/hbox/backup/cad_srv::goliath-usrhomes-2020-04-02T23:15:19 bkp_mnt/

Same for any other archive in that borg repo.

And the cd command then gives:

[root@goliath ~]# cd bkp_mnt
-bash: cd: bkp_mnt: Transport endpoint is not connected

When running with -f --debug oprions, it throws the error:

AttributeError: 'llfuse.EntryAttributes' object has no attribute 'st_birthtime_ns'

Describe the problem you're observing.

An error is thrown when I try an ls command on the folder where the mount point is, after launching borg mount with the -f option.
I used to be able to correctly mount borg archives; last time I'm sure I could is a couple of months ago (last time I needed to).
Sinc then, borg backup scripts are still giving successful messages on daily backups since last time I've been able to mount correctly.
Borg list shows the archive list correctly.

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Yes, any borg mount command, on any of the listed archives gives the result above.
The mount point folder is listed like this:

[root@goliath ~]# ls -l
ls: cannot access bkp_mnt: Input/output error
total 420
<...>
d?????????? ? ?    ?         ?            ? bkp_mnt
<...>

Include any warning/errors/backtraces from the system logs

[root@goliath ~]# borg mount /opt/hbox/backup/cad_srv::goliath-usrhomes-2020-04-02T23:15:19 bkp_mnt/ -f --debug
using builtin fallback logging configuration
35 self tests completed in 0.12 seconds
Verified integrity of /opt/hbox/backup/cad_srv/index.7349
Enter passphrase for key /opt/hbox/backup/cad_srv:
TAM-verified manifest
security: read previous location '/opt/hbox/backup/cad_srv'
security: read manifest timestamp '2020-04-06T00:00:21.564033'
security: determined newest manifest timestamp as 2020-04-06T00:00:21.564033
security: repository checks ok, allowing access
mount data cache capacity: 32 chunks
Mounting filesystem
fuse: process_archive completed in 9.8 s for archive goliath-usrhomes-2020-04-02T23:15:19
Initializing llfuse
Calling fuse_mount
Calling fuse_lowlevel_new
Calling fuse_session_add_chan
Calling fuse_session_loop
handler raised <class 'AttributeError'> exception ('llfuse.EntryAttributes' object has no attribute 'st_birthtime_ns'), terminating main loop.
Terminated main loop because request handler raised exception, re-raising..
Calling fuse_session_remove_chan
Calling fuse_session_destroy
Local Exception
Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line 4529, in main
    exit_code = archiver.run(args)
  File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line 4461, in run
    return set_ec(func(args))
  File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line 1387, in do_mount
    return self._do_mount(args)
  File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line 166, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line 1397, in _do_mount
    operations.mount(args.mountpoint, args.options, args.foreground)
  File "/usr/lib64/python3.6/site-packages/borg/fuse.py", line 351, in mount
    signal = fuse_main()
  File "/usr/lib64/python3.6/site-packages/borg/fuse.py", line 34, in fuse_main
    return llfuse.main(workers=1)
  File "src/fuse_api.pxi", line 343, in llfuse.main (src/llfuse.c:35968)
  File "src/handlers.pxi", line 80, in llfuse.fuse_getattr (src/llfuse.c:8699)
  File "src/handlers.pxi", line 81, in llfuse.fuse_getattr (src/llfuse.c:8646)
  File "/usr/lib64/python3.6/site-packages/borg/fuse.py", line 544, in getattr
    entry.st_birthtime_ns = item.get('birthtime', mtime_ns)
AttributeError: 'llfuse.EntryAttributes' object has no attribute 'st_birthtime_ns'

Platform: Linux goliath 3.10.0-1127.el7.x86_64 #1 SMP Tue Feb 18 16:39:12 EST 2020 x86_64
Linux: Red Hat Enterprise Linux Workstation 7.8 Maipo
Borg: 1.1.11  Python: CPython 3.6.8 msgpack: 0.5.6
PID: 187781  CWD: /root
sys.argv: ['/bin/borg', 'mount', '/opt/hbox/backup/cad_srv::goliath-usrhomes-2020-04-02T23:15:19', 'bkp_mnt/', '-f', '--debug']
SSH_ORIGINAL_COMMAND: None

[infectormp]

Dupe of #5064

[mzannoni]

Ok, thanks @infectormp , sorry for the spam, I didn't get to that issue when searching.
Anyhow, this RedHat has python36-llfuse version 1.0-2, so I guess it's the same issue.
Should I just wait for the next borg release, then?

[ThomasWaldmann]

The 1.1.11 code has a bug so it does not work with llfuse < 1.3.

It is already fixed and for dist-maintainers it might make sense to do a hotfix release with the patch, see #5064.

[mzannoni]

Thanks a lot @ThomasWaldmann. I think for now I can wait for the new release to land on the EPEL repo, I verified borg extract still works properly on my archives, so I'm covered in case I need to recover files.

[FelixSchwarz]

Hey * - I just noticed that bug myself. I'll check what we can do here.

@ThomasWaldmann - Please @-mention me for specific EPEL/Fedora bugs so I can fix these.

[FelixSchwarz]

new update for EPEL 7 contains the fix: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-1e5016cd77

If you can please install the version from EPEL's testing repo and leave karma in bodhi.