Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

OSXFUSE crashes/unmounts during rsync backup on encrypted directory (encfs) #55

Closed
leeze opened this Issue Sep 24, 2012 · 10 comments

Comments

Projects
None yet
2 participants

leeze commented Sep 24, 2012

I ´m trying to do an incremental backup (like time machine) using a rsync-command an encfs-encrypted directory:
The encrypted directory is located on an external USB-Drive. I mount it using encfs:

encfs -s -f --fuse-debug $ENCRYPTEDDIRECTORY /Volumes/$DECRYPTEDVOLUME

Afterwards I do the backup:

rsync -aP --link-dest=$PREVIOUSBACKUPDIRECTORY_ON_DECRYPTEDVOLUME $SOURCEDIRECTORY $CURRENTBACKUPDIRECTORY_ON_DECRYPTEDVOLUME

The first backup works fine. But when I try to do a second backup (using the previousbackupdirectory), OS X FUSE/EncFS always crashes and unmounts (after having created about several thousands of hardlinks).

It seems that OSXFUSE only crashes when rsync tries to create many hardlinks.
There are some reports that might be related to this issue (macfuse and encfs), but as far I know the problem has never been solved:

Don´t know, maybe this bug is also a problem with encfs.

I am running OS X 10.7.4, OS X FUSE 2.5.2 and EncFS 1.7.4.

Here´s the output of encfs in fuse-debug-mode:
[…]
unique: 0, opcode: LOOKUP (1), nodeid: 40564, insize: 53
LOOKUP /zehn/CD4/elaImages1/Sparrenburg/Bild1031.jpg
NODEID: 40625
unique: 0, error: 0 (Undefined error: 0), outsize: 152
unique: 2, opcode: LINK (13), nodeid: 40794, insize: 65
LINK /zehn/CD4/images44GeburtstagDaniel/123-2367_IMG.JPG
NODEID: 40803
unique: 2, error: 0 (Undefined error: 0), outsize: 152
unique: 4, opcode: FORGET (2), nodeid: 10, insize: 48
FORGET 10/1
delete: 10
unique: 3, opcode: LOOKUP (1), nodeid: 40564, insize: 53
LOOKUP /zehn/CD4/elaImages1/Sparrenburg/Bild1032.jpg
NODEID: 40627
unique: 3, error: 0 (Undefined error: 0), outsize: 152
unique: 5, opcode: GETATTR (3), nodeid: 40802, insize: 40
unique: 5, error: 0 (Undefined error: 0), outsize: 128
unique: 1, opcode: FORGET (2), nodeid: 46, insize: 48
FORGET 46/2
Assertion failed: (node->nlookup >= nlookup), function forget_node, file fuse.c, line 495.
Abort trap: 6

And here´s the output of gdb/backtrace from another crash:

Assertion failed: (node->nlookup >= nlookup), function forget_node, file fuse.c, line 495.

Program received signal SIGABRT, Aborted.
0x00007fff8387dce2 in __pthread_kill ()
(gdb) bt
#0 0x00007fff8387dce2 in __pthread_kill ()
#1 0x00007fff81f467d2 in pthread_kill ()
#2 0x00007fff81f37a7a in abort ()
#3 0x00007fff81f6a5de in __assert_rtn ()
#4 0x00000001001d3d93 in forget_node ()
#5 0x00000001001d4262 in fuse_lib_forget ()
#6 0x00000001001d9d1f in do_forget ()
#7 0x00000001001da0dd in fuse_ll_process ()
#8 0x00000001001dc214 in fuse_session_process ()
#9 0x00000001001d8fdb in fuse_session_loop ()
#10 0x00000001001d4da6 in fuse_loop ()
#11 0x00000001001dcc40 in fuse_main_common ()
#12 0x00000001001dcc88 in fuse_main_real ()
#13 0x000000010000b476 in boost::scoped_array::operator
#14 0x0000000100001308 in ?? ()

I hope this helps You to figure out what´s going wrong. If you need further information let me know.

Regards

Daniel

Owner

bfleischer commented Sep 25, 2012

Thanks for the detailed bug report. I will look into it and report back.

@ghost ghost assigned bfleischer Sep 25, 2012

leeze commented Sep 25, 2012

I just found an easier way to reproduce the bug:

First I mount the encrypted encfs-filesystem, then I copy a file into the decrypted directory and afterwards I create a hardlink in the decrypted directory:

 encfs -s -f /$CRYPTEDDIRECTORY /Volumes/decrypteddirectory

Second bash:

cd /Volumes/decrypteddirectory
cp a_file .
link a_file file_b
cd ..

Encfs/OS X Fuse does not crash immediately. But as soon as I use umount, I get the same "Assertation failed ...."-error:

umount /Volumes/decrypteddirectory
@ghost

ghost commented Sep 26, 2012

I've just upgraded to 2.5.2 and I'm also seeing a hang during an unmount operation. This typically happens when I'm running under a Release mode build, but I have seen it while running in the debugger in debug mode.

I simply mount a file system, then in Finder, unmount it and it hangs.

Owner

bfleischer commented Oct 15, 2012

@southrivertech

The hang during umount should be fixed in commit cd0621c

@leeze

So far I have not been able to reproduce the issue. Are you able to reproduce it reliably? Does the encrypted directory reside on a HFS+ volume?

Is there anything else that might be important to trigger the bug?

Owner

bfleischer commented Oct 16, 2012

@leeze

Could you do a ls -lia on a_file and file_b after linking and post the results?

leeze commented Oct 17, 2012

@bfleischer

My encrypted volume resides on HFS+Volume (not journaled).

Surprisingly I cannot reproduce the issue described in my second message (an easier way to reproduce the issue). This might be due to a system update to OS X 10.7.5. I couldn´t figure out anything else that has changed since then.

Anyway here´s the result (ls -lia):

daniels-MacBook-Pro:decrypted daniel$ ls -lia
total 88
16132114 drwxr-xr-x  5 daniel  staff    170 Oct 17 11:35 .
   26119 drwxrwxrwt@ 6 root    admin    204 Oct 17 11:36 ..
16132135 -rw-r--r--  2 daniel  staff  19059 Oct 17 11:35 file_a
16132135 -rw-r--r--  2 daniel  staff  19059 Oct 17 11:35 file_b

The volume can be unmounted correctly. Both INodes are the same.

But nevertheless I can still reliably reproduce the issue while doing the rsync-backup. After having created about 20000 hard-links encfs unmounts/crashes:
Assertion failed: (node->nlookup >= nlookup), function forget_node, file fuse.c, line 495.
Abort trap: 6

I don´t have a clue what else might trigger the bug. I suppose the only way for You to reproduce the bug is to do a large rsync-backup.

Owner

bfleischer commented Oct 22, 2012

I tried my luck with rsync and it crashes for me, too. However, manually creating millions of links in a simple loop works like a charm. I assume this is issue is caused by some kind of race condition.

Owner

bfleischer commented Oct 23, 2012

I think I fixed the issue. Actually there were two bugs causing the crash. Can you confirm that the issue is gone in this build https://dl.dropbox.com/u/70779888/OSXFUSE-2.5.3.pkg?

leeze commented Oct 24, 2012

It seems the problem has been solved. I didn´t encounter any problems during long backups anymore. Afterwards Encfs/OSXFuse unmounts correctly.

Thanks a lot!

@leeze leeze closed this Oct 24, 2012

Owner

bfleischer commented Oct 26, 2012

Thanks for the great bug report. The final version of OSXFUSE 2.5.3, that has been released today, fixes an additional bug I discovered after sending you the test build. I recommend uninstalling the test build and installing the final version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment