Under load, files appearing twice or disappearing when running ls, unless readdir is sorted

[sgielen]

I've been investigating an issue where an application I developed seemed to occasionally list files twice, or not list them at all, under load. I've troubleshooted down the stack, and can now reproduce the issue directly against macfuse 4.1.2.

Seeing as the sources aren't available anymore, I hope you're willing to debug this further :-)

The issue only occurs if readdir always calls the filler in a different order every time. In my case, my application is written in Go and the files come from a map type, which is ordered inconsistently/arbitrarily. Therefore, filler is always called in a different order. My workaround is to sort the files before calling filler, but I believe this isn't intentional behaviour in macfuse.

A test case written in C++ is shared here: https://gist.github.com/sgielen/ade2b98681840d50e70324bdfff8a171 - the comments at the top of the cpp file show how to reproduce: run ls -la test/Tsttst | wc -l in a loop and you'll see it doesn't always respond with 28 lines as expected.

Here's what I've tried so far:

• the sleep on line 197 seems to be required for reproduction? Sometimes, longer sleeps (~500 ms) make the issue occur more often.
• if the shuffle at line 198 is removed, the issue also doesn't occur.
• With different paths coming from vfs_readdir, the issue also doesn't occur -- I'm not sure which exact pattern is necessary, but I'm pretty sure it's at least related to the lengths of the paths. When I make them all the same length, the issue is not reproducible.
• it reproduces faster for me with additional load on the filesystem, e.g. running while :; do du -chs test/Tsttst; done in one or two terminals at the same time.

Could you verify that you can reproduce the issue using this testcase?

I think #564 might be related; it also has a patch attached. Perhaps a good starting point would be to see if that patch resolves the issue? :-) Thanks!

[bfleischer]

Seeing as the sources aren't available anymore, I hope you're willing to debug this further :-)

The source code for libfuse is available on https://github.com/osxfuse/fuse/commits/master.

I think #564 might be related; it also has a patch attached. Perhaps a good starting point would be to see if that patch resolves the issue?

A variant of the patch in #564 (osxfuse/fuse@06fc407) found its way in the upstream Linux libfuse codebase in libfuse version 2.9.9. I might have overlooked something, but the logic of the upstream patch seems to be flawed. File systems that are calling filler with non-zero offsets will always trigger the EIO error case. I never understood why it made its way in the upstream repository. Eventually I had to revoke the patch in version 4.0.3 in oder to address #729.

The patch in #564 is similar, but does not contain the error case. I think I will need to look into it again.

[bfleischer]

The patch from #564 does not fix the "ls returns less than 28 entries" issue.

[sgielen]

Ah, I did not realize that this part of fuse was still open source. In that case, I can try delving into it further.

Just out of curiosity, can you also reproduce the issue with the testcase on your end?

[bfleischer]

Yes, I can reproduce the issue with your C++ testcase.

The kernel extension uses offsets when querying the directory contents from user space. One explanation for the missing files could be that the cached directory entries in libfuse have been reordered between calls from the kernel extension. This might result in the kernel extension receiving the same directory entry more than once. I assume the duplicates get then filtered on a higher level of macOS. But at this point this is just a theory.

[bfleischer]

I spent some more time looking into this. The patch in #564 and the upstream libfuse patch osxfuse/fuse@06fc407 are completely fine. What lead to my confusion is an important implementation difference between Linux and macOS. telldir() returns zero for the first directory entry (at least on APFS volumes). This is the reason the readdir callback of the loopback reference file system used to call the filler function with a zero offset for the first directory entry and non-zero offsets for all subsequent directory entries. This is a bug in the loopback file system. The upstream libfuse patch simply exposed the bug in the loopback file system.

Regarding the issue with the missing or duplicate directory entries: This is not an easy fix. The VFS layer on macOS works differently than on Linux. On macOS VFS plugins are not passed any file or directory handles. As a result macFUSE needs to emulate this behavior and reuses handles. This means that two readdir calls on the same directory operate on the same libfuse directory handle and they share the same cached directory listing. With two racing readdir calls the cached directory listing in libfuse might be refreshed before the first readdir call has read the whole directory listing. This is not an issue if the ordering of the directory entries is stable. But if the ordering is not stable you might end up with missing or duplicate files.

Fixing this is difficult because the macFUSE kernel extension does not know if two readdir calls are related and should operate on the same directory handle or not. It might be possible to work around this by using cookies. Cookies are special offsets returned by readdir in the kernel. This might allow the kernel extension to differentiate between related readdir calls that need to operate on the same directory handle and unrelated readdir calls that need to use different directory handles.

TL;DR

Until this is addressed directory listings need to be stable between readdir calls.

[billziss-gh]

@bfleischer thank you for the explanation for this problem.