Use interrupts instead of closing /dev/fuse socket when a request takes too long

[tv42]

To the best of my understanding, currently if any FUSE operation takes too long (longer than daemon_timeout, default 60s), the whole filesystem goes into "broken" mode, and needs to be unmounted. I understand this is because of OS X limitations that require the kext to respond to the request, or the kext itself will be declared broken.

How about, instead of wrecking the whole mount, behave just as if the request was interrupted at the daemon_timeout time. As far as OS X is concerned, it fails with EINTR; as far the userspace FUSE server is concerned, it sees a fuse_interrupt_in request cancelling the earlier request. If the userspace FUSE server doesn't implement interrupting, it'll just process the earlier request to completion, at which point its result will get discarded -- just like if the response raced with the interrupt message.

This would avoid a lot of trouble in writing networked filesystems with OSXFUSE. Currently everyone must implement their own timeout logic underneath, so that the kext daemon_timeout never triggers.

[bfleischer]

The daemon_timeout concept stems back from the MacFUSE days. To be honest I never really questioned it, but I see your point.

I think sending interrupts instead of killing the file system could introduce data inconsistencies. OS X's caching layer (or at least the process that started the timed-out request) thinks the request has been aborted even though it might have been completed. We might end up with a situation where the data "on-disk" and "in-cache" are not in sync.

It would be interesting to see how NFS or SMB deal with timeouts.

What about the following modified approach?

After daemon_timout is up we send an interrupt to user space and give the daemon another 10 seconds to respond to the interrupt. If the request is completed or interrupted by then all is fine. If the daemon does not respond to the interrupt request in time it is considered unresponsive/dead and the file system will be killed.

[tv42]

I'm not sure what inconsistency would exposed that isn't already there from just hitting control-C. The userspace FUSE server may ignore the interrupt request and just respond to the original request, or it may respect the interrupt; whether that is triggered by a global timeout or signal shouldn't change the data consistency rules.

[tv42]

Oh and having some ultimate "you must respond in this time" daemon_timeout is probably still necessary, because of the claimed OSX design where the OSXFUSE kext needs to respond to the rest of the kernel within a limited time. I just want to make it easier for the userspace FUSE server to behave nicely; interrupts should already be wired up for normal operation, using them here avoids everyone doubleguessing daemon_timeout.

[bfleischer]

I'm not sure what inconsistency would exposed that isn't already there from just hitting control-C. The userspace FUSE server may ignore the interrupt request and just respond to the original request, or it may respect the interrupt; whether that is triggered by a global timeout or signal shouldn't change the data consistency rules.

Let's assume a process is overwriting a file on a FUSE volume. The data moves through the kernel's caching layer and then to the FUSE user space daemon.

Let's assume further that the daemon does not respond within daemon_timeout seconds. As a result we send an interrupt request to user space and return EINTR to the caching layer (and the process). From the perspective of the caching layer (and the process) the file system operation has been aborted.

This means the file cache still contains the original file content from before the interrupted write request. The user space daemon on the other hand ignores the interrupt and completes the write operation. At this point the actual "on-disk" data is different from the "in-cache" data.

Following read/write operations will operate on "in-cache" data, not the actual "on-disk" data. This could lead to file corruption.

[tv42]

I don't know much about OS X. On Linux, I'd expect one of two possibilities:

1. FOPEN_DIRECT_IO is used, there is no cache, and it's up to the FUSE server to decide how to serve further read requests.
2. kernel page cache caches the writes; there's no such concept as the FUSE server "refusing a write", the write hits the cache and the page is marked dirty; afterward, the write is transmitted to the FUSE server. It's a writeback or writethrough cache, in both variations the write hits the cache first. This is even easier to demonstrate with mmap; the write to memory happens well before the dirty page is flushed to backing store. Obviously, the page remains dirty until it has been written out.

If there's an error flushing a dirty page in the page cache, that's exactly the same scenario as a disk reporting a block write error. That's what the FUSE flush request is for, and that's why userspace needs to check close(2) return values.

Relevant bits of Linux kernel code:

• fuse_writepage_end calling mapping_set_error when flushing a dirty page cache entry fails:
  http://lxr.free-electrons.com/source/fs/fuse/file.c#L1517
• fuse_send_write_page only clearing dirty bit on successful write: http://lxr.free-electrons.com/source/fs/fuse/file.c#L999

[bfleischer]

In the newly released osxfuse 3.2.0 the kernel extension sends an interrupt request to give the file system daemon a chance to handle the timeout. If the daemon does not respond to the interrupt request within daemon_timeout seconds, the file system will be marked dead. I think this is a good compromise. File systems that handle interrupts get a proper warning.

[tv42]

This is great news!

[tv42]

I just tested this and it makes all well-behaved http://bazil.org/fuse -using filesystems do the right thing on slow requests. This means even aborting outgoing network requests and all. This is great, thank you very much!

(Well-behaved = respects Context cancellation: http://blog.golang.org/context)