% The FUSE Wire Protocol
This document tries to summarize and structure what I have learned about the FUSE (Filesystem in Userspace) protocol and Linux kernel internals during the development of gocryptfs.
The Markdown source code of this document is available at https://github.com/rfjakob/the-fuse-wire-protocol - pull requests welcome!
The rendered HTML should always be available at https://nuetzlich.net/the-fuse-wire-protocol/.
Linux Filesystem Stack
To understand how FUSE works it is important to know how the Linux filesystem stack looks like. FUSE is designed to fit seamlessly into the existing model.
unlink("/tmp/foo") on an ext4 filesystem as an example.
Like many other system calls,
unlink() operates on a file path, while
Linux interally operates on
dentry ("directory entry") structs
dentry has a pointer to an
that is filled by the filesystem (in our example, ext4).
inode struct in turn contains a list of function pointers in an
The overall structure looks like this:
The Linux VFS layer splits the path into segments. In our case,
/ (root directory)
dentry is created
at mount-time and serves as the starting point for the recursive walk:
- The VFS calls
/and receives the
- The VFS calls
tmpand receives the
- The VFS calls
unlink() functions are, in our example, implemented by the ext4 filesystem.
For a FUSE filesystem, the functions in
inode_operations are implemented in the
userspace filesystem. The FUSE module in the Linux kernel provides stub implementations
that forward the requests to the userspace filesystem and convert between kernel API and FUSE wire protocol.
Directory Entry Cache -
Translating paths to
dentry structs is a performance-critical operation. To avoid calling the
lookup() function for each segment, the Linux kernel implements a directory entry
For local filesystems like ext4, the cached entries never expire. For FUSE filesystems, the default
timeout is 1 second, but it can be set to an arbitrary value using the
entry_timeout mount option
in libfuse (see
man 8 fuse) or the
field in go-fuse.
The Linux kernel and the userspace filesystem communicate by sending messages through the
/dev/fuse device. On the kernel side, message parsing and generation is handled by the FUSE
module. On the userspace side this is usually handled by a FUSE library.
libfuse is the reference implementation and is developed
in lockstep with the kernel. Alternative FUSE libraries like
follow the developments in libfuse.
Both sides have the message format defined correspondingly in C header files. As there is no other formal specification, these header files define the building blocks of the FUSE wire protocol:
Every message from the kernel to userspace starts with the
the most interesting fields are:
opcode... the operation the kernel wants to perform (a uint32 from enum fuse_opcode)
nodeid... the file or directory to operate on (arbitrary uint64 identifier)
The opcode defines the data that follows the header. An opcode-specific struct and up to
two filenames may follow. A
RENAME message uses all of those fields and looks like this:
UNLINK message looks like this:
The go-fuse library has two nice tables listing what data follows the header for each opcode. Due to Go naming conventions, the struct names are slightly different than the C names, but the correlation should be clear enough.
nodeid field in
fuse_in_header identifies which file or directory the operation
should be performed on. The kernel has to obtain the
nodeid from the
userspace filesystem before it can perform any other operation.
The process is the same for in-kernel filesystems: See the section "The Inode Object" in https://www.kernel.org/doc/Documentation/filesystems/vfs.txt.
LOOKUP opcode allows the kernel to get a
nodeid for a filename in a directory.
LOOKUP message looks like this:
The userspace filesystem replies with the
nodeid corresponding to
the filename in the directory identified by the
nodeid in the header.
The root directory has a fixed
nodeid of 1.
nodeid is an arbitrary value that is chosen by the userspace
filesystem. The userspace filesystem must remember which file or
nodeid corresponds to.
Writing a FUSE Filesystem: a Tutorial
Joseph J. Pfeiffer Jr.
Overview of the Linux Virtual File System
Richard Gooch, Pekka Enberg
To FUSE or Not to FUSE: Performance of User-Space File Systems
Vangoor, Tarasov, Zadok; 2017