ZFS + Samba + sendfile (mmap) = issues

[Ukko-Ylijumala]

Case: production system with 10Gb Ethernet exporting ZFS-backed file shares from Samba to Windows 7 workstations. SMB2 protocol is being used (forced in smb.conf), and Samba is v4 (release).

Problem: Turning on Samba's option to use sendfile system call (zero copy path) slows transfers down, when it should speed them up.

Server has dual E5620 Xeons and 96 GB RAM. Intel 10GbE 82599EB NIC (same on client). Client in this case has dual X5670 Xeons and 24 GB RAM. Server's running kernel 3.4.6, and client is on Windows 7 SP1 (both x64).

Reference transfer speed is obtained from RAMdisk to RAMdisk copy of a 20GB file. Different Samba versions (3.6.x -> 4.0 release) were tested with different optimizations and smb.conf options to obtain best possible "raw" transfer speeds. Linux network stack had to be tuned also (send & receive window sizes mostly in addition to jumbo frames) to obtain passable 10GbE throughput (close to wire speed with iperf).

The next tests are run 5 times and averages taken. First run is discarded (to make sure the test file is cached) so we disregard disk accesses completely.

Copying the aforementioned file from server to client using RAMdisk as origin and destination results in roughly 710 MB/s throughput with sendfile. Without sendfile throughput drops to about 340 MB/s.

Next up is testing with XFS. We create a zvol, and make an XFS filesystem on top of it. We then copy the same file from RAMdisk to XFS, drop caches and do the copy again from client. Now we get around 670 MB/s with sendfile, and about 320 MB/s without sendfile.

Finally we test with ZFS. Copy the same file to ZFS dataset, then copy from client. With sendfile we now get 310 MB/s (!!), and without sendfile we get 370 MB/s (!!).

Quite interesting. ZFS behaves completely differently from RAMdisk and XFS here. What could possibly cause this discrepancy? According to sendfile's documentation, it uses mmap in the implementation, so there could be something iffy in ZFS's mmap code. Does anyone have more ideas?

Currently I'm profiling (with Oprofile and perf toolstack) what happens in the kernel when these tests are being run, but the profiling data is hard to make sense of. Next I'll take a look at the differences in mmap in XFS and ZFS, but I'm not sure if my knowledge in that area is sufficient to decipher the details.

I'll post my profiling findings in a later post, hopefully tomorrow. In the meantime, please feel free to make suggestions etc.

[behlendorf]

It's hard to say for certain without the profiling data but two possibilities come to mind.

• The ZFS Posix Layer (ZPL) does not yet implement the f_ops->sendfile() callback. Likely the VFS is just internally falling back and using f_ops->write() rather than returning and outright failure. There's no particular reason f_ops->sendfile() can't be implemented, it just hasn't happened yet.
• Alternately, if it's really doing mmap(2) I/O there is an extra data copy in those call paths to keep the ARC and page cache in sync. This can be removed once the two are unified.

[tomposmiko]

@behlendorf
Do you plan to implement these features in the close future (this year)?

[behlendorf]

@tomposmiko I'm not aware of anyone working on implementing sendfile().

[Nilix007]

This issue caught my attention and I've been looking into the source code of Linux' VFS.

IIRC the sendfile syscall is implemented via the splice_read/splice_write function callbacks in the function_operations struct. Ext4 and XFS implement both callbacks via generic functions provided by fs/splice.c. Hence, it should be possible to use these generic funtions for ZFS, too.

So, implementing sendfile should be as easy as adding the following code snippet to module/zfs/zpl_file.c: (untested so far)

const struct file_operations zpl_dir_file_operations = {
    // [...]
    .splice_read    = default_file_splice_read,
    .splice_write   = iter_file_splice_write,
}

These two callbacks make use of read_iter and write_iter VFS callbacks which are provided by ZFS.

@behlendorf Am I missing something? Are there any special things in ZFS I am not aware of?

[behlendorf]

@Nilix007 if all you need to make use of are already registered read_iter and write_iter VFS callbacks then adding the default splice callbacks may just work. One thing to be aware of is that ZFS isn't yet integrated with page cache so that might be an issue, it depends on what these generic helpers do.

It's certainly worth investigating! You might be able to find some existing test cases for this in xfstests.

[tycho]

@behlendorf I notice that 1c2358c adds the .splice_read and .splice_write functions to zpl_file_operations. Is that all that's needed for sendfile() to not suck, or is there more work needed?

[nabijaczleweli]

This is all, do_sendfile() goes through {{do_splice_direct()=>splice_direct_to_actor()+{direct_splice_actor()=>f_op->splice_write}}/splice_file_to_pipe()}=>do_splice_to()=>f_op->splice_read.