New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OSX fcntl(fd, F_NOCACHE, 1) not equivalent to O_DIRECT on Linux #48

Closed
aktau opened this Issue Feb 22, 2015 · 3 comments

Comments

Projects
None yet
3 participants
@aktau
Copy link

aktau commented Feb 22, 2015

This is probably already well known, but it bit me while planning to do some comparative benchmarks between Linux and OSX. I started with OSX and direct=1, rw=randread, but noticed that OSX was reading at 1200MB/s from a 1GB file. This was unexpected as I only have one spinning rust HDD in the MBP.

So I looked up the ways to do direct I/O on OSX. On stack overflow and the Apple mailing lists, fcntl(fd, F_NOCACHE, 1) looked to be the canonical solution. This was also implemented in fio(1) in 2011 in commit 7e8ad19. It seems that F_NOCACHE disables the page cache from that point on, but the file in quesition was already in the page cache, it will not be purged and the pages will be used.

I also commented on stack overflow: clear buffer cache on OSX with my observations. I'll copy it here as well:

It's my impression that even when turning off the cache like this (with F_NOCACHE or F_GLOBAL_NOCACHE), if there are pages of the file already in the page cache, those will still be used. I tried to test this by using fio(1) with direct=1. It seems to confirm my suspicions (I get ~1200MB/s throughput on a random read, on a spindle HDD in my MBP, not an SSD). I've confirmed with dtruss that fio(1) actually calls fcntl correctly.

After running "sudo purge" and trying the same fio(1) invocation, it's much slower. So yes, it appears that F_NOCACHE is no direct equivalent of O_DIRECT on Linux.

I'm not advocating running sudo purge as part of fio, but perhaps it could be added to the documentation of direct that the behaviour is quite different from O_DIRECT but they're both fio's way of doing direct IO. Running sudo purge is both slow, has an adverse effect on the rest of the system while (and after) it runs for obvious reasons.

Another idea I had was to forcibly re-write the file each time for reading, while having the file open with F_NOCACHE, which makes the written pages not enter the page cache (UPC). That would also be slow (possibly) but hopefully it wouldn't evict other, unrelated files.

An interesting discussion involving an Apple dev on the Apple mailing list: http://lists.apple.com/archives/filesystem-dev/2007/Sep/msg00010.html. Specifically the part mentioning that files opened with F_NOCACHE will still uses already loaded pages if they're present.

EDIT: Some train of thought rambling: but perhaps mmap + MAP_NOCACHE on OSX is a possible avenue (+ telling uses to use that for direct IO testing). The man page doesn't give me a lot of hope though, as it appears that there's not guarantess and the OS might keep things in memory if it feels like it anyway. A small reference mentioning it: http://www.qtcentre.org/threads/24733-High-performance-large-file-reading-on-OSX

@axboe

This comment has been minimized.

Copy link
Owner

axboe commented Feb 22, 2015

Thanks for doing a thorough investigation! And I agree with you, we should add some mention of having to run purge to evict any previous cache. Or at least be aware that caching effects may exist.

But it really is a horrible interface, seems strange that OSX doesn't give you finer control of what is cached and how.

@aktau

This comment has been minimized.

Copy link

aktau commented Feb 22, 2015

Thanks for doing a thorough investigation!

No problem at all. I'm glad I learned something and found a workaround in short order. At least the difference was large enough (1200MB/s vs 22MB/s i pretty noticeable).

But it really is a horrible interface, seems strange that OSX doesn't give you finer control of what is cached and how.

I agree. I've looked for a bit more but could find nothing at all. It seems purge(8) uses a kernel extension that isn't properly documented. So it doesn't seem productive to find out whether or not it could be used for fio's purposes. A bit of documentation seems like the best option at the moment.

@axboe axboe closed this Jan 15, 2016

sitsofe added a commit to sitsofe/fio that referenced this issue Aug 27, 2017

fio: refactor explicit directio calls
Create an fio_directio() helper for the explicit directio setting call
and refactor its usage to be entirely within filesetup.c by calling it
from generic_open_file(). Also make initial layout in extend_file() call
it thus keeping up with the change introduced by
6e344dc ("filesetup: keep OS_O_DIRECT
flag when pre-allocating file").

A positive side effect of this change means the following job

rm -f /tmp/fiofile; ./fio --loops=10 --filename /tmp/fiofile --bs=4k \
 --size=1M --direct=1 --name=go

that creates a brand new file will no longer report cached I/O speeds
(such as 2000MiB/s) on macOS. This was happening because macOS fio is
unable to invalidate already cached data (see
axboe#48 ) and data was being cached
during the layout phase.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
@sitsofe

This comment has been minimized.

Copy link
Collaborator

sitsofe commented Sep 8, 2017

Just for the record: I checked whether mmap + MADV_DONTNEED/MADV_FREE on at OSX 10.9 would cause pieces of files pre-cached into the UBC (Unified Buffer Cache) to be evicted/invalidated when re-read. Despite the rumour the answer is no, it will not cause instant eviction/invalidation of entries and it seems purge(8) is the only way to instantly evict things from the UBC on OSX.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment