nocache - minimize filesystem caching effects

The nocache tool tries to minimize the effect an application has on the Linux file system cache. This is done by intercepting the open and close system calls and calling posix_fadvise with the POSIX_FADV_DONTNEED parameter. Because the library remembers which pages (ie., 4K-blocks of the file) were already in file system cache when the file was opened, these will not be marked as "don't need", because other applications might need that, although they are not actively used (think: hot standby).

Use case: backup processes that should not interfere with the present state of the cache.

Install and Use

Just type make. Then, prepend ./nocache to your command:

./nocache cp -a ~/ /mnt/backup/home-$(hostname)

You can also put the nocache.so file to a location like /usr/local/lib and do LD_PRELOAD=/usr/local/lib/nocache.so. (This is what the install target will do.)

Testing

For testing purposes, I included two small tools:

• cachedel calls posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED) on the file argument. Thus, if the file is not accessed by any other application, the pages will be eradicated from the fs cache. Specifying -n will repeat the syscall the given number of times which can be useful in some circumstances (see below).
• cachestats has three modes: In quiet mode (-q), the exit status is 0 (success) if the file is fully cached. In normal mode, the number of cached vs. not-cached pages is printed. In verbose mode (-v), an actual map is printed out, where each page that is present in the cache is marked with x.

It looks like this:

$ cachestats -v ~/somefile.mp3
pages in cache: 85/114 (74.6%)  [filesize=453.5K, pagesize=4K]

cache map:
     0: |x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|
    32: |x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|
    64: |x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x| | | | | | | | | | | | |
    96: | | | | | | | | | | | | | | | | | |x|

Also, you can use vmstat 1 to view cache statistics.

Example run

Without nocache, the file will be fully cached when you copy it somewhere:

$ ./cachestats ~/file.mp3
pages in cache: 154/1945 (7.9%)  [filesize=7776.2K, pagesize=4K]
$ cp ~/file.mp3 /tmp
$ ./cachestats ~/file.mp3
pages in cache: 1945/1945 (100.0%)  [filesize=7776.2K, pagesize=4K]

With nocache, the original caching state will be preserved.

$ ./cachestats ~/file.mp3
pages in cache: 154/1945 (7.9%)  [filesize=7776.2K, pagesize=4K]
$ ./nocache cp ~/file.mp3 /tmp
$ ./cachestats ~/file.mp3
pages in cache: 154/1945 (7.9%)  [filesize=7776.2K, pagesize=4K]

Limitations

The pre-loaded library tries really hard to catch all system calls that open or close a file. This happens by "hijacking" the libc functions that wrap the actual system calls. In some cases, this may fail, for example because the application does some clever wrapping. (That is the reason why __openat_2 is defined: GNU tar uses this instead of a regular openat.)

However, since the actual fadvise calls are performed right before the file descriptor is closed, this may not happen if they are left open when the application exits, although the destructor tries to do that.

There are timing issues to consider, as well. If you consider nocache cat <file>, in most (all?) cases the cache will not be restored. For discussion and possible solutions see http://lwn.net/Articles/480930/. My experience showed that in many cases you could "fix" this by doing the posix_fadvise call twice. For both tools nocache and cachedel you can specify the number using -n, like so:

$ nocache -n 2 cat ~/file.mp3

This actually only sets the environment variable NOCACHE_NR_FADVISE to the specified value, and the shared library reads out this value. If test number 3 in t/basic.t fails, then try increasing this number until it works, e.g.:

$ env NOCACHE_NR_FADVISE=2 make test

Acknowledgements

Most of the application logic is from Tobias Oetiker's patch for rsync http://insights.oetiker.ch/linux/fadvise.html. Note however, that rsync uses sockets, so if you try a nocache rsync, only the local process will be intercepted.