Skip to content

Commit

Permalink
output: Don't use nbdkit-file-plugin cache=none when writing
Browse files Browse the repository at this point in the history
nbdkit-file-plugin flag cache=none is meant to ensure that the file
plugin does not pollute the page cache.  However it doesn't yet work
very well, especially for writing.  As you can see here:

https://gitlab.com/nbdkit/nbdkit/-/blob/048d5b9818c88355e596824355269773e5c4f6ad/plugins/file/file.c#L594

it is currently implemented by flushing the whole file (including
parallel requests), and then evicting the data from the page cache.
This implementation is naive.  (See nbdcopy copy/file-ops.c for a much
better implementation.)

This causes quite a considerable slow down when writing to a local
file (eg -os local).  The penalty varies between machines and possibly
kernels.  On my Fedora Rawhide AMD server with 12 cores and SSDs, the
slow down is under 5%.  But on my RHEL 9 Intel NUC server with 2 cores
and a hard disk, it makes a really huge difference, nearly doubling
the total time taken.

I left a note in the code that we should fix this in nbdkit and re-add
this option or similar when it is working.

I also removed the fadvise=sequential option.  This is associated with
cache=none in the nbdkit documentation, and anyway we are not doing
sequential writes.  (This makes no measurable difference in my testing)
  • Loading branch information
rwmjones committed Dec 7, 2021
1 parent b5df82a commit ac59d3b
Showing 1 changed file with 8 additions and 2 deletions.
10 changes: 8 additions & 2 deletions output/output.ml
Expand Up @@ -98,8 +98,14 @@ let output_to_local_file ?(changeuid = fun f -> f ())
let cmd = Nbdkit.add_arg cmd "file" filename in
let cmd =
if Nbdkit.version nbdkit_config >= (1, 22, 0) then (
let cmd = Nbdkit.add_arg cmd "fadvise" "sequential" in
let cmd = Nbdkit.add_arg cmd "cache" "none" in
(* nbdkit 1.28 has a very naive implementation of
* page cache eviction. We need to copy the one from
* nbdcopy copy/file-ops.c. Until then do not use
* this flag because it causes a large slow down on
* some machines. XXX
*)
(*let cmd = Nbdkit.add_arg cmd "fadvise" "sequential" in
let cmd = Nbdkit.add_arg cmd "cache" "none" in*)
cmd
)
else cmd in
Expand Down

0 comments on commit ac59d3b

Please sign in to comment.