Where qemu-img 4.0 used to say 'virtual size: 100M', the 4.1 release
now says 'virtual size: 100 MiB'.  Similarly, '5.0G' turned into '5
GiB'. But rather than worry about potential future changes to the
human-readable output, we can just use --output=json (at which point
we no longer even have to force qemu-img to the C locale, which we
were not even doing consistently).  It might be slightly more robust
to find our specific information using jq, but for now a grep of the
output json is still reliable enough for our needs.

If the '\b' is a problem on BSD, we could use '([, ]|$)' instead.  Or
that's where jq would make it easier to parse off a given number
without worrying about what comes after the number.

Signed-off-by: Eric Blake <eblake@redhat.com>
(cherry picked from commit 902f220)

On OCaml 4.10 you cannot memcpy to a String_val target since
String_val returns a const char * pointer.  In any case we should be
using caml_alloc_initialized_string.  For versions of OCaml before
August 2017 which didn't have this function, provide a replacement.

Better fix than commit 8a84d9d.
The idea for this fix is copied from libnbd.

(cherry picked from commit c27c625)

String_val now returns const char *, so it cannot be the destination
for memcpy.  The compiler itself seems to use &Byte_u (strv, 0) in
this case, so let's do that.

(cherry picked from commit 8a84d9d)

Currently we call the plugin .open callback as soon as we receive a
TCP connection:

  $ nbdkit -fv --tls=require --tls-certificates=tests/pki null \
           --run "telnet localhost 10809"
  [...]
  Trying ::1...
  Connected to localhost.
  Escape character is '^]'.
  nbdkit: debug: accepted connection
  nbdkit: debug: null: open readonly=0       ◀ NOTE
  nbdkit: null[1]: debug: newstyle negotiation: flags: global 0x3
  NBDMAGICIHAVEOPT

In plugins such as curl, guestfs, ssh, vddk and others we do a
considerable amount of work in the .open callback (such as making a
remote connection or launching an appliance).  Therefore we are
providing an easy Denial of Service / Amplification Attack for
unauthorized clients to cause a lot of work to be done for only the
cost of a simple TCP 3 way handshake.

This commit moves the call to the .open callback after the NBD
handshake has mostly completed.  In particular TLS authentication must
be complete before we will call into the plugin.

It is unlikely that there are plugins which really depend on the
current behaviour of .open (which I found surprising even though I
guess I must have written it).  If there are then we could add a new
.connect callback or similar to allow plugins to get control at the
earlier point in the connection.

After this change you can see that the .open callback is not called
from just a simple TCP connection:

  $ ./nbdkit -fv --tls=require --tls-certificates=tests/pki null \
             --run "telnet localhost 10809"
  [...]
  Trying ::1...
  Connected to localhost.
  Escape character is '^]'.
  nbdkit: debug: accepted connection
  nbdkit: null[1]: debug: newstyle negotiation: flags: global 0x3
  NBDMAGICIHAVEOPT
  xx
  nbdkit: null[1]: debug: newstyle negotiation: client flags: 0xd0a7878
  nbdkit: null[1]: error: client requested unknown flags 0xd0a7878
  Connection closed by foreign host.
  nbdkit: debug: null: unload plugin

Signed-off-by: Richard W.M. Jones <rjones@redhat.com>
(cherry picked from commit c05686f)
(cherry picked from commit e06cde0)
(cherry picked from commit f03f18a)

This structure was not freed along the non-error path, both resulting
in a memory leak and providing an easy way for clients to blow up
nbdkit servers if they enable TLS support.  Ooops.

Found by valgrind.

(cherry picked from commit baf1091)

Same as commit e90b0a4,
but for the tests.

(cherry picked from commit 4577cb3)

In discussion on a bug report we were both confused by the debug
message:

  nbdkit: python[1]: debug: connection cleanup with final status 1

What is the meaning of the status here?  Turns out it's essentially an
internal value of no significance.

Also the message as a whole is not really significant, just indicating
that the function handle_single_connection is returning, which we can
deduce anyway.

Delete the whole debug message as it's confusing.

(cherry picked from commit ae98825)

For virt-v2v I implemented Cinder support.  By design Cinder cannot
offer block-level access.  Instead you must attach a Cinder volume to
a VM running on OpenStack.  It appears as a /dev device and so no
special API is required to read or write it (for nbdkit, just use
nbdkit-file-plugin).

Glance also does not offer block-level access.

(cherry picked from commit 7d2a2aa)

We want to enable this function generally, but in specific cases we
want to ignore it:

main.c: In function ‘handle_quit’:
main.c:948:3: error: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Werror=unused-result]
   write (write_quit_fd, &c, 1);
   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
main.c: In function ‘fork_into_background’:
main.c:1057:3: error: ignoring return value of ‘chdir’, declared with attribute warn_unused_result [-Werror=unused-result]
   chdir ("/");
   ^~~~~~~~~~~

(cherry picked from commit e90b0a4)

For example this is wrong:

    B<rdelay=SECS>

but this is right:

    B<rdelay=>SECS

because "SECS" is a placeholder, not a literal string that would be
passed to nbdkit.

(cherry picked from commit 9e72434)

(cherry picked from commit 2a1705a)

(cherry picked from commit 569a57a)

(cherry picked from commit ca92062)

(cherry picked from commit c0e8b1b)

…led.

See also commit b5ce88e.

(cherry picked from commit 6ce828f)

(cherry picked from commit 226e064)

This test is quite prone to failure, especially when nbdkit is slow to
start up (ie. when running under valgrind).  This commit aims to make
the test more stable:

 - Use a PID file and wait for the subject nbdkit process to start
   before testing it.  (We don't actually need the PID because we use
   --exit-with-parent, the PID file is just an indication that the
   server is listening on the socket).

 - Remove the inner shell.  There doesn't seem to be any reason for
   this.  --exit-with-parent will wait for the shell script to exit
   instead.

 - Remove any leftover files at the start.

 - Make ‘trap’ invocation match what we do in other tests.

I ran the single test hundreds of times in a loop both with and
without valgrind and was not able to make it crash:

while make -C tests check TESTS="test-parallel-nbd.sh" >& /tmp/log ; do echo -n .; done

while make -C tests check-valgrind TESTS="test-parallel-nbd.sh" >& /tmp/log ; do echo -n .; done

(cherry picked from commit 894513e)

(cherry picked from commit 7433724)

(cherry picked from commit ef45023)

The tracebacks are compressed into a single line because we're using
PyObject_Str, but they are just about usable if not very readable.
For example you would see an error like this:

nbdkit: error: ./python-exception.py: config_complete: error: ['Traceback (most recent call last):\n', '  File "./python-exception.py", line 54, in config_complete\n    raise_error1()\n', '  File "./python-exception.py", line 48, in raise_error1\n    raise_error2()\n', '  File "./python-exception.py", line 45, in raise_error2\n    raise RuntimeError("this is the test string")\n', 'RuntimeError: this is the test string\n']

which can be read by manually unfolding the exception in an editor as:

nbdkit: error: ./python-exception.py: config_complete: error:
Traceback (most recent call last):
  File "./python-exception.py", line 54, in config_complete
    raise_error1()
  File "./python-exception.py", line 48, in raise_error1
    raise_error2()
  File "./python-exception.py", line 45, in raise_error2
    raise RuntimeError("this is the test string")
RuntimeError: this is the test string

This also fixes the Python exception test:

(1) It originally was not testing anything.  Adding ‘set -e’ fixes
that.

(2) The valgrind test is always broken because of Python itself.
Skip the test under valgrind.

(3) This now tests both simple exceptions and full tracebacks.

Tested with Python 2.7.15 & 3.6.6.

(cherry picked from commit 72c0d64)

(cherry picked from commit d31666d)

…order.

(cherry picked from commit a2c4e88)

Commit b5ce88e fixed the cases where multiple filters in front
of a plugin did not all have their .prepare callback utilized,
which (correctly) results in more output from the log filter,
  2018-08-01 15:17:45.249533 connection=2 Connect [...] fua=1
contrasted to all the existing lines we care about:
  2018-08-01 15:49:26.126100 connection=2 Write [...] fua=1 ...

Adjust the grep of the test output for FUA to not be confused by
the new output lines.

Signed-off-by: Eric Blake <eblake@redhat.com>
(cherry picked from commit 2bec560)

If there is an asynchronous exit while running
_handle_single_connection (eg. because of a signal or exit being
called from a thread) backend can become NULL at just about any time.
Make this function more robust and add a comment.

One example seen when running the tests:

$1 = (struct backend *) 0x0
(gdb) t a a bt

Thread 3 (Thread 0x7f143906dc80 (LWP 26858)):
    p=0x1256fe0, have_lock=<optimized out>) at malloc.c:4281
    listp=0x7f143a882738 <__exit_funcs>,
    run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true)
    at exit.c:108
    argv=0x7ffe58414148, init=<optimized out>, fini=<optimized out>,
    rtld_fini=<optimized out>, stack_end=0x7ffe58414138)
    at ../csu/libc-start.c:308

Thread 2 (Thread 0x7f1433fff700 (LWP 26872)):
    at ../sysdeps/unix/sysv/linux/read.c:26

Thread 1 (Thread 0x7f1438e68700 (LWP 26860)):
    sockin=<optimized out>) at connections.c:317
    at connections.c:335
    at pthread_create.c:474
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

(cherry picked from commit 99f264b)

Since plugins can only be loaded once and we print the filename
already on registration, there's no reason to print the full filename
in later messages.

(cherry picked from commit a9932f4)

nbdkit_debug implicitly prints the plugin name but not the filter
name.  This leads to confusing debug messages such as:

  nbdkit: pattern[1]: debug: prepare
  nbdkit: pattern[1]: debug: prepare

In this instance, two different prepare functions were called in
different filters, but neither filter name was shown.  The plugin name
(pattern) is shown, but that is irrelevant as prepare is a method only
present in filters.

By printing the filter name in the debug message we improve the output:

  nbdkit: pattern[1]: debug: truncate: prepare
  nbdkit: pattern[1]: debug: map: prepare

Another example of the improved output is:

  nbdkit: pattern[1]: debug: truncate: can_flush     # filter
  nbdkit: pattern[1]: debug: map: can_flush          # filter
  nbdkit: pattern[1]: debug: can_flush               # plugin

Also this changes a few filenames in messages to simple filter names.
The filenames are very long and since a filter can only be loaded once
printing the filename more than once is a waste of space.

(cherry picked from commit e7d8da6)

…plugin.

This function was completely bogus, the effect being that the
threadlocal name was always set to the name of the second filter.

Also fixes an adjacent comment.

(cherry picked from commit 8a43ae9)

…one.

These methods are not chained through the next_ops structure, so we
must do that work in src/filters.c.

This fixes the behaviour of filters when you layer multiple filters in
front of a plugin.

(cherry picked from commit b5ce88e)

In these test functions, when a test fails, the bash ‘trap’ command
causes the cleanup function to be called.  However bash does not annul
or cancel the traps when cleanup is called, so at the end of the
cleanup function when ‘exit’ is called, cleanup is called recursively.

Avoid this by cancelling the traps at the top of the cleanup function.
Also an extra debugging message is emitted here giving the value of
$status which can be useful.

(cherry picked from commit f9cbcb0)

(cherry picked from commit ba2a921)

Fix issues Eric found in the original patch:
https://www.redhat.com/archives/libguestfs/2018-July/msg00072.html

- When handling ENODEV, the caller is expecting EOPNOTSUPP to trigger
  fallback.
- ENODEV should be ignored in file_trim.

Tested only on Fedora 28 and RHEL 7.5.

Message-Id: <20180730170236.15486-1-nsoffer@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
(cherry picked from commit c59734d)

On RHEL 7.5 we need to include <linux/falloc.h> for FALLOC_FL_* macros.
Without the macros, fallocate is never used and we fall back to manual
zeroing.

Here are examples runs with this change with a local file on ext4:

$ export SOCK=/tmp/nbd.sock
$ export FILE=/var/tmp/nbd.img
$ export BLOCK=/dev/loop2

$ src/nbdkit -f plugins/file/.libs/nbdkit-file-plugin.so file=$FILE -U $SOCK
$ time qemu-img convert -n -f raw -O raw /var/tmp/fedora-27.img nbd:unix:$SOCK

real    0m13.361s
user    0m0.127s
sys     0m0.668s

$ src/nbdkit -f plugins/file/.libs/nbdkit-file-plugin.so file=$BLOCK -U $SOCK
$ time qemu-img convert -n -f raw -O raw /var/tmp/fedora-27.img nbd:unix:$SOCK

real    0m13.491s
user    0m0.129s
sys     0m0.612s

Tested on Fedora 28 and RHEL 7.5.

Message-Id: <20180730180404.18736-1-nsoffer@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
(cherry picked from commit 8be884d)