Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
streaming, tests: Fix for new, lower disk size limit in qemu > 5.2.0. 
This qemu commit reduced the maximum disk size supported from
2⁶³ - 512 bytes to 2⁶³ - 2³⁰ bytes:

  https://git.qemu.org/?p=qemu.git;a=commitdiff;h=8b1170012b1de6649c66ac1887f4df7e312abf3b

Since nbdkit was testing qemu’s older limit, we must adjust the
streaming plugin and a couple of tests which embedded this limit.

Unfortunately this is not the end of the bugs.  I discovered that
qemu-io (and possibly qemu parsing in general) uses floating point so
silently truncates everything at 53 bits.  Any large number which is
not a multiple of 2048 could be silently misinterpreted by the qemu
command line:

  https://www.mail-archive.com/qemu-devel@nongnu.org/msg770572.html

Thanks: Miroslav Rezanina, Daniel Berrangé
  • Loading branch information
@rwmjones
rwmjones committed Jan 7, 2021
1 parent ff8bc69 commit c3ec8c9
Show file tree
Hide file tree
Showing 4 changed files with 162 additions and 77 deletions.
6 changes: 2 additions & 4 deletions plugins/streaming/streaming.c
View file
Expand Up @@ -52,10 +52,8 @@ static enum { UNKNOWN_MODE, READ_MODE, WRITE_MODE } mode = UNKNOWN_MODE;
static char *filename = NULL;
static int fd = -1;

/* This is (2^63-1) & ~511. We have to round it down to the nearest
* sector size for qemu.
*/
static int64_t size = INT64_C(9223372036854775296);
/* This is 2^63 - 2^30. This is the largest disk that qemu supports. */
static int64_t size = INT64_C(9223372035781033984);

/* Flag if we have entered the unrecoverable error state because of
* a seek backwards.
Expand Down
14 changes: 14 additions & 0 deletions tests/functions.sh.in
View file
Expand Up @@ -47,6 +47,20 @@ SOEXT="@SOEXT@"
EXEEXT="@EXEEXT@"
abs_top_srcdir="@abs_top_srcdir@"

# Largest size of disk that qemu supports.
#
# Up to qemu 5.2.0 that was 2^63 - 512 (because of the requirement for
# a whole number of sectors). qemu > 5.2.0 reduced this to 2^63 - 2^30
# https://git.qemu.org/?p=qemu.git;a=commitdiff;h=8b1170012b1de6649c66ac1887f4df7e312abf3b
#
# qemu-io has further bugs which limit what we can test. See this
# thread:
# https://www.mail-archive.com/qemu-devel@nongnu.org/msg770572.html
#
# Because we might be using a 32 bit shell, express this as a constant
# rather than using $(()) calculations.
largest_qemu_disk=9223372035781033984

# cleanup_fn cmd [args]
#
# Run the command ‘cmd [args]’ when the test script exits. This is
Expand Down
111 changes: 74 additions & 37 deletions tests/test-memory-largest-for-qemu.sh
View file
Expand Up @@ -35,6 +35,7 @@

source ./functions.sh
set -e
set -x

requires qemu-io --version

Expand All @@ -44,15 +45,19 @@ rm -f $files
cleanup_fn rm -f $files

# Run nbdkit with memory plugin.
# size = (2^63-1) & ~511 which is the largest supported by qemu.
start_nbdkit -P memory-largest-for-qemu.pid -U $sock \
memory 9223372036854775296
start_nbdkit -P memory-largest-for-qemu.pid -U $sock memory $largest_qemu_disk

# Write some stuff to the beginning, middle and end.
#
# Ideally we would write the last sector only. However qemu-io has a
# bug where offsets are calculated using floating point numbers and
# silently truncated to 53 bits of precision. Reading the last 1024
# bytes happens to give the correct result here.
# https://www.mail-archive.com/qemu-devel@nongnu.org/msg770572.html
qemu-io -f raw "nbd+unix://?socket=$sock" \
-c 'w -P 1 0 512' \
-c 'w -P 2 1000000001 65536' \
-c 'w -P 3 9223372036854774784 512'
-c 'w -P 3 9223372035781032960 1024'

qemu-io -r -f raw "nbd+unix://?socket=$sock" \
-c 'r -v 0 512' | grep -E '^[[:xdigit:]]+:' > memory-largest-for-qemu.out
Expand Down Expand Up @@ -176,39 +181,71 @@ then
fi

qemu-io -r -f raw "nbd+unix://?socket=$sock" \
-c 'r -v 9223372036854774784 512' | grep -E '^[[:xdigit:]]+:' > memory-largest-for-qemu.out
if [ "$(cat memory-largest-for-qemu.out)" != "7ffffffffffffc00: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffc10: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffc20: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffc30: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffc40: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffc50: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffc60: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffc70: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffc80: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffc90: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffca0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffcb0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffcc0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffcd0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffce0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffcf0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffd00: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffd10: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffd20: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffd30: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffd40: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffd50: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffd60: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffd70: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffd80: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffd90: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffda0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffdb0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffdc0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffdd0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffde0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7ffffffffffffdf0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................" ]
-c 'r -v 9223372035781032960 1024' | grep -E '^[[:xdigit:]]+:' > memory-largest-for-qemu.out
if [ "$(cat memory-largest-for-qemu.out)" != "7fffffffbffffc00: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffc10: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffc20: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffc30: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffc40: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffc50: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffc60: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffc70: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffc80: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffc90: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffca0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffcb0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffcc0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffcd0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffce0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffcf0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffd00: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffd10: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffd20: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffd30: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffd40: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffd50: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffd60: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffd70: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffd80: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffd90: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffda0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffdb0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffdc0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffdd0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffde0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffdf0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffe00: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffe10: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffe20: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffe30: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffe40: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffe50: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffe60: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffe70: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffe80: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffe90: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffea0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffeb0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffec0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffed0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffee0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffef0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbfffff00: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbfffff10: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbfffff20: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbfffff30: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbfffff40: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbfffff50: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbfffff60: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbfffff70: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbfffff80: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbfffff90: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbfffffa0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbfffffb0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbfffffc0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbfffffd0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbfffffe0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................
7fffffffbffffff0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 ................" ]
then
echo "$0: unexpected memory:"
cat memory-largest-for-qemu.out
Expand Down

0 comments on commit c3ec8c9

Please sign in to comment.