Merge remote-tracking branch 'remotes/ericb/tags/pull-bitmaps-2020-06…

…-09' into staging

bitmaps patches for 2020-06-09

- documenation fix
- various improvements to qcow2.py program used in iotests

# gpg: Signature made Tue 09 Jun 2020 21:50:35 BST
# gpg:                using RSA key 71C2CC22B1C4602927D2F3AAA7A16B4A2527436A
# gpg: Good signature from "Eric Blake <eblake@redhat.com>" [full]
# gpg:                 aka "Eric Blake (Free Software Programmer) <ebb9@byu.net>" [full]
# gpg:                 aka "[jpeg image of size 6874]" [full]
# Primary key fingerprint: 71C2 CC22 B1C4 6029 27D2  F3AA A7A1 6B4A 2527 436A

* remotes/ericb/tags/pull-bitmaps-2020-06-09:
  iotests: Fix 291 across more file systems
  qcow2_format.py: dump bitmaps header extension
  qcow2: QcowHeaderExtension print names for extension magics
  qcow2_format: refactor QcowHeaderExtension as a subclass of Qcow2Struct
  qcow2_format.py: QcowHeaderExtension: add dump method
  qcow2_format.py: add field-formatting class
  qcow2_format.py: separate generic functionality of structure classes
  qcow2_format.py: use strings to specify c-type of struct fields
  qcow2_format.py: use modern string formatting
  qcow2_format.py: use tuples instead of lists for fields
  qcow2_format.py: drop new line printing at end of dump()
  qcow2.py: move qcow2 format classes to separate module
  qcow2.py: add licensing blurb
  qcow2.py: python style fixes
  qemu-img: Fix doc typo for 'bitmap' subcommand

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

docs/tools/qemu-img.rst

@@ -300,7 +300,7 @@ Command description:
 
   ``--disable`` to change *BITMAP* to stop recording future edits.
 
-  ``--merge`` to merge the contents of *SOURCE_BITMAP* into *BITMAP*.
+  ``--merge`` to merge the contents of the *SOURCE* bitmap into *BITMAP*.
 
   Additional options include ``-g`` which sets a non-default
   *GRANULARITY* for ``--add``, and ``-b`` and ``-F`` which select an

tests/qemu-iotests/031.out

@@ -25,7 +25,7 @@ refcount_order            4
 header_length             72
 
 Header extension:
-magic                     0x12345678
+magic                     0x12345678 (<unknown>)
 length                    31
 data                      'This is a test header extension'
 
@@ -53,7 +53,7 @@ refcount_order            4
 header_length             72
 
 Header extension:
-magic                     0x12345678
+magic                     0x12345678 (<unknown>)
 length                    31
 data                      'This is a test header extension'
 
@@ -81,12 +81,12 @@ refcount_order            4
 header_length             72
 
 Header extension:
-magic                     0xe2792aca
+magic                     0xe2792aca (Backing format)
 length                    11
 data                      'host_device'
 
 Header extension:
-magic                     0x12345678
+magic                     0x12345678 (<unknown>)
 length                    31
 data                      'This is a test header extension'
 
@@ -116,12 +116,12 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0x6803f857
+magic                     0x6803f857 (Feature table)
 length                    336
 data                      <binary>
 
 Header extension:
-magic                     0x12345678
+magic                     0x12345678 (<unknown>)
 length                    31
 data                      'This is a test header extension'
 
@@ -149,12 +149,12 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0x6803f857
+magic                     0x6803f857 (Feature table)
 length                    336
 data                      <binary>
 
 Header extension:
-magic                     0x12345678
+magic                     0x12345678 (<unknown>)
 length                    31
 data                      'This is a test header extension'
 
@@ -182,17 +182,17 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0xe2792aca
+magic                     0xe2792aca (Backing format)
 length                    11
 data                      'host_device'
 
 Header extension:
-magic                     0x6803f857
+magic                     0x6803f857 (Feature table)
 length                    336
 data                      <binary>
 
 Header extension:
-magic                     0x12345678
+magic                     0x12345678 (<unknown>)
 length                    31
 data                      'This is a test header extension'
 

tests/qemu-iotests/036.out

@@ -25,7 +25,7 @@ incompatible_features     []
 compatible_features       []
 autoclear_features        [63]
 Header extension:
-magic                     0x6803f857
+magic                     0x6803f857 (Feature table)
 length                    336
 data                      <binary>
 
@@ -37,7 +37,7 @@ incompatible_features     []
 compatible_features       []
 autoclear_features        []
 Header extension:
-magic                     0x6803f857
+magic                     0x6803f857 (Feature table)
 length                    336
 data                      <binary>
 

tests/qemu-iotests/061.out

@@ -25,7 +25,7 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0x6803f857
+magic                     0x6803f857 (Feature table)
 length                    336
 data                      <binary>
 
@@ -83,7 +83,7 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0x6803f857
+magic                     0x6803f857 (Feature table)
 length                    336
 data                      <binary>
 
@@ -139,7 +139,7 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0x6803f857
+magic                     0x6803f857 (Feature table)
 length                    336
 data                      <binary>
 
@@ -194,7 +194,7 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0x6803f857
+magic                     0x6803f857 (Feature table)
 length                    336
 data                      <binary>
 
@@ -263,7 +263,7 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0x6803f857
+magic                     0x6803f857 (Feature table)
 length                    336
 data                      <binary>
 
@@ -325,7 +325,7 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0x6803f857
+magic                     0x6803f857 (Feature table)
 length                    336
 data                      <binary>
 
@@ -354,7 +354,7 @@ refcount_order            4
 header_length             112
 
 Header extension:
-magic                     0x6803f857
+magic                     0x6803f857 (Feature table)
 length                    336
 data                      <binary>
 

tests/qemu-iotests/291

@@ -62,6 +62,8 @@ $QEMU_IO -c 'w 1M 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io
 $QEMU_IMG bitmap --disable -f $IMGFMT "$TEST_IMG" b1
 $QEMU_IMG bitmap --enable -f $IMGFMT "$TEST_IMG" b2
 $QEMU_IO -c 'w 2M 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io
+echo "Check resulting qcow2 header extensions:"
+$PYTHON qcow2.py "$TEST_IMG" dump-header-exts
 
 echo
 echo "=== Bitmap preservation not possible to non-qcow2 ==="
@@ -77,7 +79,7 @@ echo
 
 # Only bitmaps from the active layer are copied
 $QEMU_IMG convert --bitmaps -O qcow2 "$TEST_IMG.orig" "$TEST_IMG"
-$QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific
+_img_info --format-specific
 # But we can also merge in bitmaps from other layers.  This test is a bit
 # contrived to cover more code paths, in reality, you could merge directly
 # into b0 without going through tmp
@@ -87,7 +89,9 @@ $QEMU_IMG bitmap --add --merge b0 -b "$TEST_IMG.base" -F $IMGFMT \
 $QEMU_IMG bitmap --merge tmp -f $IMGFMT "$TEST_IMG" b0
 $QEMU_IMG bitmap --remove --image-opts \
     driver=$IMGFMT,file.driver=file,file.filename="$TEST_IMG" tmp
-$QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific
+_img_info --format-specific
+echo "Check resulting qcow2 header extensions:"
+$PYTHON qcow2.py "$TEST_IMG" dump-header-exts
 
 echo
 echo "=== Check bitmap contents ==="

tests/qemu-iotests/291.out

@@ -14,6 +14,25 @@ wrote 1048576/1048576 bytes at offset 1048576
 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 wrote 1048576/1048576 bytes at offset 2097152
 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+Check resulting qcow2 header extensions:
+Header extension:
+magic                     0xe2792aca (Backing format)
+length                    5
+data                      'qcow2'
+
+Header extension:
+magic                     0x6803f857 (Feature table)
+length                    336
+data                      <binary>
+
+Header extension:
+magic                     0x23852875 (Bitmaps)
+length                    24
+nb_bitmaps                2
+reserved32                0
+bitmap_directory_size     0x40
+bitmap_directory_offset   0x510000
+
 
 === Bitmap preservation not possible to non-qcow2 ===
 
@@ -24,7 +43,7 @@ qemu-img: Format driver 'raw' does not support bitmaps
 image: TEST_DIR/t.IMGFMT
 file format: IMGFMT
 virtual size: 10 MiB (10485760 bytes)
-disk size: 4.39 MiB
+cluster_size: 65536
 Format specific information:
     compat: 1.1
     compression type: zlib
@@ -44,7 +63,7 @@ Format specific information:
 image: TEST_DIR/t.IMGFMT
 file format: IMGFMT
 virtual size: 10 MiB (10485760 bytes)
-disk size: 4.48 MiB
+cluster_size: 65536
 Format specific information:
     compat: 1.1
     compression type: zlib
@@ -65,6 +84,20 @@ Format specific information:
             granularity: 65536
     refcount bits: 16
     corrupt: false
+Check resulting qcow2 header extensions:
+Header extension:
+magic                     0x6803f857 (Feature table)
+length                    336
+data                      <binary>
+
+Header extension:
+magic                     0x23852875 (Bitmaps)
+length                    24
+nb_bitmaps                3
+reserved32                0
+bitmap_directory_size     0x60
+bitmap_directory_offset   0x520000
+
 
 === Check bitmap contents ===