unable to execute QEMU command 'transaction': Bitmap already exists

[ccrssaa]

Hi

could you please explain what's wrong with following scenario:

1. initial state, VM is running, no checkpoints yet

root@d02l:~# virsh list
 Id   Name          State
-----------------------------
 3    vm-template   running

root@d02l:~# virsh checkpoint-list vm-template
 Name   Creation Time
-----------------------

2. do backup

root@d02l:~# mkdir /backup/vm-template.20210501-1
root@d02l:~# virtnbdbackup -d vm-template -l full -o /backup/vm-template.20210501-1
2021-05-01 17:55:24 INFO common - printVersion: Version: 0.18 Arguments: /usr/local/bin/virtnbdbackup -d vm-template -l full -o /backup/vm-template.20210501-1
2021-05-01 17:55:24 INFO virtnbdbackup - main: Domain has 1 disks attached which support changed block tracking.
2021-05-01 17:55:24 INFO virtnbdbackup - main: Looking for checkpoints
2021-05-01 17:55:24 INFO virtnbdbackup - main: Using checkpoint name: virtnbdbackup
2021-05-01 17:55:24 INFO virtnbdbackup - main: Temporary scratch file target directory: /var/tmp
2021-05-01 17:55:24 INFO virtnbdbackup - main: Starting backup job.
2021-05-01 17:55:25 INFO libvirthelper - startBackup: Freezed filesystems.
2021-05-01 17:55:25 INFO libvirthelper - startBackup: Thawed filesystems.
2021-05-01 17:55:25 INFO virtnbdbackup - main: Started backup job with checkpoint, saving information.
2021-05-01 17:55:25 INFO nbdhelper - version: 1.6.3
2021-05-01 17:55:25 INFO virtnbdbackup - backupDisk: NDB Endpoint socket: /var/tmp/virtnbdbackup.11541
2021-05-01 17:55:25 INFO virtnbdbackup - backupDisk: Using nbd to query extents
2021-05-01 17:55:25 INFO virtnbdbackup - backupDisk: Got 3157 extents
2021-05-01 17:55:25 INFO virtnbdbackup - backupDisk: 42949672960 bytes disk size
saving disk vda: 100%|████████████████████████████████████████████████████████████████████| 3.65G/3.65G [01:13<00:00, 49.3MB/s]
2021-05-01 17:56:39 INFO virtnbdbackup - backupDisk: Backup of disk vda finished, file: /backup/vm-template.20210501-1/vda.full.data
2021-05-01 17:56:39 INFO virtnbdbackup - backupConfig: Saving VM config to: /backup/vm-template.20210501-1/vmconfig.virtnbdbackup.xml
2021-05-01 17:56:39 INFO virtnbdbackup - main: Stopping backup task
2021-05-01 17:56:39 INFO virtnbdbackup - main: Finished
root@d02l:~#

3. now VM have checkpoint:

root@d02l:~# virsh checkpoint-list vm-template
 Name            Creation Time
--------------------------------------------
 virtnbdbackup   2021-05-01 17:55:25 +0300

root@d02l:~#

4. stop VM and start it again

root@d02l:~# pcs resource disable vm-template
root@d02l:~# virsh list
 Id   Name   State
--------------------

root@d02l:~# pcs resource enable vm-template
root@d02l:~# virsh list
 Id   Name          State
-----------------------------
 4    vm-template   running

(It's a pacemaker cluster, therefore pcs resource disable/enable instead of
virsh shutdown/start)

5. checkpoint disappeared

root@d02l:~# virsh checkpoint-list vm-template
root@d02l:~# virsh checkpoint-list vm-template
 Name   Creation Time
-----------------------


 6. and virtnbdbackup is now getting "bitmap already exists" error

root@d02l:~# mkdir /backup/vm-template.20210501-2
root@d02l:~# virtnbdbackup -d vm-template -l full -o /backup/vm-template.20210501-2
2021-05-01 18:04:39 INFO common - printVersion: Version: 0.18 Arguments: /usr/local/bin/virtnbdbackup -d vm-template -l full -o /backup/vm-template.20210501-2
2021-05-01 18:04:39 INFO virtnbdbackup - main: Domain has 1 disks attached which support changed block tracking.
2021-05-01 18:04:39 INFO virtnbdbackup - main: Looking for checkpoints
2021-05-01 18:04:39 INFO virtnbdbackup - main: Using checkpoint name: virtnbdbackup
2021-05-01 18:04:39 INFO virtnbdbackup - main: Temporary scratch file target directory: /var/tmp
2021-05-01 18:04:39 INFO virtnbdbackup - main: Starting backup job.
2021-05-01 18:04:39 INFO libvirthelper - startBackup: Freezed filesystems.
2021-05-01 18:04:40 ERROR virtnbdbackup - main: internal error: unable to execute QEMU command 'transaction': Bitmap already exists: virtnbdbackup

7. and it's surely exists

root@d02l:~# virsh domfsthaw vm-template
Thawed 1 filesystem(s)

root@d02l:~# pcs resource disable vm-template
root@d02l:~# qemu-img info /var/lib/libvirt/images.drbd0/template.qcow2
image: /var/lib/libvirt/images.drbd0/template.qcow2
file format: qcow2
virtual size: 40 GiB (42949672960 bytes)
disk size: 1.61 GiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    bitmaps:
        [0]:
            flags:
                [0]: auto
            name: virtnbdbackup
            granularity: 65536
    refcount bits: 16
    corrupt: false
    extended l2: false
root@d02l:~#

8. but virtnbdbackup doesn't work until manual removal of bitmap

root@d02l:~# qemu-img bitmap --remove /var/lib/libvirt/images.drbd0/template.qcow2 virtnbdbackup
root@d02l:~# qemu-img info /var/lib/libvirt/images.drbd0/template.qcow2
image: /var/lib/libvirt/images.drbd0/template.qcow2
file format: qcow2
virtual size: 40 GiB (42949672960 bytes)
disk size: 1.61 GiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false
root@d02l:~#

9. libvirt and qemu are more or less recent this time:

root@d02l:~# virsh version
Compiled against library: libvirt 7.0.0
Using library: libvirt 7.0.0
Using API: QEMU 7.0.0
Running hypervisor: QEMU 5.2.0

[abbbi]

it seems your setup is losing checkpoint information if resource is removed.Thats an issue with your environment rather than virtnbdbackup. Information about checkpoints and its bitmaps is stored at two places:

• libvirt
• the qcow image of the disk

virtnbdbackup attempts to create the checkpoint because according to libvirt it doesnt exist, but the qcow image still
has the bitmap for the checkpoint set.

You have to find out why your "pcs resource disable vm-template" command removes the checkpoint information but leaves
the bitmap intact..., i have not seen such things happening wiht "virsh" so i guess you need to fix your setup.

You now need to use qemu direct commands like block-dirty-bitmap-remove to remove the dangling bitmap from the qcow image information using the QMP layer (see: https://qemu-project.gitlab.io/qemu/interop/bitmaps.html)

[ccrssaa]

Thanks, going to check, probably because VMs are transient in Pacemaker

Was able to dump checkpoint to file first

virsh checkpoint-dumpxml vm-template --checkpointname virtnbdbackup > tmp.xml

then recreate it after stop/start

virsh checkpoint-create vm-template --redefine --xmlfile tmp.xml

Thinking about implementing "redefine lost transient VM checkpoints using virtnbdbackup logs " as a workaround

[abbbi]

Thanks, going to check, probably because VMs are transient in Pacemaker

yes, most probably. I think qemu-img also has an bitmap option that can remove bitmaps from qcow files.

[abbbi]

Ive pushed a small fix to master branch. Virtnbdbackup should now thaw the filesystems if starting the backup job fails with an exception, that should at least not block your filesystems within the VM in case such an situation happens again :)