copy --refresh for vm from another server fails with: Snapshot cannot be restored due to subsequent snapshot(s)

[hi-ko]

• Distribution: ubuntu
• Distribution version: 22.04.3 LTS
  • Kernel version: 5.15.0-91-generic
  • Incus version: 0.5.1
  • Storage backend in use: zfs

copy --refresh for vm to another server fails on subsequent copy

When copying a vm using copy --refresh from another server, the first attempt is successful but once changes and snapshots added the copy fails with:

incus copy incus2:test-copy1 test-copy1 --refresh
Error: Failed instance creation: Error transferring instance data: Failed migration on target: Failed creating instance on target: Snapshot "snap0" cannot be restored due to subsequent snapshot(s). Set zfs.remove_snapshots to override

incus storage volume list zpool1|grep test-copy1
| virtual-machine            | test-copy1              |             | block        | 1       |
| virtual-machine (snapshot) | test-copy1/snap0        |             | block        | 0       |
root@lxd02:~# zfs list -t snapshot |grep test-copy1
zpool1/virtual-machines/test-copy1@snapshot-snap0                               60K      -     7.11M  -
zpool1/virtual-machines/test-copy1@snapshot-snap1                               60K      -     7.11M  -
zpool1/virtual-machines/test-copy1.block@snapshot-snap0                          0B      -      640M  -

I can remove the snapshot-snap1 via zfs destroy but the snapshot will come back on next copy attempt and produces the error again. It looks like the zfs send/receive is done too early for vms?

Steps to reproduce

1. create vm on server1
   incus launch images:ubuntu/22.04/cloud test-copy1 --vm
2. Create snapshot
   incus snapshot create test-copy1
3. On server2 do initial copy
   incus copy server1:test-copy1 test-copy1 --refresh
4. On server1 do changes and create new snapshot

   incus shell test-copy1
   dd if=/dev/urandom of=testfile count=10 bs=1M;exit
   incus snapshot create test-copy1
   

5. copy angain from server2
   incus copy server1:test-copy1 test-copy1 --refresh
Error: Failed instance creation: Error transferring instance data: Failed migration on target: Failed creating instance on target: Snapshot "snap0" cannot be restored due to subsequent snapshot(s). Set zfs.remove_snapshots to override

Information to attach

incus config show  test-copy1
architecture: x86_64
config:
  image.architecture: amd64
  image.description: Ubuntu jammy amd64 (20240131_07:42)
  image.os: Ubuntu
  image.release: jammy
  image.serial: "20240131_07:42"
  image.type: disk-kvm.img
  image.variant: cloud
  volatile.base_image: b984c16267ca9d6ed22606193f6a30f394d4e7e0f76839357a67ca12027a93ec
  volatile.cloud-init.instance-id: 8e6bdd84-3dc4-4318-93e2-1b088ec50577
  volatile.eth0.host_name: tap0c081c0a
  volatile.eth0.hwaddr: 00:16:3e:4e:be:e2
  volatile.last_state.ready: "false"
  volatile.uuid: 96c7d6b8-9756-40c2-83f9-20cc65cdab0a
  volatile.uuid.generation: 96c7d6b8-9756-40c2-83f9-20cc65cdab0a
  volatile.vsock_id: "326816208"
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""

[stgraber]

Got it reproduced. Interestingly this only seems to affect virtual machines, containers don't suffer the same issue.

[stgraber]

Oh, I see the problem... working on a fix now.