Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

System snapshots might not be carried over live migrations #6363

Closed
3 tasks
OpenNebulaSupport opened this issue Oct 20, 2023 · 0 comments · Fixed by OpenNebula/docs#2725 or OpenNebula/docs#2726
Closed
3 tasks

System snapshots might not be carried over live migrations #6363

OpenNebulaSupport opened this issue Oct 20, 2023 · 0 comments · Fixed by OpenNebula/docs#2725 or OpenNebula/docs#2726
Assignees
@rsmontero @dann1
Labels
Category: KVM Priority: Normal Sponsored Status: Accepted Type: Bug
Milestone
Release 6.8.1

Comments

@OpenNebulaSupport
Copy link
Collaborator

Description
It might be possible that after a migration a system snapshot is not carried over. This results in opennebula displaying a snapshot that doesn't exist at the hypervisor level, which fails to be deleted.

To Reproduce

  • Create a VM
  • Take a system snapshot
  • Power off the VM. The issue doesn't happen without this poweroff.
  • Resume the VM
  • Live migrate the VM
  • Try to delete the system snapshot

image

Expected behavior
Snapshot should be carried over

Details

  • Version: tested on 6.4 but likely happens in every version

Additional context

VM log

Fri Oct 20 14:33:38 2023 [Z0][VM][I]: New state is ACTIVE
Fri Oct 20 14:33:38 2023 [Z0][VM][I]: New LCM state is PROLOG
Fri Oct 20 14:33:39 2023 [Z0][VM][I]: New LCM state is BOOT
Fri Oct 20 14:33:39 2023 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/10/deployment.0
Fri Oct 20 14:33:39 2023 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_context.
Fri Oct 20 14:33:39 2023 [Z0][VMM][I]: ExitCode: 0
Fri Oct 20 14:33:39 2023 [Z0][VMM][I]: Successfully execute network driver operation: pre.
Fri Oct 20 14:33:39 2023 [Z0][VMM][I]: ExitCode: 0
Fri Oct 20 14:33:39 2023 [Z0][VMM][I]: Successfully execute virtualization driver operation: /bin/mkdir -p.
Fri Oct 20 14:33:39 2023 [Z0][VMM][I]: ExitCode: 0
Fri Oct 20 14:33:39 2023 [Z0][VMM][I]: Successfully execute virtualization driver operation: /bin/cat - >/var/lib/one//datastores/0/10/vm.xml.
Fri Oct 20 14:33:39 2023 [Z0][VMM][I]: ExitCode: 0
Fri Oct 20 14:33:39 2023 [Z0][VMM][I]: Successfully execute virtualization driver operation: /bin/cat - >/var/lib/one//datastores/0/10/ds.xml.
Fri Oct 20 14:33:40 2023 [Z0][LCM][I]: VM reported RUNNING by the drivers
Fri Oct 20 14:33:40 2023 [Z0][VM][I]: New LCM state is RUNNING
Fri Oct 20 14:33:41 2023 [Z0][VMM][I]: ExitCode: 0
Fri Oct 20 14:33:41 2023 [Z0][VMM][I]: Successfully execute virtualization driver operation: deploy.
Fri Oct 20 14:33:41 2023 [Z0][VMM][I]: ExitCode: 0
Fri Oct 20 14:33:41 2023 [Z0][VMM][I]: Successfully execute network driver operation: post.
Fri Oct 20 14:33:50 2023 [Z0][VM][I]: New LCM state is HOTPLUG_SNAPSHOT
Fri Oct 20 14:33:51 2023 [Z0][VMM][I]: VM Snapshot successfully created.
Fri Oct 20 14:33:51 2023 [Z0][VM][I]: New LCM state is RUNNING
Fri Oct 20 14:33:55 2023 [Z0][VM][I]: New LCM state is SHUTDOWN_POWEROFF
Fri Oct 20 14:34:06 2023 [Z0][VMM][I]: ExitCode: 0
Fri Oct 20 14:34:06 2023 [Z0][VMM][I]: Successfully execute virtualization driver operation: shutdown.
Fri Oct 20 14:34:06 2023 [Z0][VMM][I]: ExitCode: 0
Fri Oct 20 14:34:06 2023 [Z0][VMM][I]: Successfully execute network driver operation: clean.
Fri Oct 20 14:34:06 2023 [Z0][VM][I]: New state is POWEROFF
Fri Oct 20 14:34:06 2023 [Z0][VM][I]: New LCM state is LCM_INIT
Fri Oct 20 14:34:18 2023 [Z0][VM][I]: New state is ACTIVE
Fri Oct 20 14:34:18 2023 [Z0][VM][I]: New LCM state is BOOT_POWEROFF
Fri Oct 20 14:34:18 2023 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/10/deployment.1
Fri Oct 20 14:34:20 2023 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_context.
Fri Oct 20 14:34:20 2023 [Z0][VMM][I]: ExitCode: 0
Fri Oct 20 14:34:20 2023 [Z0][VMM][I]: Successfully execute network driver operation: pre.
Fri Oct 20 14:34:20 2023 [Z0][VMM][I]: ExitCode: 0
Fri Oct 20 14:34:20 2023 [Z0][VMM][I]: Successfully execute virtualization driver operation: /bin/mkdir -p.
Fri Oct 20 14:34:20 2023 [Z0][VMM][I]: ExitCode: 0
Fri Oct 20 14:34:20 2023 [Z0][VMM][I]: Successfully execute virtualization driver operation: /bin/cat - >/var/lib/one//datastores/0/10/vm.xml.
Fri Oct 20 14:34:20 2023 [Z0][VMM][I]: ExitCode: 0
Fri Oct 20 14:34:20 2023 [Z0][VMM][I]: Successfully execute virtualization driver operation: /bin/cat - >/var/lib/one//datastores/0/10/ds.xml.
Fri Oct 20 14:34:21 2023 [Z0][VMM][I]: ExitCode: 0
Fri Oct 20 14:34:21 2023 [Z0][VMM][I]: Successfully execute virtualization driver operation: deploy.
Fri Oct 20 14:34:21 2023 [Z0][VMM][I]: ExitCode: 0
Fri Oct 20 14:34:21 2023 [Z0][VMM][I]: Successfully execute network driver operation: post.
Fri Oct 20 14:34:21 2023 [Z0][VM][I]: New LCM state is RUNNING
Fri Oct 20 14:34:37 2023 [Z0][VM][I]: New LCM state is MIGRATE
Fri Oct 20 14:34:38 2023 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_premigrate.
Fri Oct 20 14:34:39 2023 [Z0][VMM][I]: ExitCode: 0
Fri Oct 20 14:34:39 2023 [Z0][VMM][I]: Successfully execute network driver operation: pre.
Fri Oct 20 14:34:42 2023 [Z0][VMM][I]: ExitCode: 0
Fri Oct 20 14:34:42 2023 [Z0][VMM][I]: Successfully execute virtualization driver operation: migrate.
Fri Oct 20 14:34:42 2023 [Z0][VMM][I]: ExitCode: 0
Fri Oct 20 14:34:42 2023 [Z0][VMM][I]: Successfully execute network driver operation: clean.
Fri Oct 20 14:34:42 2023 [Z0][VMM][I]: ExitCode: 0
Fri Oct 20 14:34:42 2023 [Z0][VMM][I]: Successfully execute network driver operation: post.
Fri Oct 20 14:34:42 2023 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_postmigrate.
Fri Oct 20 14:34:42 2023 [Z0][VM][I]: New LCM state is RUNNING
Fri Oct 20 14:34:48 2023 [Z0][VM][I]: New LCM state is HOTPLUG_SNAPSHOT
Fri Oct 20 14:34:48 2023 [Z0][VMM][I]: Command execution failed (exit code: 1): 'if [ -x "/var/tmp/one/vmm/kvm/snapshot_delete" ]; then /var/tmp/one/vmm/kvm/snapshot_delete 0f903ff7-3d68-4b45-a09d-169f0c9e1e3c snap-0 10 ubuntu2204-kvm-qcow2-6-4-hcnii-1.test; else exit 42; fi'
Fri Oct 20 14:34:48 2023 [Z0][VMM][E]: snapshot_delete: Command "virsh --connect qemu:///system snapshot-delete 0f903ff7-3d68-4b45-a09d-169f0c9e1e3c snap-0" failed: error: Domain snapshot not found: no domain snapshot with matching name 'snap-0'
Fri Oct 20 14:34:48 2023 [Z0][VMM][I]: Could not delete snapshot snap-0 for domain 0f903ff7-3d68-4b45-a09d-169f0c9e1e3c.
Fri Oct 20 14:34:48 2023 [Z0][VMM][E]: SNAPSHOTDELETE: ERROR: snapshot_delete: Command "virsh --connect qemu:///system snapshot-delete 0f903ff7-3d68-4b45-a09d-169f0c9e1e3c snap-0" failed: error: Domain snapshot not found: no domain snapshot with matching name 'snap-0' Could not delete snapshot snap-0 for domain 0f903ff7-3d68-4b45-a09d-169f0c9e1e3c.
Fri Oct 20 14:34:48 2023 [Z0][VM][I]: New LCM state is RUNNING

VM template

{
  "VM": {
    "ID": "10",
    "UID": "0",
    "GID": "0",
    "UNAME": "oneadmin",
    "GNAME": "oneadmin",
    "NAME": "alpine-10",
    "PERMISSIONS": {
      "OWNER_U": "1",
      "OWNER_M": "1",
      "OWNER_A": "0",
      "GROUP_U": "0",
      "GROUP_M": "0",
      "GROUP_A": "0",
      "OTHER_U": "0",
      "OTHER_M": "0",
      "OTHER_A": "0"
    },
    "LAST_POLL": "1697813114",
    "STATE": "3",
    "LCM_STATE": "3",
    "PREV_STATE": "3",
    "PREV_LCM_STATE": "3",
    "RESCHED": "0",
    "STIME": "1697812417",
    "ETIME": "0",
    "DEPLOY_ID": "0f903ff7-3d68-4b45-a09d-169f0c9e1e3c",
    "MONITORING": {
      "CPU": "0.0",
      "DISKRDBYTES": "2048",
      "DISKRDIOPS": "2",
      "DISKWRBYTES": "9216",
      "DISKWRIOPS": "6",
      "DISK_SIZE": [
        {
          "ID": "0",
          "SIZE": "90"
        },
        {
          "ID": "1",
          "SIZE": "1"
        }
      ],
      "ID": "10",
      "MEMORY": "139556",
      "NETRX": "1076",
      "NETTX": "1740",
      "TIMESTAMP": "1697813114"
    },
    "TEMPLATE": {
      "AUTOMATIC_DS_REQUIREMENTS": "(\"CLUSTERS/ID\" @> 0) & (TM_MAD = \"qcow2\")",
      "AUTOMATIC_NIC_REQUIREMENTS": "(\"CLUSTERS/ID\" @> 0)",
      "AUTOMATIC_REQUIREMENTS": "(CLUSTER_ID = 0) & !(PUBLIC_CLOUD = YES) & !(PIN_POLICY = PINNED)",
      "CONTEXT": {
        "DISK_ID": "1",
        "ETH0_DNS": "",
        "ETH0_EXTERNAL": "",
        "ETH0_GATEWAY": "192.168.150.1",
        "ETH0_GATEWAY6": "",
        "ETH0_IP": "192.168.150.100",
        "ETH0_IP6": "",
        "ETH0_IP6_GATEWAY": "",
        "ETH0_IP6_METHOD": "",
        "ETH0_IP6_METRIC": "",
        "ETH0_IP6_PREFIX_LENGTH": "",
        "ETH0_IP6_ULA": "",
        "ETH0_MAC": "02:00:c0:a8:96:64",
        "ETH0_MASK": "",
        "ETH0_METHOD": "",
        "ETH0_METRIC": "",
        "ETH0_MTU": "",
        "ETH0_NETWORK": "",
        "ETH0_SEARCH_DOMAIN": "",
        "ETH0_VLAN_ID": "",
        "ETH0_VROUTER_IP": "",
        "ETH0_VROUTER_IP6": "",
        "ETH0_VROUTER_MANAGEMENT": "",
        "NETWORK": "YES",
        "SSH_PUBLIC_KEY": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYz+lkZoNyspRhrtXDKFN3cIEwN3w08mz0YGKpVDIiV0+/vgG8dAUQ70Irs3m83W9BHN+vNjKPgKcF+X+sSfxniOtavahxGCRjAhhs1IVm196C5ODbSgXVUWULdtmMHelXbLBJ8X340h/UO+eQ6eRLaRfslXUsgRqremVcvCCPz4LIuRiliGWiELAmqYcY+1zJLeg3QV2Pgn5vschM9e/A4AseKO+HnbGB/I5tnoeZT/Gc3FGfUZLNFVB2XsVGAEEzkqO8VI2msB7MCAZBHffIK6WfLIYgGP6Ha2JT1NWJU7Ncj9Xuql0ElF01VwWMDWzqc0DOiVSsTL89ugJKU6+h one",
        "TARGET": "hda"
      },
      "CPU": "0.1",
      "DISK": [
        {
          "ALLOW_ORPHANS": "FORMAT",
          "CLONE": "YES",
          "CLONE_TARGET": "SYSTEM",
          "CLUSTER_ID": "0",
          "DATASTORE": "default",
          "DATASTORE_ID": "1",
          "DEV_PREFIX": "vd",
          "DISK_ID": "0",
          "DISK_SNAPSHOT_TOTAL_SIZE": "0",
          "DISK_TYPE": "FILE",
          "DRIVER": "qcow2",
          "FORMAT": "qcow2",
          "IMAGE": "alpine",
          "IMAGE_ID": "0",
          "IMAGE_STATE": "2",
          "LN_TARGET": "NONE",
          "ORIGINAL_SIZE": "256",
          "READONLY": "NO",
          "SAVE": "NO",
          "SIZE": "256",
          "SOURCE": "/var/lib/one//datastores/1/393da3a7d5d93a4e5e50e12a1c829d19",
          "TARGET": "vda",
          "TM_MAD": "qcow2",
          "TYPE": "FILE"
        }
      ],
      "GRAPHICS": {
        "LISTEN": "0.0.0.0",
        "PORT": "5910",
        "TYPE": "vnc"
      },
      "MEMORY": "96",
      "NIC": [
        {
          "AR_ID": "0",
          "BRIDGE": "br0",
          "BRIDGE_TYPE": "linux",
          "CLUSTER_ID": "0",
          "GATEWAY": "192.168.150.1",
          "IP": "192.168.150.100",
          "MAC": "02:00:c0:a8:96:64",
          "MODEL": "virtio",
          "NAME": "NIC0",
          "NETWORK": "public",
          "NETWORK_ID": "0",
          "NIC_ID": "0",
          "SECURITY_GROUPS": "0",
          "TARGET": "one-10-0",
          "VN_MAD": "dummy"
        }
      ],
      "NIC_DEFAULT": {
        "MODEL": "virtio"
      },
      "OS": {
        "UUID": "0f903ff7-3d68-4b45-a09d-169f0c9e1e3c"
      },
      "SECURITY_GROUP_RULE": [
        {
          "PROTOCOL": "ALL",
          "RULE_TYPE": "OUTBOUND",
          "SECURITY_GROUP_ID": "0",
          "SECURITY_GROUP_NAME": "default"
        },
        {
          "PROTOCOL": "ALL",
          "RULE_TYPE": "INBOUND",
          "SECURITY_GROUP_ID": "0",
          "SECURITY_GROUP_NAME": "default"
        }
      ],
      "SNAPSHOT": [
        {
          "HYPERVISOR_ID": "snap-0",
          "NAME": "snapshot-0",
          "SNAPSHOT_ID": "0",
          "SYSTEM_DISK_SIZE": "96",
          "TIME": "1697812430"
        }
      ],
      "TEMPLATE_ID": "0",
      "TM_MAD_SYSTEM": "qcow2",
      "VMID": "10"
    },
    "USER_TEMPLATE": {
      "ARCH": "x86_64",
      "ERROR": "Fri Oct 20 14:34:48 2023: SNAPSHOTDELETE: ERROR: snapshot_delete: Command \"virsh --connect qemu:///system snapshot-delete 0f903ff7-3d68-4b45-a09d-169f0c9e1e3c snap-0\" failed: error: Domain snapshot not found: no domain snapshot with matching name 'snap-0' Could not delete snapshot snap-0 for domain 0f903ff7-3d68-4b45-a09d-169f0c9e1e3c."
    },
    "HISTORY_RECORDS": {
      "HISTORY": [
        {
          "OID": "10",
          "SEQ": "0",
          "HOSTNAME": "ubuntu2204-kvm-qcow2-6-4-hcnii-2.test",
          "HID": "1",
          "CID": "0",
          "STIME": "1697812418",
          "ETIME": "1697812458",
          "VM_MAD": "kvm",
          "TM_MAD": "qcow2",
          "DS_ID": "0",
          "PSTIME": "1697812418",
          "PETIME": "1697812419",
          "RSTIME": "1697812419",
          "RETIME": "1697812446",
          "ESTIME": "0",
          "EETIME": "0",
          "ACTION": "19",
          "UID": "0",
          "GID": "0",
          "REQUEST_ID": "6976"
        },
        {
          "OID": "10",
          "SEQ": "1",
          "HOSTNAME": "ubuntu2204-kvm-qcow2-6-4-hcnii-2.test",
          "HID": "1",
          "CID": "0",
          "STIME": "1697812458",
          "ETIME": "1697812482",
          "VM_MAD": "kvm",
          "TM_MAD": "qcow2",
          "DS_ID": "0",
          "PSTIME": "0",
          "PETIME": "0",
          "RSTIME": "1697812458",
          "RETIME": "1697812482",
          "ESTIME": "0",
          "EETIME": "0",
          "ACTION": "2",
          "UID": "0",
          "GID": "0",
          "REQUEST_ID": "1520"
        },
        {
          "OID": "10",
          "SEQ": "2",
          "HOSTNAME": "ubuntu2204-kvm-qcow2-6-4-hcnii-1.test",
          "HID": "0",
          "CID": "0",
          "STIME": "1697812477",
          "ETIME": "0",
          "VM_MAD": "kvm",
          "TM_MAD": "qcow2",
          "DS_ID": "0",
          "PSTIME": "0",
          "PETIME": "0",
          "RSTIME": "1697812482",
          "RETIME": "0",
          "ESTIME": "0",
          "EETIME": "0",
          "ACTION": "0",
          "UID": "-1",
          "GID": "-1",
          "REQUEST_ID": "-1"
        }
      ]
    }
  }
}

Progress Status

  • Code committed
  • Testing - QA
  • Documentation (Release notes - resolved issues, compatibility, known issues)
@OpenNebulaSupport OpenNebulaSupport added this to the Release 6.8.1 milestone Oct 20, 2023
dann1 added a commit that referenced this issue Oct 24, 2023
@dann1
B #6363: 64 patch
2cb4c09
dann1 added a commit to dann1/docs that referenced this issue Oct 25, 2023
@dann1
B OpenNebula/one#6363: Update resolved issues
715b1ff
@dann1 dann1 linked a pull request Oct 25, 2023 that will close this issue
Update resolved issues OpenNebula/docs#2725
Merged
dann1 added a commit to dann1/docs that referenced this issue Oct 25, 2023
@dann1
B OpenNebula/one#6363: Update resolved issues
02aae71
@dann1 dann1 linked a pull request Oct 25, 2023 that will close this issue
B OpenNebula/one#6363: Update resolved issues OpenNebula/docs#2726
Merged
dann1 added a commit that referenced this issue Oct 25, 2023
@dann1
B #6363: Preserve shared flag
236aea3
rsmontero added a commit to OpenNebula/docs that referenced this issue Oct 26, 2023
@dann1 @rsmontero
B OpenNebula/one#6363: Update resolved issues (#2725)
ea28fda 
Co-authored-by: Ruben S. Montero <rsmontero@opennebula.org>
rsmontero pushed a commit to OpenNebula/docs that referenced this issue Oct 26, 2023
@dann1
B OpenNebula/one#6363: Update resolved issues (#2726)
a3d6093
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rsmontero rsmontero

@dann1 dann1

Labels
Category: KVM Priority: Normal Sponsored Status: Accepted Type: Bug
Projects
None yet
Milestone
Release 6.8.1
Development

Successfully merging a pull request may close this issue.

Update resolved issues dann1/docs
B OpenNebula/one#6363: Update resolved issues dann1/docs
3 participants
@rsmontero @dann1 @OpenNebulaSupport