Replies: 16 comments
|
I had introduced the kernel panic with the following command "echo c > /proc/sysrq-trigger" The kvm host went to alert state and no ha entry was created for the vm under "select * from op_ha_work" 
With the fix #13373 The host goes into down state and the vm ha is triggered Could you please provide the command you used to trigger the kernel panic and also the values of the global settings |
|
Hi @kiranchavala , the kernel panic was triggered in the exact same way you did it 
To clarify, the issue is not that HA fails to trigger; the logs clearly show it attempting to restart the VM on another host. The actual problem is that after the first attempt fails, the process stops after the second try and gives up entirely, even though the VM remains offline. Let me know if you need anything else or if there's something you think i'm doing wrong |
|
@gusmef thanks for the update In my lab , the HA doesn't start for the vm and host stays in alert state What is the host status in your case ? Also what is value of the global setting force.ha |
|
@kiranchavala the global settings are The host status was "Alert". I think we somehow tricked the HA into starting by declaring the "dead" host as degraded (manually from the web gui). |
|
Hi All, I am also facing the same issue for my cloudstack environment 4.20.3.0 with ceph storage. Before, VM HA will not work after for the heartbeat detection i added the NFS storage for this, Now VM HA will work, But IF host have an 20 VMs most of the VMs are restarted to another hosts, But few VMs are stucked at stopping state, same I checked DB it shows the HA done not going to take second retry, But still those VMs are stucked for stopping state, Need to recover manually. Please check for this |
|
could you please let us know the nfs version and qemu versions from the kvm host |
|
According to this comment there are issues with the recent qemu and nfs version |
Regarding the comment shared, i don't think it's the exact same situation, the first attempt to migrate fails (and that's ok, there's still the nfs lock from the dead host) but then cloudstack should try again to migrate the vm, and it doesn't. It stops at attempt 2/5 because there's a mismatch between the work.updated and the vm.updated counters, but no one updated manually the vm. |
But I am using ceph for all VMs . NFS is only for an heartbeat. I have used one my management server to create NFS mount and connected to all the compute hosts. |
|
@kiranchavala root@ltchl1pacsmgm01:~# Command 'qemu-system-x86_64' not found, but can be installed with: root@ltchl1pacsmgm02:~# -bash: qemu-system-x86_64: command not found root@ltchl1pacsmgm03:~# -bash: qemu-system-x86_64: command not found root@ltchl1pacscom01:~# QEMU emulator version 8.2.2 (Debian 1:8.2.2+ds-0ubuntu1.16) root@ltchl1pacscom02:~# nfsstat -m QEMU emulator version 8.2.2 (Debian 1:8.2.2+ds-0ubuntu1.16) root@ltchl1pacscom02:~# nfsstat -m QEMU emulator version 8.2.2 (Debian 1:8.2.2+ds-0ubuntu1.16) |
|
Hi @kiranchavala Important: NFS is not used as VM primary storage in our environment. We also use NFSv4.2 for CloudStack secondary storage. However, all user VM root and data disks are on Ceph RBD. Test performed At the time of one test, there were 9 user VMs running on this host. Actual behavior: For example, during one 9-VM test, 6 VMs restarted successfully and the following VMs remained in Management-server log observations For an affected VM, the HA stop/network cleanup workflow encountered a database transaction failure: For other affected VMs, HA logs: The HA work is initially rescheduled. During the next HA attempt, CloudStack detects that the VM state has changed from Then the HA work is completed: However, the VM remains in The MariaDB Galera cluster was healthy after the test: Current counters on one database node were: These Galera counters are cumulative and were not captured before and immediately after every HA test, so we cannot confirm that they are directly caused by this specific HA event. However, the CloudStack management-server logs clearly show Since the source host was hard powered off, this stale RBD lock is expected. However, CloudStack does not complete the HA recovery because the VM remains in The storage backend is different, but the final behavior appears similar: after an initial HA recovery failure, CloudStack stops retrying and completes the HA work even though the VM remains unavailable. |
|
cc @andrijapanicsb @sureshanaparti your thoughts |
|
@kiranchavala Now, I solved that issue. I added the NFS primary storage only for KVMHA Heartbeat. After that still Host HA is not working, Then we disable the Host HA and Only worked now for VM HA. I tried and many thing for this stucked VMs how to recover but we cant able to do that? Because, Many private cloud have an same issue during HA some VMs are stucked |
|
@muthukrishnang1100 I am not able to reproduce the issue in my lab on the recent build of Cloudstack 4.22.1 with nfs storage If you can reproduce the issue consistently on 4.22.1. Please let us know http://packages.shapeblue.com/cloudstack/upstream/ Also I am not sure if galera mysql is offically supported with cloudsack https://docs.cloudstack.apache.org/en/4.22.1.0/releasenotes/compat.html#software-requirements |
|
@kiranchavala But, I am using Ceph and NFS for HA heartbeat only. |
Uh oh!
There was an error while loading. Please reload this page.
problem
We're trying VM HA on cloudstack. We disabled HOST HA (see issue 13371) and then simulated a crash on an host with a "fake" kernel panic. The investigators correctly found the host dead and put it in alert/disconnected state.
The management servers then tried to start the vms on another host
The job fails (expectedly i would say) because of the nfs write lock held from the dead host.
But when retrying the vm restart we get
The restart process didn't even try a second time because of the mismatch in the update counter between the vm and the work (20 vs 17). The final state of the vm (without manual intervention) is Stopped.
versions
Cloudstack version: 4.22.1.0, with management server running on ubuntu22 and agent running on oracle linux 9
Shared nfsv4 (4.2) storage
The steps to reproduce the bug
What to do about it?
No response
All reactions