CloudStack 4.19.3 KVM Agent Slow Operations on Ubuntu 24.04

[shakibshalim74]

Hello CloudStack Community,

We are facing an issue with one of our KVM hosts running the CloudStack Agent on Ubuntu 24.04.4 LTS.

Our environment is:

CloudStack: 4.19.3.0
Management Server: Ubuntu 22.04
Database Server: Ubuntu 22.04
KVM/CloudStack Agent Host: Ubuntu 24.04
Hypervisor: KVM/libvirt
Storage: Ceph

The issue is occurring on some KVM hosts. VM operations such as creation, start, stop, migration, and VNC/console access are taking unusually long or timing out.

The CloudStack Agent service is running normally, and the management server is still receiving ping responses from the host. However, the agent logs show repeated delays while accessing libvirt storage pools:

Trying to fetch storage pool 9a2687c4-d8ca-31dc-8dc6-c1e990e1690e from libvirt
Trying to fetch storage pool abed4ab1-27de-393d-8413-8efb1698fffc from libvirt
Trying to fetch storage pool a2d455c6-68cb-303f-a7fa-287e62a5be9c from libvirt

These messages are appearing repeatedly across multiple agentRequest-Handler threads, with delays of several seconds to tens of seconds.

We also see:

Couldn't retrieve free memory, returning -1.

From the host, the basic libvirt connection works:

virsh -c qemu:///system list --all

The host currently has approximately 502 GiB RAM, with more than 350 GiB available, and the root filesystem is only around 4% utilized, so there does not appear to be a local memory or filesystem capacity issue.

We would appreciate guidance on whether this could be related to CloudStack 4.19.3 compatibility with Ubuntu 24.04/libvirt, or whether there are known issues with the CloudStack KVM agent accessing Ceph-backed libvirt storage pools.

Could anyone suggest the recommended troubleshooting steps or confirm whether this OS/version combination is supported?

Thanks in advance for your help.

[weizhouapache]

@shakibshalim74
please refer to #12450 (comment)

[RyanP-Testing]

Hello @shakibshalim74,

Thank you for the detailed report — this is a really helpful description of the issue. I've analyzed your logs and the environment details, and I believe I can pinpoint what's happening here.

Root Cause Analysis

The primary issue is a known performance bug in libvirt on Ubuntu 24.04.

The version of libvirt that ships with Ubuntu 24.04 LTS (versions 10.0.0, 10.2.0, and 10.4.0) has a confirmed performance regression that causes the CloudStack Agent to hang or become extremely slow when accessing storage pools, particularly with Ceph-backed pools . This perfectly matches your symptoms: the agent repeatedly tries to fetch storage pools with delays of several seconds to tens of seconds.

The "Couldn't retrieve free memory" warning is a secondary issue likely caused by the agent struggling to parse memory statistics on Ubuntu 24.04, and should resolve once the primary libvirt issue is fixed .

The Fix

The good news is this problem is already well-documented in the CloudStack community. Users have confirmed that libvirt version 10.6.0 resolves the performance issues . After upgrading, the agent behaves correctly and the slowdowns disappear.

Here's my recommended approach:

1. Identify the specific libvirt version on your affected KVM hosts:

   dpkg -l | grep libvirt

2. Upgrade libvirt to version 10.6.0 or newer. Since Ubuntu 24.04's default repositories may not yet include this version, you may need to:

   • Check if a backport is available from the official Ubuntu repositories
   • Build from source, or
   • Consider using a different distribution like Debian 12 (Bookworm) for new hosts, which uses libvirt 9.0.0 and does not exhibit this issue

3. Restart the CloudStack Agent after the upgrade:

   sudo systemctl restart cloudstack-agent

Temporary Workaround

If you cannot upgrade libvirt immediately, you can configure a daily cron job to restart the CloudStack Agent (e.g., during a maintenance window). The community has confirmed this clears the stuck state and restores normal operation .

Important Context

The issue is not CloudStack 4.19.3 compatibility with Ubuntu 24.04 in general — it's specifically a libvirt regression. The CloudStack agent itself works fine once the underlying libvirt issue is addressed .

Please let me know what libvirt version you find on your hosts, and whether this solution resolves the problem for you. The community would also appreciate any feedback on whether libvirt 10.6.0 resolves the issue in your Ceph-backed environment.

Regards,
Ryan P.
Cloud &DevOps Engineer
AccuWeb.Cloud

[shakibshalim74]

Hello Ryan,

Thank you for the detailed information.

We have migrated all production VMs from one of our hypervisor hosts, except for one test VM, and we would like to upgrade libvirt from 10.0.0 to 10.6.0 on this host for testing.

Could you please share the recommended and tested procedure for upgrading libvirt from 10.0.0 to 10.6.0 on Ubuntu 24.04, especially for CloudStack 4.19.3 with Ceph RBD storage?

We would also like to know the best practice for performing this upgrade on a production server.

Thanks for your guidance.

Regards
Shakib

[nickaccuweb]

Hello Shakibshalim,

Since the host has already been evacuated, this is a good time to test libvirt 10.6.

Please note that Ubuntu 24.04 officially ships with libvirt 10.0.x. Libvirt 10.6 has been tested by the CloudStack community for this issue, but it is not yet an official Ubuntu 24.04 production update.

For testing, I recommend:

1. Disable the host in CloudStack so no new VMs are deployed.
2. Stop the remaining test VM.
3. Backup:

/etc/libvirt/
/etc/cloudstack/agent/
/etc/ceph/
4. Upgrade libvirt 10.6 using the Noble-specific test PPA used during the issue investigation.
5. Restart libvirt and the CloudStack Agent.
6. Test VM start/stop, console, migration, volume operations, and Ceph RBD access.

For production, I recommend testing one host first for a few days. If everything remains stable, you can then upgrade the remaining hosts one by one during a maintenance window.

I do not recommend mixing Ubuntu 24.10 repositories with Ubuntu 24.04 or upgrading all production hosts at once.

Regards,