cloudstack-agent issues nonsensical commands as root

[almostobvious]

problem

So I have a problem adding a host which made me take a deep dive in the logs. I was horrified to see so many commands being issued by cloudstack-agent that are completely wrong and inappropriate for the system I'm running.

So on a Rocky linux 9.6, the agent tries to use iptables (the actual firewall is firewalld), looks for nfs.service (actual service is nfs-server.service), and does a bunch of stuff overriding system configuration as if I am on Ubuntu and not RhEl derivative.

Unsurprisingly, this doesn't work and any attempt to add the host fails with about few 1000s of lines of logs.

Using
cloudstack-common-4.20.1.0-1.noarch
cloudstack-agent-4.20.1.0-1.noarch

The agent ultimately get stuck in a loop - slice of the log below.

2025-06-13 14:22:08,741 INFO  [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Fetching host CPU arch
2025-06-13 14:22:08,746 INFO  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) Host uses control group [cgroup2fs].
2025-06-13 14:22:08,746 INFO  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) Calculating the max shares of the host.
2025-06-13 14:22:08,747 INFO  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) The max shares of the host is [14400].
2025-06-13 14:22:08,791 WARN  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) Execution of process [113893] for command [sudo grep InitiatorName= /etc/iscsi/initiatorname.iscsi ] failed.
2025-06-13 14:22:08,791 WARN  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) Process [113893] for command [sudo grep InitiatorName= /etc/iscsi/initiatorname.iscsi ] encountered the error: [grep: /etc/iscsi/initiatorname.iscsi: No such file or directory].
2025-06-13 14:22:08,802 WARN  [utils.script.Script] (Agent-Handler-1:[]) (logid:) Execution of process [113899] for command [/bin/bash -c virt-v2v --version ] failed.
2025-06-13 14:22:08,802 WARN  [utils.script.Script] (Agent-Handler-1:[]) (logid:) Process [113899] for command [/bin/bash -c virt-v2v --version ] encountered the error: [127].
2025-06-13 14:22:08,813 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:[]) (logid:) Attempting to create storage pool f142354b-9aee-486d-9be0-625921e37195 (Filesystem) in libvirt
2025-06-13 14:22:08,813 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:[]) (logid:) Found existing defined storage pool f142354b-9aee-486d-9be0-625921e37195, using it.
2025-06-13 14:22:08,813 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:[]) (logid:) Trying to fetch storage pool f142354b-9aee-486d-9be0-625921e37195 from libvirt
2025-06-13 14:22:09,848 INFO  [cloud.agent.Agent] (Agent-Handler-2:[]) (logid:) Process agent startup answer, agent [id: 0, uuid: null, name: null] connected to the server
2025-06-13 14:22:09,848 ERROR [cloud.agent.Agent] (Agent-Handler-2:[]) (logid:) Error parsing task java.lang.NullPointerException
        at java.base/java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
        at java.base/java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
        at java.base/java.util.Properties.put(Properties.java:1301)
        at java.base/java.util.Properties.setProperty(Properties.java:229)
        at com.cloud.agent.dao.impl.PropertiesStorage.persist(PropertiesStorage.java:55)
        at com.cloud.agent.AgentShell.setPersistentProperty(AgentShell.java:215)
        at com.cloud.agent.Agent.setUuid(Agent.java:423)
        at com.cloud.agent.Agent.processStartupAnswer(Agent.java:688)
        at com.cloud.agent.Agent.processResponse(Agent.java:932)
        at com.cloud.agent.Agent$ServerHandler.doTask(Agent.java:1220)
        at com.cloud.utils.nio.Task.call(Task.java:83)
        at com.cloud.utils.nio.Task.call(Task.java:29)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

2025-06-13 14:22:09,936 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:) Attempting to create storage pool 22594f15-9678-347e-9627-0f935db15e9f (NetworkFilesystem) in libvirt
2025-06-13 14:22:09,939 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:) Found existing defined storage pool 22594f15-9678-347e-9627-0f935db15e9f, using it.
2025-06-13 14:22:09,947 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:) Trying to fetch storage pool 22594f15-9678-347e-9627-0f935db15e9f from libvirt
2025-06-13 14:22:10,069 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-1:[]) (logid:) Attempting to create storage pool 1cfe58ff-e09e-3b31-ae0d-449aab610592 (NetworkFilesystem) in libvirt
2025-06-13 14:22:10,072 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-1:[]) (logid:) Found existing defined storage pool 1cfe58ff-e09e-3b31-ae0d-449aab610592, using it.
2025-06-13 14:22:10,089 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-1:[]) (logid:) Trying to fetch storage pool 1cfe58ff-e09e-3b31-ae0d-449aab610592 from libvirt

versions

The versions of ACS, hypervisors, storage, network etc..

non-working host:
Rocky 9.6, 5.14.0-570.21.1.el9_6.x86_64
libvirt-10.10.0-7.3.el9_6.x86_64
qemu-kvm-9.1.0-15.el9_6.4.x86_64
cloudstack-common-4.20.1.0-1.noarch
cloudstack-agent-4.20.1.0-1.noarch

Also have working hosts which run :
Rocky 9.5, 5.14.0-503.35.1.el9_5.x86_64
cloudstack-common-4.20.0.0-1.x86_64
cloudstack-agent-4.20.0.0-1.x86_64
libvirt-10.5.0-7.5.el9_5.x86_64
qemu-kvm-9.0.0-10.el9_5.2.x86_64

The steps to reproduce the bug

1. Clean install Rocky 9.6
2. Follow instructions on https://docs.cloudstack.apache.org/en/latest/installguide/configuration.html#adding-a-host which is terribly outdated, so also you need to look at https://docs.cloudstack.apache.org/en/latest/quickinstallationguide/qig.html to get things going
3. Try to add the host
   ...

What to do about it?

No response

[DaanHoogland]

@almostobvious , this sounds like we don’t fully support rocky 9.6 . I think earlier rocky 9 was tested to work so I wonder at what version it broke. Do you have any idea?

[almostobvious]

Hi Daan - I have a successful install on 9.5 with cloudstack-agent 4.20.0.0-1.x86_64, but mind, that was not exactly straightforward either as the docs / instructions for setting up the host are ancient and incomplete. In fact, I could only get that working by not following the "Add a host" in the 4.20 docs which seems to be stuck on decade old versions of centos, but instead using "Quick Installation Guide" for at least the main waypoints (QIG makes some very oversimplified assumptions about networking).

Rocky is bug-for-bug compatible with RHEL so you can expect RHEL 9.6 to fail too. Or perhaps it's the new cloudstack-agent version 4.20.1.0-1.noarch that is the culprit.

Question - how can I install cloudstack-agent 4.20.0.0-1.noarch? It seems the repo only has 4.20.1.0-1.x86_64 there - or am I looking at the wrong repo?

I've now been using ACS for a few months and I am slowly finding ways to make it useful. But if you ask me what is the one thing that needs attention in ACS - it's the documentation. In fact, I would stop all development for a tick or two and focus on testing and updating the documentation so that a capable engineer can a) get things done and b) understand what they are doing. And also, that the docs relate to reasonable OS versions people will actually use and have some semblance of support / security updates.

I wonder why is documentation so neglected (is this deliberate?). It just makes for a horrible user experience.

[almostobvious]

Ok so it only took me ~10 hours to add a host, but I got there.

1. I had to use cmk to clean up previous attempts at primary storage etc for now non-existent host. This was a bit confusing as I thought it was the cause of issues (but it wasn't). Still, this can't be properly done from the web ui because it lacks a whole raft of options - e.g. you can't put storagepool to maintenance.

In my case I had to do:

enable storagemaintenance id=<old / orphaned storage id>
delete storagepool id=22594f15-9678-347e-9627-0f935db15e9f

Which finally allowed me to get rid of orphaned storage.

2. I uninstalled cloudstack-agent. Then cleaned up all config by finding and deleting a whole bunch of stuff that gets left behind after uninstall (and I was suspecting of causing havoc, but it really didn't). I used following (and some good judgement) to find what to delete

 find /usr/ -name "*libvirt*"
 find /etc/ -name "*libvirt*"
 find /var/ -name "*libvirt*"
 find /usr/ -name "*cloudstack*"
 find /etc/ -name "*cloudstack*"
 find /var/ -name "*cloudstack*"

3. reboot, and then force installation of previous agent version that worked for me on RHEL/Rocky 9.5:

yum install cloudstack-agent-4.20.0.0-1

4. Now I went back and followed Quick install guide KVM config section here https://docs.cloudstack.apache.org/en/latest/quickinstallationguide/qig.html#kvm-configuration

5. Now make sure nfs-server is running and set up correctly:

// verify /etc/exports as follows:
/exports/secondary *(rw,async,no_root_squash,no_subtree_check)
/exports/primary *(rw,async,no_root_squash,no_subtree_check)

// ensure correct ownership
 chown nobody:nobody /exports/primary/
 chown nobody:nobody /exports/secondary/

// nfs server check
systemctl enable --now nfs-server
exportfs -r

6. Start / enable agent

systemctl enable cloudstack-agent
systemctl start cloudstack-agent

7. Now I used CMK to create host:

add host zoneid=<id> clusterid=<id> hypervisor=KVM username=root password=<pass> podid=<podid> url=http://<new host ip>

That took a little time, but succeeded.

I am now pretty sure the issue is with the new agent build 4.20.1.0-1 (whether or not it's down to the spurious ubuntu / paleo stuff it seems to be doing on RHEL I can't tell).