Instance not taking IP address correctly

[Noelantogerorge]

ISSUE TYPE

• Other

COMPONENT NAME

VR,UI,NETWORK

CLOUDSTACK VERSION

Cloudstack on 4.18

CONFIGURATION

Advance networking

OS / ENVIRONMENT

All KVM Host and cloudstack is on Ubundu 22.04

SUMMARY

I have 3 host machine. All are bare metal servers and running ubuntu 22.04. Host A first machine , Host B second machine and Host C third machine. We have 2 network LAN Isolated network and Public Shared network. As all VM I need 2 nic Lan and public I had attached 2 network while creating the VM's
All instance running on Host A is working fine. I can get LAN network and Public network ip on VM os and everthing works fine
When I bring up a instance on Host B it only take Public network. LAN network will not get ip assigned on Instance OS
When I bring up a instance on Host C it won't take IP for public and Lan network.
I might have gone wrong in somewhere. Please help me out to figure the issue

STEPS TO REPRODUCE

I don't know is this a bug or my configuration issue. I had followed the officical document to configure 3 KVM host and create 1 isolated netork and 1 shared network. Try to bring up VM on each host and see do the VM OS has taken the IP address

I am attaching the network configuration and brctl of 3 host machine below for more clarity

EXPECTED RESULTS

VM created on 3 host should get VR dhcp assigned IP address

ACTUAL RESULTS

Not working as expected. VM not taking IP address correctly...
Host A.txt
Host B.txt
Host C.txt

[kiranchavala]

Hi @Noelantogerorge

which network did you select as default (isolated or shared) when you were deploying the vm

[Noelantogerorge]

@kiranchavala
Thanks for looking on to it
Shared network was selected as default while creating the instance

[Noelantogerorge]

Hi any input on the same... I think its a configuration issue from myside but I tried reconfiguring third host several times but same issue

[Noelantogerorge]

@kiranchavala

I am getting below logs on agent.log.. I checked managemnet log but couldn't get any ERROR on that

tail -f /var/log/cloudstack/agent/agent.log
2023-07-17 13:12:19,174 INFO [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:5962b42d) Processing agent ready command, agent id = 9
2023-07-17 13:12:19,175 INFO [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:5962b42d) Set agent id 9
2023-07-17 13:12:19,177 INFO [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:5962b42d) Processed new management server list: 192.168.10.18@static
2023-07-17 13:12:19,177 INFO [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:5962b42d) Ready command is processed for agent id = 9
2023-07-17 13:12:57,276 WARN [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) (logid:0aac4bde) Couldn't retrieve free memory, returning -1.
2023-07-17 13:13:58,473 WARN [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:ca0d3ff1) Couldn't retrieve free memory, returning -1.
2023-07-17 13:14:00,217 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:1bf5ec1e) Trying to fetch storage pool d7ccf3f5-1a6f-3be9-91b6-15455289b700 from libvirt
2023-07-17 13:14:00,242 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:1bf5ec1e) Asking libvirt to refresh storage pool d7ccf3f5-1a6f-3be9-91b6-15455289b700
2023-07-17 13:14:06,877 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-4:null) (logid:329649a3) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not available. Transformations will not be applied.
2023-07-17 13:14:06,878 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-4:null) (logid:329649a3) Groovy scripting engine is not initialized. Data transformation skipped.
2023-07-17 13:14:26,793 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:6ecf772e) Trying to fetch storage pool d7ccf3f5-1a6f-3be9-91b6-15455289b700 from libvirt
2023-07-17 13:14:26,887 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:6ecf772e) Trying to fetch storage pool d7ccf3f5-1a6f-3be9-91b6-15455289b700 from libvirt
2023-07-17 13:14:26,965 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-3:null) (logid:6ecf772e) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is not available. Transformations will not be applied.
2023-07-17 13:14:26,966 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-3:null) (logid:6ecf772e) Groovy scripting engine is not initialized. Data transformation skipped.
2023-07-17 13:14:28,850 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-3:null) (logid:6ecf772e) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not available. Transformations will not be applied.
2023-07-17 13:14:28,851 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-3:null) (logid:6ecf772e) Groovy scripting engine is not initialized. Data transformation skipped.
2023-07-17 13:14:59,590 WARN [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) (logid:c16d8ba2) Couldn't retrieve free memory, returning -1.
2023-07-17 13:15:48,989 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:24d779e0) Trying to fetch storage pool d7ccf3f5-1a6f-3be9-91b6-15455289b700 from libvirt
2023-07-17 13:16:00,744 WARN [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:dfd5950e) Couldn't retrieve free memory, returning -1.

2023-07-17 13:17:02,052 WARN [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) (logid:66ba85d3) Couldn't retrieve free memory, returning -1.
2023-07-17 13:18:01,334 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:52bcb2d1) Trying to fetch storage pool d7ccf3f5-1a6f-3be9-91b6-15455289b700 from libvirt
2023-07-17 13:18:01,362 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:52bcb2d1) Asking libvirt to refresh storage pool d7ccf3f5-1a6f-3be9-91b6-15455289b700
2023-07-17 13:18:03,318 WARN [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:cb023a6c) Couldn't retrieve free memory, returning -1.
2023-07-17 13:19:04,666 WARN [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) (logid:4baef0dc) Couldn't retrieve free memory, returning -1.
2023-07-17 13:20:05,930 WARN [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-1:null) (logid:2258d40c) Couldn't retrieve free memory, returning -1.
2023-07-17 13:21:07,246 WARN [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-2:null) (logid:dfa991e7) Couldn't retrieve free memory, returning -1.
2023-07-17 13:22:08,558 WARN [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:null) (logid:c1be80a5) Couldn't retrieve free memory, returning -1.
2023-07-17 13:23:09,870 WARN [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:c2885951) Couldn't retrieve free memory, returning -1.

[Noelantogerorge]

@weizhouapache @kiranchavala Please share your views....

[weizhouapache]

@Noelantogerorge
if vm console works, can you manually assign the vlan/public IPs inside the VMs, and check the connectivity with other VMs in same network ?

[zap51]

@Noelantogerorge,
Please help me with some more specifics here. I see this is advanced networking, your traffic labels are cloudbr0 and cloudbr1. I'm assuming Management goes over cloudbr0, Guest & Public goes over cloudbr1. Is this right?

AFAIK, your instance shouldn't loose the configured IP no matter static or dynamically configured. Did you get a chance to login through the Web Console and inspect the IPs? Perhaps you may try reaching gateways of your network.

Please help me with the below.

1. Are we making use of VLANs in your network? If yes, please share.
2. Please also share # brctl show when instance is migrated to HostB and HostC.
3. Please try creating an additional VM, try migrating to other host and test the reachability from accessible VM over shared and isolated network. # ip n would show the MAC address table, you should be worried when you see FAILED as the issue lies in the physical network or the configuration of yours.

Thanks

[Noelantogerorge]

@Noelantogerorge if vm console works, can you manually assign the vlan/public IPs inside the VMs, and check the connectivity with other VMs in same network ?

@weizhouapache I tried with that but I am not getting any communication. Even a ping to gateway IP of shared network and isolated network is down

[Noelantogerorge]

@Noelantogerorge, Please help me with some more specifics here. I see this is advanced networking, your traffic labels are cloudbr0 and cloudbr1. I'm assuming Management goes over cloudbr0, Guest & Public goes over cloudbr1. Is this right?

AFAIK, your instance shouldn't loose the configured IP no matter static or dynamically configured. Did you get a chance to login through the Web Console and inspect the IPs? Perhaps you may try reaching gateways of your network.

Please help me with the below.

1. Are we making use of VLANs in your network? If yes, please share.
2. Please also share # brctl show when instance is migrated to HostB and HostC.
3. Please try creating an additional VM, try migrating to other host and test the reachability from accessible VM over shared and isolated network. # ip n would show the MAC address table, you should be worried when you see FAILED as the issue lies in the physical network or the configuration of yours.

Thanks

@zap51 When I migrate VM to Host A or Host B it is getting cloudstack assigned IP address and Instances works fine.
I am facing the issue only on Host C. I have attached brctl of host C on first comment. I think that KVM is having some issue.
I tried reconfiguring the host multiple times but end in same situation. I think some configuration might gone wrong on it.

[zap51]

@Noelantogerorge We might need some more information here. It rather appears as a network configuration issue to me. Please try asking this on the Cloudstack users mailing list at https://lists.apache.org/list.html?users@cloudstack.apache.org

Thanks

[Noelantogerorge]

@weizhouapache @zap51
After server reconfiguration my Instances on KVM host C is gettig public IP address whereas LAN network is still down.
After starting Instance and logged in to console and try ifconfig
I can see dhcp ip address of cloudstack VR get assigned but still IP interface is not assign IP but NIC is up.
Checked agent log and management log but couldn't find any error

[Noelantogerorge]

Closing this as issue identifiend as VLAN ID getting blocked on my switch. After allowing all Vlan ID on all of my switch its works perfectly