console not working after system vm has been recreated on a different cluster

[sbrueseke]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Console Proxy

CLOUDSTACK VERSION

4.19.1.2

CONFIGURATION

Management server on Ubuntu 22.04 LTS
KVM hypervisor on Ubuntu 22.04 LTS

OS / ENVIRONMENT

SUMMARY

We run a multiple cluster installation. After deleting console proxy vm, Cloudstack recreated a new system vm on antoher cluster.
After that console access for instances running on a different cluster than the console proxy vm is running on, did not work anymore. But after stopping and starting an instance, console access was working again.

STEPS TO REPRODUCE

1. detroy console proxy system vm
2. let CS recreate the console proxy system vm and hope it will be created on another cluster
3. try to open a console on an instance running on a different cluster as the console proxy is running on (not working)
4. stop and start this instance and try again (working again)
5. try to open the console on an instance running on the same cluster as the console proxy is running on (working)

EXPECTED RESULTS

console should work on all instances even after CS recreated it on another cluster

ACTUAL RESULTS

console is only working on instances running on the same cluster as the console proxy system vm. After stop and start of an instance, console is working again.

[rohityadavcloud]

cc @nvazquez @weizhouapache @DaanHoogland to help review/triage if it's a regression or some kind of limitation we should address in next 4.19.2/4.20.1

[weizhouapache]

@sbrueseke
can you share some network information of the clusters ?
if possible, please share the output of "ip route" in the CPVM as well

[DaanHoogland]

@sbrueseke are the clusters in the same pod? (I don't think it relevant, but it might)
is there a recent update involved that changed behaviour, that you know off?

[sbrueseke]

Clusters are in the same pod.

Cluster A
root@pc-kvm-1:~# ip route
default via 10.10.1.1 dev cloudbr0 proto static
10.10.1.0/24 dev cloudbr0 proto kernel scope link src 10.10.1.21
10.10.7.0/24 dev bond2 proto kernel scope link src 10.10.7.21
169.254.0.0/16 dev cloud0 proto kernel scope link src 169.254.0.1

Cluster B
root@pc-kvm-6:~# ip route
default via 10.10.1.1 dev cloudbr0 proto static
10.10.1.0/24 dev cloudbr0 proto kernel scope link src 10.10.1.26
10.10.7.0/24 dev bond2 proto kernel scope link src 10.10.7.26
169.254.0.0/16 dev cloud0 proto kernel scope link src 169.254.0.1

[weizhouapache]

Clusters are in the same pod.

Cluster A root@pc-kvm-1:~# ip route default via 10.10.1.1 dev cloudbr0 proto static 10.10.1.0/24 dev cloudbr0 proto kernel scope link src 10.10.1.21 10.10.7.0/24 dev bond2 proto kernel scope link src 10.10.7.21 169.254.0.0/16 dev cloud0 proto kernel scope link src 169.254.0.1

Cluster B root@pc-kvm-6:~# ip route default via 10.10.1.1 dev cloudbr0 proto static 10.10.1.0/24 dev cloudbr0 proto kernel scope link src 10.10.1.26 10.10.7.0/24 dev bond2 proto kernel scope link src 10.10.7.26 169.254.0.0/16 dev cloud0 proto kernel scope link src 169.254.0.1

@sbrueseke
looks like simple setup which should work.

any firewall rules on the kvm hosts ?

[sbrueseke]

@weizhouapache no firewall on hosts

[weizhouapache]

@weizhouapache no firewall on hosts

can you also check these steps ?

• on the kvm host (another cluster), run "virsh domdisplay " or "virsh vncdisplay "

root@kvm1:~# virsh domdisplay i-2-1192-VM
vnc://10.0.xxx.xxx:2

• in CPVM, try ping and telnet

root@v-296-VM:~# ping -c1 10.0.xxx.xxx
# if it does not work, can you share output of  `ip route`  in the CPVM ?

root@v-296-VM:~# telnet 10.0.xxx.xxx 5902
Trying 10.0.xxx.xxx...
Connected to 10.0.xxx.xxx.
Escape character is '^]'.
RFB 003.008

[sbrueseke]

root@pc-kvm-1:~$ virsh domdisplay i-21-283-VM
vnc://10.10.1.21:1

root@v-529-VM:~# ping -c1 10.10.1.21
PING 10.10.1.21 (10.10.1.21): 56 data bytes
64 bytes from 10.10.1.21: icmp_seq=0 ttl=64 time=0.291 ms
--- 10.10.1.21 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.291/0.291/0.291/0.000 ms

root@v-529-VM:~# telnet 10.10.1.21 5901
Trying 10.10.1.21...
Connected to 10.10.1.21.
Escape character is '^]'.
RFB 003.008

[weizhouapache]

root@pc-kvm-1:~$ virsh domdisplay i-21-283-VM vnc://10.10.1.21:1

root@v-529-VM:~# ping -c1 10.10.1.21 PING 10.10.1.21 (10.10.1.21): 56 data bytes 64 bytes from 10.10.1.21: icmp_seq=0 ttl=64 time=0.291 ms --- 10.10.1.21 ping statistics --- 1 packets transmitted, 1 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.291/0.291/0.291/0.000 ms

root@v-529-VM:~# telnet 10.10.1.21 5901 Trying 10.10.1.21... Connected to 10.10.1.21. Escape character is '^]'. RFB 003.008

@sbrueseke
Has the vm been running for long time, let's say more than 1 year ?

[sbrueseke]

i-21-283-VM was created in June this year and last updated on July 16th. Are you able to reproduce this issue on your installation? Everything worked fine, before we recreated the console proxy VM and it was put on another cluster.