Migrating router with public interface fails

[marcowi]

ISSUE TYPE

• Bug Report

COMPONENT NAME

VR,QEMU

CLOUDSTACK VERSION

4.11.1

CONFIGURATION

Both hosts have identical configuration:
cloudbr0 - private
cloudbr1 - storage
cloudbr2 - guest (VLAN 2101)
cloudbr3 - public (VLAN 99)

OS / ENVIRONMENT

SUMMARY

Migrating a virtual router from one host to another results in error "org.libvirt.LibvirtException: Cannot get interface MTU on 'brbond2-99': no such device".
Digging into it shows that the qemu hook (https://github.com/apache/cloudstack/blob/master/agent/bindir/libvirtqemuhook.in) iterating over the interfaces; including the public one which has the name brbond3-99. In this iteration it gets the guest interface name from the agent.properties and concatenates it with the vlan-id 99 so it results in brbond2-99.

STEPS TO REPRODUCE

Migrate a router with a public interface which has a vlan.

EXPECTED RESULTS

Since the public interface is needed on the other host as well one should handle the public interface the same but with the public.network.device value from agent.properties.

ACTUAL RESULTS

Public interfaces are treated as guest interfaces.

[rohityadavcloud]

Do you have bonded nics on the destination host? @marcowi what are the names of your private, public and guest network devices of the kvm hosts (as configured in agent.properties)?

[marcowi]

Yes, they are bonded on every host and the hosts are identical in hard- and software.

agent.properties:
private.network.device=cloudbr0
public.network.device=cloudbr3
guest.network.device=cloudbr2

[rohityadavcloud]

@PaulAngus @DagSonsteboSB any comments on this?

[PaulAngus]

how are you assigning these vlans (99 and 2101) ? as you have a public, this suggests advanced networking. the way have written you configuration suggest to me that you have hardcoded these VLANs on the hosts. for advanced networking, this should not be the case. They should be connected to trunked switch ports, and CloudStack allowed to apply the tags. This sounds like CloudStack is now trying to add VLAN tags to already VLAN tagged interfaces.

[marcowi]

It is advanced networking and the 99 is configured in Home>Infrastructure>Zones>myzone>eno5>Public as vlan://99 for the range of public IPs.
The 2101 is taken out of the vlan-range configured in Home>Infrastructure>Zones>myzone>eno4>Guest

If you mean hard coded in term of configuring it on the host itself then no, they are not.

[DagSonsteboSB]

@marcowi - how are the bonds configured on your host? I.e. what are the names? Can you do a "brctl show" on your KVM host and send the output?

I suspect this has to do with the underlying bond naming - CloudStack only has a limited list of terms it looks for - and if your bonds are called "brbondXXX" then this would not be included. To my knowledge the list is as follows:

cloudstack/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java

Line 1293 in 65f31f1

String [] _ifNamePatterns = {

[marcowi]

You are probably right @DagSonsteboSB
This is what brctl show outputs

bridge name bridge id STP enabled interfaces
brbond2-2101 8000.40f2e9266694 no bond2.2101
brbond3-99 8000.40f2e9266695 no bond3.99
cloud0 8000.fe00a9fe02aa no vnet0
cloudbr0 8000.40f2e9266692 no bond0
cloudbr1 8000.40f2e9266693 no bond1
cloudbr2 8000.40f2e9266694 no bond2
cloudbr3 8000.40f2e9266695 no bond3

And this is /etc/network/interfaces

#management
auto eno2
iface eno2 inet manual
bond-master bond0

auto enp139s0f0
iface enp139s0f0 inet manual
bond-master bond0

auto bond0
iface bond0 inet manual
bond-mode active-backup
bond-miimon 100
bond-slaves none

auto cloudbr0
iface cloudbr0 inet static
bridge_ports bond0
bridge_fd 5
bridge_stp off
bridge_maxwait 1
address 192.168.0.2
netmask 255.255.255.0

#storage
auto eno3
iface eno3 inet manual
bond-master bond1

auto enp139s0f1
iface enp139s0f1 inet manual
bond-master bond1

auto bond1
iface bond1 inet manual
bond-mode active-backup
bond-miimon 100
bond-slaves none

auto cloudbr1
iface cloudbr1 inet static
bridge_ports bond1
bridge_fd 5
bridge_stp off
bridge_maxwait 1
address 172.16.0.2
netmask 255.255.255.0

#guest
auto eno4
iface eno4 inet manual
bond-master bond2

auto enp144s0f0
iface enp144s0f0 inet manual
bond-master bond2

auto bond2
iface bond2 inet manual
bond-mode active-backup
bond-miimon 100
bond-slaves none

auto cloudbr2
iface cloudbr2 inet static
bridge_ports bond2
bridge_fd 5
bridge_stp off
bridge_maxwait 1
address 10.0.0.2
netmask 255.0.0.0

#public
auto eno5
iface eno5 inet manual
bond-master bond3

auto enp144s0f1
iface enp144s0f1 inet manual
bond-master bond3

auto bond3
iface bond3 inet manual
bond-mode active-backup
bond-miimon 100
bond-slaves none

auto cloudbr3
iface cloudbr3 inet manual
bridge_ports bond3
bridge_fd 5
bridge_stp off
bridge_maxwait 1

auto enp134s0f0
iface enp134s0f0 inet static
address xxx.xxx.xxx.xxx
netmask xxx.xxx.xxx.xxx
gateway xxx.xxx.xxx.xxx
dns-nameservers xxx.xxx.xxx.xxx

[DagSonsteboSB]

@marcowi your setup actually looks OK - your bridges all backed by devices named "bondXX" which should be OK. Just spun up the following and got the same "brbondXX" as you.

[root@C7KVM3 ~]# brctl show
bridge name bridge id STP enabled interfaces
brbond1-512 8000.061a74000927 no bond1.512
vnet0
vnet3
brbond1-7 8000.061a74000927 no bond1.7
vnet2
cloud0 8000.fe00a9fe016a no vnet1
cloudbr0 8000.06d9320107f4 no bond0
cloudbr1 8000.061a74000927 no bond1
virbr0 8000.525400162f56 yes virbr0-nic

[marcowi]

Any suggestions on how to proceed?

[rohityadavcloud]

@marcowi were you able to resolve the issue, it seems environment specific. Make sure via traffic labels the network devices exist on all hosts.