Several public IP with load balancer rules overwrite each other

[KuasarCloud]

ISSUE TYPE

• Bug Report

COMPONENT NAME

VR: LOAD BALANCER RULE

CLOUDSTACK VERSION

4.15.1.0

CONFIGURATION

SUMMARY

When having at least 2 public IPs configured with load balancer rules, the configuration overwrite each other and
only one set of rules on 1 public IP are usable.

STEPS TO REPRODUCE

1. Assign 2 public IPs
2. Create 3 rules on IP one
3. Create 1 rule on IP two

Then the rules on step 2 disappear, I could test it entering to the haproxy.cfg file and realized they're gone

EXPECTED RESULTS

Having multiple IP each with multiple load balancer rules

[weizhouapache]

@KuasarCloud could you please give more details ? for example

• cloudstack version
• hypervisor
• vpc or network, and vpc/network offerings
• state of vm, network or vpc.

I cannot reproduce the issue in main. here is the haproxy.cfg

listen 10_0_55_141-22
	bind 10.0.55.141:22
	balance roundrobin
	server 10_0_55_141-22_0 192.168.2.177:22 check
	 
listen 10_0_55_142-23
	bind 10.0.55.142:23
	balance roundrobin
	server 10_0_55_142-23_0 192.168.2.177:23 check

[weizhouapache]

@KuasarCloud
could you please give more details on this issue ?

[KuasarCloud]

Hi @weizhouapache,

Thanks for your prompt reply, here are some details

• Hyp: KVM
• Cloudstack Version and VR: 4.15.0.1
• VPC: Redundant VPC offering
• Network: DefaultIsolatedNetworkOfferingForVpcNetworks
• state of vm, network or vpc: enabled

These is my current haproxy.conf with 1 IP and 3 listen options (rules)

listen 172_x_x_x-xxxx
bind 172.x.x.x:xxx
balance roundrobin
server 172_x_x_x-xxx_0 172.x.x.x:xxxx check

listen 172_x_x_x-xxx
bind 172.x.x.x:xxx
balance roundrobin
server 172_x_x_x-xxx_0 172.x.x.x:xxx check
server 172_x_x_x-xxx_1 172.x.x.x:xxx check
server 172_x_x_x-xxx_2 172.x.x.x:xxx check

listen 172_x_x_x-xxx
bind 172.x.x.x:xxx
balance roundrobin
server 172_x_x_x-xxx_0 172.x.x.x:xxx check
server 172_x_x_x-xxx_1 172.x.x.x:xxx check
server 172_x_x_x-xxx_2 172.x.x.x:xxx check
mode http
option httpclose

But in the UI I have two defined different IP with lb rules each, the first one with the ones as defined above and the second is not in the config but when I update (add/remove member) on my second ip, the above config disappear and got replaced with the second ip lb rules.

[weizhouapache]

Hi @weizhouapache,

Thanks for your prompt reply, here are some details

• Hyp: KVM
• Cloudstack Version and VR: 4.15.0.1
• VPC: Redundant VPC offering
• Network: DefaultIsolatedNetworkOfferingForVpcNetworks
• state of vm, network or vpc: enabled

These is my current haproxy.conf with 1 IP and 3 listen options (rules)

listen 172_x_x_x-xxxx bind 172.x.x.x:xxx balance roundrobin server 172_x_x_x-xxx_0 172.x.x.x:xxxx check

listen 172_x_x_x-xxx bind 172.x.x.x:xxx balance roundrobin server 172_x_x_x-xxx_0 172.x.x.x:xxx check server 172_x_x_x-xxx_1 172.x.x.x:xxx check server 172_x_x_x-xxx_2 172.x.x.x:xxx check

listen 172_x_x_x-xxx bind 172.x.x.x:xxx balance roundrobin server 172_x_x_x-xxx_0 172.x.x.x:xxx check server 172_x_x_x-xxx_1 172.x.x.x:xxx check server 172_x_x_x-xxx_2 172.x.x.x:xxx check mode http option httpclose

But in the UI I have two defined different IP with lb rules each, the first one with the ones as defined above and the second is not in the config but when I update (add/remove member) on my second ip, the above config disappear and got replaced with the second ip lb rules.

@KuasarCloud
I still cannot reproduce the issue.

are you using the official 4.15.0.0 or 4.15.1.0 ?
do you use 172.x for both public ip and private ip in vpc ?

[KuasarCloud]

Official 4.15.1.0

Internal vpc private ip starts with 172 and public IP coincidentally start with 172 as well but are different CIDR.

[weizhouapache]

Official 4.15.1.0

Internal vpc private ip starts with 172 and public IP coincidentally start with 172 as well but are different CIDR.

@KuasarCloud ok.

can you reproduce the issue and share the following information ?
(1) last LoadBalancerConfigCommand in /var/log/cloudstack/agent/agent.log
(2) last processed json file in VR: ls -rt /var/cache/cloud/processed/load_balancer.json.*.gz |tail -n1 |xargs zcat

[yadvr]

Does this affect 4.16/main branch? @KuasarCloud can you test 4.16 snapshot builds/packages from http://download.cloudstack.org/testing/nightly/latest/

[KuasarCloud]

@weizhouapache: find the info requested

The previous configuration was overwritten

• /var/log/cloudstack/agent/agent.log

2021-10-08 09:44:07,124 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:139f6f30) Expected 6 answers while executing IpAssocVpcCommand but received 2
2021-10-08 09:44:08,121 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:139f6f30) Expected 6 answers while executing IpAssocVpcCommand but received 2
2021-10-08 09:44:08,199 INFO [cloud.network.HAProxyConfigurator] (agentRequest-Handler-5:null) (logid:139f6f30) Haproxy mode http enabled
2021-10-08 09:44:10,300 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-5:null) (logid:139f6f30) Expected 1 answers while executing LoadBalancerConfigCommand but received 2
2021-10-08 09:44:10,352 INFO [cloud.network.HAProxyConfigurator] (agentRequest-Handler-2:null) (logid:139f6f30) Haproxy mode http enabled
2021-10-08 09:44:12,798 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-2:null) (logid:139f6f30) Expected 1 answers while executing LoadBalancerConfigCommand but received 2
2021-10-08 09:44:13,708 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-4:null) (logid:139f6f30) Expected 6 answers while executing IpAssocVpcCommand but received 2
2021-10-08 09:44:14,769 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:139f6f30) Expected 6 answers while executing IpAssocVpcCommand but received 2
2021-10-08 09:44:26,003 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:49a7954c) Trying to fetch storage pool 3f7ecd1c-466e-3741-b176-0d3e29a2dfdc from libvirt
2021-10-08 09:44:26,007 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:49a7954c) Asking libvirt to refresh storage pool 3f7ecd1c-466e-3741-b176-0d3e29a2dfdc
2021-10-08 09:44:31,254 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:1f2a0b5a) Expected 6 answers while executing IpAssocVpcCommand but received 2
2021-10-08 09:44:32,171 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:1f2a0b5a) Expected 6 answers while executing IpAssocVpcCommand but received 2
2021-10-08 09:44:32,222 INFO [cloud.network.HAProxyConfigurator] (agentRequest-Handler-5:null) (logid:1f2a0b5a) Haproxy mode http enabled
2021-10-08 09:44:33,900 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-5:null) (logid:1f2a0b5a) Expected 1 answers while executing LoadBalancerConfigCommand but received 2
2021-10-08 09:44:33,940 INFO [cloud.network.HAProxyConfigurator] (agentRequest-Handler-4:null) (logid:1f2a0b5a) Haproxy mode http enabled
2021-10-08 09:44:35,590 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-4:null) (logid:1f2a0b5a) Expected 1 answers while executing LoadBalancerConfigCommand but received 2
2021-10-08 09:44:36,535 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-2:null) (logid:1f2a0b5a) Expected 6 answers while executing IpAssocVpcCommand but received 2
2021-10-08 09:44:37,509 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:1f2a0b5a) Expected 6 answers while executing IpAssocVpcCommand but received 2

• ls -rt /var/cache/cloud/processed/load_balancer.json.*.gz |tail -n1 |xargs zcat

{
"rules":[
{
"configuration":[
"global",
"\tlog 127.0.0.1:3914 local0 warning",
"\tmaxconn 4096",
"\tmaxpipes 1024",
"\tchroot /var/lib/haproxy",
"\tuser haproxy",
"\tgroup haproxy",
"\tdaemon",
"\t ",
"defaults",
"\tlog global",
"\tmode tcp",
"\toption dontlognull",
"\tretries 3",
"\toption redispatch",
"\toption forwardfor",
"\toption forceclose",
"\ttimeout connect 5000",
"\ttimeout client 50000",
"\ttimeout server 50000",
"\nlisten stats_on_private\n\tbind 169.254.195.x:xxxx\n\tmode http\n\toption httpclose\n\tstats enable\n\tstats uri /admin?stats\n\tstats realm Haproxy\ Statistics\n\tstats auth admin1:xxxxxx\n",
"\t ",
"listen 172_x_x_x-xxxx",
"\tbind 172.x.x.x:xxxx",
"\tbalance roundrobin",
"\tserver 172_x_x_x-xxxx_0 172.x.x.x:xxxx check cookie 172_x_x_x-xxxx",
"\tserver 172_x_x_x-xxxx_1 172.x.x.x:xxxx check cookie 172_x_x_x-xxxx",
"\tcookie cs insert ",
"\tmode http",
"\toption httpclose",
"\t ",
"\t "
],
"tmp_cfg_file_path":"/etc/haproxy/",
"tmp_cfg_file_name":"haproxy.cfg.new.1633704272222",
"add_rules":[
"172.x.xx:xxxx:"
],
"remove_rules":[

     ],
     "stat_rules":[
        "169.254.195.x:8081:0/0:,"
     ],
     "router_ip":"172.x.x.x"
  }

],
"type":"loadbalancer",
"delete_from_processed_cache":false
}

[weizhouapache]

@KuasarCloud
can you find LoadBalancerConfigCommand in management-server.log ?

can you test with 4.15.2.0 ?

[KuasarCloud]

Can I do a apt upgrade cloudstack-management to test 4.15.2.0?

Find it below

{
"com.cloud.agent.api.routing.LoadBalancerConfigCommand":{
"loadBalancers":[
{
"uuid":"a6588c9a-4d8b-4349-bf89-a29b4efbbf9e",
"srcIp":"172.x.x.x",
"srcPort":"xxxx",
"protocol":"tcp",
"lbProtocol":"tcp",
"algorithm":"roundrobin",
"revoked":"false",
"alreadyAdded":"false",
"inline":"false",
"destinations":[
{
"destIp":"172.xx.x.x",
"destPort":"xxxx",
"revoked":"true",
"alreadyAdded":"false"
},
{
"destIp":"172.x.x.x",
"destPort":"xxxx",
"revoked":"false",
"alreadyAdded":"false"
}
],
"stickinessPolicies":[
{
"methodName":"LbCookie",
"params":[
{
"t":"cookie-name",
"u":"cs"
}
]
}
]
}
],
"lbStatsVisibility":"link-local",
"lbStatsPublicIP":"172.x.x.x",
"lbStatsPrivateIP":"169.254.195.x",
"lbStatsGuestIP":"172.x.x.x",
"lbStatsPort":"xxxx",
"lbStatsSrcCidrs":"0/0",
"lbStatsAuth":"admin1:xxxxx",
"lbStatsUri":"/admin?stats",
"maxconn":"4096",
"keepAliveEnabled":"false",
"nic":{
"deviceId":"3",
"networkRateMbps":"200",
"defaultNic":"false",
"pxeDisable":"true",
"nicUuid":"516d3831-2dc9-41fe-ad0c-93241799b8cf",
"dpdkEnabled":"false",
"uuid":"fba79ab8-309c-4006-93fa-10b1c92b8385",
"ip":"172.x.x.x",
"netmask":"255.255.255.0",
"gateway":"172.x.x.x",
"mac":"02:00:78:a3:00:05",
"broadcastType":"Vxlan",
"type":"Guest",
"broadcastUri":"vxlan://xxxx",
"isolationUri":"vxlan://xxxx",
"isSecurityGroupEnabled":"false",
"name":"cloudbr0"
},
"vpcId":"1",
"accessDetails":{
"router.name":"r-5-VM",
"router.guest.ip":"172.x.x.x",
"router.ip":"169.254.195.x",
"zone.network.type":"Advanced"
},
"wait":"0",
"bypassHostMaintenance":"false"
}
}