listLoadBalancerRuleInstances API fails when VM has a removed NIC in the same network #7174
Description
ISSUE TYPE
- Bug Report
COMPONENT NAME
API
CLOUDSTACK VERSION
4.16.1+ , maybe earlier as well
CONFIGURATION
N/A
OS / ENVIRONMENT
N/A
SUMMARY
listLoadBalancerRuleInstances API fails when VM has a removed NIC in the same network. This could happen in the cases when a VM is removed from the VPC tier and then added to it again. Later when a LB is created using this VM for the VPC, listLoadBalancerRuleInstances would fail.
Possibly due to this method which list instances with JOIN and and may return removed NIC entries https://github.com/apache/cloudstack/blob/main/engine/schema/src/main/java/com/cloud/vm/dao/UserVmDaoImpl.java#L345-L352
It doesn't affect LB rule as such but just affect list API.
STEPS TO REPRODUCE
Steps mentioned below
DB state
MariaDB [cloud]> SELECT * FROM nics WHERE instance_id=(SELECT id FROM vm_instance WHERE uuid='620d7d83-533d-45e6-a8d6-88abcc149b6e');
+----+--------------------------------------+-------------+-------------------+-------------+---------------+----------+---------+---------------+------------+------+--------------+----------+--------------------------+----------------+-----------+---------------------+---------------+-------------+-------------+---------+---------------------+---------------------+-------------+----------+--------------+-------------+
| id | uuid | instance_id | mac_address | ip4_address | netmask | gateway | ip_type | broadcast_uri | network_id | mode | state | strategy | reserver_name | reservation_id | device_id | update_time | isolation_uri | ip6_address | default_nic | vm_type | created | removed | ip6_gateway | ip6_cidr | secondary_ip | display_nic |
+----+--------------------------------------+-------------+-------------------+-------------+---------------+----------+---------+---------------+------------+------+--------------+----------+--------------------------+----------------+-----------+---------------------+---------------+-------------+-------------+---------+---------------------+---------------------+-------------+----------+--------------+-------------+
| 12 | bd0183b1-0604-499d-9ef5-4bd3ffc70f2f | 6 | 02:00:33:77:00:03 | 10.1.1.169 | 255.255.255.0 | 10.1.1.1 | Ip4 | NULL | 205 | Dhcp | Deallocating | Start | ExternalGuestNetworkGuru | NULL | 1 | 2023-02-06 06:09:40 | NULL | NULL | 0 | User | 2023-02-06 06:09:40 | 2023-02-06 11:35:36 | NULL | NULL | 0 | 1 |
| 21 | 5a627371-3cbf-48dd-86f9-4bf972fb0d3e | 6 | 02:00:50:f1:00:02 | NULL | NULL | NULL | Ip4 | vlan://1368 | 204 | Dhcp | Reserved | Start | ExternalGuestNetworkGuru | NULL | 0 | 2023-02-06 11:34:04 | vlan://1368 | NULL | 1 | User | 2023-02-06 11:34:04 | NULL | NULL | NULL | 0 | 1 |
| 22 | 736371b1-f5e4-4327-af7d-238e70ae5ebf | 6 | 02:00:21:c5:00:04 | 10.1.1.59 | 255.255.255.0 | 10.1.1.1 | Ip4 | vlan://1367 | 205 | Dhcp | Reserved | Start | ExternalGuestNetworkGuru | NULL | 1 | 2023-02-06 11:35:54 | vlan://1367 | NULL | 0 | User | 2023-02-06 11:35:54 | NULL | NULL | NULL | 0 | 1 |
+----+--------------------------------------+-------------+-------------------+-------------+---------------+----------+---------+---------------+------------+------+--------------+----------+--------------------------+----------------+-----------+---------------------+---------------+-------------+-------------+---------+---------------------+---------------------+-------------+----------+--------------+-------------+
3 rows in set (0.00 sec)
2023-02-06 11:38:42,135 DEBUG [c.c.a.ApiServlet] (qtp396883763-17:ctx-08f485df) (logid:bd584e95) ===START=== 10.0.3.251 -- GET id=12625eed-7b69-443a-b755-a9e2fad6940e&vmidipmap[0].vmid=620d7d83-533d-45e6-a8d6-88abcc149b6e&vmidipmap[0].vmip=10.1.1.59&command=assignToLoadBalancerRule&response=json
2023-02-06 11:38:42,140 DEBUG [c.c.a.ApiServer] (qtp396883763-17:ctx-08f485df ctx-1e844f80) (logid:bd584e95) CIDRs from which account 'Acct[17fbff81-a395-11ed-804d-1e0009000324-admin] -- Account {"id": 2, "name": "admin", "uuid": "17fbff81-a395-11ed-804d-1e0009000324"}' is allowed to perform API calls: 0.0.0.0/0,::/0
2023-02-06 11:38:42,152 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-29:ctx-a25d3b90 job-89) (logid:6ca0570b) Add job-89 into job monitoring
2023-02-06 11:38:42,155 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp396883763-17:ctx-08f485df ctx-1e844f80) (logid:bd584e95) submit async job-89, details: AsyncJobVO: {id:89, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.user.loadbalancer.AssignToLoadBalancerRuleCmd, cmdInfo: {"vmidipmap[0].vmid":"620d7d83-533d-45e6-a8d6-88abcc149b6e","response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"247","id":"12625eed-7b69-443a-b755-a9e2fad6940e","vmidipmap[0].vmip":"10.1.1.59","ctxDetails":"{\"interface com.cloud.network.rules.FirewallRule\":\"12625eed-7b69-443a-b755-a9e2fad6940e\"}","ctxAccountId":"2","uuid":"12625eed-7b69-443a-b755-a9e2fad6940e","cmdEventType":"LB.ASSIGN.TO.RULE"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 32985499829028, completeMsid: null, lastUpdated: null, lastPolled: null, created: null, removed: null}
2023-02-06 11:38:42,156 DEBUG [c.c.a.ApiServlet] (qtp396883763-17:ctx-08f485df ctx-1e844f80) (logid:bd584e95) ===END=== 10.0.3.251 -- GET id=12625eed-7b69-443a-b755-a9e2fad6940e&vmidipmap[0].vmid=620d7d83-533d-45e6-a8d6-88abcc149b6e&vmidipmap[0].vmip=10.1.1.59&command=assignToLoadBalancerRule&response=json
2023-02-06 11:38:42,157 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-29:ctx-a25d3b90 job-89) (logid:dabb8445) Executing AsyncJobVO: {id:89, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.user.loadbalancer.AssignToLoadBalancerRuleCmd, cmdInfo: {"vmidipmap[0].vmid":"620d7d83-533d-45e6-a8d6-88abcc149b6e","response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"247","id":"12625eed-7b69-443a-b755-a9e2fad6940e","vmidipmap[0].vmip":"10.1.1.59","ctxDetails":"{\"interface com.cloud.network.rules.FirewallRule\":\"12625eed-7b69-443a-b755-a9e2fad6940e\"}","ctxAccountId":"2","uuid":"12625eed-7b69-443a-b755-a9e2fad6940e","cmdEventType":"LB.ASSIGN.TO.RULE"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 32985499829028, completeMsid: null, lastUpdated: null, lastPolled: null, created: null, removed: null}
2023-02-06 11:38:42,173 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-29:ctx-a25d3b90 job-89 ctx-d195db5c) (logid:dabb8445) Adding VM instance {id: "6", name: "i-2-6-VM", uuid: "620d7d83-533d-45e6-a8d6-88abcc149b6e", type="User"} to the load balancer pool
2023-02-06 11:38:42,194 DEBUG [o.a.c.n.t.AdvancedNetworkTopology] (API-Job-Executor-29:ctx-a25d3b90 job-89 ctx-d195db5c) (logid:dabb8445) APPLYING VPC IP RULES
2023-02-06 11:38:42,197 DEBUG [o.a.c.n.t.BasicNetworkTopology] (API-Job-Executor-29:ctx-a25d3b90 job-89 ctx-d195db5c) (logid:dabb8445) Applying vpc ip association in network Ntwk[205|Guest|11]
2023-02-06 11:38:42,215 DEBUG [c.c.a.t.Request] (API-Job-Executor-29:ctx-a25d3b90 job-89 ctx-d195db5c) (logid:dabb8445) Seq 1-158751886864811237: Sending { Cmd , MgmtId: 32985499829028, via: 1(ref-trl-4355-k-M7-abhishek-kumar-kvm1), Ver: v1, Flags: 100001, [{"com.cloud.agent.api.routing.IpAssocVpcCommand":{"ipAddresses":[{"accountId":"2","publicIp":"10.0.53.224","sourceNat":"false","add":"true","oneToOneNat":"false","firstIP":"true","broadcastUri":"vlan://51","vlanGateway":"10.0.48.1","vlanNetmask":"255.255.240.0","vifMacAddress":"1e:00:c4:00:00:03","networkRate":"200","trafficType":"Public","networkName":"cloudbr1","newNic":"false","isPrivateGateway":"false","nicTO":{"deviceId":"0","networkRateMbps":"-1","defaultNic":"false","pxeDisable":"false","details":{"MacAddressChanges":"true","PromiscuousMode":"false","ForgedTransmits":"true","MacLearning":"false"},"dpdkEnabled":"false","gateway":"10.0.48.1","mac":"1e:00:c4:00:00:03","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://51","isSecurityGroupEnabled":"false","name":"cloudbr1"}}],"accessDetails":{"router.name":"r-3-VM","router.guest.ip":"10.0.53.223","network.public.last.ip":"false","router.ip":"169.254.128.211","zone.network.type":"Advanced"},"wait":"0","bypassHostMaintenance":"false"}}] }
2023-02-06 11:38:42,336 DEBUG [c.c.a.ApiServlet] (qtp396883763-23:ctx-65287e60) (logid:5510388f) ===START=== 10.0.3.251 -- GET jobId=dabb8445-540d-432e-ab0d-7044b7b0e4ed&command=queryAsyncJobResult&response=json
2023-02-06 11:38:42,340 DEBUG [c.c.a.ApiServer] (qtp396883763-23:ctx-65287e60 ctx-67a69ba8) (logid:5510388f) CIDRs from which account 'Acct[17fbff81-a395-11ed-804d-1e0009000324-admin] -- Account {"id": 2, "name": "admin", "uuid": "17fbff81-a395-11ed-804d-1e0009000324"}' is allowed to perform API calls: 0.0.0.0/0,::/0
2023-02-06 11:38:42,351 DEBUG [c.c.a.ApiServlet] (qtp396883763-23:ctx-65287e60 ctx-67a69ba8) (logid:5510388f) ===END=== 10.0.3.251 -- GET jobId=dabb8445-540d-432e-ab0d-7044b7b0e4ed&command=queryAsyncJobResult&response=json
...
2023-02-06 11:38:46,509 ERROR [c.c.a.ApiServer] (qtp396883763-17:ctx-e72603b7 ctx-eca3fb69) (logid:191355ef) unhandled exception executing api command: [Ljava.lang.String;@72b7f33b
java.lang.IndexOutOfBoundsException: Index 1 out of bounds for length 1
at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
at java.base/java.util.Objects.checkIndex(Objects.java:372)
at java.base/java.util.ArrayList.get(ArrayList.java:459)
at org.apache.cloudstack.api.command.user.loadbalancer.ListLoadBalancerRuleInstancesCmd.execute(ListLoadBalancerRuleInstancesCmd.java:133)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:772)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:596)
at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:321)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:134)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:131)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:93)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:829)
- Deploy a VPC with its tier
- Deploy a VM into this tier and with one additional network
- Remove VM's nic for the VPC tier
- Add VPC tier network again. In DB, VM will have 3 entries now in the cloud.nics table
- Acquire a new public IP for the VPC
- Add a new LB rule using the VM for the acquired public IP. Observe failure in the UI and logs
EXPECTED RESULTS
No error seen in the UI and logs. listLoadBalancerRuleInstances returns result.
ACTUAL RESULTS
Error seen as shown in the above screenshot/logs