[4.11.2]: rolling upgrade of VR results in duplicate VRs #2880

resmo · 2018-10-05T09:40:27Z

ISSUE TYPE

Bug Report

COMPONENT NAME

VR

CLOUDSTACK VERSION

4.11.2

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

2018-10-05 11:24:43,634 INFO  [jobs.impl.AsyncJobMonitor] (API-Job-Executor-1:ctx-2f1d201c job-32445) Add job-32445 into job monitoring
2018-10-05 11:24:43,637 DEBUG [jobs.impl.AsyncJobManagerImpl] (API-Job-Executor-1:ctx-2f1d201c job-32445) Executing AsyncJobVO {id:32445, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.user.network.RestartNetworkCmd, cmdInfo: {"makeredundant":"false","cleanup":"true","response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"1145358","id":"48df88b1-c2e2-432d-811a-681a2be74dde","ctxDetails":"{\"interface com.cloud.network.Network\":\"48df88b1-c2e2-432d-811a-681a2be74dde\"}","ctxAccountId":"2","uuid":"48df88b1-c2e2-432d-811a-681a2be74dde","cmdEventType":"NETWORK.RESTART","_":"1538731483586"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 345050582394, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2018-10-05 11:24:43,664 DEBUG [engine.orchestration.NetworkOrchestrator] (API-Job-Executor-1:ctx-2f1d201c job-32445 ctx-dd2ddecb) Restarting network 207...
2018-10-05 11:24:43,665 DEBUG [engine.orchestration.NetworkOrchestrator] (API-Job-Executor-1:ctx-2f1d201c job-32445 ctx-dd2ddecb) Performing rolling restart of routers of network Ntwk[207|Guest|8]
2018-10-05 11:24:43,677 DEBUG [engine.orchestration.NetworkOrchestrator] (API-Job-Executor-1:ctx-2f1d201c job-32445 ctx-dd2ddecb) Asking VirtualRouter to implemenet Ntwk[207|Guest|8]
2018-10-05 11:24:43,727 DEBUG [network.router.NetworkHelperImpl] (API-Job-Executor-1:ctx-2f1d201c job-32445 ctx-dd2ddecb) Allocating the VR with id=1604 in datacenter com.cloud.dc.DataCenterVO$$EnhancerByCGLIB$$2f0df801@1 with the hypervisor type VMware
2018-10-05 11:24:43,749 DEBUG [network.router.NetworkHelperImpl] (API-Job-Executor-1:ctx-2f1d201c job-32445 ctx-dd2ddecb) Adding nic for Virtual Router in Guest network Ntwk[207|Guest|8]
2018-10-05 11:24:43,749 DEBUG [network.router.NetworkHelperImpl] (API-Job-Executor-1:ctx-2f1d201c job-32445 ctx-dd2ddecb) Adding nic for Virtual Router in Control network 
2018-10-05 11:24:43,756 DEBUG [engine.orchestration.NetworkOrchestrator] (API-Job-Executor-1:ctx-2f1d201c job-32445 ctx-dd2ddecb) Found existing network configuration for offering [Network Offering [3-Control-System-Control-Network]: Ntwk[202|Control|3]
2018-10-05 11:24:43,756 DEBUG [engine.orchestration.NetworkOrchestrator] (API-Job-Executor-1:ctx-2f1d201c job-32445 ctx-dd2ddecb) Releasing lock for Acct[dff128d0-4e01-11e4-bb66-0050569e64b8-system]
2018-10-05 11:24:43,764 DEBUG [network.router.NetworkHelperImpl] (API-Job-Executor-1:ctx-2f1d201c job-32445 ctx-dd2ddecb) Adding nic for Virtual Router in Public network 
2018-10-05 11:24:43,772 DEBUG [engine.orchestration.NetworkOrchestrator] (API-Job-Executor-1:ctx-2f1d201c job-32445 ctx-dd2ddecb) Found existing network configuration for offering [Network Offering [1-Public-System-Public-Network]: Ntwk[200|Public|1]
2018-10-05 11:24:43,772 DEBUG [engine.orchestration.NetworkOrchestrator] (API-Job-Executor-1:ctx-2f1d201c job-32445 ctx-dd2ddecb) Releasing lock for Acct[dff128d0-4e01-11e4-bb66-0050569e64b8-system]
2018-10-05 11:24:43,783 INFO  [network.router.NetworkHelperImpl] (API-Job-Executor-1:ctx-2f1d201c job-32445 ctx-dd2ddecb) Use same MAC as previous RvR, the MAC is 1e:00:6b:00:00:95
2018-10-05 11:24:43,785 DEBUG [cloud.vm.VirtualMachineManagerImpl] (API-Job-Executor-1:ctx-2f1d201c job-32445 ctx-dd2ddecb) Allocating entries for VM: VM[DomainRouter|r-1604-VM]
2018-10-05 11:24:43,791 DEBUG [cloud.vm.VirtualMachineManagerImpl] (API-Job-Executor-1:ctx-2f1d201c job-32445 ctx-dd2ddecb) Allocating nics for VM[DomainRouter|r-1604-VM]
2018-10-05 11:24:43,793 DEBUG [engine.orchestration.NetworkOrchestrator] (API-Job-Executor-1:ctx-2f1d201c job-32445 ctx-dd2ddecb) Allocating nic for vm VM[DomainRouter|r-1604-VM] in network Ntwk[207|Guest|8] with requested profile NicProfile[0-0-null-10.1.1.1-vlan://1976
2018-10-05 11:24:43,808 DEBUG [cloud.network.NetworkModelImpl] (API-Job-Executor-1:ctx-2f1d201c job-32445 ctx-dd2ddecb) Service SecurityGroup is not supported in the network id=207
2018-10-05 11:24:43,810 DEBUG [engine.orchestration.NetworkOrchestrator] (API-Job-Executor-1:ctx-2f1d201c job-32445 ctx-dd2ddecb) Allocating nic for vm VM[DomainRouter|r-1604-VM] in network Ntwk[202|Control|3] with requested profile null
2018-10-05 11:24:43,819 DEBUG [engine.orchestration.NetworkOrchestrator] (API-Job-Executor-1:ctx-2f1d201c job-32445 ctx-dd2ddecb) Allocating nic for vm VM[DomainRouter|r-1604-VM] in network Ntwk[200|Public|1] with requested profile NicProfile[0-0-null-185.27.109.13-vlan://96
2018-10-05 11:24:43,830 DEBUG [cloud.vm.VirtualMachineManagerImpl] (API-Job-Executor-1:ctx-2f1d201c job-32445 ctx-dd2ddecb) Allocating disks for VM[DomainRouter|r-1604-VM]
2018-10-05 11:24:43,845 DEBUG [cloud.vm.VirtualMachineManagerImpl] (API-Job-Executor-1:ctx-2f1d201c job-32445 ctx-dd2ddecb) Allocation completed for VM: VM[DomainRouter|r-1604-VM]
2018-10-05 11:24:43,866 DEBUG [network.router.NetworkHelperImpl] (API-Job-Executor-1:ctx-2f1d201c job-32445 ctx-dd2ddecb) Starting router VM[DomainRouter|r-1604-VM]
2018-10-05 11:24:43,880 DEBUG [jobs.impl.AsyncJobManagerImpl] (API-Job-Executor-1:ctx-2f1d201c job-32445 ctx-dd2ddecb) Sync job-32446 execution on object VmWorkJobQueue.1604
2018-10-05 11:27:51,739 DEBUG [network.router.NetworkHelperImpl] (API-Job-Executor-1:ctx-2f1d201c job-32445 ctx-dd2ddecb) Attempting to destroy router 1547
2018-10-05 11:27:51,755 DEBUG [db.Transaction.Transaction] (API-Job-Executor-1:ctx-2f1d201c job-32445 ctx-dd2ddecb) Rolling back the transaction: Time = 2 Name =  API-Job-Executor-1; called by -TransactionLegacy.rollback:890-TransactionLegacy.removeUpTo:833-TransactionLegacy.close:657-Transaction.execute:43-Transaction.execute:47-AsyncJobManagerImpl.submitAsyncJob:231-VirtualMachineManagerImpl.stopVmThroughJobQueue:4505-VirtualMachineManagerImpl.advanceStop:1600-VirtualMachineManagerImpl.advanceExpunge:513-VirtualMachineManagerImpl.advanceExpunge:502-VirtualMachineManagerImpl.expunge:491-NetworkHelperImpl.destroyRouter:253
2018-10-05 11:27:51,766 WARN  [jobs.impl.AsyncJobManagerImpl] (API-Job-Executor-1:ctx-2f1d201c job-32445 ctx-dd2ddecb) Unable to schedule async job for command com.cloud.vm.VmWorkStop, unexpected exception.
2018-10-05 11:27:51,777 ERROR [cloud.api.ApiAsyncJobDispatcher] (API-Job-Executor-1:ctx-2f1d201c job-32445) Unexpected exception while executing org.apache.cloudstack.api.command.user.network.RestartNetworkCmd
2018-10-05 11:27:51,779 DEBUG [jobs.impl.AsyncJobManagerImpl] (API-Job-Executor-1:ctx-2f1d201c job-32445) Complete async job-32445, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unable to schedule async job for command com.cloud.vm.VmWorkStop, unexpected exception."}
2018-10-05 11:27:51,780 DEBUG [jobs.impl.AsyncJobManagerImpl] (API-Job-Executor-1:ctx-2f1d201c job-32445) Publish async job-32445 complete on message bus
2018-10-05 11:27:51,780 DEBUG [jobs.impl.AsyncJobManagerImpl] (API-Job-Executor-1:ctx-2f1d201c job-32445) Wake up jobs related to job-32445
2018-10-05 11:27:51,780 DEBUG [jobs.impl.AsyncJobManagerImpl] (API-Job-Executor-1:ctx-2f1d201c job-32445) Update db status for job-32445
2018-10-05 11:27:51,781 DEBUG [jobs.impl.AsyncJobManagerImpl] (API-Job-Executor-1:ctx-2f1d201c job-32445) Wake up jobs joined with job-32445 and disjoin all subjobs created from job- 32445
2018-10-05 11:27:51,788 DEBUG [jobs.impl.AsyncJobManagerImpl] (API-Job-Executor-1:ctx-2f1d201c job-32445) Done executing org.apache.cloudstack.api.command.user.network.RestartNetworkCmd for job-32445
2018-10-05 11:27:51,788 INFO  [jobs.impl.AsyncJobMonitor] (API-Job-Executor-1:ctx-2f1d201c job-32445) Remove job-32445 from job monitoring

2018-10-05 11:27:51,766 WARN  [jobs.impl.AsyncJobManagerImpl] (API-Job-Executor-1:ctx-2f1d201c job-32445 ctx-dd2ddecb) Unable to schedule async job for command com.cloud.vm.VmWorkStop, unexpected exception.
javax.persistence.EntityExistsException: Entity already exists: 
        at com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1434)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:235)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:231)
        at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
        at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
        at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:231)
        at com.cloud.vm.VirtualMachineManagerImpl.stopVmThroughJobQueue(VirtualMachineManagerImpl.java:4505)
        at com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:1600)
        at com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:513)
        at com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:502)
        at com.cloud.vm.VirtualMachineManagerImpl.expunge(VirtualMachineManagerImpl.java:491)
        at com.cloud.network.router.NetworkHelperImpl.destroyRouter(NetworkHelperImpl.java:253)
        at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.destroyRouter(VirtualNetworkApplianceManagerImpl.java:350)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
        at com.sun.proxy.$Proxy240.destroyRouter(Unknown Source)
        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2954)
        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853)
        at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
        at com.sun.proxy.$Proxy230.restartNetwork(Unknown Source)
        at org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '32447' for key 'PRIMARY'
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        ... 73 more

2018-10-05 11:27:51,777 ERROR [cloud.api.ApiAsyncJobDispatcher] (API-Job-Executor-1:ctx-2f1d201c job-32445) Unexpected exception while executing org.apache.cloudstack.api.command.user.network.RestartNetworkCmd
com.cloud.utils.exception.CloudRuntimeException: Unable to schedule async job for command com.cloud.vm.VmWorkStop, unexpected exception.
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:247)
        at com.cloud.vm.VirtualMachineManagerImpl.stopVmThroughJobQueue(VirtualMachineManagerImpl.java:4505)
        at com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:1600)
        at com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:513)
        at com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:502)
        at com.cloud.vm.VirtualMachineManagerImpl.expunge(VirtualMachineManagerImpl.java:491)
        at com.cloud.network.router.NetworkHelperImpl.destroyRouter(NetworkHelperImpl.java:253)
        at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.destroyRouter(VirtualNetworkApplianceManagerImpl.java:350)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
        at com.sun.proxy.$Proxy240.destroyRouter(Unknown Source)
        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2954)
        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853)
        at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
        at com.sun.proxy.$Proxy230.restartNetwork(Unknown Source)
        at org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

STEPS TO REPRODUCE

restartNetwork cleanup=true

EXPECTED RESULTS

single upgraded VR

ACTUAL RESULTS

2 VR for the same entwork claiming the same IP

The text was updated successfully, but these errors were encountered:

rohityadavcloud · 2018-10-05T10:42:39Z

@resmo restart of network with cleanup=true triggers the rolling restart of a single isolated network, in case of errors you may end up with more than one VR for that network and restart required flag will be true for that network. The next restart will destroy any/all the older VRs.

resmo · 2018-10-05T11:49:44Z

The problem with duplicated VRs are the result, the issue is the exception I appended to the bug description

resmo · 2018-10-05T11:54:17Z

I can not upgrade routers after this bug, not even stopping VRs... rolling back to 4.11.1 and try to reproduce

resmo · 2018-10-05T12:34:47Z

I am able to reproduce. This time with a already deleted router and run restartNetwork cleanup=true 😞

rohityadavcloud · 2018-10-05T12:55:53Z

I was unable to reproduce with latest 4.11. Let me test again, look at the code around the exceptions and get back to you @resmo. It would help if we can get access to your test env or more details on your setup. /cc @PaulAngus

resmo · 2018-10-08T14:40:09Z

I think, I am able to reproduce this issue, I even run into it in 4.11.1:

- remember a running router's internal name
- stop management service
- delete router from vmware vsphere
- start management service
- (router state is still running in cloudstack even it does not exist anymore)
- restart network with cleanup=true of the network of this router (before the missing report has finished 3 times)

rohityadavcloud · 2018-10-08T14:47:47Z

Thanks @resmo I'll have a look.

borisstoyanov · 2018-10-09T11:12:53Z

@resmo I've tried your steps, but I couldn't reproduce the problem you're having, can you think of any other detail that may help reproducing it.

rohityadavcloud · 2018-10-16T05:21:59Z

Moved this to 4.11.3.0/future milestone as this with #2900 admin can opt for legacy behaviour

resmo · 2018-10-23T09:52:30Z

@rhtyd still hitting this exception in rc3 with network.rolling.restart=false

Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '32468' for key 'PRIMARY'
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	... 75 more

😢

rohityadavcloud · 2018-10-23T09:59:53Z

Thanks for sharing @resmo let me investigate and get back to you

rohityadavcloud · 2018-10-27T20:16:26Z

I'm still unable to reproduce, likely the case happens due to concurrency issue when the VR is stopped/started simultaneous probably by the VM power sync manager. Ping @resmo can you test the following:

Set the force stop on destroy global setting to true, and try again.
Are you able to consistently reproduce the issue?
Can you try latest 4.11 with rolling restart that force stops (poweroffs) the VRs before destroying them?

resmo · 2018-10-29T14:04:27Z

@rhtyd Did set the force top on destroy to true and was not able to reproduce anymore, however, still seeing "initiate guest OS shutdown" on vSphere. I wonder, if this setting make any difference, I should see a "Power Off virtual machine" on vSphere, right?

PaulAngus · 2018-10-29T14:46:36Z

@resmo i believe Rohit corrected that here: #2926

rohityadavcloud · 2018-10-29T15:57:39Z

Thanks for sharing @resmo, as @PaulAngus replied the power off issue during rolling restart has been fixed now as well.

rohityadavcloud · 2018-10-30T04:18:13Z

I'll close this issue based on confirmation from @resmo, the poweroff PR has been merged as well. Please reopen if you can reproduce this issue again in future.

rohityadavcloud added this to the 4.11.2.0 milestone Oct 5, 2018

rohityadavcloud added the component:virtual-router label Oct 5, 2018

rohityadavcloud self-assigned this Oct 5, 2018

rohityadavcloud mentioned this issue Oct 15, 2018

network: Allow ability to disable rolling restart feature #2900

Merged

5 tasks

rohityadavcloud modified the milestones: 4.11.2.0, 4.11.3.0 Oct 16, 2018

rohityadavcloud modified the milestones: 4.11.3.0, 4.11.2.0 Oct 24, 2018

rohityadavcloud mentioned this issue Oct 24, 2018

network: on rolling restart force stop old routers #2926

Merged

5 tasks

rohityadavcloud closed this as completed Oct 30, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[4.11.2]: rolling upgrade of VR results in duplicate VRs #2880

[4.11.2]: rolling upgrade of VR results in duplicate VRs #2880

resmo commented Oct 5, 2018 •

edited

Loading

rohityadavcloud commented Oct 5, 2018

resmo commented Oct 5, 2018

resmo commented Oct 5, 2018

resmo commented Oct 5, 2018 •

edited

Loading

rohityadavcloud commented Oct 5, 2018 •

edited

Loading

resmo commented Oct 8, 2018

rohityadavcloud commented Oct 8, 2018

borisstoyanov commented Oct 9, 2018

rohityadavcloud commented Oct 16, 2018

resmo commented Oct 23, 2018

rohityadavcloud commented Oct 23, 2018

rohityadavcloud commented Oct 27, 2018

resmo commented Oct 29, 2018 •

edited

Loading

PaulAngus commented Oct 29, 2018

rohityadavcloud commented Oct 29, 2018

rohityadavcloud commented Oct 30, 2018

[4.11.2]: rolling upgrade of VR results in duplicate VRs #2880

[4.11.2]: rolling upgrade of VR results in duplicate VRs #2880

Comments

resmo commented Oct 5, 2018 • edited Loading

ISSUE TYPE

COMPONENT NAME

CLOUDSTACK VERSION

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

rohityadavcloud commented Oct 5, 2018

resmo commented Oct 5, 2018

resmo commented Oct 5, 2018

resmo commented Oct 5, 2018 • edited Loading

rohityadavcloud commented Oct 5, 2018 • edited Loading

resmo commented Oct 8, 2018

rohityadavcloud commented Oct 8, 2018

borisstoyanov commented Oct 9, 2018

rohityadavcloud commented Oct 16, 2018

resmo commented Oct 23, 2018

rohityadavcloud commented Oct 23, 2018

rohityadavcloud commented Oct 27, 2018

resmo commented Oct 29, 2018 • edited Loading

PaulAngus commented Oct 29, 2018

rohityadavcloud commented Oct 29, 2018

rohityadavcloud commented Oct 30, 2018

resmo commented Oct 5, 2018 •

edited

Loading

resmo commented Oct 5, 2018 •

edited

Loading

rohityadavcloud commented Oct 5, 2018 •

edited

Loading

resmo commented Oct 29, 2018 •

edited

Loading