Rolling back the transaction: Time = 12 Name - Unexpected exception Problem with getting the ec attribute #3842
Description
ISSUE TYPE
- Other
COMPONENT NAME
CLOUDSTACK VERSION
CONFIGURATION
OS / ENVIRONMENT
SUMMARY
We have had a stable CloudStack operating for over 18 months. Unfortunately after a loss of power we cannot restart the CS Managers properly.
We have multiple copies of the configuration DB and we believe that none of those is corrupt.
Our initial belief was that the issue was related to the version of Java installed so we changed between 3 different versions - always getting the same result.
We also tested on different servers, changing the version of CloudStack (and Phyton) and the result is always the same.
We include portions of the log where we see anomalies, hoping someone will guide us.
[o.e.j.w.StandardDescriptorProcessor] (main:null) (logid:) NO JSP Support for /client, did not find org.eclipse.jetty.jsp.JettyJspServlet
DEBUG [c.c.u.d.T.Transaction] (secstorage-1:ctx-f3dfd638) (logid:2f779524) Rolling back the transaction: Time = 12 Name = secstorage-1; called by -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:174-ExposeInvocationInterceptor.invoke:92-ReflectiveMethodInvocation.proceed:185-JdkDynamicAopProxy.invoke:212-$Proxy185.persist:-1-AsyncJobManagerImpl.getPseudoJob:184-AsyncJobExecutionContext.registerPseudoExecutionContext:194-SystemVmLoadScanner$1.runInContext:79
2020-01-27 09:39:59,927 WARN [c.c.v.SystemVmLoadScanner] (secstorage-1:ctx-f3dfd638) (logid:2f779524) Unexpected exception Problem with getting the ec attribute
com.cloud.utils.exception.CloudRuntimeException: Problem with getting the ec attribute
at com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1439)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy185.persist(Unknown Source)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.getPseudoJob(AsyncJobManagerImpl.java:184)
at org.apache.cloudstack.framework.jobs.AsyncJobExecutionContext.registerPseudoExecutionContext(AsyncJobExecutionContext.java:194)
at com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:79)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Can not set long field org.apache.cloudstack.framework.jobs.impl.AsyncJobVO.id to java.math.BigInteger
at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:167)
at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:171)
at sun.reflect.UnsafeLongFieldAccessorImpl.set(UnsafeLongFieldAccessorImpl.java:102)
... 30 more
STEPS TO REPRODUCE
EXPECTED RESULTS
ACTUAL RESULTS