Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade from server v0.59.1 to v0.63.0 fails #3960

Closed
jmreicha opened this issue Mar 14, 2016 · 13 comments
Closed

Upgrade from server v0.59.1 to v0.63.0 fails #3960

jmreicha opened this issue Mar 14, 2016 · 13 comments
Labels
kind/question Issues that just require an answer. No code change needd

Comments

@jmreicha
Copy link

Rancher Version: server v0.63.0

Docker Version: 1.10.1

OS: CoreOS 970.1.0

Steps to Reproduce:

Start the rancher server container and tail the logs

Results:

2016-03-14 18:29:45,782 INFO    [main] [ConsoleStatus] [1/32] [0ms] [0ms] Loading bootstrap
2016-03-14 18:29:46,248 INFO    [main] [ConsoleStatus] [2/32] [466ms] [466ms] Loading config-defaults
18:34:50.352 [main] WARN  o.eclipse.jetty.webapp.WebAppContext - Failed startup of context o.e.j.w.WebAppContext@29e3db80{/,file:/usr/share/cattle/cb6bab72d3fdd8af16f1c72bb3366250/,STARTING}{file:/usr/share/cattle/cb6bab72d3fdd8af16f1c72bb3366250/}
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase-core' defined in URL [jar:file:/usr/share/cattle/cb6bab72d3fdd8af16f1c72bb3366250/WEB-INF/lib/cattle-app-config-0.5.0-SNAPSHOT.jar!/META-INF/cattle/system/spring-system-context.xml]: Invocation of init method failed; nested exception is liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by fb256133d901 (fe80:0:0:0:42:acff:fe11:2%26) since 3/14/16 6:25 PM
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1512) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:296) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:628) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932) ~[spring-context-3.2.4.RELEASE.jar:3.2.4.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479) ~[spring-context-3.2.4.RELEASE.jar:3.2.4.RELEASE]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:184) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:157) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:348) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:358) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:333) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:152) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:97) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:75) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:56) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at io.cattle.platform.context.SpringContext.<init>(SpringContext.java:11) ~[cattle-iaas-system-0.5.0-SNAPSHOT.jar:na]
    at io.cattle.platform.context.WebContextLoaderListener.constructCloudStackSpringContext(WebContextLoaderListener.java:15) ~[cattle-iaas-system-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:51) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:800) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444) ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294) ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349) ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342) ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505) ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.server.Server.start(Server.java:387) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.server.Server.doStart(Server.java:354) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
    at io.cattle.platform.launcher.jetty.Main.main(Main.java:145) [cb6bab72d3fdd8af16f1c72bb3366250/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_95]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_95]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_95]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_95]
    at io.cattle.platform.launcher.Main.run(Main.java:188) [cb6bab72d3fdd8af16f1c72bb3366250/:na]
    at io.cattle.platform.launcher.Main.main(Main.java:251) [cb6bab72d3fdd8af16f1c72bb3366250/:na]
Caused by: liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by fb256133d901 (fe80:0:0:0:42:acff:fe11:2%26) since 3/14/16 6:25 PM
    at liquibase.lockservice.StandardLockService.waitForLock(StandardLockService.java:168) ~[liquibase-core-3.1.0.jar:na]
    at liquibase.Liquibase.update(Liquibase.java:189) ~[liquibase-core-3.1.0.jar:na]
    at liquibase.Liquibase.update(Liquibase.java:181) ~[liquibase-core-3.1.0.jar:na]
    at liquibase.integration.spring.SpringLiquibase.performUpdate(SpringLiquibase.java:342) ~[liquibase-core-3.1.0.jar:na]
    at liquibase.integration.spring.SpringLiquibase.afterPropertiesSet(SpringLiquibase.java:299) ~[liquibase-core-3.1.0.jar:na]
    at io.cattle.platform.liquibase.Loader.afterPropertiesSet(Loader.java:56) ~[cattle-framework-db-loader-0.5.0-SNAPSHOT.jar:na]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1571) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1509) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    ... 44 common frames omitted
18:34:50.369 [main] WARN  o.e.j.u.component.AbstractLifeCycle - FAILED o.e.j.w.WebAppContext@29e3db80{/,file:/usr/share/cattle/cb6bab72d3fdd8af16f1c72bb3366250/,UNAVAILABLE}{file:/usr/share/cattle/cb6bab72d3fdd8af16f1c72bb3366250/}: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase-core' defined in URL [jar:file:/usr/share/cattle/cb6bab72d3fdd8af16f1c72bb3366250/WEB-INF/lib/cattle-app-config-0.5.0-SNAPSHOT.jar!/META-INF/cattle/system/spring-system-context.xml]: Invocation of init method failed; nested exception is liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by fb256133d901 (fe80:0:0:0:42:acff:fe11:2%26) since 3/14/16 6:25 PM
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase-core' defined in URL [jar:file:/usr/share/cattle/cb6bab72d3fdd8af16f1c72bb3366250/WEB-INF/lib/cattle-app-config-0.5.0-SNAPSHOT.jar!/META-INF/cattle/system/spring-system-context.xml]: Invocation of init method failed; nested exception is liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by fb256133d901 (fe80:0:0:0:42:acff:fe11:2%26) since 3/14/16 6:25 PM
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1512) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:296) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:628) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932) ~[spring-context-3.2.4.RELEASE.jar:3.2.4.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479) ~[spring-context-3.2.4.RELEASE.jar:3.2.4.RELEASE]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:184) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:157) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:348) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:358) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:333) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:152) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:97) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:75) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:56) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at io.cattle.platform.context.SpringContext.<init>(SpringContext.java:11) ~[cattle-iaas-system-0.5.0-SNAPSHOT.jar:na]
    at io.cattle.platform.context.WebContextLoaderListener.constructCloudStackSpringContext(WebContextLoaderListener.java:15) ~[cattle-iaas-system-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:51) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:800) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444) ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294) ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349) ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342) ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505) ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.server.Server.start(Server.java:387) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.server.Server.doStart(Server.java:354) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
    at io.cattle.platform.launcher.jetty.Main.main(Main.java:145) [cb6bab72d3fdd8af16f1c72bb3366250/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_95]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_95]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_95]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_95]
    at io.cattle.platform.launcher.Main.run(Main.java:188) [cb6bab72d3fdd8af16f1c72bb3366250/:na]
    at io.cattle.platform.launcher.Main.main(Main.java:251) [cb6bab72d3fdd8af16f1c72bb3366250/:na]
Caused by: liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by fb256133d901 (fe80:0:0:0:42:acff:fe11:2%26) since 3/14/16 6:25 PM
    at liquibase.lockservice.StandardLockService.waitForLock(StandardLockService.java:168) ~[liquibase-core-3.1.0.jar:na]
    at liquibase.Liquibase.update(Liquibase.java:189) ~[liquibase-core-3.1.0.jar:na]
    at liquibase.Liquibase.update(Liquibase.java:181) ~[liquibase-core-3.1.0.jar:na]
    at liquibase.integration.spring.SpringLiquibase.performUpdate(SpringLiquibase.java:342) ~[liquibase-core-3.1.0.jar:na]
    at liquibase.integration.spring.SpringLiquibase.afterPropertiesSet(SpringLiquibase.java:299) ~[liquibase-core-3.1.0.jar:na]
    at io.cattle.platform.liquibase.Loader.afterPropertiesSet(Loader.java:56) ~[cattle-framework-db-loader-0.5.0-SNAPSHOT.jar:na]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1571) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1509) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    ... 44 common frames omitted
18:34:50.415 [main] WARN  o.e.j.u.component.AbstractLifeCycle - FAILED org.eclipse.jetty.server.Server@75df5f59: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase-core' defined in URL [jar:file:/usr/share/cattle/cb6bab72d3fdd8af16f1c72bb3366250/WEB-INF/lib/cattle-app-config-0.5.0-SNAPSHOT.jar!/META-INF/cattle/system/spring-system-context.xml]: Invocation of init method failed; nested exception is liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by fb256133d901 (fe80:0:0:0:42:acff:fe11:2%26) since 3/14/16 6:25 PM
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase-core' defined in URL [jar:file:/usr/share/cattle/cb6bab72d3fdd8af16f1c72bb3366250/WEB-INF/lib/cattle-app-config-0.5.0-SNAPSHOT.jar!/META-INF/cattle/system/spring-system-context.xml]: Invocation of init method failed; nested exception is liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by fb256133d901 (fe80:0:0:0:42:acff:fe11:2%26) since 3/14/16 6:25 PM
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1512) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:296) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:628) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932) ~[spring-context-3.2.4.RELEASE.jar:3.2.4.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479) ~[spring-context-3.2.4.RELEASE.jar:3.2.4.RELEASE]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:184) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:157) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:348) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:358) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:333) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:152) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:97) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:75) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:56) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at io.cattle.platform.context.SpringContext.<init>(SpringContext.java:11) ~[cattle-iaas-system-0.5.0-SNAPSHOT.jar:na]
    at io.cattle.platform.context.WebContextLoaderListener.constructCloudStackSpringContext(WebContextLoaderListener.java:15) ~[cattle-iaas-system-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:51) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:800) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444) ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294) ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349) ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342) ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505) ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) ~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.server.Server.start(Server.java:387) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) ~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.server.Server.doStart(Server.java:354) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
    at io.cattle.platform.launcher.jetty.Main.main(Main.java:145) [cb6bab72d3fdd8af16f1c72bb3366250/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_95]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_95]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_95]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_95]
    at io.cattle.platform.launcher.Main.run(Main.java:188) [cb6bab72d3fdd8af16f1c72bb3366250/:na]
    at io.cattle.platform.launcher.Main.main(Main.java:251) [cb6bab72d3fdd8af16f1c72bb3366250/:na]
Caused by: liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by fb256133d901 (fe80:0:0:0:42:acff:fe11:2%26) since 3/14/16 6:25 PM
    at liquibase.lockservice.StandardLockService.waitForLock(StandardLockService.java:168) ~[liquibase-core-3.1.0.jar:na]
    at liquibase.Liquibase.update(Liquibase.java:189) ~[liquibase-core-3.1.0.jar:na]
    at liquibase.Liquibase.update(Liquibase.java:181) ~[liquibase-core-3.1.0.jar:na]
    at liquibase.integration.spring.SpringLiquibase.performUpdate(SpringLiquibase.java:342) ~[liquibase-core-3.1.0.jar:na]
    at liquibase.integration.spring.SpringLiquibase.afterPropertiesSet(SpringLiquibase.java:299) ~[liquibase-core-3.1.0.jar:na]
    at io.cattle.platform.liquibase.Loader.afterPropertiesSet(Loader.java:56) ~[cattle-framework-db-loader-0.5.0-SNAPSHOT.jar:na]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1571) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1509) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    ... 44 common frames omitted
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase-core' defined in URL [jar:file:/usr/share/cattle/cb6bab72d3fdd8af16f1c72bb3366250/WEB-INF/lib/cattle-app-config-0.5.0-SNAPSHOT.jar!/META-INF/cattle/system/spring-system-context.xml]: Invocation of init method failed; nested exception is liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by fb256133d901 (fe80:0:0:0:42:acff:fe11:2%26) since 3/14/16 6:25 PM
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1512)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:296)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:628)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479)
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:184)
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:157)
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:348)
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:358)
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:333)
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:152)
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:97)
    at org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37)
    at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:75)
    at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:56)
    at io.cattle.platform.context.SpringContext.<init>(SpringContext.java:11)
    at io.cattle.platform.context.WebContextLoaderListener.constructCloudStackSpringContext(WebContextLoaderListener.java:15)
    at org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:51)
    at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:800)
    at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444)
    at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791)
    at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294)
    at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349)
    at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
    at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741)
    at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
    at org.eclipse.jetty.server.Server.start(Server.java:387)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
    at org.eclipse.jetty.server.Server.doStart(Server.java:354)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at io.cattle.platform.launcher.jetty.Main.main(Main.java:145)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at io.cattle.platform.launcher.Main.run(Main.java:188)
    at io.cattle.platform.launcher.Main.main(Main.java:251)
Caused by: liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by fb256133d901 (fe80:0:0:0:42:acff:fe11:2%26) since 3/14/16 6:25 PM
    at liquibase.lockservice.StandardLockService.waitForLock(StandardLockService.java:168)
    at liquibase.Liquibase.update(Liquibase.java:189)
    at liquibase.Liquibase.update(Liquibase.java:181)
    at liquibase.integration.spring.SpringLiquibase.performUpdate(SpringLiquibase.java:342)
    at liquibase.integration.spring.SpringLiquibase.afterPropertiesSet(SpringLiquibase.java:299)
    at io.cattle.platform.liquibase.Loader.afterPropertiesSet(Loader.java:56)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1571)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1509)
    ... 44 more
18:34:50.470 [main] ERROR ConsoleStatus - Startup Failed [316117ms]
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase-core' defined in URL [jar:file:/usr/share/cattle/cb6bab72d3fdd8af16f1c72bb3366250/WEB-INF/lib/cattle-app-config-0.5.0-SNAPSHOT.jar!/META-INF/cattle/system/spring-system-context.xml]: Invocation of init method failed; nested exception is liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by fb256133d901 (fe80:0:0:0:42:acff:fe11:2%26) since 3/14/16 6:25 PM
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1512) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:296) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:628) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932) ~[spring-context-3.2.4.RELEASE.jar:3.2.4.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479) ~[spring-context-3.2.4.RELEASE.jar:3.2.4.RELEASE]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:184) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:157) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:348) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:358) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:333) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:152) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:97) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:75) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:56) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at io.cattle.platform.context.SpringContext.<init>(SpringContext.java:11) ~[cattle-iaas-system-0.5.0-SNAPSHOT.jar:na]
    at io.cattle.platform.context.WebContextLoaderListener.constructCloudStackSpringContext(WebContextLoaderListener.java:15) ~[cattle-iaas-system-0.5.0-SNAPSHOT.jar:na]
    at org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:51) ~[cattle-framework-spring-module-0.5.0-SNAPSHOT.jar:na]
    at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:800) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444) ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294) ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349) ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342) ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505) ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) ~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.server.Server.start(Server.java:387) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) ~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.server.Server.doStart(Server.java:354) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
    at io.cattle.platform.launcher.jetty.Main.main(Main.java:145) ~[cb6bab72d3fdd8af16f1c72bb3366250/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_95]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_95]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_95]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_95]
    at io.cattle.platform.launcher.Main.run(Main.java:188) [cb6bab72d3fdd8af16f1c72bb3366250/:na]
    at io.cattle.platform.launcher.Main.main(Main.java:251) [cb6bab72d3fdd8af16f1c72bb3366250/:na]
Caused by: liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by fb256133d901 (fe80:0:0:0:42:acff:fe11:2%26) since 3/14/16 6:25 PM
    at liquibase.lockservice.StandardLockService.waitForLock(StandardLockService.java:168) ~[liquibase-core-3.1.0.jar:na]
    at liquibase.Liquibase.update(Liquibase.java:189) ~[liquibase-core-3.1.0.jar:na]
    at liquibase.Liquibase.update(Liquibase.java:181) ~[liquibase-core-3.1.0.jar:na]
    at liquibase.integration.spring.SpringLiquibase.performUpdate(SpringLiquibase.java:342) ~[liquibase-core-3.1.0.jar:na]
    at liquibase.integration.spring.SpringLiquibase.afterPropertiesSet(SpringLiquibase.java:299) ~[liquibase-core-3.1.0.jar:na]
    at io.cattle.platform.liquibase.Loader.afterPropertiesSet(Loader.java:56) ~[cattle-framework-db-loader-0.5.0-SNAPSHOT.jar:na]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1571) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1509) ~[spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    ... 44 common frames omitted
STARTUP FAILED [316124] ms

Expected:

Server should start up normally? Or is there some other step that needs to be run that I am missing?
@deniseschannon
Copy link

Your database is locked. It sometimes occurs during upgrade when the container hasn't released the lock on the DB.

http://docs.rancher.com/rancher/faqs/troubleshooting/#databaselock

@deniseschannon deniseschannon added the kind/question Issues that just require an answer. No code change needd label Mar 14, 2016
@deniseschannon
Copy link

If have you have an extremely large DB, you might run into this issue:

#3949

@jmreicha
Copy link
Author

@deniseschannon Thanks for the link. This is a new setup so the database should be relatively small.

I followed the instructions to go in to the container and release the lock but after doing that it looks like the lock gets recreated right away and rancher seems to get stuck in the same place?

Here's what I did:

mysql> select * from DATABASECHANGELOGLOCK;
+----+--------+---------------------+--------------------------------------------+
| ID | LOCKED | LOCKGRANTED         | LOCKEDBY                                   |
+----+--------+---------------------+--------------------------------------------+
|  1 |       | 2016-03-14 19:13:06 | 6f88da59c81a (fe80:0:0:0:42:acff:fe11:2%4) |
+----+--------+---------------------+--------------------------------------------+
1 row in set (0.00 sec)

mysql> update DATABASECHANGELOGLOCK set LOCKED="", LOCKGRANTED=null, LOCKEDBY=null where ID=1;
Query OK, 1 row affected (0.01 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> select * from DATABASECHANGELOGLOCK;
+----+--------+---------------------+--------------------------------------------+
| ID | LOCKED | LOCKGRANTED         | LOCKEDBY                                   |
+----+--------+---------------------+--------------------------------------------+
|  1 |       | 2016-03-14 19:16:22 | df629a02bc85 (fe80:0:0:0:42:acff:fe11:2%8) |
+----+--------+---------------------+--------------------------------------------+
1 row in set (0.00 sec)

Maybe I am missing something stupid?

@jmreicha
Copy link
Author

One more piece of information to add. I am using a mounted volume in the container, not a data container.

@jmreicha
Copy link
Author

I ran the update command again and it appears to have released the lock but then when I try to start/restart the rancher-server container it gets stuck at [main] [ConsoleStatus] [2/32] [466ms] [466ms] Loading config-defaults again. When I look in the container it looks like a new lock has been created.

@ict4ngo
Copy link

ict4ngo commented Mar 14, 2016

affected too

@deniseschannon
Copy link

@ict4ngo Are you having issues with the databaselock or stuck at the "loading config-defaults"?

@ict4ngo
Copy link

ict4ngo commented Mar 14, 2016

@deniseschannon exactly stuck at loading config-default the rancher-server containers after some times rancher-server container dies and throw this in the logs :

21:00:06.172 [main] ERROR ConsoleStatus - Startup Failed [304792ms]
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase-core' defined in URL [jar:file:/usr/share/cattle/c592f277c49e73822b95285c83f2d9cc/WEB-INF/lib/cattle-app-config-0.5.0-SNAPSHOT.jar!/META-INF/cattle/system/spring-db-loader-context.xml]: Invocation of init method failed; nested exception is liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by fb6d599b5caf (fe80:0:0:0:42:acff:fe11:2%10) since 3/14/16 8:16 PM

@ict4ngo
Copy link

ict4ngo commented Mar 14, 2016

interesting I try 3 - 4 times to unlock without any effect (lock was coming back all the time) but the fifth time it worked

@jmreicha
Copy link
Author

I saw that v0.63.1 was released so tried it but am getting a new error now after running the DB commands to release the lock. There is a ton of output so I just made a gist for it. https://gist.github.com/jmreicha/b3a21e6f94bce5057f7e.

@deniseschannon
Copy link

@jmreicha Please see the notes in the release notes of hte issue that you are hitting.
https://github.com/rancher/rancher/releases/tag/v0.63.1

@jmreicha
Copy link
Author

@deniseschannon Oh sorry! I missed that, I will try again and post results.

@jmreicha
Copy link
Author

@deniseschannon The instructions you posted seem to have fixed my original issue. Thanks!

@jmreicha jmreicha mentioned this issue Mar 15, 2016
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/question Issues that just require an answer. No code change needd
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ict4ngo @jmreicha @deniseschannon