Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The 1.0 branch doesn't work with cassandra-1.0.10 #44

Closed
pcn opened this issue Jul 6, 2012 · 5 comments
Closed

The 1.0 branch doesn't work with cassandra-1.0.10 #44

pcn opened this issue Jul 6, 2012 · 5 comments

Comments

@pcn
Copy link

pcn commented Jul 6, 2012

What's the last known working version of cassandra, priam, and tomcat? Trying to use the current 1.0 checkout, e49e963, trying to load that into a tomcat6 server results in errors like this:

log4j: Finished configuring.
 INFO 03:13:16,382 Calling URL API: http://169.254.169.254/latest/meta-data/placement/availability-zone returns: us-east-1e
 INFO 03:13:16,384 Calling URL API: http://169.254.169.254/latest/meta-data/public-hostname returns: ec2-174-129-69-127.compute-1.amazonaws.com
 INFO 03:13:16,385 Calling URL API: http://169.254.169.254/latest/meta-data/public-ipv4 returns: 174.129.69.127
 INFO 03:13:16,386 Calling URL API: http://169.254.169.254/latest/meta-data/instance-id returns: i-a21ac6da
 INFO 03:13:16,387 Calling URL API: http://169.254.169.254/latest/meta-data/instance-type returns: c1.medium
Jul 6, 2012 3:13:17 AM org.apache.catalina.core.StandardContext start
SEVERE: Error listenerStart
Jul 6, 2012 3:13:17 AM org.apache.catalina.core.StandardContext start
SEVERE: Context [/Priam] startup failed due to previous errors
Jul 6, 2012 3:13:17 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/Priam] appears to have started a thread named [com.google.inject.internal.util.$Finalizer] but has failed to stop it. This is very likely to create a memory
 leak.
Jul 6, 2012 3:13:17 AM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/Priam] created a ThreadLocal with key of type [null] (value [com.google.inject.internal.InjectorImpl$1@4d898115]) and a value of type [java.lang.Object[]] (
value [[Ljava.lang.Object;@7e79b177]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
log4j: log4j called after unloading, see http://logging.apache.org/log4j/1.2/faq.html#unload.
java.lang.IllegalStateException: Class invariant violation
        at org.apache.log4j.LogManager.getLoggerRepository(LogManager.java:199)
        at org.apache.log4j.LogManager.getLogger(LogManager.java:228)
        at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:73)
        at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:243)
        at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:255)
        at com.netflix.priam.utils.Throttle.<clinit>(Throttle.java:32)
        at sun.misc.Unsafe.ensureClassInitialized(Native Method)
        at sun.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(UnsafeFieldAccessorFactory.java:25)
        at sun.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:122)
        at java.lang.reflect.Field.acquireFieldAccessor(Field.java:918)
        at java.lang.reflect.Field.getFieldAccessor(Field.java:899)
        at java.lang.reflect.Field.set(Field.java:657)
        at org.apache.catalina.loader.WebappClassLoader.clearReferencesStaticFinal(WebappClassLoader.java:2023)
        at org.apache.catalina.loader.WebappClassLoader.clearReferences(WebappClassLoader.java:1883)
        at org.apache.catalina.loader.WebappClassLoader.stop(WebappClassLoader.java:1787)
        at org.apache.catalina.loader.WebappLoader.stop(WebappLoader.java:738)
        at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4812)
        at org.apache.catalina.core.StandardContext.start(StandardContext.java:4675)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:546)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:905)
        at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:740)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:500)
        at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1277)
        at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:321)
        at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
        at org.apache.catalina.core.StandardHost.start(StandardHost.java:785)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
        at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:445)
        at org.apache.catalina.core.StandardService.start(StandardService.java:519)
        at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:581)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)

I'm muddling along with what's in master, but so much is broken that it's not particularly useful.

If there's a last known-good commit that I should be using, I can start from there.

Thanks

-Peter

@jasobrown
Copy link
Contributor

Hmmm, I haven't seen this before. We are running the latest build(s) of the 1.0 branch internally here and haven't had problems like the one you are seeing. Also, we're starting to the trunk version for cassandra 1. (in test), as well.

Is this the only exception that gets thrown, or are there others? Also, are you adding in any extra jars/config to the webapp?

Also, maybe setting the tomcat debugging level really low may help print out some extra info we can use to help debug,

@pcn
Copy link
Author

pcn commented Jul 16, 2012

It's been an active week, and my half-broken priam-1.1 magically recovered 2 nodes that died on Saturday - Hooray!

I'm pretty ignorant of java, so I'm 100% sure that I just ran mvn with the recommended flags from the docs. Would the version of Tomcat I'm using matter?

@jasobrown
Copy link
Contributor

We're using Tomcat 6.x here, and that has been stable for a long time with Priam. A google search yielded this link :http://stackoverflow.com/questions/6564553/tomcat-6-log4j-1-2-16-java-lang-illegalstateexception-class-invariant-viol . How are you deploying the Priam webapp to Tomcat? Do you bounce tomcat each time, or just redeploy the war to a running Tomcat process?

@pcn
Copy link
Author

pcn commented Jul 18, 2012

I've tried clearing the property referenced in stackoverflow - it gets rid of the invariant error message. In my case that message happens when the server has detected an error and has already unloaded it, so I'm still without priam in this case.

My deployment, when testing, is to shut down tomcat6, delete all the state I find in /var/lib/tomcat6/webapp/Priam* and one or two other places I can't recall right now, then replace with a new war and start up tomcat again. When it's died once this seems to be minimum to get a working system.

@pcn pcn closed this as completed Aug 14, 2012
@pcn
Copy link
Author

pcn commented Aug 14, 2012

It turns out that updating dependencies in maven has fixed this. I assume the gradle script in the current master branch would do the same.

Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants