Tomcat redeploying issues

[sophiaraw]

I am running a Grails application (grails-2.0.4) with the RabbitMQ native plugin and deploying on Tomcat 7.

On first time deployment or after restarting tomcat all works well, but as soon as I re-deploy my app without restarting tomcat, I get the following errors: Seems like my channels aren't being closed on undeployment.

Mar 03, 2015 12:56:33 PM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load com.rabbitmq.client.impl.ContentHeaderPropertyReader. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1600)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
at com.rabbitmq.client.AMQP$BasicProperties.(AMQP.java:1660)
at com.rabbitmq.client.impl.AMQImpl.readContentHeaderFrom(AMQImpl.java:3511)
at com.rabbitmq.client.impl.CommandAssembler.consumeHeaderFrame(CommandAssembler.java:102)
at com.rabbitmq.client.impl.CommandAssembler.handleFrame(CommandAssembler.java:159)
at com.rabbitmq.client.impl.AMQCommand.handleFrame(AMQCommand.java:87)
at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:89)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:552)
at java.lang.Thread.run(Thread.java:745)

Mar 03, 2015 12:56:33 PM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load com.rabbitmq.client.impl.ChannelManager$1. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1600)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
at com.rabbitmq.client.impl.ChannelManager.scheduleShutdownProcessing(ChannelManager.java:106)
at com.rabbitmq.client.impl.ChannelManager.handleSignal(ChannelManager.java:100)
at com.rabbitmq.client.impl.AMQConnection.finishShutdown(AMQConnection.java:735)
at com.rabbitmq.client.impl.AMQConnection.shutdown(AMQConnection.java:708)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:569)
at java.lang.Thread.run(Thread.java:745)

Mar 03, 2015 12:56:33 PM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load org.codehaus.groovy.runtime.ArrayUtil. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1600)
at java.lang.ClassLoader.loadClass(ClassLoader.java:412)
at org.codehaus.groovy.runtime.callsite.CallSiteClassLoader.loadClass(CallSiteClassLoader.java:48)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at org.codehaus.groovy.reflection.ClassLoaderForClassArtifacts.loadClass(ClassLoaderForClassArtifacts.java:58)
at com.budjb.rabbitmq.converter.MapMessageConverter$canConvertTo.call(Unknown Source)
at com.budjb.rabbitmq.converter.MessageConverterManager.convertFromBytes(MessageConverterManager.groovy:73)
at com.budjb.rabbitmq.converter.MessageConverterManager$convertFromBytes.call(Unknown Source)
at com.budjb.rabbitmq.consumer.ConsumerAdapter.convertMessage(ConsumerAdapter.groovy:579)
at com.budjb.rabbitmq.consumer.ConsumerAdapter.this$2$convertMessage(ConsumerAdapter.groovy)
at com.budjb.rabbitmq.consumer.ConsumerAdapter$this$2$convertMessage.callCurrent(Unknown Source)
at com.budjb.rabbitmq.consumer.ConsumerAdapter.processMessage(ConsumerAdapter.groovy:448)
at com.budjb.rabbitmq.consumer.ConsumerAdapter.this$2$processMessage(ConsumerAdapter.groovy)
at com.budjb.rabbitmq.consumer.ConsumerAdapter$this$2$processMessage.callCurrent(Unknown Source)
at com.budjb.rabbitmq.consumer.ConsumerAdapter.deliverMessage(ConsumerAdapter.groovy:417)
at com.budjb.rabbitmq.consumer.ConsumerAdapter.this$2$deliverMessage(ConsumerAdapter.groovy)
at com.budjb.rabbitmq.consumer.ConsumerAdapter$this$2$deliverMessage.call(Unknown Source)
at com.budjb.rabbitmq.consumer.ConsumerAdapter$RabbitConsumer.handleDelivery(ConsumerAdapter.groovy:88)
at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:144)
at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:95)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Mar 03, 2015 12:56:33 PM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load org.codehaus.groovy.runtime.callsite.CallSiteArray. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1600)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
at org.codehaus.groovy.runtime.callsite.CallSiteClassLoader.loadClass(CallSiteClassLoader.java:45)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at org.codehaus.groovy.reflection.ClassLoaderForClassArtifacts.loadClass(ClassLoaderForClassArtifacts.java:58)
at com.budjb.rabbitmq.converter.MapMessageConverter$canConvertTo.call(Unknown Source)
at com.budjb.rabbitmq.converter.MessageConverterManager.convertFromBytes(MessageConverterManager.groovy:73)
at com.budjb.rabbitmq.converter.MessageConverterManager$convertFromBytes.call(Unknown Source)
at com.budjb.rabbitmq.consumer.ConsumerAdapter.convertMessage(ConsumerAdapter.groovy:579)
at com.budjb.rabbitmq.consumer.ConsumerAdapter.this$2$convertMessage(ConsumerAdapter.groovy)
at com.budjb.rabbitmq.consumer.ConsumerAdapter$this$2$convertMessage.callCurrent(Unknown Source)
at com.budjb.rabbitmq.consumer.ConsumerAdapter.processMessage(ConsumerAdapter.groovy:448)
at com.budjb.rabbitmq.consumer.ConsumerAdapter.this$2$processMessage(ConsumerAdapter.groovy)
at com.budjb.rabbitmq.consumer.ConsumerAdapter$this$2$processMessage.callCurrent(Unknown Source)
at com.budjb.rabbitmq.consumer.ConsumerAdapter.deliverMessage(ConsumerAdapter.groovy:417)
at com.budjb.rabbitmq.consumer.ConsumerAdapter.this$2$deliverMessage(ConsumerAdapter.groovy)
at com.budjb.rabbitmq.consumer.ConsumerAdapter$this$2$deliverMessage.call(Unknown Source)
at com.budjb.rabbitmq.consumer.ConsumerAdapter$RabbitConsumer.handleDelivery(ConsumerAdapter.groovy:88)
at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:144)
at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:95)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Caught an exception when recovering topology Caught an exception while recovering queue activity.event: channel is already closed due to clean channel shutdown; protocol method: #method<channel.close>(reply-code=200, reply-text=OK, class-id=0, method-id=0)
com.rabbitmq.client.TopologyRecoveryException: Caught an exception while recovering queue activity.event: channel is already closed due to clean channel shutdown; protocol method: #method<channel.close>(reply-code=200, reply-text=OK, class-id=0, method-id=0)
at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.recoverQueues(AutorecoveringConnection.java:510)
at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.recoverEntities(AutorecoveringConnection.java:468)
at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.beginAutomaticRecovery(AutorecoveringConnection.java:411)
at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.access$000(AutorecoveringConnection.java:52)
at com.rabbitmq.client.impl.recovery.AutorecoveringConnection$1.shutdownCompleted(AutorecoveringConnection.java:351)
at com.rabbitmq.client.impl.ShutdownNotifierComponent.notifyListeners(ShutdownNotifierComponent.java:75)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:574)
at java.lang.Thread.run(Thread.java:745)

Where am I going wrong?
Sophia

[budjb]

Is this causing a malfunctio after the redeploy, or is this just a concern about log clutter? What version of the plugin are you using?

[sophiaraw]

Thanks for you reply. It is not causing a malfunction, but I am worried about memory leaking. I am using rabbitmq-native:3.0.4

Sophia

[budjb]

I'd try stopping the rabbit context in the bootstrap. Something like this:

import com.budjb.rabbitmq.RabbitContext

class BootStrap {
    RabbitContext rabbitContext

    def destroy = {
        rabbitContext.stop()
    }
}

If you'd be willing to try it, I'd love to know if that helps.

[sophiaraw]

Thanks, that sorted me out! All working now

[budjb]

Thanks for trying it out. I'll add that in for the next release.

[budjb]

This is fixed in the 3.1 release.

[pvangeel]

I know this issue has been closed, however I am running in to a similar issue on v 3.1.2

When a consumer takes too long, a shutdown will cause a lot of unexpected behaviour where it can no longer be autowired when consuming messages.

I think the right way would be to do a shutdown on the consumer instead of a stop as it would do a graceful shutdown and wait for the threadpool to shutdown.

Is there a way that this can be configured? I tried it in my Bootstrap.groovy but I noticed the destroy method of the plugin (which does the 'stop') runs earlier.