13. Analyzing production environments

Starting with Sun's JDK 1.5 it is quite easy to get information about a badly behaving application in a production environment using the JVM tools jmap and jstack. Jmap fetches a heap dump from the VM the application is running in and jstack fetches the thread dumps. These information can be taken from the server the application server is running on and analyzed offline. Even without remote access to the server you still can get these dumps from your customer. Many Monitoring tools usually need some kind of remote access.

Imagine a customer's web application with a lot of concurrent users accessing the system and the system is under heavy load. The system is only accessed during office hours. But you have noticed the load of the system stays high in the night although nobody is accessing it. What can you do?

First of all you can fetch several thread dumps either using jstack or using kill -QUIT if you're using an older VM than 1.5. Using these dumps you can easily detect long running (looping?) threads which might cause the load. You should fetch about five dumps and wait several seconds between fetching, lets say 20 seconds. The command line with jstack then looks something like this

jstack <pid> > dump.log 

for the first dump and

jstack  >> dump.log 

for the additional dumps. You then can use the TDA - Thread Dump Analyzer to analyze the thread dumps to get an idea what is currently happening in your application. TDA will try to give you some hints about what might be wrong in the dumps (e.g. a lot of threads are waiting for the garbage collector) like in the screenshot below.

High load without anybody accessing the application usually either means the garbage collector is running endlessly because the application is very low on memory and the garbage collector is unable to free enough memory (TDA will give you a hint on this) or there are some threads (or just one) looping and running endlessly. TDA tries to filter out all idle threads to make it easier for you to find really running threads which are either running or are waiting for some external resources (it will give you a hint regarding the last issue). You need to enable these filters in the filter settings. You might also have a look at the TDA help for additional information.

For letting TDA search for long running threads mark the threads dumps you want to analyze and choose the "Find long running" threads option. TDA will then search for threads it finds in the dumps at least n-times, whereas you define n in the setting dialog for the detection. It will then present the result added to the dump tree. You should enable the idle-threads filter to filter out unimportant threads. In the screenshot below you can see a long running thread which is running in an endless loop.