Skip to content
presentation for JDayLviv 2015, Thread dumps on JVM
HTML
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.gitignore
Heap.png
JVM_middleware_interactions.png
MemoryInJVM.png
Threaddumps-on-JVM.html
commands
comparisonBetweenThreadsAndThreadsWithLocks.png
java_pid_java_tools.png
jps.png
readme.adoc
td1
td2
td3
td4
td5
threads
threadsAndLocks

readme.adoc

Thread-dumps on JVM @ JDay Lviv

Promise

How can you take a threaddump?
Print threads?
What is a safepoint and why you should care?
Which tools can help and how?
And when you have the threaddump…​ what’s in there?
We’ll talk threads and locks. Dead and alive alike.
We’ll talk CLI and GUI (tools, of course).
We’ll talk about thread lifecycle.
And naming. Oh, the naming!
Example rich talk, though rather aimed at beginners.

JVM overview

Image showing where threads work and from where they originate from

Threads

  1. instances of java.lang.Thread

  2. have associated native thread

  3. have names (or are named Thread#nextNumberInSequence)

  4. have local variables, accessible only to given thread

Important
Thread lifecycle

Enum: java.lang.Thread.State:

NEW

Thread state for a thread which has not yet started.

RUNNABLE

Thread state for a runnable thread.

BLOCKED

Thread state for a thread blocked waiting for a monitor lock.

WAITING

A thread in the waiting state is waiting for another thread to perform a particular action.

TIMED_WAITING

A thread that is waiting for another thread to perform an action for up to a specified waiting time is in this state.

TERMINATED

A thread that has exited is in this state.

Thread.getState() gives you that.

Native thread gets created in NEW and reclaimed upon thread termination.

Caution
RUNNABLE means OK, right?

Thread limits?

Are there any?

What limits the threads?

JVM and RAM

Let’s add more for just one JVM:

Memory organization in JVM

See limit, Bash built-in. In my case:

➜  ThreaddumpsOnJVM git:(master) ✗ limit
cputime         unlimited
filesize        unlimited
datasize        unlimited
stacksize       8MB
coredumpsize    0kB
memoryuse       unlimited
maxproc         63635
descriptors     1024
memorylocked    64kB
addressspace    unlimited
maxfilelocks    unlimited
sigpending      63635
msgqueue        819200
nice            0
rt_priority     0
rt_time         unlimited

Interesting limits are stacksize, descriptors and maxfilelocks when speaking about threads.

How can you take a threaddump?

  1. kill -3 JAVA_PID_HERE

About kill command

kill

send a signal to a process

kill -9 111
kill -s SIGKILL 111
kill -KILL 111

May be a shell built-in AND a command: which kill
Essentially a wrapper around kill syscall.

  1. kill -L - no such option on my Ubuntu/Bash

    1. kill -l

  2. /bin/kill -L

    1. /bin/kill -l - same as built-in

Tip
Further reading about kill

Excellent pieces of information, especially the first one:

About signals

/bin/kill -L

Default? TERM

man --section 7 signals | grep SIGQUIT
man --section 7 signals | grep Core

Caution
Java and signals
-Xrs

disables default signals for Java applications

PID of a Java process?

  1. ps aux | grep java

  2. pgrep java

  3. jps -l

    jps

    Lists the instrumented Java Virtual Machines (JVMs) on the target
    system. This command is experimental and unsupported.

  4. jcmd

    jcmd

    sends diagnostic command requests to a running JVM.

Image shows comparison of ways above
Image shows how Java tools offer Java PIDs
Warning
jps output is empty?
  1. which user is running the Java process you are looking for?

  2. ls -l /tmp/hsperfdata_YourUserHere/

  3. check the -Djava.io.tmpdir flag or start JVM setting it to =/someDir.

How can you take a threaddump?

  1. kill -3 JAVA_PID

  2. jstack JAVA_PID

    jstack

    Prints Java thread stack traces for a Java process, core file,
    or remote debug server. This command is experimental and unsupported.

Tip
Poor man’s debugger? jstack in a loop…​ However, can pinpoint live-locks!

How can you take a threaddump?

  1. kill -3 JAVA_PID

  2. jstack JAVA_PID

  3. Ctrl+\

  4. jconsole

  5. jvisualvm

Caution
Dump location?
  1. Your process' console

  2. Current directory

  3. JVM settings

    1. different JVMs, different flags

  4. System settings?

    1. man core

    2. cat /proc/sys/kernel/core_pattern

Printing threads then?

jcmd

Going native…​

whichThreadIsIt.sh

ps -mo lwp,c -p 14222

Safepoints

What is a safepoint and why you should care?

Thread dump then

And when you have the threaddump…​ what’s in there?

  1. JVM and Java info

  2. Thread dump info

  3. Stack traces of threads

  4. Heap breakdown

Comparing printing threads

With -l we have also lock information, who holds what:

Shows difference between just printing threads and printing them with lock info via `-l`

Commands used during presentation

jconsole
ctrl+break

kill -3 PID
kill -s 3 PID
kill -QUIT PID

type kill
man kill

man -k signal

man --section 7 signals | grep SIGQUIT
man --section 7 signals | grep Core

man core

vim /proc/sys/kernel/core_pattern


ps | grep java

pgrep java
pgrep -u root java
pgrep -u root,tammo java

jps -q


jps
jps -l
jps -m
jcmd
jps -v
jcmd -l

ls /tmp/hsperfdata_tammo/
while true; do; ls /tmp/hsperfdata_tammo/; done


jcmd -options
jcmd 12769 help
jcmd 12769 help Thread.print
jcmd 12769 Thread.print

jcmd 12769 Thread.print > threads
jcmd 12769 Thread.print -l > threadsAndLocks

jcmd 14222 Thread.print -l > threadsAndLocks

jstack 14222 > threaddump

man jcmd
man jinfo
man jstack
jvisualvm -J-Xverbose:gc

for i in $(seq 1 5); do; jstack 16002 > td$i; echo $i; sleep 5; done;

jconsole
jvisualvm

Besides link already used and added earlier, few others caught my eye:

Caution
If you started process in the background and killed it’s terminal, process might get killed. If it ain’t, then
to redirect it’s output is no easy task. Still, it’s feasible.
You can’t perform that action at this time.