Toolkit of simple scripts useful for managing Hadoop
Python Shell Perl
Switch branches/tags
Nothing to show
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.


haatkit is a toolkit of simple scripts that are useful when working with a Hadoop cluster, or probably any system primarily made up of a lot of different Java processes.

It was inspired by (but shares no code with) the maatkit suite of tools for working with MySQL.

To be included in haatkit, a tool should be:

  • no more than a few hundred lines of python or bash
  • require no dependencies that aren't installed on a stock RHEL5 system
  • follow the "Unix philosophy": -- easy to connect with other tools via pipes -- return reasonable exit codes on failure -- use command line flags rather than configuration

Tools currently included


Colorizes certain regular expressions inside a log. For example:

$ hk-colorize "{r'WARN|FATAL|Exception': 'red', r'blk_\d+': 'yellow,bold'}"

If you would like to page through the colorized output, 'less -R' is recommended.

hk-jgrep, hk-jkill

Equivalents of pgrep/pkill for Java processes. For example:

$ hk-jkill -9 HRegionServer


$ cat /proc/$(hk-jgrep NameNode)/limits


The JDK jstack utility can be very finicky. In particular, if the executing user is not the same as the user who ran the JVM, or if the version of jstack comes from a slightly different version of JVM than is running the process, it will return incorrect results, and encourage the user to run 'jstack -F'. This rarely produces a useful stack trace.

hk-jstack determines the correct version of jstack and uses sudo to become the correct user to acquire the dump.


$ hk-jstack $(hk-jgrep NameNode)


Filters a jstack of a Hadoop daemon to cut out "uninteresting" threads. Uninteresting threads are things like threads in threadpools that are currently just waiting for something to do, or IPC handlers that have no call to process.


$ hk-jstack $(hk-jgrep NameNode) | hk-interesting-threads


Takes in a jstack and outputs a summary of which threads are waiting on which locks, and where those locks are held.


$ hk-jstack $(hk-jgrep HRegionServer) | hk-lock-contention
1 waiting on 0x000000060bdbe6f0 (java.util.HashMap)
  held at org.apache.hadoop.hbase.regionserver.HRegionServer.metrics(