Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Toolkit of simple scripts useful for managing Hadoop

branch: master

Fetching latest commit…

Octocat-spinner-32-eaf2f5

Cannot retrieve the latest commit at this time

Octocat-spinner-32 bin
Octocat-spinner-32 LICENSE.txt
Octocat-spinner-32 README.md
README.md

haatkit

haatkit is a toolkit of simple scripts that are useful when working with a Hadoop cluster, or probably any system primarily made up of a lot of different Java processes.

It was inspired by (but shares no code with) the maatkit suite of tools for working with MySQL.

To be included in haatkit, a tool should be:

  • no more than a few hundred lines of python or bash
  • require no dependencies that aren't installed on a stock RHEL5 system
  • follow the "Unix philosophy": -- easy to connect with other tools via pipes -- return reasonable exit codes on failure -- use command line flags rather than configuration

Tools currently included

hk-colorize

Colorizes certain regular expressions inside a log. For example:

$ hk-colorize "{r'WARN|FATAL|Exception': 'red', r'blk_\d+': 'yellow,bold'}"

If you would like to page through the colorized output, 'less -R' is recommended.

hk-jgrep, hk-jkill

Equivalents of pgrep/pkill for Java processes. For example:

$ hk-jkill -9 HRegionServer

or:

$ cat /proc/$(hk-jgrep NameNode)/limits

hk-jstack

The JDK jstack utility can be very finicky. In particular, if the executing user is not the same as the user who ran the JVM, or if the version of jstack comes from a slightly different version of JVM than is running the process, it will return incorrect results, and encourage the user to run 'jstack -F'. This rarely produces a useful stack trace.

hk-jstack determines the correct version of jstack and uses sudo to become the correct user to acquire the dump.

Usage:

$ hk-jstack $(hk-jgrep NameNode)

hk-interesting-threads

Filters a jstack of a Hadoop daemon to cut out "uninteresting" threads. Uninteresting threads are things like threads in threadpools that are currently just waiting for something to do, or IPC handlers that have no call to process.

Usage:

$ hk-jstack $(hk-jgrep NameNode) | hk-interesting-threads

hk-lock-contention

Takes in a jstack and outputs a summary of which threads are waiting on which locks, and where those locks are held.

Usage:

$ hk-jstack $(hk-jgrep HRegionServer) | hk-lock-contention
1 waiting on 0x000000060bdbe6f0 (java.util.HashMap)
  held at org.apache.hadoop.hbase.regionserver.HRegionServer.metrics(HRegionServer.java:1164)
Something went wrong with that request. Please try again.