KittenWhisker

KittenWhisker is a tool to facilitate the performance debugging of Spark applications. It generates symbol files for JIT compiled methods of Drivers and Executors which can be further used by Linux perf tools.

The below is an example of FlameGraph generated by KittenWhisker (Sorry for missing the original SVG file, just do it intentionally to protect some credential infomation)

I will show how to configure KittenWhisker and use KittenWhisker.

Use KittenWhisker

The workflow of KittenWhisker is as following:

1. By providing java options through spark-submit, KittenWhisker is attached to driver and executor processes as a java agent

2. In the java agent, it will (1) fork a new process triggering linux perf tool to sample the CPU usage specific to the Driver/Executor process; and (2) generate symbol files for JIT methods with perf-map-agent by Johannes Rudolph

3. Eventually, the generated files are uploaded to the HDFS-compatible system

4. Users can collect the uploaded data and do further analysis, e.g. generating FlameGraph.

Run Spark Application with KittenWhisker

1. Compose your perf command

   The result from KittenWhisker is based on Linux's perf-event, so you have to specify the perf command for KittenWhisker to run. For example, you can save the following command in a file named perf.conf: (more usage of perf can be found in Brendan's blog post )

   sudo perf record -F 99 -g sleep 120

   Note that, sleep 120 will make perf to sample the usage for 2 mins, and KittenWhisker will insert -p and -o right before sleep command, which specifies the PID to be monitored and the name of output file respectively.

2. compose spark-submit command in start_app_cmd.sh. The below is an example of the command

spark-submit --master yarn-client --class perf.RunBenchmark --files
 /home/zhunan/code/kittenwhisker/perf.conf --driver-class-path $JAVA_HOME/lib/tools.jar --jars
  /home/zhunan/code/kittenwhisker/target/kittenwhisker-0.1-SNAPSHOT-jar-with-dependencies.jar,
  $JAVA_HOME/lib/tools.jar --conf spark.driver.extraJavaOptions=-javaagent:/home/zhunan/code/
  kittenwhisker/target/kittenwhisker-0.1-SNAPSHOT-jar-with-dependencies.jar=waitingLength=200000,
  targetDirectory=/flameperf/ --conf "spark.executor.extraJavaOptions=-javaagent:
  ./happysparking-0.1-SNAPSHOT-jar-with-dependencies.jar=waitingLength=200000,
  targetDirectory=/flameperf/" --conf "spark.executor.extraClassPath=./tools.jar" --driver-memory
   16g --executor-memory 20g --executor-cores 8 --num-executors 4
    /home/zhunan/code/spark-benchmark/target/scala-2.11/spark-benchmark.jar
     sql --benchmark perf.TPCDS --database db1
      --path /tpcds/ --executionMode parquet -i 1 --outputDir /outputresults/ --reportFormat parquet

In this command, we did the following things

• --files /home/zhunan/code/kittenwhisker/perf.conf: upload the file containing perf command perf.conf to every executors/driver;

• --driver-class-path $JAVA_HOME/lib/tools.jar: specify the driver class path to make Sun's tool jar to be accessible by driver process;

• --jars /home/zhunan/code/kittenwhisker/target/kittenwhisker-0.1-SNAPSHOT-jar-with-dependencies.jar, $JAVA_HOME/lib/tools.jar: upload KittenWhisker's jar and Sun's tools.jar to the executors' class paths;

• --conf spark.driver.extraJavaOptions=-javaagent:/home/zhunan/code/ kittenwhisker/target/kittenwhisker-0.1-SNAPSHOT-jar-with-dependencies.jar=waitingLength=200000, targetDirectory=/flameperf/: we specify java options for the driver to make KittenWhisker attached to the Driver process as a java agent; In this option, we specified two parameters for KittenWhisker, waitingLength and targetDirectory. waitingLength will make KittenWhisker wait for a period with a specified length (in ms), which is useful to try to measure only after your JVM is warmed up. targetDirectory is the directory in a HDFS-compatible system which is to store the files generated by KittenWhisker.

• --conf "spark.executor.extraJavaOptions=-javaagent: ./happysparking-0.1-SNAPSHOT-jar-with-dependencies.jar=waitingLength=200000, targetDirectory=/flameperf/": similar to the last one, we specify java options for the executors

• --conf "spark.executor.extraClassPath=./tools.jar": specifies tools.jar should be included in the executors' class path

• Others: the others are Spark's resource-related and application specific parameters.

3. Run Spark application with KittenWhisker

   now you can run your spark application with KittenWhisker.

   ./start_test.sh zhunan /flameperf ./

   The first parameter of the script is the name to login to the nodes in your cluster; the second one is the directory in the HDFS-compatible system to save the generated files; the third parameter specifies the local location to download data from HDFS.

   This script will grant yarn user the sudo permission to run perf and chmod commands, since Linux's perf tool requires that. After the application is finished, the permission would be revoked.

4. Produce Flamegraph with KittenWhisker

   With one line command, you can generate Flame graphs for all processes in your spark applications.

   ./generate_flamegraph.sh ./flameperf

   the only parameter is the local directory path saving the downloaded data.

Configure KittenWhisker

Prepare Slaves File

Before you install KittenWhisker, you have to prepare a slaves files containing IP address of all nodes in your cluster (or all nodes which can be used to run Spark Drivers/Executors)

Clone KittenWhisker repository and install it

1. First you have to clone the repo from github

git clone https://github.com/CodingCat/KittenWhisker.git

2. Then you can compile it with maven (ensure that you installed cmake beforehand)

cd KittenWhisker; mvn package

Install Depended Tools

Then you need to install several tools depended by KittenWhisker

1. Install Linux's Perf tool in every node of the cluster

   Depends on your OS, you have various ways to do it, e.g. in Ubuntu, you can do it by

   sudo apt-get install linux-tools-common linux-tools-generic linux-tools-`uname -r`
   

   You can also install via a helper script provided by KittenWhisker, run the following command in the root path of KittenWhisker

   ./install_packages.sh username_to_login_to_machines 
   

   This script will read slaves file, login to each machine specified in the file with the username you provided and install perf tool. (Assuming the machine is with Ubuntu OS)

2. Install JDK debugging symbols (if you are using OpenJDK)

   If you are using OpenJDK, JDK debugging symbols are used to interpret the JVM runtime function names. To install it, you can run

   sudo apt-get install openjdk-8-dbg
   

   Or with the helper script in KittenWhisker, you can simply do

   ./install_packages.sh username_to_login_to_machines 
   

   with the assumption that you are using Ubuntu.

3. Install Flamegraph tool (just need to be done in the machine you currently login in )

   Install Flamegraph tool is as simple as clone Brendan's repo and setup environment variable for KittenWhisker. Do the following

   git clone https://github.com/brendangregg/FlameGraph.git
   
   cd FlameGraph; echo "export FLAMEGRAPH_DIR=`pwd`" >> ~/.bashrc
   

Acknowledgement

This work cannot be done without the existing tools from other developers. It uses perf-map-agent by Johannes Rudolph, and it also grabs some code from Min Zhou's perfj. The flame graph generation part is, of course, based on Brendan Gregg's FlameGraph Tool

Disclaimer

It is NOT a software released by Microsoft and the code is very experimental! You will use it at your own risk. You have been warned!