KittenWhisker is a tool to facilitate the performance debugging of Spark applications.
It generates symbol files for JIT compiled methods of Drivers and Executors which can be further
used by Linux
The below is an example of FlameGraph generated by KittenWhisker (Sorry for missing the original SVG file, just do it intentionally to protect some credential infomation)
The workflow of KittenWhisker is as following:
By providing java options through
spark-submit, KittenWhisker is attached to driver and executor processes as a java agent
In the java agent, it will (1) fork a new process triggering linux perf tool to sample the CPU usage specific to the Driver/Executor process; and (2) generate symbol files for JIT methods with perf-map-agent by Johannes Rudolph
Eventually, the generated files are uploaded to the HDFS-compatible system
Users can collect the uploaded data and do further analysis, e.g. generating FlameGraph.
Run Spark Application with KittenWhisker
Compose your perf command
The result from KittenWhisker is based on Linux's perf-event, so you have to specify the
perfcommand for KittenWhisker to run. For example, you can save the following command in a file named
perf.conf: (more usage of perf can be found in Brendan's blog post )
sudo perf record -F 99 -g sleep 120
sleep 120will make perf to sample the usage for 2 mins, and KittenWhisker will insert
-oright before sleep command, which specifies the PID to be monitored and the name of output file respectively.
compose spark-submit command in start_app_cmd.sh. The below is an example of the command
spark-submit --master yarn-client --class perf.RunBenchmark --files /home/zhunan/code/kittenwhisker/perf.conf --driver-class-path $JAVA_HOME/lib/tools.jar --jars /home/zhunan/code/kittenwhisker/target/kittenwhisker-0.1-SNAPSHOT-jar-with-dependencies.jar, $JAVA_HOME/lib/tools.jar --conf spark.driver.extraJavaOptions=-javaagent:/home/zhunan/code/ kittenwhisker/target/kittenwhisker-0.1-SNAPSHOT-jar-with-dependencies.jar=waitingLength=200000, targetDirectory=/flameperf/ --conf "spark.executor.extraJavaOptions=-javaagent: ./happysparking-0.1-SNAPSHOT-jar-with-dependencies.jar=waitingLength=200000, targetDirectory=/flameperf/" --conf "spark.executor.extraClassPath=./tools.jar" --driver-memory 16g --executor-memory 20g --executor-cores 8 --num-executors 4 /home/zhunan/code/spark-benchmark/target/scala-2.11/spark-benchmark.jar sql --benchmark perf.TPCDS --database db1 --path /tpcds/ --executionMode parquet -i 1 --outputDir /outputresults/ --reportFormat parquet
In this command, we did the following things
--files /home/zhunan/code/kittenwhisker/perf.conf: upload the file containing perf command perf.conf to every executors/driver;
--driver-class-path $JAVA_HOME/lib/tools.jar: specify the driver class path to make Sun's tool jar to be accessible by driver process;
--jars /home/zhunan/code/kittenwhisker/target/kittenwhisker-0.1-SNAPSHOT-jar-with-dependencies.jar, $JAVA_HOME/lib/tools.jar: upload KittenWhisker's jar and Sun's tools.jar to the executors' class paths;
--conf spark.driver.extraJavaOptions=-javaagent:/home/zhunan/code/ kittenwhisker/target/kittenwhisker-0.1-SNAPSHOT-jar-with-dependencies.jar=waitingLength=200000, targetDirectory=/flameperf/: we specify java options for the driver to make KittenWhisker attached to the Driver process as a java agent; In this option, we specified two parameters for KittenWhisker, waitingLength and targetDirectory.
waitingLengthwill make KittenWhisker wait for a period with a specified length (in ms), which is useful to try to measure only after your JVM is warmed up.
targetDirectoryis the directory in a HDFS-compatible system which is to store the files generated by KittenWhisker.
--conf "spark.executor.extraJavaOptions=-javaagent: ./happysparking-0.1-SNAPSHOT-jar-with-dependencies.jar=waitingLength=200000, targetDirectory=/flameperf/": similar to the last one, we specify java options for the executors
--conf "spark.executor.extraClassPath=./tools.jar": specifies tools.jar should be included in the executors' class path
Others: the others are Spark's resource-related and application specific parameters.
Run Spark application with KittenWhisker
now you can run your spark application with KittenWhisker.
./start_test.sh zhunan /flameperf ./
The first parameter of the script is the name to login to the nodes in your cluster; the second one is the directory in the HDFS-compatible system to save the generated files; the third parameter specifies the local location to download data from HDFS.
This script will grant
yarnuser the sudo permission to run perf and chmod commands, since Linux's perf tool requires that. After the application is finished, the permission would be revoked.
Produce Flamegraph with KittenWhisker
With one line command, you can generate Flame graphs for all processes in your spark applications.
the only parameter is the local directory path saving the downloaded data.
Prepare Slaves File
Before you install KittenWhisker, you have to prepare a
slaves files containing IP address of all nodes
in your cluster (or all nodes which can be used to run Spark Drivers/Executors)
Clone KittenWhisker repository and install it
- First you have to clone the repo from github
git clone https://github.com/CodingCat/KittenWhisker.git
- Then you can compile it with maven (ensure that you installed
cd KittenWhisker; mvn package
Install Depended Tools
Then you need to install several tools depended by
Install Linux's Perf tool in every node of the cluster
Depends on your OS, you have various ways to do it, e.g. in Ubuntu, you can do it by
sudo apt-get install linux-tools-common linux-tools-generic linux-tools-`uname -r`
You can also install via a helper script provided by KittenWhisker, run the following command in the root path of KittenWhisker
This script will read slaves file, login to each machine specified in the file with the username you provided and install perf tool. (Assuming the machine is with Ubuntu OS)
Install JDK debugging symbols (if you are using OpenJDK)
If you are using OpenJDK, JDK debugging symbols are used to interpret the JVM runtime function names. To install it, you can run
sudo apt-get install openjdk-8-dbg
Or with the helper script in KittenWhisker, you can simply do
with the assumption that you are using Ubuntu.
Install Flamegraph tool (just need to be done in the machine you currently login in )
Install Flamegraph tool is as simple as clone Brendan's repo and setup environment variable for KittenWhisker. Do the following
git clone https://github.com/brendangregg/FlameGraph.git cd FlameGraph; echo "export FLAMEGRAPH_DIR=`pwd`" >> ~/.bashrc
This work cannot be done without the existing tools from other developers. It uses perf-map-agent by Johannes Rudolph, and it also grabs some code from Min Zhou's perfj. The flame graph generation part is, of course, based on Brendan Gregg's FlameGraph Tool
It is NOT a software released by Microsoft and the code is very experimental! You will use it at your own risk. You have been warned!