Switch branches/tags
Nothing to show
Clone or download
brendangregg Merge pull request #161 from pcdv/master
Add stackcollapse for java error stacks in logs
Latest commit f857ebc Oct 16, 2018
Type Name Latest commit message Commit time
Failed to load latest commit information.
demos add mixedmode example Oct 11, 2015
dev Fix spelling of stackcollapse in comments Oct 16, 2018
docs USAGE message and license file reference May 18, 2013
test add another test source (vertx) Aug 17, 2017
.travis.yml Travis CI test Jul 5, 2016
README.md [#138] Add stackcollapse script for visual studio profiles Jul 28, 2017
aix-perf.pl AIX stack probes and stack collapse script Dec 28, 2016
difffolded.pl add -s for diffing profiles including hex numbers Nov 4, 2014
example-dtrace-stacks.txt more examples Dec 8, 2015
example-dtrace.svg more examples Dec 8, 2015
example-perf-stacks.txt.gz more examples Dec 8, 2015
example-perf.svg more examples Dec 8, 2015
files.pl Merge pull request #128 from milianw/loop-files Apr 7, 2017
flamegraph.pl Add a Flame Chart mode which orders horizontally by (call) time Aug 5, 2018
jmaps use existing AGENT_HOME environment variable if present Aug 9, 2017
pkgsplit-perf.pl update header Aug 11, 2017
range-perf.pl range-perf.pl: add options for different timestamps Feb 24, 2017
record-test.sh update tests Aug 17, 2017
stackcollapse-aix.pl AIX stack probes and stack collapse script Dec 28, 2016
stackcollapse-bpftrace.pl Add stackcollapse-bpftrace.pl Oct 16, 2018
stackcollapse-elfutils.pl stackcollapse tool for elfutils stack Oct 11, 2015
stackcollapse-gdb.pl stackcollapse-gdb: Do not forget the last sample Oct 11, 2015
stackcollapse-go.pl Fix spelling of stackcollapse in comments Oct 16, 2018
stackcollapse-instruments.pl Support floating point numbers in instruments CSV Dec 6, 2015
stackcollapse-java-exceptions.pl Ignore "waiting on" in middle of stack Feb 2, 2018
stackcollapse-jstack.pl Fix spelling of stackcollapse in comments Oct 16, 2018
stackcollapse-ljp.awk Fix spelling of stackcollapse in comments Oct 16, 2018
stackcollapse-perf-sched.awk Be less brittle to different arguments to perf script Jan 21, 2016
stackcollapse-perf.pl Fix spelling of stackcollapse in comments Oct 16, 2018
stackcollapse-pmc.pl Add FreeBSD hwpmc callgraph parser Jun 3, 2014
stackcollapse-recursive.pl Recursive call filter: allow floating point values and better regex h… Jul 23, 2015
stackcollapse-sample.awk Allow stacks to be captured from /usr/bin/sample on macOS Mar 9, 2017
stackcollapse-stap.pl Fix spelling of stackcollapse in comments Oct 16, 2018
stackcollapse-vsprof.pl [#138] Add stackcollapse script for visual studio profiles Jul 28, 2017
stackcollapse-vtune.pl Update stackcollapse-vtune.pl Sep 7, 2016
stackcollapse-xdebug.php Add GPL license Apr 13, 2018
stackcollapse.pl Fix spelling of stackcollapse in comments Oct 16, 2018
test.sh update tests Aug 17, 2017


Flame Graphs visualize profiled code

Main Website: http://www.brendangregg.com/flamegraphs.html

Example (click to zoom): Example

Other sites:

Flame graphs can be created in three steps:

  1. Capture stacks
  2. Fold stacks
  3. flamegraph.pl

1. Capture stacks

Stack samples can be captured using Linux perf_events, FreeBSD pmcstat (hwpmc), DTrace, SystemTap, and many other profilers. See the stackcollapse-* converters.

Linux perf_events

Using Linux perf_events (aka "perf") to capture 60 seconds of 99 Hertz stack samples, both user- and kernel-level stacks, all processes:

# perf record -F 99 -a -g -- sleep 60
# perf script > out.perf

Now only capturing PID 181:

# perf record -F 99 -p 181 -g -- sleep 60
# perf script > out.perf


Using DTrace to capture 60 seconds of kernel stacks at 997 Hertz:

# dtrace -x stackframes=100 -n 'profile-997 /arg0/ { @[stack()] = count(); } tick-60s { exit(0); }' -o out.kern_stacks

Using DTrace to capture 60 seconds of user-level stacks for PID 12345 at 97 Hertz:

# dtrace -x ustackframes=100 -n 'profile-97 /pid == 12345 && arg1/ { @[ustack()] = count(); } tick-60s { exit(0); }' -o out.user_stacks

60 seconds of user-level stacks, including time spent in-kernel, for PID 12345 at 97 Hertz:

# dtrace -x ustackframes=100 -n 'profile-97 /pid == 12345/ { @[ustack()] = count(); } tick-60s { exit(0); }' -o out.user_stacks

Switch ustack() for jstack() if the application has a ustack helper to include translated frames (eg, node.js frames; see: http://dtrace.org/blogs/dap/2012/01/05/where-does-your-node-program-spend-its-time/). The rate for user-level stack collection is deliberately slower than kernel, which is especially important when using jstack() as it performs additional work to translate frames.

2. Fold stacks

Use the stackcollapse programs to fold stack samples into single lines. The programs provided are:

  • stackcollapse.pl: for DTrace stacks
  • stackcollapse-perf.pl: for Linux perf_events "perf script" output
  • stackcollapse-pmc.pl: for FreeBSD pmcstat -G stacks
  • stackcollapse-stap.pl: for SystemTap stacks
  • stackcollapse-instruments.pl: for XCode Instruments
  • stackcollapse-vtune.pl: for Intel VTune profiles
  • stackcollapse-ljp.awk: for Lightweight Java Profiler
  • stackcollapse-jstack.pl: for Java jstack(1) output
  • stackcollapse-gdb.pl: for gdb(1) stacks
  • stackcollapse-go.pl: for Golang pprof stacks
  • stackcollapse-vsprof.pl: for Microsoft Visual Studio profiles

Usage example:

For perf_events:
$ ./stackcollapse-perf.pl out.perf > out.folded

For DTrace:
$ ./stackcollapse.pl out.kern_stacks > out.kern_folded

The output looks like this:

unix`_sys_sysenter_post_swapgs 1401
unix`_sys_sysenter_post_swapgs;genunix`close 5
unix`_sys_sysenter_post_swapgs;genunix`close;genunix`closeandsetf 85
unix`_sys_sysenter_post_swapgs;genunix`close;genunix`closeandsetf;c2audit`audit_closef 26
unix`_sys_sysenter_post_swapgs;genunix`close;genunix`closeandsetf;c2audit`audit_setf 5
unix`_sys_sysenter_post_swapgs;genunix`close;genunix`closeandsetf;genunix`audit_getstate 6
unix`_sys_sysenter_post_swapgs;genunix`close;genunix`closeandsetf;genunix`audit_unfalloc 2
unix`_sys_sysenter_post_swapgs;genunix`close;genunix`closeandsetf;genunix`closef 48

3. flamegraph.pl

Use flamegraph.pl to render a SVG.

$ ./flamegraph.pl out.kern_folded > kernel.svg

An advantage of having the folded input file (and why this is separate to flamegraph.pl) is that you can use grep for functions of interest. Eg:

$ grep cpuid out.kern_folded | ./flamegraph.pl > cpuid.svg

Provided Examples

Linux perf_events

An example output from Linux "perf script" is included, gzip'd, as example-perf-stacks.txt.gz. The resulting flame graph is example-perf.svg:


You can create this using:

$ gunzip -c example-perf-stacks.txt.gz | ./stackcollapse-perf.pl --all | ./flamegraph.pl --color=java --hash > example-perf.svg

This shows my typical workflow: I'll gzip profiles on the target, then copy them to my laptop for analysis. Since I have hundreds of profiles, I leave them gzip'd!

Since this profile included Java, I used the flamegraph.pl --color=java palette. I've also used stackcollapse-perf.pl --all, which includes all annotations that help flamegraph.pl use separate colors for kernel and user level code. The resulting flame graph uses: green == Java, yellow == C++, red == user-mode native, orange == kernel.

This profile was from an analysis of vert.x performance. The benchmark client, wrk, is also visible in the flame graph.


An example output from DTrace is also included, example-dtrace-stacks.txt, and the resulting flame graph, example-dtrace.svg:


You can generate this using:

$ ./stackcollapse.pl example-stacks.txt | ./flamegraph.pl > example.svg

This was from a particular performance investigation: the Flame Graph identified that CPU time was spent in the lofs module, and quantified that time.


See the USAGE message (--help) for options:

USAGE: ./flamegraph.pl [options] infile > outfile.svg

--title       # change title text
--width       # width of image (default 1200)
--height      # height of each frame (default 16)
--minwidth    # omit smaller functions (default 0.1 pixels)
--fonttype    # font type (default "Verdana")
--fontsize    # font size (default 12)
--countname   # count type label (default "samples")
--nametype    # name type label (default "Function:")
--colors      # set color palette. choices are: hot (default), mem, io,
              # wakeup, chain, java, js, perl, red, green, blue, aqua,
              # yellow, purple, orange
--hash        # colors are keyed by function name hash
--cp          # use consistent palette (palette.map)
--reverse     # generate stack-reversed flame graph
--inverted    # icicle graph
--negate      # switch differential hues (blue<->red)
--help        # this message

./flamegraph.pl --title="Flame Graph: malloc()" trace.txt > graph.svg

As suggested in the example, flame graphs can process traces of any event, such as malloc()s, provided stack traces are gathered.

Consistent Palette

If you use the --cp option, it will use the $colors selection and randomly generate the palette like normal. Any future flamegraphs created using the --cp option will use the same palette map. Any new symbols from future flamegraphs will have their colors randomly generated using the $colors selection.

If you don't like the palette, just delete the palette.map file.

This allows your to change your colorscheme between flamegraphs to make the differences REALLY stand out.


Say we have 2 captures, one with a problem, and one when it was working (whatever "it" is):

cat working.folded | ./flamegraph.pl --cp > working.svg
# this generates a palette.map, as per the normal random generated look.

cat broken.folded | ./flamegraph.pl --cp --colors mem > broken.svg
# this svg will use the same palette.map for the same events, but a very
# different colorscheme for any new events.

Take a look at the demo directory for an example: