Overview

This is a set of tools used to find memory leaks in long-running applications. These tools use =perf= to instrument all memory allocations/deallocations. These traces can then be analyzed to find allocated memory that was not properly cleaned up.

This was written to find memory leaks in emacs daemon sessions. The resulting bugs and mailing list posts:

• https://lists.gnu.org/archive/html/emacs-devel/2015-09/msg00814.html
• http://debbugs.gnu.org/cgi/bugreport.cgi?bug=21509
• http://debbugs.gnu.org/cgi/bugreport.cgi?bug=21556
• http://debbugs.gnu.org/cgi/bugreport.cgi?bug=21623

Some of the tools are emacs-specific, but some are not. Note that this is all fairly rough, and the user would want to understand how each tool works to be able to use it effectively. This is also not well-documented yet, but the links above, and the text below show examples.

Tools

Non-emacs-specific

reademacsvar.sh

Exports some shell variables that all the scripts use to do their thing. Mostly this is paths, etc.

plotmem.sh

Used to generate a realtime plot of memory usage of a particular process. This lets us see the leaks as they happen.

probes.sh

Creates perf probes that we care about. This is all allocation/deallocations, and anything else we like.

record_alloc.sh

Runs perf record to record all allocations/deallocations.

parse_script.pl

Reads the output of perf script, following all allocations. The output is a list of all unexpected memory operations (calling free() on a pointer that didn’t come from malloc() for instance) and a list of all potentially leaky memory (anything that was allocated but not freed). If we started logging some time after the process has started, and finished logging before the process has finished then both of these will have false positives:

• We could see a free() of memory that was allocated before we started logging
• We can report a leak for something that was free()-ed after we stopped logging

So take all output with a grain of salt, and use your best judgement. The backtraces come from perf. It reports code addresses, and not line numbers. You can get the line numbers line this:

addr2line -e LIBRARY ADDRESS

where LIBRARY and ADDRESS come from the perf backtrace.

plotleaks.sh

Takes the output of parse_script.pl, and makes a plot of potential leak sizes vs input line number. This is useful to quickly see the leaks. For instance, let’s say the emacs session we’re tracing leaks 3072 bytes each time a new frame is created, and that we created 10 frames in a row while running perf record. The plot this script produces would then show 10 points at 3072 evenly spaced through time. Leaks at the start of the session are most likely to be true ones (there was time to free() the memory), so I generally follow up anything that leaked lots of memory at the start.

follow_alloc.pl

Reads the output of perf script, and filters out all memory operations that do not refer to a particular allocation size. This is useful to focus on particular allocations identified by plotleaks.sh. So if plotleaks.sh shows lots of leaks of size 3072, we use this to cut down the log to show only the leaks we care about.

Emacs-specific

daemon.sh

Starts up a new emacs daemon.

client.sh

Creates a new client frame.

kill.sh

Kills the emacs daemon.

loopclient.sh

Repeatedly creates/destroys a client frame.

show_stderr.sh

Shows the STDERR output of a process (emacs in this case). This is useful because the emacs daemon redirects its STDERR to /dev/null, but gdb printing commands such as pr and pp write to STDERR, and we want to see this output.

General notes

When running perf record, the perf process can be overloaded and drop events as a result (an error message says this). A larger buffer can help (-m option) at the expense of using more RAM. It also helps to make smaller logs (fewer things to record, fp backtrace generation instead of DWARF; see below).

perf can generate backtraces in two ways:

• Using the frame-pointer. This is the preferred method, but it only works for functions that have a frame pointer. All gcc optimization levels strip this out, so rebuild with -fno-omit-frame-pointer if possible.
• Using DWARF debug information. This doesn’t require frame pointers, but needs debug info. Another down side is that this generates much bigger perf logs, and perf is more likely to drop events. Currently perf has a bug in that it’s not able to read the split debug information in Debian packages, so you need this patch: http://lkml.iu.edu/hypermail/linux/kernel/1509.0/04006.html

If the binary application being instrumented is rebuilt, probes into that application need to be deleted and re-added.

Example

A usage example appears here.

License

Released into the public domain. Do whatever you like.