Functracer is a debugging tool. It collects backtraces, arguments and return value of functions specified in a plugin. It works with optimized (-O2) code, where debugging symbols are available either in the application binaries themselves, or in separate debug files.
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.



                 A specialized, tracing-based debugging tool

                Copyright (C) 2007-2012 by Nokia Corporation

 0. Authors
 1. Introduction
 2. Where can I find it
 3. How does it work
 4. How to use
 5. Where does it work
 6. Performance impact
 7. Other effects
 8. Documentation
 9. Issues and Limitations
 10. To Do
 11. License

0. Authors

Functracer has been developed by:

Anderson Lizardo <>
Allan Bezerra <>
Bruna Moreira <>
Bruno Abinader <>
Ricardo Guimaraes <>
Thiago Santos <>
Andris Zeila <>
Eero Tamminen <>

1. Introduction

Functracer is a specialized, tracing-based debugging tool. It collects
backtraces, arguments and return value of functions specified in a plugin. It
works with optimized (-O2) code, where debugging symbols are available either
in the application binaries themselves, or in separate debug files.

Currently, functracer has the following plugins implemented:
 "memory" (used by default, when no plugin is specified),

For more details, see the README.plugins document located in src/modules
(inside sources) or /usr/share/doc/functracer (when installed).

2. Where can I find it

3. How does it work

Using software breakpoints, just like GDB and ltrace. A breakpoint is inserted
on the entrypoint of selected functions specified in the plugins. When one of
them is called, the breakpoint is hit and the resource address, size and
callchain backtrace are collected and reported.

The functracer output format is independent of function type is tracked. The
function format is a generic one and it was designed to be uniform/compatible
with any function. This is need to do a post-processing tool
(functracer-postproc) be independent from them.

There is a need to identify following things from the resource allocation
- timestamp
- exact function name
- resource ID (address, object ID, FD, thread ID)
- resource size
- backtrace

And following from the resource free functions:
- timestamp
- exact function name
- resource ID (to match it)
- backtrace

Some extra considerations for these:
- If user does not need timestamp or backtrace (depth=0), they can be left out
  from the output.
- Resource size is not relevant for threads or FDs, for those it can be
  set/output as 1.
- Memset/copy and probably many user defined operations are not resource
  alloc/free operations, for those the "resource ID" can be zero.

Formats could be for resource allocation functions:
  <index>. [timestamp] <function name>(<size>) = <resource ID>
  [backtrace functions]

And for resource free functions:
  <index>. [timestamp] <function name>(<resource ID>)
  [backtrace functions]

For examble, on the memory plugin the resource allocation functions (e.g.
calloc, malloc, memalign, realloc) are tracked and functracer reports the

 function size   resource
   name    |      address 
    |      |        |
    v      v        v
0. malloc(456) = 0x0896adb0
   0x0804868c            ---
   0xb7e87118              |-> backtrace lines
   0xb7d94e6d            ---

For functions that perform both resource allocation and destruction (e.g.
realloc), it will be produced two separate events, first the destruction event,
then the allocation one. For example, if source code does:

realloc(0x001ae840, 16) = 0x001ae858

Functracer will produce the following trace events:

0. realloc(0x001ae840)       <- destruction event
1. realloc(16) = 0x001ae858  <- allocation event
   [backtrace lines]

4. How to use

Output from "functracer --help":

Usage: functracer [OPTION...] PROGRAM [ARGS...]
Run PROGRAM and track selected functions.

  -b, --free-backtraces      enable backtraces for free() function
  -d, --debug                maximum debug level
  -e, --track=PLUGIN         which set of functions to track
  -f, --file                 use a file to save backtraces instead of dump to
  -i, --time                 print absolute timestamps for every event
  -l, --path=DIR             dump reports to a custom location (defaults to
  -p, --pid=PID              which PID to track
  -r, --resolve-name         enable symbol name resolution
  -s, --start                enable tracking memory from beginning
  -t, --depth=NUMBER         maximum backtrace depth
  -v, --verbose              Show internal events
  -h, --help                 Give this help list
      --usage                Give a short usage message
  -V, --version              Print program version

You can control the dumping of output files. To use it, just send SIGUSR1
to the tracked application (NOT functracer itself) to toggle the tracing on/off.
Tracing is not enabled by default on application start, unless "-s" option is
used. The control logic is:
  - If not tracing:
    - SIGUSR1 starts tracing
  - If tracing:
    - SIGUSR1 stops tracing and dumps backtraces

The output file is created by default in the current user home directory. This
can be changed using the -l option. The file is named "<PID>-<n>.rtrace.txt",
where <PID> is the process ID (for multithreaded applications it is the TGID of
the thread group, and events for all threads are reported on the same file) and
<n> is the "step number" used for numbering and not overwriting trace files
created for the same PID. 

The built-in symbol resolving in functracer (option "-r") has severe limitations
compared to the postprocessor symbol resolving (functracer-postproc):
- it doesn't resolve static or inline function names.
- it does not resolve source file name and line number.
- it does not do demangling for C++ symbol names
  (low-level symbol names into human readable names).
- it can sometimes crash the traced process.

To see the list of process invocations, just type the following where the trace
files were saved:
$ grep ^Process *.rtrace.txt

Note that messages in *.rtrace.txt file about processes use "process/thread"
instead of just "process". This is a ptrace limitation, because it does not
distinguish between thread and process IDs.

5. Where does it work

It works with ELF based Linux systems running on 32-bit x86 or ARM architecture.
Other architectures might be added in future if the required architecture specific
code is written for them.

6. Performance impact

Functracer sets breakpoints to selected functions to be able to trace their
use. This can have quite noticeable impact on the traced program performance,
even when the Functracer reporting and backtracing is not enabled.

Unless one wants to trace the whole program life-cycle, one should (re-)attach
to a program only during the use-cases that one is interested about and
terminate functracer with CTRL-C when the use-case ends.

7. Other effects

Program being slowed down may also effect its behavior slightly (timers, thread
synchronization etc.), but if the program is robust (e.g. checks for error
codes returned by system calls, accounts for timeout conditions etc.), it
should still function normally. Additionally, ptrace() changes unix signaling
semantics slightly, but in most cases this should not affect anything either.

8. Documentation

Functracer is documented by means of a manpage, an internals document,
and several READMEs:

- The manpage describes briefly how to use functracer and its options. It is
  located in the man directory and the help2man tool is used to build it.
- The "Functraces Internals", located in the doc directory, describes all the
  inner workings. There are files in pdf and plain text format already. To
  convert the document from plain text to html and pdf formats, the asciidoc
  tool is required.
- src/modules/README.plugins, describes current plugins and how to write
  new plugins.
- src/modules/TODO.plugins, describes missing plugin features and
  contains some hints on how they could be implemented.
- testsuite/README, describes how to run the test-suite and add more
  tests to it.

9. Issues and Limitations

Not all functions can be tracked by plugins. This happens for two reasons:
- Currently the function names are matched against the library export table.
  This means that only functions exported by libraries are visible, and that
  one should use the global name, that may have been "mangled" by toolchain
  headers etc (e.g. "__open" instead of just "open").

- The SSOL approach has some restrictions regarding at which instructions
  a breakpoint can be inserted. In later releases, it was implemented a
  instruction rewriting mechanism that covers all functions tracked by current

  Some instructions do not work well when single-stepped out of place, because
  their behavior is location dependent (e.g. they use the PC register value for
  calculations, or write to it). Therefore, it is necessary to rewrite these
  instructions so that they become "SSOL safe". This consists of replacing the
  location dependent register operand (PC, for instance), with a one having no
  side-effects (e.g. R0). After the instruction is safely single-stepped, the
  result is written back to the correct register before the process continues.

  In functracer, for sake of simplicity, it only rewrites those instructions
  that occur in breakpoints inserted by current functracer plugins. Functracer
  only inserts breakpoints on the beginning of functions, therefore there are
  only a few instructions that appear there.  Ideally, for the -a option, all
  (or most) instructions should be supported. For this one has to write code
  to recognize and rewrite other instructions.

  The concept and the branch displacement macro were based on kprobes code from
  the Linux kernel. The kernel code, on the other hand, has support for
  rewriting almost all instructions on ARM.

  Also note that for x86, no instruction rewriting rules were implemented (yet),
  because so far none of the current plugins inserted breakpoints at "SSOL
  unsafe" instructions.

10. To Do

- Add a new option "trace-children" for optional tracking to children
  processes. Functracer slows down the process(es) quite a bit so it may be
  nice to have that as optional similarly to ltrace/strace/valgrind.

  To implement this, it is need to verify if the event is creating a new
  process or a thread by reading the file /proc/pid/status. There is a existent
  code on process.c that read the process Tgid: get_tgid(). If tgid is 
  different of current pid, so we have a thread otherwise is a new process.
  See details in add_process() in process.c file.

- Add support for reading complex data of functions parameters like strings,
  arrays, etc. (for more details, see TODO.plugins).

- Add support for tracing syscalls instead of library function calls
  to avoid issues raising from "mangled" library function names and
  accidentally missing some of them.  This would be mainly useful for
  file descriptor and shared memory tracking.

- Add support for resolving STT_GNU_IFUNC / IRESOLVE functions
  (e.g. strlen in Glibc) to be able to track all Glibc functions. See:

- Handle items listed by: fgrep -e FIXME -e TODO $(find -type f)

11. License

This program is free software; you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Freea Software
Foundation; either version 2 of the License, or (at your option) any later