add CFI directives for reliable stack unwinding

[vicuna]

Original bug ID: 5314
Reporter: @ygrek
Assigned to: @mshinwell
Status: closed (set by @xavierleroy on 2013-08-31T10:48:38Z)
Resolution: fixed
Priority: normal
Severity: feature
Version: 3.12.0
Fixed in version: 3.13.0+dev
Category: ~DO NOT USE (was: OCaml general)
Related to: #5334
Monitored by: tgazagna meurer "Pascal Cuoq" @glondu thelema @hcarty gerd @yakobowski

Bug description

Currently in many cases stack of ocaml programs cannot be unwinded because there is no usual frame pointer and generic debuggers and profilers (e.g. gdb,oprofile) are often confused. But using DWARF2 CFI directives it is possible to add static information for frame address calculation. These directives are understood by GNU assembler and stored in .debug_frame or .eh_frame sections, which can be later used by debugging tools. Patch attached for x86 and amd64 code emitters. Here are some examples :

$ cat test.ml

let really_crash () =
print_endline (Obj.magic 0 : string);
print_endline "oops"

let rec crash_here1 = function
| 0 -> really_crash (); 0
| n -> crash_here2 (n - 1) + 1
and crash_here2 = function
| 0 -> (try really_crash (); 0 with _ -> 0)
| n -> crash_here1 (n - 1) + 1

let () =
let n = crash_here1 5 in
exit n

On x86 - original ocaml 3.12.1 :

$ /opt/ocaml-3.12.1/bin/ocamlopt.opt -g -inline 0 -S test.ml -o test
$ gdb -batch -ex 'set pagination 0' -ex 'set interactive-mode off' -ex 'r' -ex 'bt' -ex 'q' --args ./test

Program received signal SIGSEGV, Segmentation fault.
0x0804a370 in camlPervasives__output_string_1191 ()
#0 0x0804a370 in camlPervasives__output_string_1191 ()
#1 0x0804a76c in camlPervasives__print_endline_1274 ()
#2 0x08049a3a in camlTest__really_crash_1030 ()
#3 0x08049a96 in camlTest__crash_here2_1032 ()
#4 0xbffff1c8 in ?? ()
#5 0x0804aebc in main ()

After the patch :

$ /opt/ocaml-3.12.1-cfi/bin/ocamlopt.opt -g -inline 0 -S test.ml -o test
$ gdb -batch -ex 'set pagination 0' -ex 'set interactive-mode off' -ex 'r' -ex 'bt' -ex 'q' --args ./test

Program received signal SIGSEGV, Segmentation fault.
0x0804a370 in camlPervasives__output_string_1191 ()
#0 0x0804a370 in camlPervasives__output_string_1191 ()
#1 0x0804a76c in camlPervasives__print_endline_1274 ()
#2 0x08049a3a in camlTest__really_crash_1030 ()
#3 0x08049a96 in camlTest__crash_here2_1032 ()
#4 0x08049abd in camlTest__crash_here1_1031 ()
#5 0x08049a5d in camlTest__crash_here2_1032 ()
#6 0x08049abd in camlTest__crash_here1_1031 ()
#7 0x08049a5d in camlTest__crash_here2_1032 ()
#8 0x08049abd in camlTest__crash_here1_1031 ()
#9 0x08049b06 in camlTest__entry ()
#10 0x08049791 in caml_program ()
#11 0x08057a72 in caml_start_program ()
#12 0x00000000 in ?? ()

On amd64 situation is better (stack seems to be always fully unwinded), but frames are still not detected correctly and backtrace contains some garbage :

$ cat test.ml
let func2 x =
if x * x mod 111 = 0 then
raise Not_found

let func1 x y =
for i = x to y do
try
func2 i
with exn -> print_endline "exn"
done

let () =
func1 100 200

$ /opt/ocaml-3.12.1/bin/ocamlopt.opt -g -inline 0 -S test.ml -o test
$ gdb -batch -ex 'set interactive-mode off' -ex 'b camlTest__func2_1030' -ex 'r' -ex 'bt' -ex 'kill' -ex 'q' --args ./test
Breakpoint 1 at 0x403550

Breakpoint 1, 0x0000000000403550 in camlTest__func2_1030 ()
#0 0x0000000000403550 in camlTest__func2_1030 ()
#1 0x00000000004035f2 in camlTest__func1_1032 ()
#2 0x00007fffffffe570 in ?? ()
#3 0x00000000004035d7 in camlTest__func1_1032 ()
#4 0x00000000000000c9 in ?? ()
#5 0x0000000000000191 in ?? ()
#6 0x0000000000411b75 in caml_start_program ()
#7 0x000000000040365a in camlTest__entry ()
#8 0x00000000000003e8 in ?? ()
#9 0x0000000000403219 in caml_program ()
#10 0x0000000000029011 in ?? ()
#11 0x0000000000411b3e in caml_start_program ()
#12 0x0000000000000000 in ?? ()

After the patch :
$ /opt/ocaml-3.12.1-cfi/bin/ocamlopt.opt -g -inline 0 -S test.ml -o test
$ gdb -batch -ex 'set interactive-mode off' -ex 'b camlTest__func2_1030' -ex 'r' -ex 'bt' -ex 'kill' -ex 'q' --args ./test
Breakpoint 1 at 0x403550

Breakpoint 1, 0x0000000000403550 in camlTest__func2_1030 ()
#0 0x0000000000403550 in camlTest__func2_1030 ()
#1 0x00000000004035f2 in camlTest__func1_1032 ()
#2 0x000000000040365a in camlTest__entry ()
#3 0x0000000000403219 in caml_program ()
#4 0x0000000000411b3e in caml_start_program ()
#5 0x0000000000000000 in ?? ()

Additional information

BTW, looking at asmcomp/i386/emit.mlp I am a bit puzzled : line 650, in branch Lop(Iintoffloat) : stack is growing, but stack_offset is decremented - is this correct? (I couldn't trigger storing the result to stack so maybe it doesn't matter in practice).

File attachments

• ocaml-3.11-cfi.patch
• ocaml-3.11.2-cfi.patch
• ocaml-3.12-cfi.patch
• ocaml-3.12.1-cfi.patch
• ocaml-3.12.1-cfi2.patch

[vicuna]

Comment author: @ygrek

Patch updated :

• configure test whether assembler accepts CFI directives
• annotate important runtime functions (now it is possible to see where caml_call_gc comes from)
• correct CFA for exception handlers

[vicuna]

Comment author: @ygrek

Same patch for 3.11.2 (for testing - easy to apply on top of debian package)

[vicuna]

Comment author: @ygrek

Patches updated:

• caml_c_call annotation was wrong, reverted

[vicuna]

Comment author: @ygrek

Documentation for CFI directives : http://sourceware.org/binutils/docs/as/CFI-directives.html
Useful example at http://www.logix.cz/michal/devel/gas-cfi/
The patch is being used for quite some time already and seems to work pretty well. Note that implementation depends on the way ocamlopt emits code, so it shortcuts some calculations (e.g. for exceptions).
Probably an important missing piece is ARM support but I do not know ARM assembly at all.

[vicuna]

Comment author: tgazagna

I've tested the patch as well and it works pretty well. An other missing piece is OSX support, but as on this architecture, the shipped version of as is very old (1.*), it doesn't support CFI directives. I guess generating directly .eh_sections and .debug_section is too much a pain ...

[vicuna]

Comment author: thelema

I've been using this final patch on a couple systems and it greatly increases the information available to me from stack traces. It makes the technique of poor man's profiling (http://poormansprofiler.org/) usable, instead of looking at just a few functions on the stack.

[vicuna]

Comment author: @ygrek

see #5487