Symbolizing backtraces

in the event that mod_pagespeed crashes, and you are inspecting the backtrace of a release build, the .so file is stripped, so not a lot of information is available.

example backtrace

(gdb) bt
#0  0x00007f54cfd95625 in raise () from /lib64/libc.so.6
#1  0x00007f54cfd96e05 in abort () from /lib64/libc.so.6
#2  0x00007f54cad1fa5d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6
#3  0x00007f54cad1dbe6 in ?? () from /usr/lib64/libstdc++.so.6
#4  0x00007f54cad1dc13 in std::terminate() () from /usr/lib64/libstdc++.so.6
#5  0x00007f54cad1dd0e in __cxa_throw () from /usr/lib64/libstdc++.so.6
#6  0x00007f54cacc2837 in std::__throw_logic_error(char const*) () from /usr/lib64/libstdc++.so.6
#7  0x00007f54cacfde59 in ?? () from /usr/lib64/libstdc++.so.6
#8  0x00007f54cacfdf33 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&) () from /usr/lib64/libstdc++.so.6
#9  0x00007f54c9fce073 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#10 0x00007f54c9effb9a in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#11 0x00007f54c9f01e34 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#12 0x00007f54c9f0b3a4 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#13 0x00007f54d1860a58 in ap_send_error_response ()
#14 0x00007f54d1848506 in ap_read_request ()
#15 0x00007f54d18600e0 in ?? ()
#16 0x00007f54d185c148 in ap_run_process_connection ()
#17 0x00007f54d1869052 in ?? ()
#18 0x00007f54d00fe9d1 in start_thread () from /lib64/libpthread.so.0
#19 0x00007f54cfe4b8fd in clone () from /lib64/libc.so.6

notice the relevant lines

#9  0x00007f54c9fce073 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#10 0x00007f54c9effb9a in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#11 0x00007f54c9f01e34 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#12 0x00007f54c9f0b3a4 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so

The first step is to get the version (release number) operating system (deb/rpm) and if the package is 32 or 64 bits from the crash reporter.

In order to symbolize, we need an unstripped copy of the .so (should be uploaded to the "releases" on github, or ask a googler to retrieve it from the build machine if it's not there), and the base address of the library.

To retrieve the base address in gdb, there are two ways.

info proc mappings

will display where the libraries are mapped (showing the base address), but requires /proc to be available.

if /proc is not mounted, then we can use info sharedlibrary. info sharedlibrary has "from" and "to", where the "from" address is the start of the .text section. To compute the offset between the "from" address and the base address, we can use readelf or rabin2.

$ readelf -S  ./libmod_pagespeed.so | grep .text
  [12] .text             PROGBITS         000000000007e980  0007e980

$ rabin2 -S ./libmod_pagespeed.so | grep .text
idx=11 vaddr=0x0007e980 paddr=0x0007e980 sz=4311238 vsz=4311238 perm=--r-x name=.text

so the base address will be the "from" address - the address of the .text section.

in this case, the base address is 7f54c9e80000

[1] pry(main)> 0x00007f54c9ef8840 - 0x00078840
=> 140002141405184
[2] pry(main)> _.to_s 16
=> "7f54c9e80000"

Now that you have the base address, an unstripped binary, and the addresses in the backtrace, it's simple to symbolize.

You can use radare2 or addr2line.

example scripts using both are here

addresses = [0x00007f54c9fce073, 0x00007f54c9effb9a,
             0x00007f54c9f01e34, 0x00007f54c9f0b3a4]

base_address = 0x7f54c9e80000

addresses.each{|address|
  offset = "0x#{(address - base_address).to_s 16}" # note must be hex
  exec = "addr2line -f -e ./libmod_pagespeed.so #{offset} --demangle"
  puts `#{exec}`
}

with output

$ ruby ./bt.rb
net_instaweb::GoogleUrl::GoogleUrl(char const*)
:?
net_instaweb::InstawebContext::MakeRequestUrl(net_instaweb::RewriteOptions const&, request_rec*)
:?
net_instaweb::InstawebHandler::InstawebHandler(request_rec*)
:?
net_instaweb::(anonymous namespace)::instaweb_out_filter(ap_filter_t*, apr_bucket_brigade*)
mod_instaweb.cc:?

require 'r2pipe'

addresses = [0x00007f54c9fce073, 0x00007f54c9effb9a,
             0x00007f54c9f01e34, 0x00007f54c9f0b3a4]

base_address = 0x7f54c9e80000

r2p = R2Pipe.new "./libmod_pagespeed.so"

addresses.each{|address|
  offset = address - base_address"
  puts r2p.cmd "fd #{offset}"
}

with output

$ ruby ./bt.rb
sym.net_instaweb::GoogleUrl::Relativize + 851
sym.net_instaweb::InstawebContext::MakeRequestUrl + 282
sym.net_instaweb::InstawebHandler::InstawebHandler + 436
sym.net_instaweb::_anonymousnamespace_::instaweb_out_filter + 1012

the output from radare2 seems a bit better than from addr2line.