LeakSanitizer gives bogus or incomplete stacktraces #870

delcypher · 2017-10-17T11:24:57Z

It seems that LeakSanitizer will sometimes give bogus or incomplete stacktraces when using fast_unwind_on_malloc=1.

For context see Z3Prover/z3#1297 . This where I first ran into the issue.

Reproduction steps

This is using Clang 5.0 on Arch Linux

git clone https://github.com/Z3Prover/z3.git 
cd z3
git checkout 01f642a6f37e2a25cbfe45f0c48aa9341989028d
# Remove fixes to memory leaks so we can see them again
git revert  c093e6d4b9c1a9a3d97099fab492e89227fc3715 09ea370ea33f64e54f9cd837cd4e9f66ebe933df
mkdir build
cd build
# Set compiler environment variables. It is important they are exported because
# the CMake build invokes CMake again and there we want to use the same compiler
# and flags.
export CC=clang
export CXX=clang++
export CFLAGS="-fsanitize=address -fno-omit-frame-pointer"
export CXXFLAGS="${CFLAGS}"
cmake -DCMAKE_BUILD_TYPE=Debug -G Ninja ../
ninja
# Now build API example
ninja c_example

Run with `fast_unwind_on_malloc`

LSAN_OPTIONS=fast_unwind_on_malloc=1 examples/c_example_build_dir/c_example 
...
=================================================================
==6550==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 513 byte(s) in 1 object(s) allocated from:
    #0 0x557f2bc88181 in __interceptor_malloc (/home/dan/dev/z3/lsan_test/examples/c_example_build_dir/c_example+0xe6181)
    #1 0x7f6a576430a8 in operator new(unsigned long) /build/gcc-multilib/src/gcc/libstdc++-v3/libsupc++/new_op.cc:50
    #2 0x61600023d137  (<unknown module>)

SUMMARY: AddressSanitizer: 513 byte(s) leaked in 1 allocation(s).

This stacktrace is unhelpful due to it being incomplete, but not misleading. However in the original bug report I also observed this. Note in that bug report I was using Clang 3.9. I might have had UBSan enabled too (-fsanitize=undefined). I don't remember but I'm compiling now to check.

UPDATE: I tried building with UBSan as well as ASan with Clang 5.0 too and this doesn't reproduce the bogus stack trace.

=================================================================
==141==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 513 byte(s) in 1 object(s) allocated from:
    #0 0x5086a0 in operator new(unsigned long) (/home/user/z3_build/examples/c_example_build_dir/c_example+0x5086a0)
    #1 0x7fffdedff87c in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::reserve(unsigned long) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0x11f87c)
    #2 0x7fffefdce0ce in _fini (/home/user/z3_build/libz3.so.4.5+0x10d6c0ce)

This stacktrace is really misleading (i.e. bogus). I realised fairly quickly when I used gdb to step into the _fini implementation for libz3.so that the assembly for that function definitely wasn't allocating a std::string.

Run without `fast_unwind_on_malloc`

LSAN_OPTIONS=fast_unwind_on_malloc=0 examples/c_example_build_dir/c_example
...
=================================================================
==6565==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 513 byte(s) in 1 object(s) allocated from:
    #0 0x55d6f17ed181 in __interceptor_malloc (/home/dan/dev/z3/lsan_test/examples/c_example_build_dir/c_example+0xe6181)
    #1 0x7f90da6210a8 in operator new(unsigned long) /build/gcc-multilib/src/gcc/libstdc++-v3/libsupc++/new_op.cc:50
    #2 0x7f90da6b934d in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::reserve(unsigned long) /build/gcc-multilib/src/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:293
    #3 0x7f90da6acb07 in std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::overflow(int) /build/gcc-multilib/src/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/sstream.tcc:114
    #4 0x7f90da6b76f8 in std::basic_streambuf<char, std::char_traits<char> >::xsputn(char const*, long) /build/gcc-multilib/src/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/streambuf.tcc:98
    #5 0x7f90da6a76fb in std::basic_streambuf<char, std::char_traits<char> >::sputn(char const*, long) /build/gcc-multilib/src/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/streambuf:451
    #6 0x7f90da6a76fb in void std::__ostream_write<char, std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*, long) /build/gcc-multilib/src/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/ostream_insert.h:50
    #7 0x7f90da6a76fb in std::basic_ostream<char, std::char_traits<char> >& std::__ostream_insert<char, std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*, long) /build/gcc-multilib/src/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/ostream_insert.h:101
    #8 0x7f90da6a7a98 in std::basic_ostream<char, std::char_traits<char> >& std::operator<< <std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*) /build/gcc-multilib/src/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/ostream:561
    #9 0x7f90df0ed87b in smtparser::error_prefix(proto_expr*) /home/dan/dev/z3/src/src/parsers/smt/smtparser.cpp:805:23
    #10 0x7f90df0f074f in void smtparser::set_error<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, char const*>(char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, char const*, proto_expr*) /home/dan/dev/z3/src/src/parsers/smt/smtparser.cpp:830:9
    #11 0x7f90df0eee68 in smtparser::make_sort(proto_expr*, obj_ref<sort, ast_manager>&) /home/dan/dev/z3/src/src/parsers/smt/smtparser.cpp:1700:17
    #12 0x7f90df0f8817 in smtparser::declare_fun(proto_expr*) /home/dan/dev/z3/src/src/parsers/smt/smtparser.cpp:2062:14
    #13 0x7f90df0db79e in smtparser::declare_funs(proto_expr*) /home/dan/dev/z3/src/src/parsers/smt/smtparser.cpp:1898:18
    #14 0x7f90df0d3ef8 in smtparser::make_benchmark(symbol&, proto_expr* const*) /home/dan/dev/z3/src/src/parsers/smt/smtparser.cpp:975:22
    #15 0x7f90df0ceccd in smtparser::parse(proto_expr_parser&) /home/dan/dev/z3/src/src/parsers/smt/smtparser.cpp:786:26
    #16 0x7f90df0ce549 in smtparser::parse_stream(std::istream&) /home/dan/dev/z3/src/src/parsers/smt/smtparser.cpp:649:16
    #17 0x7f90df0cb08d in smtparser::parse_string(char const*) /home/dan/dev/z3/src/src/parsers/smt/smtparser.cpp:669:16
    #18 0x7f90df25ec40 in Z3_parse_smtlib_string /home/dan/dev/z3/src/src/api/api_parsers.cpp:66:44
    #19 0x55d6f182bd43 in parser_example5 /home/dan/dev/z3/src/examples/c/test_capi.c:1796:9
    #20 0x55d6f1836035 in main /home/dan/dev/z3/src/examples/c/test_capi.c:2860:5
    #21 0x7f90da93bf69 in __libc_start_main (/usr/lib/libc.so.6+0x20f69)
    #22 0x55d6f172d409 in _start (/home/dan/dev/z3/lsan_test/examples/c_example_build_dir/c_example+0x26409)

SUMMARY: AddressSanitizer: 513 byte(s) leaked in 1 allocation(s).

The text was updated successfully, but these errors were encountered:

yugr · 2017-10-17T12:21:04Z

Note that this is a known limitation and already covered in 4-th question of Asan FAQ. As a workaround you can build your software with -fno-omit-frame-pointer.

delcypher · 2017-10-17T12:33:24Z

@yugr . I think you read my issue too quickly ;)

I've already seen that FAQ.
I am building with -fno-omit-frame-pointer.
My main issue is the bogus stacktrace (see _fini)

yugr · 2017-10-17T13:48:18Z

@delcypher Ok, sorry for being superficial here. My guess is that problem is caused by this frame

 #1 0x7f90da6210a8 in operator new(unsigned long) /build/gcc-multilib/src/gcc/libstdc++-v3/libsupc++/new_op.cc:50

It comes from precompiled libstdc++ and thus does not preserve frame pointer which causes fast unwinder to stop. The fact that it breaks differently on different versions of Clang probly makes sense - unwinder just reads random garbage from rsp[0] which may have arbitrary value depending on surrounding code.

eugenis · 2017-10-20T21:04:44Z

That's a common issue in sanitizer deployment, and the usual fix is -shared-libstdc++. System libstdc++/libc++ is normally built w/o frame pointers, and fast_unwind_on_malloc=0 is way too slow.

…

On Tue, Oct 17, 2017 at 6:48 AM, Yury Gribov ***@***.***> wrote: @delcypher <https://github.com/delcypher> Ok, sorry for being superficial here. My guess is that problem is caused by this frame #1 0x7f90da6210a8 in operator new(unsigned long) /build/gcc-multilib/src/gcc/libstdc++-v3/libsupc++/new_op.cc:50 It comes from precompiled libstdc++ and thus does not preserve frame pointer which causes fast unwinder to stop. The fact that it breaks it differently on different versions of Clang probly makes sense - unwinder just reads random garbage from rsp[0] which may have arbitrary value depending on surrounding code. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#870 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAZuSsE5FQCsWldSWbTkthdVEAsVoWkHks5stLAlgaJpZM4P7_kF> .

delcypher · 2017-10-20T21:34:53Z

@yugr @eugenis Thanks for explaining. It makes sense now.

Perhaps something about this should be added to the FAQ?

eugenis · 2017-10-20T21:56:13Z

Done, right after the malloc question. https://github.com/google/sanitizers/wiki/AddressSanitizer

…

On Fri, Oct 20, 2017 at 2:34 PM, Dan Liew ***@***.***> wrote: @yugr <https://github.com/yugr> @eugenis <https://github.com/eugenis> Thanks for explaining. It makes sense now. Perhaps something about this should be added to the FAQ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#870 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAZuSmC9KU7mBhgQ4XaDmy5FBxRvekQtks5suRH-gaJpZM4P7_kF> .

delcypher · 2017-10-20T22:00:02Z

Thanks. Although in my case I'm already using a shared libstdc++ anyway. I'll close this issue as the underlying issue makes sense to me and is documented.

gsauthof · 2018-10-13T09:27:01Z

Direct FAQ link for the ASAN_OPTIONS=fast_unwind_on_malloc=0 workaround:

https://github.com/google/sanitizers/wiki/AddressSanitizer#faq

(ran into this on Fedora 27 with gcc with/without -fno-omit-frame-pointer)

…sanitizers (google/sanitizers#870) FossilOrigin-Name: b9ce20f5d7e27c417b041c445dfbd0dd8996fcbeb6a8cebb2579263fe0287f07

delcypher closed this as completed Oct 20, 2017

PiotrSikora mentioned this issue Oct 9, 2019

build: statically link C++ stdlib in ASan builds. envoyproxy/envoy#8555

Closed

achamayou mentioned this issue Oct 9, 2020

LeakSanitiser break on some shutdowns in Daily build microsoft/CCF#1701

Open

silasdb added a commit to silasdb/janechat that referenced this issue Oct 20, 2021

By using -fno-omit-frame-pointer we generate better stack traces for …

6874b1c

…sanitizers (google/sanitizers#870) FossilOrigin-Name: b9ce20f5d7e27c417b041c445dfbd0dd8996fcbeb6a8cebb2579263fe0287f07

jiridanek mentioned this issue Jun 10, 2022

Function qd_server_config_free does not free the pn_data_t reference conn_props skupperproject/skupper-router#556

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LeakSanitizer gives bogus or incomplete stacktraces #870

LeakSanitizer gives bogus or incomplete stacktraces #870

delcypher commented Oct 17, 2017 •

edited

yugr commented Oct 17, 2017 •

edited

delcypher commented Oct 17, 2017

yugr commented Oct 17, 2017 •

edited

eugenis commented Oct 20, 2017 via email

delcypher commented Oct 20, 2017

eugenis commented Oct 20, 2017 via email

delcypher commented Oct 20, 2017

gsauthof commented Oct 13, 2018

LeakSanitizer gives bogus or incomplete stacktraces #870

LeakSanitizer gives bogus or incomplete stacktraces #870

Comments

delcypher commented Oct 17, 2017 • edited

Reproduction steps

Run with fast_unwind_on_malloc

Run without fast_unwind_on_malloc

yugr commented Oct 17, 2017 • edited

delcypher commented Oct 17, 2017

yugr commented Oct 17, 2017 • edited

eugenis commented Oct 20, 2017 via email

delcypher commented Oct 20, 2017

eugenis commented Oct 20, 2017 via email

delcypher commented Oct 20, 2017

gsauthof commented Oct 13, 2018

delcypher commented Oct 17, 2017 •

edited

Run with `fast_unwind_on_malloc`

Run without `fast_unwind_on_malloc`

yugr commented Oct 17, 2017 •

edited

yugr commented Oct 17, 2017 •

edited