Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

-fsanitize=fuzzer is not handling exceptions correctly #2328

Closed
jonathanmetzman opened this issue Apr 16, 2019 · 31 comments

Comments

Projects
None yet
5 participants
@jonathanmetzman
Copy link
Contributor

commented Apr 16, 2019

The only case I've verified where this happens is jsonnet.
However, it looks like the same issue occurs with freeimage (load_from_memory_fuzzer), libsass (data_context_fuzzer), and opencv (imdecode_fuzzer)

To reproduce, download my patch and run these commands:

git apply jsonnet.txt
python infra/helper.py build_fuzzers jsonnet
python infra/helper.py check_build jsonnet
...
AddressSanitizer:DEADLYSIGNAL
=================================================================
==36==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7f236d5b6d10 bp 0x00000090605b sp 0x7ffd7f950d68 T0)
==36==The signal is caused by a READ memory access.
==36==Hint: address points to the zero page.
SCARINESS: 10 (null-deref)
ERROR: 100% of fuzz targets seem to be broken. See the list above for a detailed information.
Check build failed

This is a stacktrace that I captured slightly before the crash occurs.

#0  __asan_handle_no_return () at /src/llvm/projects/compiler-rt/lib/asan/asan_rtl.cc:598
#1  0x00000000005f7084 in (anonymous namespace)::Parser::parseTerminalBracketsOrUnary (this=0x6110000000c0) at /src/jsonnet/core/parser.cpp:591
#2  0x00000000005e5321 in (anonymous namespace)::Parser::parse (this=0x7fffffffcec0, max_precedence=<optimized out>) at /src/jsonnet/core/parser.cpp:898
#3  0x00000000005e49f8 in jsonnet_parse (alloc=<optimized out>, tokens=...) at /src/jsonnet/core/parser.cpp:1093
#4  0x00000000005caed7 in jsonnet_evaluate_snippet_aux (vm=<optimized out>, filename=<optimized out>, snippet=<optimized out>, error=<optimized out>, kind=<optimized out>) at /src/jsonnet/core/libjsonnet.cpp:492
#5  0x00000000005ca76d in jsonnet_evaluate_snippet (vm=0x60f000000040, filename=0x89c900 <.str> "", snippet=0x7fffffffdaa1 "", error=0x7fffffffd9c0) at /src/jsonnet/core/libjsonnet.cpp:667
#6  0x00000000005bbcce in ConvertJsonnetToJson(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) () at convert_jsonnet_fuzzer.cc:24
#7  0x00000000005bc011 in LLVMFuzzerTestOneInput () at convert_jsonnet_fuzzer.cc:40
#8  0x00000000004c2f35 in ExecuteCallback () at /src/llvm/projects/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:529
#9  0x00000000004c4e54 in ReadAndExecuteSeedCorpora () at /src/llvm/projects/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:729
#10 0x00000000004c5426 in Loop () at /src/llvm/projects/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:779
#11 0x00000000004b437b in FuzzerDriver () at /src/llvm/projects/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:776
#12 0x00000000004de3d3 in main () at /src/llvm/projects/compiler-rt/lib/fuzzer/FuzzerMain.cpp:19

The crash does not occur when -lFuzzingEngine is used instead of -fsanitize=fuzzer

The source line (parser.cpp:591) causing the crash contains:
throw StaticError(tok.location, "unexpected end of file.");

I think this crash has something to do with exceptions.

@jonathanmetzman jonathanmetzman changed the title -fsanitize=fuzzer breaks on some targets. libFuzzer sometimes breaks when -fsanitize=fuzzer is used. Apr 16, 2019

@jonathanmetzman

This comment has been minimized.

Copy link
Contributor Author

commented Apr 16, 2019

@jonathanmetzman jonathanmetzman changed the title libFuzzer sometimes breaks when -fsanitize=fuzzer is used. libFuzzer/ASAN sometimes breaks when -fsanitize=fuzzer is used. Apr 16, 2019

@jonathanmetzman jonathanmetzman changed the title libFuzzer/ASAN sometimes breaks when -fsanitize=fuzzer is used. libFuzzer sometimes breaks when -fsanitize=fuzzer is used. Apr 16, 2019

@jonathanmetzman

This comment has been minimized.

Copy link
Contributor Author

commented Apr 16, 2019

This happens with and without ASAN

@jonathanmetzman

This comment has been minimized.

Copy link
Contributor Author

commented Apr 16, 2019

Actually I think the real top two frames are:

#0  0x00007ffff729f1f0 in _Unwind_RaiseException () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#1  0x00000000006ade17 in __cxa_throw ()

and not __asan_handle_no_return

All of the projects I mentioned above use exceptions btw
https://github.com/opencv/opencv/search?q=throw&unscoped_q=throw
https://github.com/sass/libsass/search?q=throw&unscoped_q=throw
https://github.com/imazen/freeimage/search?q=throw&unscoped_q=throw

@jonathanmetzman

This comment has been minimized.

Copy link
Contributor Author

commented Apr 17, 2019

I have a small reproducer on host (without oss-fuzz). But I'm not sure it reveals the exact problem.

Steps:
Download my fuzz target source:
fuzzer.txt

Then run these commands:

CC=$HOME/path/to/fresh/clang CXX=$HOME/path/to/fresh/clang++ cmake -DLIBCXX_ENABLE_SHARED=OFF -DLIBCXX_ENABLE_STATIC_ABI_LIBRARY=ON -DLIBCXXABI_ENABLE_SHARED=OFF -DLLVM_TARGETS_TO_BUILD="X86" -G "Ninja" -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="compiler-rt;clang;libcxx;libcxxabi" ../llvm
ninja
mv fuzzer.txt fuzzer.cc
./bin/clang++ -fsanitize=fuzzer fuzzer.cc -stdlib=libc++ -o fuzzer
./fuzzer
terminating with uncaught exception of type char const*
==164295== ERROR: libFuzzer: deadly signal
UndefinedBehaviorSanitizer:DEADLYSIGNAL
==164295==ERROR: UndefinedBehaviorSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7fb2d3d294d8 bp 0x7ffff6fa6890 sp 0x7ffff6fa6720 T164295)
==164295==The signal is caused by a READ memory access.
==164295==Hint: address points to the zero page.
Segmentation fault

It seems like libFuzzer shouldn't be crashing when bubbling up an exception which is what appears to be happening in this case.

I'm not sure this reproducer is accurate though since the fuzz target will run when compiled with -lFuzzingEngine, it doesn't quit because of an uncaught exception, but in this case catching const char* causes the problem to go away.

I'll keep poking around and will try to come up with something better.

@kcc

This comment has been minimized.

Copy link
Contributor

commented Apr 18, 2019

guess: we may need to remove -fno-exceptions from the build flags.
(though I don't understand why)

@jonathanmetzman

This comment has been minimized.

Copy link
Contributor Author

commented Apr 18, 2019

From libFuzzer's build?

@kcc

This comment has been minimized.

Copy link
Contributor

commented Apr 18, 2019

scratch that. The example from #2328 (comment) throws exception of one type and catches an exception of another type, i.e. the exception goes undetected into libFuzzer, which causes it to abort with an error message. I think it's a valid behavior. No?

@jonathanmetzman

This comment has been minimized.

Copy link
Contributor Author

commented Apr 18, 2019

I think you're correct, my reproducer is no good. But i think there's still a legitimate issue here

@jonathanmetzman

This comment has been minimized.

Copy link
Contributor Author

commented Apr 18, 2019

Actually, the stacktrace in my example isn't exactly what I expect:

==164295== ERROR: libFuzzer: deadly signal
UndefinedBehaviorSanitizer:DEADLYSIGNAL
==164295==ERROR: UndefinedBehaviorSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7fb2d3d294d8 bp 0x7ffff6fa6890 sp 0x7ffff6fa6720 T164295)
==164295==The signal is caused by a READ memory access.
==164295==Hint: address points to the zero page.
Segmentation fault

It almost looks like two error handlers are being invoked?

If one compiles the same testcase without -stdlib=libc++ the stacktrace is actually what one expects:

./fuzzer
INFO: Seed: 507092804
INFO: Loaded 1 modules   (8 inline 8-bit counters): 8 [0x708080, 0x708088), 
INFO: Loaded 1 PC tables (8 PCs): 8 [0x4de148,0x4de1c8), 
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
terminate called after throwing an instance of 'char const*'
==252857== ERROR: libFuzzer: deadly signal
    #0 0x4c50d0  (/usr/local/google/home/metzman/llvm-project/build/fuzzer+0x4c50d0)
    #1 0x427a38  (/usr/local/google/home/metzman/llvm-project/build/fuzzer+0x427a38)
    #2 0x414723  (/usr/local/google/home/metzman/llvm-project/build/fuzzer+0x414723)
    #3 0x7fad0ae230bf  (/lib/x86_64-linux-gnu/libpthread.so.0+0x110bf)
    #4 0x7fad0a481fce  (/lib/x86_64-linux-gnu/libc.so.6+0x32fce)
    #5 0x7fad0a4833f9  (/lib/x86_64-linux-gnu/libc.so.6+0x343f9)
    #6 0x7fad0b3bf826  (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0x8c826)
    #7 0x7fad0b3c5a39  (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0x92a39)
    #8 0x7fad0b3c5a94  (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0x92a94)
    #9 0x7fad0b3c5ce7  (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0x92ce7)
    #10 0x4c64f7  (/usr/local/google/home/metzman/llvm-project/build/fuzzer+0x4c64f7)
    #11 0x415b50  (/usr/local/google/home/metzman/llvm-project/build/fuzzer+0x415b50)
    #12 0x41735d  (/usr/local/google/home/metzman/llvm-project/build/fuzzer+0x41735d)
    #13 0x4179c5  (/usr/local/google/home/metzman/llvm-project/build/fuzzer+0x4179c5)
    #14 0x40dd8a  (/usr/local/google/home/metzman/llvm-project/build/fuzzer+0x40dd8a)
    #15 0x428182  (/usr/local/google/home/metzman/llvm-project/build/fuzzer+0x428182)
    #16 0x7fad0a46f2b0  (/lib/x86_64-linux-gnu/libc.so.6+0x202b0)
    #17 0x407959  (/usr/local/google/home/metzman/llvm-project/build/fuzzer+0x407959)

NOTE: libFuzzer has rudimentary signal handlers.
      Combine libFuzzer with AddressSanitizer or similar for better crash reports.
SUMMARY: libFuzzer: deadly signal
MS: 0 ; base unit: 0000000000000000000000000000000000000000


artifact_prefix='./'; Test unit written to ./crash-da39a3ee5e6b4b0d3255bfef95601890afd8070

I don't know if these issues are related at all though so maybe this is totally unrelated.

@kcc

This comment has been minimized.

Copy link
Contributor

commented Apr 18, 2019

I'd say this is something else. Handling uncaught exceptions is probably not that urgent.

@jonathanmetzman

This comment has been minimized.

Copy link
Contributor Author

commented Apr 18, 2019

ACK

@morehouse

This comment has been minimized.

Copy link
Member

commented Apr 19, 2019

On my machine the opposite happened: with libc++ I get a stack trace from libFuzzer, but with libstdc++ I get the ASan report with no stack trace.

With libstdc++, a segfault triggers during unwinding. With libc++, abort() is called. No clue why, maybe there's a bug in my version of libstdc++.

libFuzzer routes SEGVs to the sanitizer to handle but handles SIGABORT itself, hence the different reports.

@morehouse

This comment has been minimized.

Copy link
Member

commented Apr 19, 2019

In any case, I don't think this is a libfuzzer or -fsanitize=fuzzer issue.

@morehouse morehouse closed this Apr 19, 2019

@jonathanmetzman

This comment has been minimized.

Copy link
Contributor Author

commented Apr 19, 2019

I don't understand what's the resolution of this issue.
How can we stop targets from erroring out with -fsanitize=fuzzer when those targets didn't error out with libFuzzingEngine.a

@morehouse

This comment has been minimized.

Copy link
Member

commented Apr 19, 2019

Maybe I misunderstand, but shouldn't they be fixing the uncaught exceptions?

@jonathanmetzman

This comment has been minimized.

Copy link
Contributor Author

commented Apr 19, 2019

I think these exceptions are being caught properly by the code (maybe I misunderstand). I think this was reflected by the behavior of -lFuzzingEngine, but is not the case with -fsanitize=fuzzer
For example in #2336 an exception is thrown here and should be caught here. I think that exception shouldn't bubble up right?

@jonathanmetzman jonathanmetzman changed the title libFuzzer sometimes breaks when -fsanitize=fuzzer is used. -fsanitize=fuzzer is not handling exceptions correctly Apr 19, 2019

@kcc

This comment has been minimized.

Copy link
Contributor

commented Apr 19, 2019

Clearly, if an exception is properly caught in the target we should not crash.
we may need to start from creating a smaller reproducer that reflects the problem.
(I can't promise to get to it for at least a few days...)

@kcc

This comment has been minimized.

Copy link
Contributor

commented Apr 19, 2019

(I mean, to get to creating a minimized repro. Dealing with a minimized repro might be easy)

@jonathanmetzman

This comment has been minimized.

Copy link
Contributor Author

commented Apr 19, 2019

Clearly, if an exception is properly caught in the target we should not crash.

Yes, I think this is the case, libFuzzer is incorrectly crashing on correctly handled exceptions.

(I mean, to get to creating a minimized repro. Dealing with a minimized repro might be easy)

I'll see what I can do about this, but there seems to be another layer to this problem that I miss by throwing an exception and catching it in the same file.

@morehouse

This comment has been minimized.

Copy link
Member

commented Apr 20, 2019

I believe this has something to do with the libfuzzer runtime (built by https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-clang/checkout_build_install_llvm.sh) using libgcc_s.so rather than compiler-rt.

Unwinding with -lFuzzingEngine:

#0  __interceptor__Unwind_RaiseException ()
    at /src/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:345
#1  0x00007ffff7911957 in __cxa_throw () from /usr/local/lib/libc++abi.so.1
#2  0x00000000005df2ee in (anonymous namespace)::Parser::parseTerminalBracketsOrUnary (this=0x6110000000c0) at /src/jsonnet/core/parser.cpp:591

Unwinding with -fsanitize=fuzzer:

#0  0x00007ffff6dacd10 in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#1  0x00007ffff6dadd32 in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#2  0x00007ffff6daf283 in _Unwind_RaiseException ()
   from /lib/x86_64-linux-gnu/libgcc_s.so.1
#3  0x00007ffff7911957 in __cxa_throw () from /usr/local/lib/libc++abi.so.1
#4  0x00000000005f4f0e in (anonymous namespace)::Parser::parseTerminalBracketsOrUnary (this=0x6110000000c0) at /src/jsonnet/core/parser.cpp:591

Notice that the exception goes through the ASan runtime for the first trace but throught libgcc for the second trace.

I will look into this more on Monday.

@jonathanmetzman

This comment has been minimized.

Copy link
Contributor Author

commented Apr 22, 2019

Thanks Matt!

@morehouse

This comment has been minimized.

Copy link
Member

commented Apr 23, 2019

Update: this is related to the private libc++ libFuzzer uses. If I disable it, -fsanitize=fuzzer works.

Not sure why yet, still looking...

@morehouse

This comment has been minimized.

Copy link
Member

commented Apr 24, 2019

The problem is that exception-related code in libc++ lives in the global namespace, not the versioned namespace. So libFuzzer's trick of creating a private "version" of libc++ actually exposes some of the exception interface (e.g., __cxa_throw) in the final libclang_rt.fuzzer_no_main.a library.

So if a fuzz target uses exceptions, it will pick up and use the exception interface from libclang_rt.fuzzer_no_main.a which may handle exceptions differently than the C++ runtime library the binary was compiled for. Hence the segfaults during unwinding.

It looks like we can prevent the exception interface from showing up in libclang_rt.fuzzer_no_main.a if we disable exceptions when building libFuzzer's private libc++ and libc++abi. I've sent a patch for review here: https://reviews.llvm.org/D61053.

@jonathanmetzman

This comment has been minimized.

Copy link
Contributor Author

commented Apr 24, 2019

Thanks Matt!

Btw, what do you mean fuzzer_no_main.a? I think we only link against the regular RT. What is fuzzer_no_main even for?

@morehouse

This comment has been minimized.

Copy link
Member

commented Apr 24, 2019

Actually I'm not sure. Presumably its the same library but without main, though AFAICT we don't use it. I'll look into whether we can just remove it from the builds.

But the exception thing applies to both fuzzer_no_main.a and fuzzer.a.

@kcc

This comment has been minimized.

Copy link
Contributor

commented Apr 24, 2019

IIRC, we need fuzzer_no_main to be able to unittest libFuzzer by linking the gtest's main.

llvm-git-migration pushed a commit to llvm/llvm-project that referenced this issue Apr 25, 2019

[compiler-rt] Build custom libc++abi without exceptions.
Summary:
Since neither compiler-rt nor the libc++ we build use exceptions, we
don't need libc++abi to have them either.

This resolves an issue where libFuzzer's private libc++ contains
implementations for __cxa_throw and friends, causing fuzz targets built
with their own C++ library to segfault during exception unwinding.

See google/oss-fuzz#2328.

Reviewers: phosek, EricWF, kcc

Reviewed By: phosek

Subscribers: kcc, dberris, mgorny, christof, llvm-commits, metzman

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D61053

llvm-svn: 359218

dtzWill pushed a commit to llvm-mirror/compiler-rt that referenced this issue Apr 25, 2019

[compiler-rt] Build custom libc++abi without exceptions.
Summary:
Since neither compiler-rt nor the libc++ we build use exceptions, we
don't need libc++abi to have them either.

This resolves an issue where libFuzzer's private libc++ contains
implementations for __cxa_throw and friends, causing fuzz targets built
with their own C++ library to segfault during exception unwinding.

See google/oss-fuzz#2328.

Reviewers: phosek, EricWF, kcc

Reviewed By: phosek

Subscribers: kcc, dberris, mgorny, christof, llvm-commits, metzman

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D61053

git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@359218 91177308-0d34-0410-b5e6-96231b3b80d8

chapuni pushed a commit to llvm-project/llvm-project-20170507 that referenced this issue Apr 25, 2019

[compiler-rt] Build custom libc++abi without exceptions.
Summary:
Since neither compiler-rt nor the libc++ we build use exceptions, we
don't need libc++abi to have them either.

This resolves an issue where libFuzzer's private libc++ contains
implementations for __cxa_throw and friends, causing fuzz targets built
with their own C++ library to segfault during exception unwinding.

See google/oss-fuzz#2328.

Reviewers: phosek, EricWF, kcc

Reviewed By: phosek

Subscribers: kcc, dberris, mgorny, christof, llvm-commits, metzman

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D61053

chapuni pushed a commit to llvm-project/llvm-project-submodule that referenced this issue Apr 25, 2019

[compiler-rt] Build custom libc++abi without exceptions.
Summary:
Since neither compiler-rt nor the libc++ we build use exceptions, we
don't need libc++abi to have them either.

This resolves an issue where libFuzzer's private libc++ contains
implementations for __cxa_throw and friends, causing fuzz targets built
with their own C++ library to segfault during exception unwinding.

See google/oss-fuzz#2328.

Reviewers: phosek, EricWF, kcc

Reviewed By: phosek

Subscribers: kcc, dberris, mgorny, christof, llvm-commits, metzman

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D61053
@jonathanmetzman

This comment has been minimized.

Copy link
Contributor Author

commented Apr 25, 2019

Matt's patch landed. Should we roll clang forward manually or wait until Chromium's clang is uptakes the patch? Thoughts @inferno-chromium ?

@Dor1s

This comment has been minimized.

Copy link
Collaborator

commented Apr 25, 2019

We also need to revert 8b34fd8

@inferno-chromium

This comment has been minimized.

Copy link
Collaborator

commented Apr 26, 2019

@jonathanmetzman - maybe if we can roll and verify with bad example project that regular crashes work. i am always scared of such things to roll to random revision, but i think @Dor1s and @oliverchang has done this quite a lot of times. also, this is blocking quite a few projects no ? and breaking msan significantly ?

@Dor1s

This comment has been minimized.

Copy link
Collaborator

commented Apr 26, 2019

Besides local testing, I'd suggest rolling the change in the morning and kicking off base-images build. After that, all project builds will happen, and you would need to keep an eye on the builder to make sure they look good. If they look bad, you can quickly revert the roll and kick off re-build once again, that way we won't have 2 consecutive build failures and won't spam all of our users.

@jonathanmetzman

This comment has been minimized.

Copy link
Contributor Author

commented Apr 30, 2019

Roll seems to have been successful.
All projects I knew of that broke with LIB_FUZZING_ENGINE because of this problem now work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.