[infra] some changes #1

oliverchang · 2016-08-30T17:02:20Z

Remove requirement of specifying dockerfile in Jenkinsfile. Assume it
is in the oss-fuzz library directory if it isn't specified.
Try to copy any *.dict or *.options files to output directory.

- Remove requirement of specifying dockerfile in Jenkinsfile. Assume it is always in the library directory. - Try to copy any *.dict or *.options files to output directory.

mikea · 2016-08-30T18:14:23Z

infra/libfuzzer-pipeline.groovy

@@ -65,6 +63,7 @@ def call(body) {
          writeFile file: "$wsPwd/${sanitizer}.rev", text: revText
          echo "revisions: $revText"

+          def dockerfile = "$workspace/oss-fuzz/$projectName/Dockerfile"


This won't work for projects that decide to keep their fuzzers and Dockerfiles in the source tree, right?

I have a small example: https://github.com/mikea/ossfuzz-example/blob/master/fuzz/Jenkinsfile

Ah, didn't know that was a supported use case already.

I've changed it so that dockerfile is still supported but optional.

For dicts/options, I can't think of a clean way of specifying where they exist if they're in an upstream repo and not oss-fuzz. In this case, maybe we'll just have to get them to copy to /out manually in build.sh

mikea · 2016-08-30T18:52:25Z

infra/libfuzzer-pipeline.groovy

+"
+          // Copy dict and options files
+          try {
+            sh "cp $workspace/oss-fuzz/$projectName/*.dict $out/"


cp ... || true

is more concise than try/catch.

fuzzers fail with: ================================================================= �[1m�[31m==18057==ERROR: AddressSanitizer: global-buffer-overflow on address 0x000000b8a144 at pc 0x0000007ae0ca bp 0x7fff2b91a4d0 sp 0x7fff2b91a4c8 �[1m�[0m�[1m�[34mWRITE of size 4 at 0x000000b8a144 thread T0�[1m�[0m #0 0x7ae0c9 in fuzzer::TracePC::HandleInit(unsigned int*, unsigned int*) /src/libfuzzer/FuzzerTracePC.cpp:49:8 #1 0x7bcab9 in __sanitizer_cov_trace_pc_guard_init /src/libfuzzer/FuzzerTracePC.cpp:286:15 #2 0x5156bf in sancov.module_ctor (/out/curl_fuzzer+0x5156bf) #3 0x88c1cc in __libc_csu_init (/out/curl_fuzzer+0x88c1cc) #4 0x7f4ab7aed7be in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x207be) #5 0x41fd78 in _start (/out/curl_fuzzer+0x41fd78)

There seem to be leaks in icu tools: LD_LIBRARY_PATH=../lib:../stubdata:../tools/ctestfw:$LD_LIBRARY_PATH ../bin/genrb --usePoolBundle -k -i ./out/build/icudt58l -s /src/icu/source/data/locales -d ./out/build/icudt58l ar.txt ================================================================= �[1m�[31m==7719==ERROR: LeakSanitizer: detected memory leaks �[1m�[0m �[1m�[34mDirect leak of 64 byte(s) in 1 object(s) allocated from: �[1m�[0m #0 0x4d3368 in malloc /src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:64 #1 0x5154fb in ucbuf_open /src/icu/source/tools/toolutil/ucbuf.cpp:469:36 #2 0x50d4b3 in main /src/icu/source/tools/gendict/gendict.cpp:316:19 #3 0x7ff77f72582f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

The corresponding github issue has more detail on the problem. This code is not in any meaningful state to be committed -- it contains a bunch of hacks I had to make to get the binary to build statically -- libunwind is linked in (via folly), and I had to make changes to include liblzma as well (which libunwind depends on) as well as fix some other unrelated cmake issues. Reproduction instructions below. First, build it with `python infra/helper.py build_fuzzers --sanitizer address proxygen` Then, check the build (it will pass): `python infra/helper.py check_build proxygen` Then, run the fuzzer: `python infra/helper.py run_fuzzer proxygen ProxygenHTTP1xFuzzer` It will fail in about ~30 seconds with the below trace. If the fuzzer goes down another path, I've had it reproduce easily by pulling up the shell, running `base64 -d`, pushing that input to a file, and then running the fuzzer directly on that file. ``` terminating with uncaught exception of type folly::ConversionError: Non-digit character found: "OPY" AddressSanitizer:DEADLYSIGNAL ================================================================= ==11==ERROR: AddressSanitizer: ABRT on unknown address 0x00000000000b (pc 0x7fc166eb7428 bp 0x000000000001 sp 0x7ffcff0a6c78 T0) SCARINESS: 10 (signal) #0 0x7fc166eb7427 in gsignal (/lib/x86_64-linux-gnu/libc.so.6+0x35427) google#1 0x7fc166eb9029 in abort (/lib/x86_64-linux-gnu/libc.so.6+0x37029) google#2 0x9ab7fa in abort_message (/out/ProxygenHTTP1xFuzzer+0x9ab7fa) google#3 0x9b009d in demangling_terminate_handler() (/out/ProxygenHTTP1xFuzzer+0x9b009d) google#4 0x9ab282 in std::__terminate(void (*)()) (/out/ProxygenHTTP1xFuzzer+0x9ab282) google#5 0x9adebd in __cxxabiv1::call_terminate(bool, _Unwind_Exception*) (/out/ProxygenHTTP1xFuzzer+0x9adebd) google#6 0x9ade60 in __cxxabiv1::scan_eh_tab(__cxxabiv1::(anonymous namespace)::scan_results&, _Unwind_Action, bool, _Unwind_Exception*, _Unwind_Context*) (/out/ProxygenHTTP1xFuzzer+0x9ade60) google#7 0x9ad5c6 in __gxx_personality_v0 (/out/ProxygenHTTP1xFuzzer+0x9ad5c6) google#8 0x7fc16725c262 in _Unwind_RaiseException (/lib/x86_64-linux-gnu/libgcc_s.so.1+0x10262) google#9 0x9acfd6 in __cxa_throw (/out/ProxygenHTTP1xFuzzer+0x9acfd6) google#10 0x5cedab in void folly::throw_exception<folly::ConversionError>(folly::ConversionError&&) /src/proxygen/proxygen/_build/deps/include/folly/lang/Exception.h:36:3 google#11 0x5eea03 in _ZZN5folly2toItEENSt3__19enable_ifIXntsr3std7is_sameINS_5RangeIPKcEET_EE5valueES7_E4typeES6_ENKUlNS_14ConversionCodeEE_clESA_ /src/proxygen/proxygen/_build/deps/include/folly/Conv.h:1581:26 google#12 0x5eea03 in _ZN5folly15expected_detail30expected_detail_ExpectedHelper14ExpectedHelper12thenOrThrow_IRNS0_15ExpectedStorageINS_5RangeIPKcEENS_14ConversionCodeELNS0_11StorageTypeE1EEENS_6detail18CheckTrailingSpaceEZNS_2toItEENSt3__19enable_ifIXntsr3std7is_sameIS8_T_EE5valueESI_E4typeES8_EUlS9_E_NS_8ExpectedINS_4UnitES9_EEvLb0ELi0EEET2_OSI_OT0_OT1_ /src/proxygen/proxygen/_build/deps/include/folly/Expected.h:610:5 google#13 0x5ed846 in _ZNR5folly8ExpectedINS_5RangeIPKcEENS_14ConversionCodeEE11thenOrThrowINS_6detail18CheckTrailingSpaceEZNS_2toItEENSt3__19enable_ifIXntsr3std7is_sameIS4_T_EE5valueESD_E4typeES4_EUlS5_E_EEDTclclsr3stdE7declvalISD_EEclL_ZNSB_7declvalIRS4_EEDTclsr3std3__1E9__declvalISD_ELi0EEEvEEEEOSD_OT0_ /src/proxygen/proxygen/_build/deps/include/folly/Expected.h:1226:16 google#14 0x5ed846 in _ZN5folly2toItEENSt3__19enable_ifIXntsr3std7is_sameINS_5RangeIPKcEET_EE5valueES7_E4typeES6_ /src/proxygen/proxygen/_build/deps/include/folly/Conv.h:1579:8 google#15 0x5ed445 in proxygen::ParseURL::parseAuthority() /src/proxygen/proxygen/lib/utils/ParseURL.cpp:155:15 google#16 0x5ec5e9 in proxygen::ParseURL::parseNonFully() /src/proxygen/proxygen/lib/utils/ParseURL.cpp:140:8 google#17 0x5ea87a in proxygen::ParseURL::parse() /src/proxygen/proxygen/lib/utils/ParseURL.cpp:96:5 google#18 0x59a99c in proxygen::ParseURL::init(folly::Range<char const*>) /src/proxygen/proxygen/lib/utils/ParseURL.h:34:5 google#19 0x58add1 in proxygen::ParseURL::ParseURL(folly::Range<char const*>) /src/proxygen/proxygen/lib/utils/ParseURL.h:28:5 google#20 0x58add1 in proxygen::ParseURL proxygen::HTTPMessage::setURL<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&) /src/proxygen/proxygen/lib/http/HTTPMessage.h:202:14 google#21 0x5795f0 in proxygen::HTTP1xCodec::onHeadersComplete(unsigned long) /src/proxygen/proxygen/lib/http/codec/HTTP1xCodec.cpp:976:31 google#22 0x58e2a3 in proxygen::HTTP1xCodec::onHeadersCompleteCB(proxygen::http_parser*, char const*, unsigned long) /src/proxygen/proxygen/lib/http/codec/HTTP1xCodec.cpp:1315:19 google#23 0x5f2222 in proxygen::http_parser_execute(proxygen::http_parser*, proxygen::http_parser_settings const*, char const*, unsigned long) /src/proxygen/proxygen/external/http_parser/http_parser_cpp.cpp:1868:17 google#24 0x577783 in proxygen::HTTP1xCodec::onIngress(folly::IOBuf const&) /src/proxygen/proxygen/lib/http/codec/HTTP1xCodec.cpp:200:26 google#25 0x55b728 in unsigned long proxygen::parse<proxygen::HTTP1xCodec>(proxygen::HTTP1xCodec*, unsigned char const*, unsigned int, int, std::__1::function<bool ()>) /src/proxygen/proxygen/lib/http/codec/test/TestUtils.h:57:23 google#26 0x55a6ef in LLVMFuzzerTestOneInput /src/proxygen/proxygen/fuzzers/ProxygenHTTP1xFuzzer.cpp:23:3 google#27 0x460771 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm/projects/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:554:15 google#28 0x45fe95 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool*) /src/llvm/projects/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:470:3 google#29 0x462247 in fuzzer::Fuzzer::MutateAndTestOne() /src/llvm/projects/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:696:19 google#30 0x462fe5 in fuzzer::Fuzzer::Loop(std::Fuzzer::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) /src/llvm/projects/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:832:5 google#31 0x450dd8 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm/projects/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:825:6 google#32 0x47b442 in main /src/llvm/projects/compiler-rt/lib/fuzzer/FuzzerMain.cpp:19:10 google#33 0x7fc166ea282f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f) google#34 0x424468 in _start (/out/ProxygenHTTP1xFuzzer+0x424468) AddressSanitizer can not provide additional info. SUMMARY: AddressSanitizer: ABRT (/lib/x86_64-linux-gnu/libc.so.6+0x35427) in gsignal ==11==ABORTING MS: 3 CrossOver-ChangeBit-EraseBytes-; base unit: 467e3fbf38770044dea566169b84c60cb269c89a 0x43,0x4f,0x50,0x59,0x20,0x43,0x3a,0x4f,0x50,0x59,0x2f,0x4f,0xff,0xa,0x43,0x4f,0x50,0x59,0x20,0x2f,0x4f,0xff,0xa,0x43,0x4f,0x50,0x59,0x20,0x2f,0x6f,0x20,0xff,0x50,0x59,0x0,0x43,0xcf,0x50,0x59,0xff,0x4f,0x50,0x33, COPY C:OPY/O\xff\x0aCOPY /O\xff\x0aCOPY /o \xffPY\x00C\xcfPY\xffOP3 artifact_prefix='./'; Test unit written to ./crash-2cd12847d55ebf08ec2eaee4e814c52e545e7f92 Base64: Q09QWSBDOk9QWS9P/wpDT1BZIC9P/wpDT1BZIC9vIP9QWQBDz1BZ/09QMw== ``` This seems wrong to me since the code is inside a try/catch block: https://github.com/facebook/proxygen/blob/master/proxygen/lib/utils/ParseURL.cpp#L159-L164 I can confirm that the same input does not crash the fuzzer in the version being built on oss-fuzz trunk (and indeed it would have also reported this as a test-case by now given it's been running over a day)

* Upgrade to LLVM-12 and enable undefined sanitizer (#1) * Upgrade to LLVM-12 * Enable undefined sanitizer * Fix missing scope

Due to upstream changes, the Git fuzzers must now link against common-main.o; however, this breaks the build in two ways: 1) Linking with common-main.o causes main() to have multiple definitions, one in common-main.o and one from the fuzzing engine. 2) To avoid google#1, the Git Makefile specifically excludes common-main.o from the fuzzer build rule. To work around these issues, we can override FUZZ_CXXFLAGS (add "-Wl,--allow-multiple-definition" to fix google#1) and LIB_FUZZING_ENGINE (add "common-main.o" to fix google#2). Once we can get a Makefile fix into Git's upstream, we can remove the override for LIB_FUZZING_ENGINE. However, this change causes `check_build` to fail for honggfuzz, and we have not yet been able to diagnose the reason. So for now, we also need to limit our engines to afl and libfuzzer.

Due to upstream changes, the Git fuzzers must now link against common-main.o; however, this breaks the build in two ways: 1) Linking with common-main.o causes main() to have multiple definitions, one in common-main.o and one from the fuzzing engine. 2) To avoid #1, the Git Makefile specifically excludes common-main.o from the fuzzer build rule. To work around these issues, we can override FUZZ_CXXFLAGS (add "-Wl,--allow-multiple-definition" to fix #1) and LIB_FUZZING_ENGINE (add "common-main.o" to fix #2). Once we can get a Makefile fix into Git's upstream, we can remove the override for LIB_FUZZING_ENGINE. However, this change causes `check_build` to fail for honggfuzz, and we have not yet been able to diagnose the reason. So for now, we also need to limit our engines to afl and libfuzzer.

Due to upstream changes, the Git fuzzers must now link against common-main.o; however, this breaks the build in two ways: 1) Linking with common-main.o causes main() to have multiple definitions, one in common-main.o and one from the fuzzing engine. 2) To avoid google#1, the Git Makefile specifically excludes common-main.o from the fuzzer build rule. To work around these issues, we can override FUZZ_CXXFLAGS (add "-Wl,--allow-multiple-definition" to fix google#1) and LIB_FUZZING_ENGINE (add "common-main.o" to fix google#2). Once we can get a Makefile fix into Git's upstream, we can remove the override for LIB_FUZZING_ENGINE. However, this change causes `check_build` to fail for honggfuzz, and we have not yet been able to diagnose the reason. So for now, we also need to limit our engines to afl and libfuzzer.

native go fuzzing: Restrict fuzzer lookup to *.go Projects containing words starting with Fuzz in non-go files break compile_native_go_fuzzer with the error: sed: -e expression #1, char 15: unknown option to 's' Signed-off-by: Paulo Gomes <pjbgf@linux.com> Signed-off-by: Paulo Gomes <pjbgf@linux.com>

Following the addition of some fuzzing tests ([#1](u-root/u-root#2528), [#2](u-root/u-root#2535), [#3](u-root/u-root#2536)) in the u-root project and [preparations for this integration](u-root/u-root#2543). You can check out its [website](https://u-root.org/) for more info on the project. Signed-off-by: Fabian Wienand <fabian.wienand@9elements.com> Signed-off-by: Fabian Wienand <fabian.wienand@9elements.com>

@oliverchang

cc @oliverchang @alan32liu after #9100 and #8448 After compiling locally, I can see that `./SystemSan ./target_dns -dict=vuln.dict` crashes in a few seconds with ``` ===BUG DETECTED: Arbitrary domain name resolution=== ===Domain resolved: .f.z=== ===DNS request type: 0, class: 256=== ==315== ERROR: libFuzzer: deadly signal #0 0x539131 in __sanitizer_print_stack_trace /src/llvm-project/compiler-rt/lib/asan/asan_stack.cpp:87:3 #1 0x457c48 in fuzzer::PrintStackTrace() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerUtil.cpp:210:5 #2 0x43c923 in fuzzer::Fuzzer::CrashCallback() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:233:3 #3 0x7fa57940041f (/lib/x86_64-linux-gnu/libpthread.so.0+0x1441f) (BuildId: 7b4536f41cdaa5888408e82d0836e33dcf436466) #4 0x7fa5793ff7db in send (/lib/x86_64-linux-gnu/libpthread.so.0+0x137db) (BuildId: 7b4536f41cdaa5888408e82d0836e33dcf436466) #5 0x503ba4 in __interceptor_send /src/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:6802:17 #6 0x7fa578abf462 (/lib/x86_64-linux-gnu/libresolv.so.2+0xb462) (BuildId: 4519041bde5b859c55798ac0745b0b6199cb7d94) #7 0x7fa578abbc43 in __res_context_query (/lib/x86_64-linux-gnu/libresolv.so.2+0x7c43) (BuildId: 4519041bde5b859c55798ac0745b0b6199cb7d94) #8 0x7fa578abc8ed in __res_context_search (/lib/x86_64-linux-gnu/libresolv.so.2+0x88ed) (BuildId: 4519041bde5b859c55798ac0745b0b6199cb7d94) #9 0x7fa578ad2cc1 (/lib/x86_64-linux-gnu/libnss_dns.so.2+0x2cc1) (BuildId: 3fac4ec397ba8e8938fe298f103113f315465130) #10 0x7fa578ad2e8b in _nss_dns_gethostbyname3_r (/lib/x86_64-linux-gnu/libnss_dns.so.2+0x2e8b) (BuildId: 3fac4ec397ba8e8938fe298f103113f315465130) #11 0x7fa578ad2f41 in _nss_dns_gethostbyname2_r (/lib/x86_64-linux-gnu/libnss_dns.so.2+0x2f41) (BuildId: 3fac4ec397ba8e8938fe298f103113f315465130) #12 0x7fa5792fdc9d in gethostbyname2_r (/lib/x86_64-linux-gnu/libc.so.6+0x130c9d) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee) #13 0x7fa5792d179e (/lib/x86_64-linux-gnu/libc.so.6+0x10479e) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee) #14 0x7fa5792d2f58 in getaddrinfo (/lib/x86_64-linux-gnu/libc.so.6+0x105f58) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee) #15 0x4d93ac in getaddrinfo /src/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:2667:13 #16 0x56c8d9 in LLVMFuzzerTestOneInput /out/SystemSan/target_dns.cpp:35:11 #17 0x43dec3 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:611:15 #18 0x43d6aa in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:514:3 #19 0x43ed79 in fuzzer::Fuzzer::MutateAndTestOne() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:757:19 #20 0x43fa45 in fuzzer::Fuzzer::Loop(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocator<fuzzer::SizedFile> >&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:895:5 #21 0x42edaf in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:912:6 #22 0x458402 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10 #23 0x7fa5791f1082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee) #24 0x41f7ed in _start (/out/SystemSan/target_dns+0x41f7ed) NOTE: libFuzzer has rudimentary signal handlers. Combine libFuzzer with AddressSanitizer or similar for better crash reports. SUMMARY: libFuzzer: deadly signal MS: 2 CrossOver-ManualDict- DE: "f.z"-; base unit: ac3478d69a3c81fa62e60f5c3696165a4e5e6ac4 0x66,0x2e,0x7a, f.z artifact_prefix='./'; Test unit written to ./crash-926813b2d6adde373f96a10594a5314951588384 Base64: Zi56 ``` You can also try ``` echo -n f.z > toto ./SystemSan ./target_dns toto ``` Co-authored-by: Oliver Chang <oliverchang@users.noreply.github.com> Co-authored-by: jonathanmetzman <31354670+jonathanmetzman@users.noreply.github.com>

@oliverchang

cc @oliverchang @alan32liu after google#9100 and google#8448 After compiling locally, I can see that `./SystemSan ./target_dns -dict=vuln.dict` crashes in a few seconds with ``` ===BUG DETECTED: Arbitrary domain name resolution=== ===Domain resolved: .f.z=== ===DNS request type: 0, class: 256=== ==315== ERROR: libFuzzer: deadly signal #0 0x539131 in __sanitizer_print_stack_trace /src/llvm-project/compiler-rt/lib/asan/asan_stack.cpp:87:3 google#1 0x457c48 in fuzzer::PrintStackTrace() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerUtil.cpp:210:5 google#2 0x43c923 in fuzzer::Fuzzer::CrashCallback() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:233:3 google#3 0x7fa57940041f (/lib/x86_64-linux-gnu/libpthread.so.0+0x1441f) (BuildId: 7b4536f41cdaa5888408e82d0836e33dcf436466) google#4 0x7fa5793ff7db in send (/lib/x86_64-linux-gnu/libpthread.so.0+0x137db) (BuildId: 7b4536f41cdaa5888408e82d0836e33dcf436466) google#5 0x503ba4 in __interceptor_send /src/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:6802:17 google#6 0x7fa578abf462 (/lib/x86_64-linux-gnu/libresolv.so.2+0xb462) (BuildId: 4519041bde5b859c55798ac0745b0b6199cb7d94) google#7 0x7fa578abbc43 in __res_context_query (/lib/x86_64-linux-gnu/libresolv.so.2+0x7c43) (BuildId: 4519041bde5b859c55798ac0745b0b6199cb7d94) google#8 0x7fa578abc8ed in __res_context_search (/lib/x86_64-linux-gnu/libresolv.so.2+0x88ed) (BuildId: 4519041bde5b859c55798ac0745b0b6199cb7d94) google#9 0x7fa578ad2cc1 (/lib/x86_64-linux-gnu/libnss_dns.so.2+0x2cc1) (BuildId: 3fac4ec397ba8e8938fe298f103113f315465130) google#10 0x7fa578ad2e8b in _nss_dns_gethostbyname3_r (/lib/x86_64-linux-gnu/libnss_dns.so.2+0x2e8b) (BuildId: 3fac4ec397ba8e8938fe298f103113f315465130) google#11 0x7fa578ad2f41 in _nss_dns_gethostbyname2_r (/lib/x86_64-linux-gnu/libnss_dns.so.2+0x2f41) (BuildId: 3fac4ec397ba8e8938fe298f103113f315465130) google#12 0x7fa5792fdc9d in gethostbyname2_r (/lib/x86_64-linux-gnu/libc.so.6+0x130c9d) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee) google#13 0x7fa5792d179e (/lib/x86_64-linux-gnu/libc.so.6+0x10479e) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee) google#14 0x7fa5792d2f58 in getaddrinfo (/lib/x86_64-linux-gnu/libc.so.6+0x105f58) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee) google#15 0x4d93ac in getaddrinfo /src/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:2667:13 google#16 0x56c8d9 in LLVMFuzzerTestOneInput /out/SystemSan/target_dns.cpp:35:11 google#17 0x43dec3 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:611:15 google#18 0x43d6aa in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:514:3 google#19 0x43ed79 in fuzzer::Fuzzer::MutateAndTestOne() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:757:19 google#20 0x43fa45 in fuzzer::Fuzzer::Loop(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocator<fuzzer::SizedFile> >&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:895:5 google#21 0x42edaf in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:912:6 google#22 0x458402 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10 google#23 0x7fa5791f1082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee) google#24 0x41f7ed in _start (/out/SystemSan/target_dns+0x41f7ed) NOTE: libFuzzer has rudimentary signal handlers. Combine libFuzzer with AddressSanitizer or similar for better crash reports. SUMMARY: libFuzzer: deadly signal MS: 2 CrossOver-ManualDict- DE: "f.z"-; base unit: ac3478d69a3c81fa62e60f5c3696165a4e5e6ac4 0x66,0x2e,0x7a, f.z artifact_prefix='./'; Test unit written to ./crash-926813b2d6adde373f96a10594a5314951588384 Base64: Zi56 ``` You can also try ``` echo -n f.z > toto ./SystemSan ./target_dns toto ``` Co-authored-by: Oliver Chang <oliverchang@users.noreply.github.com> Co-authored-by: jonathanmetzman <31354670+jonathanmetzman@users.noreply.github.com>

We've still got an issue with crashes on the urllib3 requests test that uses the mock HTTP server. Fix #9958 to handle port mapping errors didn't resolve it. I got a feeling there's an ordering issue. Looking at the error logs [https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=56500#c2](https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=56500#c2) there appears to be an issue where we're throwing exceptions before the coverage completes. ``` === Uncaught Python exception: === -- | MaxRetryError: HTTPConnectionPool(host='localhost', port=8011): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f4cdf33d1f0>: Failed to establish a new connection: [Errno 101] Network is unreachable')) | Traceback (most recent call last): | File "fuzz_requests.py", line 109, in TestOneInput | File "urllib3/_request_methods.py", line 118, in request | File "urllib3/_request_methods.py", line 217, in request_encode_body | File "urllib3/poolmanager.py", line 433, in urlopen | File "urllib3/connectionpool.py", line 874, in urlopen | File "urllib3/connectionpool.py", line 874, in urlopen | File "urllib3/connectionpool.py", line 874, in urlopen | File "urllib3/connectionpool.py", line 844, in urlopen | File "urllib3/util/retry.py", line 505, in increment | MaxRetryError: HTTPConnectionPool(host='localhost', port=8011): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f4cdf33d1f0>: Failed to establish a new connection: [Errno 101] Network is unreachable')) | | INFO: Instrumenting 3854 functions... | INFO: Instrumentation complete. | ==10674== ERROR: libFuzzer: fuzz target exited | #0 0x7f4ce0bac694 in __sanitizer_print_stack_trace /src/llvm-project/compiler-rt/lib/ubsan/ubsan_diag_standalone.cpp:31:3 | #1 0x7f4ce0b2df48 in fuzzer::PrintStackTrace() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerUtil.cpp:210:5 | #2 0x7f4ce0b12cdc in fuzzer::Fuzzer::ExitCallback() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:250:3 | #3 0x7f4ce09068a6 in __run_exit_handlers /build/glibc-SzIz7B/glibc-2.31/stdlib/exit.c:108:8 | #4 0x7f4ce0906a5f in exit /build/glibc-SzIz7B/glibc-2.31/stdlib/exit.c:139:3 | #5 0x7f4ce03b2c78 in libpython3.8.so.1.0 | #6 0x7f4ce03b76cf in libpython3.8.so.1.0 | #7 0x403ad2 in fuzz_requests.pkg | #8 0x403e67 in fuzz_requests.pkg | #9 0x7f4ce08e4082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/libc-start.c:308:16 | #10 0x40249d in fuzz_requests.pkg | | SUMMARY: libFuzzer: fuzz target exited ``` This is an attempted fix inspired by the requests [fuzz_server.py](https://github.com/google/oss-fuzz/blob/master/projects/requests/fuzz_server.py) where the lifecycle of the test thread is managed within the server. Since the web server is created at the start of `TestOneInput` I don't expect there to be any timing issues or thread initialisation issues.

This should fix the following issue: ``` Step #1: ---> 22f958f99cac Step #1: Step 2/4 : RUN git clone --depth 1 --branch master https://github.com/quodlibet/mutagen Step #1: ---> Running in 77c762382f3f Step #1: �[91mCloning into 'mutagen'... Step #1: �[0m�[91mwarning: Could not find remote branch master to clone. Step #1: fatal: Remote branch master not found in upstream origin Step #1: The command '/bin/sh -c git clone --depth 1 --branch master https://github.com/quodlibet/mutagen' returned a non-zero code: 128 Finished Step #1 ERROR ERROR: build step 1 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 128 Step #1: �[0m ```

This should fix the following issue: ``` Step google#1: ---> 22f958f99cac Step google#1: Step 2/4 : RUN git clone --depth 1 --branch master https://github.com/quodlibet/mutagen Step google#1: ---> Running in 77c762382f3f Step google#1: �[91mCloning into 'mutagen'... Step google#1: �[0m�[91mwarning: Could not find remote branch master to clone. Step google#1: fatal: Remote branch master not found in upstream origin Step google#1: The command '/bin/sh -c git clone --depth 1 --branch master https://github.com/quodlibet/mutagen' returned a non-zero code: 128 Finished Step google#1 ERROR ERROR: build step 1 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 128 Step google#1: �[0m ```

``` Step google#1: Step 3/5 : RUN pip3 install soupsieve html5lib lxml Step google#1: ---> Running in 95c0ff529f3c Step google#1: Collecting soupsieve Step google#1: Downloading https://files.pythonhosted.org/packages/4c/f3/038b302fdfbe3be7da016777069f26ceefe11a681055ea1f7817546508e3/soupsieve-2.5-py3-none-any.whl Step google#1: Collecting html5lib Step google#1: Downloading https://files.pythonhosted.org/packages/6c/dd/a834df6482147d48e225a49515aabc28974ad5a4ca3215c18a882565b028/html5lib-1.1-py2.py3-none-any.whl (112kB) Step google#1: Collecting lxml Step google#1: Downloading https://files.pythonhosted.org/packages/2b/b4/bbccb250adbee490553b6a52712c46c20ea1ba533a643f1424b27ffc6845/lxml-5.1.0.tar.gz (3.8MB) Step google#1: Installing build dependencies: started Step google#1: Installing build dependencies: finished with status 'done' Step google#1: Getting requirements to build wheel: started Step google#1: Getting requirements to build wheel: finished with status 'error' Step google#1: �[91m ERROR: Command errored out with exit status 1: Step google#1: command: /usr/local/bin/python3.8 /usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpffkvuryp Step google#1: cwd: /tmp/pip-install-kbqwq5f7/lxml Step google#1: Complete output (4 lines): Step google#1: <string>:67: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html Step google#1: Building lxml version 5.1.0. Step google#1: Building without Cython. Step google#1: Error: Please make sure the libxml2 and libxslt development packages are installed. ```

``` Step #1: Step 3/5 : RUN pip3 install soupsieve html5lib lxml Step #1: ---> Running in 95c0ff529f3c Step #1: Collecting soupsieve Step #1: Downloading https://files.pythonhosted.org/packages/4c/f3/038b302fdfbe3be7da016777069f26ceefe11a681055ea1f7817546508e3/soupsieve-2.5-py3-none-any.whl Step #1: Collecting html5lib Step #1: Downloading https://files.pythonhosted.org/packages/6c/dd/a834df6482147d48e225a49515aabc28974ad5a4ca3215c18a882565b028/html5lib-1.1-py2.py3-none-any.whl (112kB) Step #1: Collecting lxml Step #1: Downloading https://files.pythonhosted.org/packages/2b/b4/bbccb250adbee490553b6a52712c46c20ea1ba533a643f1424b27ffc6845/lxml-5.1.0.tar.gz (3.8MB) Step #1: Installing build dependencies: started Step #1: Installing build dependencies: finished with status 'done' Step #1: Getting requirements to build wheel: started Step #1: Getting requirements to build wheel: finished with status 'error' Step #1: �[91m ERROR: Command errored out with exit status 1: Step #1: command: /usr/local/bin/python3.8 /usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpffkvuryp Step #1: cwd: /tmp/pip-install-kbqwq5f7/lxml Step #1: Complete output (4 lines): Step #1: <string>:67: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html Step #1: Building lxml version 5.1.0. Step #1: Building without Cython. Step #1: Error: Please make sure the libxml2 and libxslt development packages are installed. ```

Base PR apache/brpc#2420 ; NOTE: I can't enable memory sanitizer due to ```log BAD BUILD: /tmp/not-out/tmpmptlk01q/fuzz_esp seems to have either startup crash or exit: /tmp/not-out/tmpmptlk01q/fuzz_esp -rss_limit_mb=2560 -timeout=25 -seed=1337 -runs=4 < /dev/null Uninitialized bytes in MemcmpInterceptorCommon at offset 15 inside [0x7030000000f0, 19) ==428==WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x682b90 in __interceptor_memcmp /src/llvm-project/compiler-rt/lib/msan/../sanitizer_common/sanitizer_common_interceptors.inc:892:10 #1 0x7fa8ef4cf62a in google::protobuf::SimpleDescriptorDatabase::DescriptorIndex<std::pair<void const*, int> >::FindLastLessOrEqual(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/tmp/not-out/tmpmptlk01q/lib/libprotobuf.so.17+0x15062a) (BuildId: 64affeb0f489ae4bcea211ed99e1eca15ff97d68) #2 0x7fa8ef4d259f in google::protobuf::SimpleDescriptorDatabase::DescriptorIndex<std::pair<void const*, int> >::AddSymbol(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::pair<void const*, int>) (/tmp/not-out/tmpmptlk01q/lib/libprotobuf.so.17+0x15359f) (BuildId: 64affeb0f489ae4bcea211ed99e1eca15ff97d68) #3 0x7fa8ef4d2a15 in google::protobuf::SimpleDescriptorDatabase::DescriptorIndex<std::pair<void const*, int> >::AddFile(google::protobuf::FileDescriptorProto const&, std::pair<void const*, int>) (/tmp/not-out/tmpmptlk01q/lib/libprotobuf.so.17+0x153a15) (BuildId: 64affeb0f489ae4bcea211ed99e1eca15ff97d68) #4 0x7fa8ef4cebef in google::protobuf::EncodedDescriptorDatabase::Add(void const*, int) (/tmp/not-out/tmpmptlk01q/lib/libprotobuf.so.17+0x14fbef) (BuildId: 64affeb0f489ae4bcea211ed99e1eca15ff97d68) #5 0x7fa8ef499f43 in google::protobuf::DescriptorPool::InternalAddGeneratedFile(void const*, int) (/tmp/not-out/tmpmptlk01q/lib/libprotobuf.so.17+0x11af43) (BuildId: 64affeb0f489ae4bcea211ed99e1eca15ff97d68) #6 0x7fa8ef49281d in protobuf_google_2fprotobuf_2fapi_2eproto::AddDescriptorsImpl() (/tmp/not-out/tmpmptlk01q/lib/libprotobuf.so.17+0x11381d) (BuildId: 64affeb0f489ae4bcea211ed99e1eca15ff97d68) ``` Signed-off-by: Arjun Singh <ajsinghyadav00@gmail.com>

[infra] some changes

4005c0d

- Remove requirement of specifying dockerfile in Jenkinsfile. Assume it is always in the library directory. - Try to copy any *.dict or *.options files to output directory.

mikea reviewed Aug 30, 2016
View reviewed changes

Allow overriding dockerfile.

6ef5202

mikea reviewed Aug 30, 2016
View reviewed changes

Some cleanup.

c88ab46

mikea merged commit bcdafbe into google:master Aug 30, 2016

mikea mentioned this pull request Oct 10, 2016

libchewing: avoid /out/.libs #8

Merged

This was referenced Dec 8, 2016

Add fuzzer scripts for botan #150

Merged

Leak detection doesn't work #9

Closed

inferno-chromium mentioned this pull request Jan 11, 2017

Build tor fuzzing binaries with static libevent linkage #267

Merged

This was referenced Jan 27, 2017

Update NSS project to use the build.sh and fuzzers provided by NSS #316

Merged

NSS startup crash #321

Closed

This was referenced Feb 5, 2017

Startup crash notion is confusing in performance analyzer #351

Closed

Fix UBSAN error with function argument casting. #362

Merged

This was referenced Feb 17, 2017

Add additional undefined behaviors #232

Closed

Disable UBSan vptr on harfbuzz, enable UBSan for other features. #405

Merged

This was referenced Mar 10, 2017

Large control file does not seem to get deleted on error during merging #445

Closed

libFuzzer should show deadly signal type or just let sanitizer handle the stack its own way. #447

Closed

dgreid mentioned this pull request Oct 11, 2017

Add cras fuzzer #880

Merged

bilboed mentioned this pull request Nov 2, 2017

Specify depth for identical testcase detection #944

Closed

inferno-chromium mentioned this pull request May 7, 2018

h2o: fuzz target crashes on startup. #1368

Closed

jonathanmetzman pushed a commit that referenced this pull request May 29, 2018

[libwebp] Add fuzz target #1 (#1448)

5d153be

tmatth pushed a commit to tmatth/oss-fuzz that referenced this pull request Oct 22, 2018

[libwebp] Add fuzz target google#1 (google#1448)

5d0ad66

inferno-chromium mentioned this pull request Oct 22, 2018

bzip2: Add bzip2 to oss-fuzz #1887

Merged

inferno-chromium mentioned this pull request Dec 8, 2018

Add implementation for perfetto project. #2011

Merged

inferno-chromium mentioned this pull request Dec 28, 2018

[libldac] Disable MSan #2058

Merged

Leo-Neat mentioned this pull request Jan 14, 2020

"helper.py reproduce" fails with AttributeError #3229

Closed

DavidKorczynski pushed a commit that referenced this pull request Jun 21, 2021

[lldb-eval] Upgrade to LLVM-12 and enable undefined sanitizer (#5952)

f43633e

* Upgrade to LLVM-12 and enable undefined sanitizer (#1) * Upgrade to LLVM-12 * Enable undefined sanitizer * Fix missing scope

jonathanmetzman mentioned this pull request Aug 16, 2021

Difference in coverage and normal fuzz builds? #6226

Open

yangcheng1203 mentioned this pull request Sep 29, 2021

【libexif】Timout in mnote_olympus_entry_get_value #6537

Closed

rlohning mentioned this pull request Dec 21, 2021

AFL: several builds started failing 15th December #7021

Closed

bobfriesenhahn mentioned this pull request Dec 27, 2021

Getc/getc_unlocked are claimed to return uninitialized data from setvbuf private buffer #7071

Open

steadmon mentioned this pull request Jun 7, 2022

git: workaround new requirement to use common-main #7818

Merged

oliverchang mentioned this pull request Nov 28, 2022

infra: refine fuzztest work #8992

Merged

hypSSLAB mentioned this pull request May 18, 2023

tcpdump-4.7.3 libpcap-1.7.3 problem from snprintf(buf, bufsize, “%subopt:%u”, 128) try to access 0x80 #10347

Closed

DavidKorczynski mentioned this pull request Oct 5, 2023

infra: base-runner: coverage: set max parallel jobs to be half of CPU count #10277

Merged

maflcko mentioned this pull request Apr 4, 2024

infra: Roll OUR_LLVM_REVISION #11714

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[infra] some changes #1

[infra] some changes #1

oliverchang commented Aug 30, 2016 •

edited

mikea Aug 30, 2016

oliverchang Aug 30, 2016

mikea Aug 30, 2016

oliverchang Aug 30, 2016

[infra] some changes #1

[infra] some changes #1

Conversation

oliverchang commented Aug 30, 2016 • edited

mikea Aug 30, 2016

Choose a reason for hiding this comment

oliverchang Aug 30, 2016

Choose a reason for hiding this comment

mikea Aug 30, 2016

Choose a reason for hiding this comment

oliverchang Aug 30, 2016

Choose a reason for hiding this comment

oliverchang commented Aug 30, 2016 •

edited