Segmentation fault when compiling with libfuzzer and lto (x86_64) #25961

maflcko · 2022-08-30T19:08:24Z

Steps to reproduce:

Install fresh Ubuntu Jammy operating system
export DEBIAN_FRONTEND=noninteractive && apt update && apt install curl wget htop git vim ccache -y && git clone https://github.com/bitcoin/bitcoin.git bitcoin-core && cd bitcoin-core && apt install build-essential libtool autotools-dev automake pkg-config bsdmainutils python3-zmq libevent-dev libboost-dev libsqlite3-dev libdb++-dev clang llvm libc++-dev libc++abi-dev -y && ./autogen.sh && ./configure CC='clang -flto' CXX='clang++ -flto' --enable-fuzz --with-sanitizers=fuzzer && make -j$(nproc)

Output:

  AR       minisketch/libminisketch.a
  AR       libbitcoin_wallet.a
  CXXLD    test/fuzz/fuzz
clang: error: unable to execute command: Segmentation fault (core dumped)

The same does not happen when compiling "normally" (dropping --with-sanitizers=fuzzer).

The text was updated successfully, but these errors were encountered:

maflcko · 2022-08-30T19:25:15Z

Likely unrelated, but interestingly, with clang-12 it always fails and prints error adding symbols: archive has no index; run ranlib to add one. To reproduce:

Fresh install of Focal or Jammy
export DEBIAN_FRONTEND=noninteractive && apt update && apt install curl wget htop git vim ccache -y && git clone https://github.com/bitcoin/bitcoin.git bitcoin-core && cd bitcoin-core && apt install build-essential libtool autotools-dev automake pkg-config bsdmainutils python3-zmq libevent-dev libboost-dev libsqlite3-dev clang-12 llvm-12 -y && ./autogen.sh && ./configure CC='clang-12 -flto' CXX='clang++-12 -flto' --enable-fuzz && make -j$(nproc)

maflcko · 2022-08-30T19:30:32Z

clang-13 seems to be working 🤷‍♂️

maflcko · 2022-08-31T08:12:27Z

Going back to clang-14 and using different linkers:

-fuse-ld=ld: Segfault
-fuse-ld=gold: Segfault
-fuse-ld=mold (1.4): Segfault
-fuse-ld=lld: Segfault with error below

lld error:

PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.	Program arguments: /usr/bin/ld.lld -pie -z relro --hash-style=gnu --build-id --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o test/fuzz/fuzz /lib/x86_64-linux-gnu/Scrt1.o /lib/x86_64-linux-gnu/crti.o /usr/bin/../lib/gcc/x86_64-linux-gnu/11/crtbeginS.o -L/usr/bin/../lib/gcc/x86_64-linux-gnu/11 -L/usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../lib64 -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib64 -L/usr/lib/llvm-14/bin/../lib -L/lib -L/usr/lib -plugin-opt=mcpu=x86-64 -plugin-opt=O2 --whole-archive /usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.fuzzer-x86_64.a --no-whole-archive --whole-archive /usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.fuzzer_interceptors-x86_64.a --no-whole-archive -lstdc++ --whole-archive /usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.ubsan_standalone-x86_64.a --no-whole-archive --dynamic-list=/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.ubsan_standalone-x86_64.a.syms --whole-archive /usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.ubsan_standalone_cxx-x86_64.a --no-whole-archive --dynamic-list=/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.ubsan_standalone_cxx-x86_64.a.syms -z relro -z now -z separate-code wallet/test/fuzz/test_fuzz_fuzz-coinselection.o wallet/test/fuzz/test_fuzz_fuzz-notifications.o test/fuzz/fuzz-addition_overflow.o test/fuzz/fuzz-addrman.o test/fuzz/fuzz-asmap.o test/fuzz/fuzz-asmap_direct.o test/fuzz/fuzz-autofile.o test/fuzz/fuzz-banman.o test/fuzz/fuzz-base_encode_decode.o test/fuzz/fuzz-bech32.o test/fuzz/fuzz-bitdeque.o test/fuzz/fuzz-block.o test/fuzz/fuzz-block_header.o test/fuzz/fuzz-blockfilter.o test/fuzz/fuzz-bloom_filter.o test/fuzz/fuzz-buffered_file.o test/fuzz/fuzz-chain.o test/fuzz/fuzz-checkqueue.o test/fuzz/fuzz-coins_view.o test/fuzz/fuzz-connman.o test/fuzz/fuzz-crypto.o test/fuzz/fuzz-crypto_aes256.o test/fuzz/fuzz-crypto_aes256cbc.o test/fuzz/fuzz-crypto_chacha20.o test/fuzz/fuzz-crypto_chacha20_poly1305_aead.o test/fuzz/fuzz-crypto_common.o test/fuzz/fuzz-crypto_diff_fuzz_chacha20.o test/fuzz/fuzz-crypto_hkdf_hmac_sha256_l32.o test/fuzz/fuzz-crypto_poly1305.o test/fuzz/fuzz-cuckoocache.o test/fuzz/fuzz-decode_tx.o test/fuzz/fuzz-descriptor_parse.o test/fuzz/fuzz-deserialize.o test/fuzz/fuzz-eval_script.o test/fuzz/fuzz-fee_rate.o test/fuzz/fuzz-fees.o test/fuzz/fuzz-flatfile.o test/fuzz/fuzz-float.o test/fuzz/fuzz-golomb_rice.o test/fuzz/fuzz-hex.o test/fuzz/fuzz-http_request.o test/fuzz/fuzz-i2p.o test/fuzz/fuzz-integer.o test/fuzz/fuzz-key.o test/fuzz/fuzz-key_io.o test/fuzz/fuzz-kitchen_sink.o test/fuzz/fuzz-load_external_block_file.o test/fuzz/fuzz-locale.o test/fuzz/fuzz-merkleblock.o test/fuzz/fuzz-message.o test/fuzz/fuzz-miniscript.o test/fuzz/fuzz-minisketch.o test/fuzz/fuzz-muhash.o test/fuzz/fuzz-multiplication_overflow.o test/fuzz/fuzz-net.o test/fuzz/fuzz-net_permissions.o test/fuzz/fuzz-netaddress.o test/fuzz/fuzz-netbase_dns_lookup.o test/fuzz/fuzz-node_eviction.o test/fuzz/fuzz-p2p_transport_serialization.o test/fuzz/fuzz-parse_hd_keypath.o test/fuzz/fuzz-parse_iso8601.o test/fuzz/fuzz-parse_numbers.o test/fuzz/fuzz-parse_script.o test/fuzz/fuzz-parse_univalue.o test/fuzz/fuzz-policy_estimator.o test/fuzz/fuzz-policy_estimator_io.o test/fuzz/fuzz-pow.o test/fuzz/fuzz-prevector.o test/fuzz/fuzz-primitives_transaction.o test/fuzz/fuzz-process_message.o test/fuzz/fuzz-process_messages.o test/fuzz/fuzz-protocol.o test/fuzz/fuzz-psbt.o test/fuzz/fuzz-random.o test/fuzz/fuzz-rbf.o test/fuzz/fuzz-rolling_bloom_filter.o test/fuzz/fuzz-rpc.o test/fuzz/fuzz-script.o test/fuzz/fuzz-script_assets_test_minimizer.o test/fuzz/fuzz-script_bitcoin_consensus.o test/fuzz/fuzz-script_descriptor_cache.o test/fuzz/fuzz-script_flags.o test/fuzz/fuzz-script_format.o test/fuzz/fuzz-script_interpreter.o test/fuzz/fuzz-script_ops.o test/fuzz/fuzz-script_sigcache.o test/fuzz/fuzz-script_sign.o test/fuzz/fuzz-scriptnum_ops.o test/fuzz/fuzz-secp256k1_ec_seckey_import_export_der.o test/fuzz/fuzz-secp256k1_ecdsa_signature_parse_der_lax.o test/fuzz/fuzz-signature_checker.o test/fuzz/fuzz-signet.o test/fuzz/fuzz-socks5.o test/fuzz/fuzz-span.o test/fuzz/fuzz-spanparsing.o test/fuzz/fuzz-string.o test/fuzz/fuzz-strprintf.o test/fuzz/fuzz-system.o test/fuzz/fuzz-timedata.o test/fuzz/fuzz-torcontrol.o test/fuzz/fuzz-transaction.o test/fuzz/fuzz-tx_in.o test/fuzz/fuzz-tx_out.o test/fuzz/fuzz-tx_pool.o test/fuzz/fuzz-txorphan.o test/fuzz/fuzz-txrequest.o test/fuzz/fuzz-utxo_snapshot.o test/fuzz/fuzz-validation_load_mempool.o test/fuzz/fuzz-versionbits.o -lpthread libtest_util.a libtest_fuzz.a libbitcoin_node.a libbitcoin_wallet.a libbitcoin_common.a libbitcoin_util.a libbitcoin_consensus.a crypto/.libs/libbitcoin_crypto_base.a crypto/.libs/libbitcoin_crypto_sse41.a crypto/.libs/libbitcoin_crypto_avx2.a crypto/.libs/libbitcoin_crypto_x86_shani.a libbitcoin_cli.a ./.libs/libunivalue.a leveldb/.libs/libleveldb.a crc32c/.libs/libcrc32c.a crc32c/.libs/libcrc32c_sse42.a leveldb/.libs/libmemenv.a secp256k1/.libs/libsecp256k1.a minisketch/libminisketch.a minisketch/libminisketch_clmul.a -levent -levent_pthreads -levent -lsqlite3 -lstdc++ -lm --no-as-needed -lpthread -lrt -lm -ldl -lgcc_s -lgcc -lpthread -lc -lgcc_s -lgcc /usr/bin/../lib/gcc/x86_64-linux-gnu/11/crtendS.o /lib/x86_64-linux-gnu/crtn.o
1.	Running pass 'Function Pass Manager' on module 'ld-temp.o'.
2.	Running pass 'Live DEBUG_VALUE analysis' on function '@_Z22minisketch_fuzz_target4SpanIKhE'
 #0 0x00007fa9acecad01 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0xe3fd01)
 #1 0x00007fa9acec8a3e llvm::sys::RunSignalHandlers() (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0xe3da3e)
 #2 0x00007fa9acecb236 (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0xe40236)
 #3 0x00007fa9abb72520 (/lib/x86_64-linux-gnu/libc.so.6+0x42520)
 #4 0x00007fa9ad4e75fc (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0x145c5fc)
 #5 0x00007fa9ad4dcc32 (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0x1451c32)
 #6 0x00007fa9ad4dca33 LiveDebugValues::InstrRefBasedLDV::depthFirstVLocAndEmit(unsigned int, llvm::DenseMap<llvm::LexicalScope const*, llvm::DILocation const*, llvm::DenseMapInfo<llvm::LexicalScope const*, void>, llvm::detail::DenseMapPair<llvm::LexicalScope const*, llvm::DILocation const*> > const&, llvm::DenseMap<llvm::LexicalScope const*, llvm::SmallSet<llvm::DebugVariable, 4u, std::less<llvm::DebugVariable> >, llvm::DenseMapInfo<llvm::LexicalScope const*, void>, llvm::detail::DenseMapPair<llvm::LexicalScope const*, llvm::SmallSet<llvm::DebugVariable, 4u, std::less<llvm::DebugVariable> > > > const&, llvm::DenseMap<llvm::LexicalScope const*, llvm::SmallPtrSet<llvm::MachineBasicBlock*, 4u>, llvm::DenseMapInfo<llvm::LexicalScope const*, void>, llvm::detail::DenseMapPair<llvm::LexicalScope const*, llvm::SmallPtrSet<llvm::MachineBasicBlock*, 4u> > >&, llvm::SmallVector<llvm::SmallVector<std::pair<llvm::DebugVariable, LiveDebugValues::DbgValue>, 8u>, 8u>&, LiveDebugValues::ValueIDNum**, LiveDebugValues::ValueIDNum**, llvm::SmallVectorImpl<LiveDebugValues::VLocTracker>&, llvm::MachineFunction&, llvm::DenseMap<llvm::DebugVariable, unsigned int, llvm::DenseMapInfo<llvm::DebugVariable, void>, llvm::detail::DenseMapPair<llvm::DebugVariable, unsigned int> >&, llvm::TargetPassConfig const&) (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0x1451a33)
 #7 0x00007fa9ad4df216 LiveDebugValues::InstrRefBasedLDV::ExtendRanges(llvm::MachineFunction&, llvm::MachineDominatorTree*, llvm::TargetPassConfig*, unsigned int, unsigned int) (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0x1454216)
 #8 0x00007fa9ad2512be llvm::MachineFunctionPass::runOnFunction(llvm::Function&) (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0x11c62be)
 #9 0x00007fa9ad0053c0 llvm::FPPassManager::runOnFunction(llvm::Function&) (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0xf7a3c0)
#10 0x00007fa9ad00c9b3 llvm::FPPassManager::runOnModule(llvm::Module&) (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0xf819b3)
#11 0x00007fa9ad005f66 llvm::legacy::PassManagerImpl::run(llvm::Module&) (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0xf7af66)
#12 0x00007fa9ae3f0aff (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0x2365aff)
#13 0x00007fa9ae3efcdc llvm::lto::backend(llvm::lto::Config const&, std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream> > > (unsigned int)>, unsigned int, llvm::Module&, llvm::ModuleSummaryIndex&) (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0x2364cdc)
#14 0x00007fa9ae3e4889 llvm::lto::LTO::runRegularLTO(std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream> > > (unsigned int)>) (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0x2359889)
#15 0x00007fa9ae3e4083 llvm::lto::LTO::run(std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream> > > (unsigned int)>, std::function<llvm::Expected<std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream> > > (unsigned int)> > (unsigned int, llvm::StringRef)>) (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0x2359083)
#16 0x00000000005ca433 lld::elf::BitcodeCompiler::compile() (/usr/bin/ld.lld+0x5ca433)
#17 0x0000000000550206 void lld::elf::LinkerDriver::compileBitcodeFiles<llvm::object::ELFType<(llvm::support::endianness)1, true> >(bool) (/usr/bin/ld.lld+0x550206)
#18 0x000000000054d05d lld::elf::LinkerDriver::link(llvm::opt::InputArgList&) (/usr/bin/ld.lld+0x54d05d)
#19 0x0000000000541ac4 lld::elf::LinkerDriver::linkerMain(llvm::ArrayRef<char const*>) (/usr/bin/ld.lld+0x541ac4)
#20 0x000000000053fd97 lld::elf::link(llvm::ArrayRef<char const*>, llvm::raw_ostream&, llvm::raw_ostream&, bool, bool) (/usr/bin/ld.lld+0x53fd97)
#21 0x000000000048b235 (/usr/bin/ld.lld+0x48b235)
#22 0x000000000048a998 main (/usr/bin/ld.lld+0x48a998)
#23 0x00007fa9abb59d90 (/lib/x86_64-linux-gnu/libc.so.6+0x29d90)
#24 0x00007fa9abb59e40 __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e40)
#25 0x000000000048a4d5 _start (/usr/bin/ld.lld+0x48a4d5)
clang: error: unable to execute command: Segmentation fault (core dumped)

fanquake · 2022-08-31T08:12:31Z

I think this might be a resource, or other local issue, as I can't reproduce the failure using the steps outlined in the op. One suggestion I'd make would be to use ./configure CC=clang CXX=clang++ --enable-fuzz --with-sanitizers=fuzzer --enable-lto, to save the manual -flto additions, and to ensure you end up with -flto in your ldflags (although it is present). Can you provide any other more verbose output from a failure?

maflcko · 2022-08-31T08:13:34Z

I think this might be a resource, or other local issue

I was using 16 GB of RAM. Do you have more than that?

fanquake · 2022-08-31T08:19:11Z

I was using 16 GB or RAM. Do you have more than that?

The machine I'm using has 32GB, but docker should be limited to only using 15GB. Will test further.

maflcko · 2022-08-31T08:29:29Z

Still happening with 32GB of RAM and 128GB of swap. Let me try a different CPU ...

  CXXLD    test/fuzz/fuzz
clang: error: unable to execute command: Segmentation fault (core dumped)
clang: error: linker command failed due to signal (use -v to see invocation)
make[2]: *** [Makefile:7337: test/fuzz/fuzz] Error 254
make[2]: Leaving directory '/root/bitcoin-core/src'
make[1]: *** [Makefile:18926: all-recursive] Error 1
make[1]: Leaving directory '/root/bitcoin-core/src'
make: *** [Makefile:824: all-recursive] Error 1
# lscpu | grep 'AMD ' ; free -h ; df -h ./
Model name:                      AMD EPYC Processor
               total        used        free      shared  buff/cache   available
Mem:            30Gi       351Mi        26Gi       4.0Mi       3.7Gi        29Gi
Swap:          127Gi          0B       127Gi
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       338G  133G  191G  42% /

fanquake · 2022-08-31T09:08:25Z

Have recreated the issue on x86_64 linux hardware, using LLVM 14, but still not on aarch64.

maflcko · 2022-08-31T09:13:09Z

Yes, this works on aarch64 for me as well.

maflcko · 2022-08-31T10:05:12Z

Trying a depends build seems to work fine as well:

export CC=clang && export CXX=clang++ && export DEBIAN_FRONTEND=noninteractive && apt update && apt install curl wget htop git vim ccache -y && git clone https://github.com/bitcoin/bitcoin.git --depth=1 ./bitcoin-core && cd bitcoin-core && apt install clang llvm build-essential libtool autotools-dev automake pkg-config bsdmainutils python3-zmq make automake cmake curl g++-multilib libtool binutils-gold bsdmainutils pkg-config python3 patch bison lld mold -y && ( cd depends && make NO_QT=1 LTO=1 NO_WALLET=1 NO_ZMQ=1 NO_UPNP=1 NO_NATPMP=1 -j $(nproc) ) && ./autogen.sh && CONFIG_SITE="$PWD/depends/x86_64-pc-linux-gnu/share/config.site" ./configure --enable-fuzz --with-sanitizers=fuzzer && make -j $(nproc)

Edit: The same command with --enable-debug added, again results in a segmentation fault

Edit2: The same command with DEBUG=1 added to depends, results in:

  CXXLD    test/fuzz/fuzz
/usr/bin/ld: error: LLVM gold plugin: <unknown>:0: Undefined temporary symbol .Ltmp298032

clang: error: linker command failed with exit code 1 (use -v to see invocation)

maflcko · 2022-08-31T14:14:51Z

clang-13 seems to be working man_shrugging

clang-16 (16.0.0-++20220831042109+45c1ce321dce-1~~exp1~~20220831042209.392) as well

maflcko · 2022-09-01T15:19:26Z

Not sure what to do here. This is almost certainly a compiler bug, given that the behaviour is reproducible among some versions of clang, but not others.

maflcko added the Bug label Aug 30, 2022

fanquake added the Build system label Aug 30, 2022

maflcko changed the title ~~Segmentation fault when compiling with libfuzzer and lto~~ Segmentation fault when compiling with libfuzzer and lto (x86_64) Aug 31, 2022

maflcko mentioned this issue Aug 31, 2022

/usr/bin/ld.gold: fatal error: LLVM gold plugin: <unknown>:0: Undefined temporary symbol .Ltmp265928 ossf/fuzz-introspector#48

Closed

maflcko closed this as completed Sep 1, 2022

maflcko added the Upstream label Sep 1, 2022

This was referenced Sep 9, 2022

Seg fault when building with --fsanitize=introspector ossf/fuzz-introspector#504

Open

try clang-16 ossf/fuzz-introspector#507

Open

maflcko mentioned this issue Apr 20, 2023

bitcoin-core: enable LTO (take 2) google/oss-fuzz#10123

Merged

bitcoin locked and limited conversation to collaborators Sep 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Segmentation fault when compiling with libfuzzer and lto (x86_64) #25961

Segmentation fault when compiling with libfuzzer and lto (x86_64) #25961

maflcko commented Aug 30, 2022

maflcko commented Aug 30, 2022

maflcko commented Aug 30, 2022

maflcko commented Aug 31, 2022

fanquake commented Aug 31, 2022

maflcko commented Aug 31, 2022 •

edited

fanquake commented Aug 31, 2022

maflcko commented Aug 31, 2022 •

edited

fanquake commented Aug 31, 2022

maflcko commented Aug 31, 2022

maflcko commented Aug 31, 2022 •

edited

maflcko commented Aug 31, 2022

maflcko commented Sep 1, 2022

Segmentation fault when compiling with libfuzzer and lto (x86_64) #25961

Segmentation fault when compiling with libfuzzer and lto (x86_64) #25961

Comments

maflcko commented Aug 30, 2022

maflcko commented Aug 30, 2022

maflcko commented Aug 30, 2022

maflcko commented Aug 31, 2022

fanquake commented Aug 31, 2022

maflcko commented Aug 31, 2022 • edited

fanquake commented Aug 31, 2022

maflcko commented Aug 31, 2022 • edited

fanquake commented Aug 31, 2022

maflcko commented Aug 31, 2022

maflcko commented Aug 31, 2022 • edited

maflcko commented Aug 31, 2022

maflcko commented Sep 1, 2022

maflcko commented Aug 31, 2022 •

edited

maflcko commented Aug 31, 2022 •

edited

maflcko commented Aug 31, 2022 •

edited