Skip to content

Commit

Permalink
ci: build Valgrind (3.21) from source
Browse files Browse the repository at this point in the history
I've been using this branch for some time, for working Valgrind CI jobs
on aarch64. Benefits include:
* Valgrind CI jobs work across x86_64 & aarch64.
* Can use latest (hopefully less buggy) Valgrind, rather than whatever
  the distro happens to package.
* No need to "bless" a specific compiler for use with Valgrind, (current
  discussion includes switching from Clang to GCC).
* Valgrind from source seems to run significantly faster compared to running
  the system package. i.e, when fuzzing under Valgrind:

Master:
```bash
asmap_direct with args
Done 646 runs in 155 second(s)
....
addrman_deserialize with args
Done 2944 runs in 2875 second(s)
```

vs running this branch:
```bash
asmap_direct with args
Done 646 runs in 23 second(s)
...
addrman_deserialize with args
Done 2944 runs in 413 second(s)
```

This is also being seen in the qa-assets repo: bitcoin-core/qa-assets#136 (comment).
For example, the `descriptor_parse` target under Valgrind currently takes:
`Done 6304 runs in 12971 second(s)`
however [with this branch, it takes](https://cirrus-ci.com/task/4623075795271680?):
`Done 6304 runs in 4609 second(s)`.

Running the native_valgrind CI (master, aarch64):
```bash
test/sighash_tests.cpp(120): Entering test case "sighash_test"
==21957== Source and destination overlap in memcpy(0x871e4b0, 0x871e4b0, 36)
==21957==    at 0x488CFA0: __GI_memcpy (in /usr/libexec/valgrind/vgpreload_memcheck-arm64-linux.so)
==21957==    by 0x8F7F63: CTxIn::operator=(CTxIn const&) (transaction.h:74)
==21957==    by 0x93F96B: SignatureHashOld(CScript, CTransaction const&, unsigned int, int) (sighash_tests.cpp:76)
==21957==    by 0x93EF1F: sighash_tests::sighash_test::test_method() (sighash_tests.cpp:138)
==21957==    by 0x93EB73: sighash_tests::sighash_test_invoker() (sighash_tests.cpp:120)
==21957==    by 0x36CF47: boost::detail::function::void_function_invoker0<void (*)(), void>::invoke(boost::detail::function::function_buffer&) (function_template.hpp:117)
==21957==    by 0x25B367: boost::function0<void>::operator()() const (function_template.hpp:763)
==21957==    by 0x2D6647: boost::detail::forward::operator()() (execution_monitor.ipp:1388)
==21957==    by 0x2D627F: boost::detail::function::function_obj_invoker0<boost::detail::forward, int>::invoke(boost::detail::function::function_buffer&) (function_template.hpp:137)
==21957==    by 0x2D0393: boost::function0<int>::operator()() const (function_template.hpp:763)
==21957==    by 0x234A6B: int boost::detail::do_invoke<boost::shared_ptr<boost::detail::translator_holder_base>, boost::function<int ()> >(boost::shared_ptr<boost::detail::translator_holder_base> const&, boost::function<int ()> const&) (execution_monitor.ipp:301)
==21957==    by 0x1F7277: boost::execution_monitor::catch_signals(boost::function<int ()> const&) (execution_monitor.ipp:903)
==21957==
{
   <insert_a_suppression_name_here>
   Memcheck:Overlap
   fun:__GI_memcpy
   fun:_ZN5CTxInaSERKS_
   fun:_ZL16SignatureHashOld7CScriptRK12CTransactionji
   fun:_ZN13sighash_tests12sighash_test11test_methodEv
   fun:_ZN13sighash_testsL20sighash_test_invokerEv
   fun:_ZN5boost6detail8function22void_function_invoker0IPFvvEvE6invokeERNS1_15function_bufferE
   fun:_ZNK5boost9function0IvEclEv
   fun:_ZN5boost6detail7forwardclEv
   fun:_ZN5boost6detail8function21function_obj_invoker0INS0_7forwardEiE6invokeERNS1_15function_bufferE
   fun:_ZNK5boost9function0IiEclEv
   fun:_ZN5boost6detail9do_invokeINS_10shared_ptrINS0_22translator_holder_baseEEENS_8functionIFivEEEEEiRKT_RKT0_
   fun:_ZN5boost17execution_monitor13catch_signalsERKNS_8functionIFivEEE
}
```

vs running this branch:
```bash
real	118m55.057s
```

Disadvantages includes:
* Becoming slightly more of a package manager in the CI.

Related to the discussion in
bitcoin#27444.
  • Loading branch information
fanquake committed Jul 4, 2023
1 parent f08d914 commit 83266cd
Show file tree
Hide file tree
Showing 3 changed files with 11 additions and 6 deletions.
6 changes: 3 additions & 3 deletions ci/test/00_setup_env_native_fuzz_with_valgrind.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,13 @@ export LC_ALL=C.UTF-8

export CI_IMAGE_NAME_TAG="debian:bookworm"
export CONTAINER_NAME=ci_native_fuzz_valgrind
export PACKAGES="clang llvm libclang-rt-dev libevent-dev libboost-dev libsqlite3-dev valgrind"
export PACKAGES="libc6-dbg clang llvm libclang-rt-dev libevent-dev libboost-dev libsqlite3-dev"
export BUILD_VALGRIND="true"
export NO_DEPENDS=1
export RUN_UNIT_TESTS=false
export RUN_FUNCTIONAL_TESTS=false
export RUN_FUZZ_TESTS=true
export FUZZ_TESTS_CONFIG="--valgrind"
export GOAL="install"
# Temporarily pin dwarf 4, until using Valgrind 3.20 or later
export BITCOIN_CONFIG="--enable-fuzz --with-sanitizers=fuzzer CC=clang CXX=clang++ CFLAGS='-gdwarf-4' CXXFLAGS='-gdwarf-4'"
export BITCOIN_CONFIG="--enable-fuzz --with-sanitizers=fuzzer CC=clang CXX=clang++"
export CCACHE_SIZE=200M
6 changes: 3 additions & 3 deletions ci/test/00_setup_env_native_valgrind.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,10 @@ export LC_ALL=C.UTF-8

export CI_IMAGE_NAME_TAG="debian:bookworm"
export CONTAINER_NAME=ci_native_valgrind
export PACKAGES="valgrind clang llvm libclang-rt-dev python3-zmq libevent-dev libboost-dev libdb5.3++-dev libminiupnpc-dev libnatpmp-dev libzmq3-dev libsqlite3-dev"
export PACKAGES="libc6-dbg clang llvm libclang-rt-dev python3-zmq libevent-dev libboost-dev libdb5.3++-dev libminiupnpc-dev libnatpmp-dev libzmq3-dev libsqlite3-dev"
export BUILD_VALGRIND="true"
export USE_VALGRIND=1
export NO_DEPENDS=1
export TEST_RUNNER_EXTRA="--exclude feature_init,rpc_bind,feature_bind_extra" # Excluded for now, see https://github.com/bitcoin/bitcoin/issues/17765#issuecomment-602068547
export GOAL="install"
# Temporarily pin dwarf 4, until using Valgrind 3.20 or later
export BITCOIN_CONFIG="--enable-zmq --with-incompatible-bdb --with-gui=no CC=clang CXX=clang++ CFLAGS='-gdwarf-4' CXXFLAGS='-gdwarf-4'" # TODO enable GUI
export BITCOIN_CONFIG="--enable-zmq --with-incompatible-bdb --with-gui=no CC=clang CXX=clang++" # TODO enable GUI
5 changes: 5 additions & 0 deletions ci/test/01_base_install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,11 @@ if [ -n "$PIP_PACKAGES" ]; then
fi
fi

if [[ ${BUILD_VALGRIND} == "true" ]]; then
git clone --depth=1 https://sourceware.org/git/valgrind.git -b VALGRIND_3_21_0 "${BASE_SCRATCH_DIR}"/valgrind
cd "${BASE_SCRATCH_DIR}"/valgrind/ && ./autogen.sh && ./configure --prefix=/usr && make install "$MAKEJOBS"
fi

if [[ ${USE_MEMORY_SANITIZER} == "true" ]]; then
git clone --depth=1 https://github.com/llvm/llvm-project -b llvmorg-16.0.6 "${BASE_SCRATCH_DIR}"/msan/llvm-project

Expand Down

0 comments on commit 83266cd

Please sign in to comment.