Skip to content

build: Introduce CMake-based build system - #30454

Merged
fanquake merged 68 commits into
bitcoin:masterfrom
hebasto:240716-cmake
Aug 28, 2024
Merged

fanquake merged 68 commits into
bitcoin:masterfrom
hebasto:240716-cmake

Conversation

@hebasto

@hebasto hebasto commented Jul 16, 2024 •

Copy link
Copy Markdown
Member

This PR introduces a new CMake-based build system, which is a drop-in replacement for the current Autotools-based build system.

ML announcement: https://groups.google.com/g/bitcoindev/c/hgKkfQWzrTo

As discussed during the recent CoreDev meetup in April, the switch from Autotools to CMake is intended to happen as soon as possible after branching 28.x off, which means that 29.0 will be built using CMake.

This PR branch is essentially the staging branch, with every change reviewed and tested by a group of contributors, including (in alphabetical order):

Reviewing in a separate staging repo was suggested in #27060 (comment).

The accompanying changes to the OSS-Fuzz project are available in hebasto/oss-fuzz#8.

Please refer to the build options parity table. The "auto" value is no longer available; non-default values must be specified explicitly. Additionally, the new default values have been chosen to suit the everyday build experience for the majority of developers.

System requirements for using the CMake-based build system:

  • CMake >= 3.22 (if not available in your system's repository, it can be downloaded from https://cmake.org/download/)
  • a build tool of your choice:
    • any Make (GNU Make is no longer a requirement); GNU Make is still required to build depends
    • Ninja (https://ninja-build.org/)
    • MSBuild
    • Xcode

A note for Windows users: The default installation of the latest version of MSVC 17.10.4 includes both CMake 3.28.3 and the vcpkg package manager).


We, the build system developers, kindly ask reviewers to refrain from making suggestions that are not directly related to the migration process or can be implemented separately. Bugs in the scripts and errors in the updated documentation should be the focus of this PR. Please be advised that comments not aligned with this PR's goal may be ignored.

Thank you all for your understanding.

@DrahtBot

DrahtBot commented Jul 16, 2024 •

Copy link
Copy Markdown
Contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type Reviewers
ACK vasild, TheCharlatan, maflcko, sipsorcery, pablomartin4btc, i-am-yuvi, theuni, fanquake
Concept ACK Sjors
Stale ACK whitslack

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #30717 (rpc: Add test-only RPCs under -test=<option> flag by Prabhat1308)
  • #30685 (build: Mark x86_64-linux-gnu release binaries as CET-enabled by hebasto)
  • #30634 (ci: Use clang-19 from apt.llvm.org by maflcko)
  • #30619 (doc: Change nproc in docs to getconf _NPROCESSORS_ONLN by l0rinc)
  • #30527 (Bump python minimum supported version to 3.10 by maflcko)
  • #30465 (depends: Set CMAKE_SYSTEM_VERSION for CMake builds by hebasto)
  • #30043 (net: Replace libnatpmp with built-in PCP+NATPMP implementation by laanwj)
  • #29881 (guix: use GCC 13 to builds releases by fanquake)
  • #29868 (Reintroduce external signer support for Windows by hebasto)
  • #25573 ([POC] guix: produce a fully -static-pie bitcoind by fanquake)
  • #19461 (multiprocess: Add bitcoin-gui -ipcconnect option by ryanofsky)
  • #19460 (multiprocess: Add bitcoin-wallet -ipcconnect option by ryanofsky)
  • #10102 (Multiprocess bitcoin by ryanofsky)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@maflcko

maflcko commented Jul 16, 2024

Copy link
Copy Markdown
Member

The only differences from the staging branch are:

I'd say the section can be removed (or moved to a separate comment), because when this will be merged, I presume many more differences will accumulate. Even looking at the outstanding ports (https://github.com/bitcoin/bitcoin/issues?q=label%3A%22Needs+CMake+port%22+is%3Aclosed) right now, there are some.

@maflcko

maflcko commented Jul 16, 2024

Copy link
Copy Markdown
Member

Please refer to the build options parity table.

Missing link/reference?

@hebasto

hebasto commented Jul 16, 2024

Copy link
Copy Markdown
Member Author

Please refer to the build options parity table.

Missing link/reference?

Thanks! Added.

Comment thread src/crypto/sha256.cpp
// Distributed under the MIT software license, see the accompanying
// file COPYING or http://www.opensource.org/licenses/mit-license.php.

#include <config/bitcoin-config.h> // IWYU pragma: keep

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In commit ff36846

Could you remind me what's going on here, please? I assume these aren't needed because we're meant to be adding compile definitions to specific libs. But from what I can tell, we're not actually adding them to sha256.cpp? So... is this currently using an un-optimized sha2?

Generally I'd feel more comfortable turning these into opt-outs, I think, to avoid accidentally building without optims.

For the sake of review/sanity, I think for this commit it's worth documenting (in the commit message) exactly which defines are affected and where in CMake they've been added.

@hebasto hebasto Jul 16, 2024 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But from what I can tell, we're not actually adding them to sha256.cpp? So... is this currently using an un-optimized sha2?

Please note the PUBLIC keyword, which propagates the definition as a usage requirement:

target_compile_definitions(bitcoin_crypto_arm_shani PUBLIC ENABLE_ARM_SHANI)

and

target_link_libraries(bitcoin_crypto PRIVATE bitcoin_crypto_arm_shani)

where that usage requirement is actually applied.

The same approach works for other definitions used in src/crypto/sha256.cpp.

For the sake of review/sanity, I think for this commit it's worth documenting (in the commit message) exactly which defines are affected and where in CMake they've been added.

Sure thing! I'll update the commit message during the next branch update.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One can see that they are indeed disabled now in https://corecheck.dev/bitcoin/bitcoin/pulls/30454 (coverage report + benchmarks).

Shouldn't autotools be nuked in this pull request, assuming that cmake and autotools are apparently incompatible, as designed now?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please see hebasto#269.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@theuni @maflcko

Can you confirm please that the recent push resolved this issue?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm https://corecheck.dev/bitcoin/bitcoin/pulls/30454 is down again

It is up now :)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm https://corecheck.dev/bitcoin/bitcoin/pulls/30454 is down again

It is up now :)

Nice. The output there looks better now. But I haven't yet tried locally.

Comment thread src/crypto/CMakeLists.txt Resolved
@hebasto

hebasto commented Jul 16, 2024

Copy link
Copy Markdown
Member Author

A couple of functional tests-related commits have been split to #30463, as suggested by @fanquake offline.

@maflcko

maflcko commented Jul 17, 2024

Copy link
Copy Markdown
Member

which means that 29.0 will be built using CMake.

Could add the 29.0 milestone?

@hebasto hebasto added this to the 29.0 milestone Jul 17, 2024
@hebasto

hebasto commented Jul 17, 2024

Copy link
Copy Markdown
Member Author

which means that 29.0 will be built using CMake.

Could add the 29.0 milestone?

Done.

@hebasto
hebasto deleted the 240716-cmake branch August 28, 2024 20:07
fanquake added a commit that referenced this pull request Aug 29, 2024
e78551b doc: Fix typo in `build-unix.md` (Hennadii Stepanov)

Pull request description:

  Addresses #30454 (comment).

ACKs for top commit:
  davidgumberg:
    ACK e78551b
  l0rinc:
    ACK e78551b
  tdb3:
    ACK e78551b

Tree-SHA512: a312a7718ac3fe4a9a204518f1306f4d4622b9c11e027f26960d15ac2e2380682604890a2ae4b874a28e0798f243f8226475cda3a8f468a1aa98df7ab26eea5f
@maflcko

maflcko commented Aug 30, 2024

Copy link
Copy Markdown
Member

ci msan fuzz looks broken, see #30760

@ismaelsadeeq ismaelsadeeq left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Post-merge Tested ACK 4105129

Tested on M2 Pro, Sonoma 14.1.1

Clang version 18.1.7

CMake version 3.29.5

Configuration and compilation completed smoothly without any issues.

Started bitcoind without any issue and tested a few RPCs.

I also ran unit tests and benchmarks, all ran smoothly.

@maflcko maflcko left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Submitting my review comments, now that this has been merged.

Comment thread CMakeLists.txt
Comment on lines +11 to +16
if(POLICY CMP0141)
# MSVC debug information format flags are selected by an abstraction.
# We want to use the CMAKE_MSVC_DEBUG_INFORMATION_FORMAT variable
# to select the MSVC debug information format.
cmake_policy(SET CMP0141 NEW)
endif()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit in a2317e2: Maybe add a comment that this can be removed after cmake_minimum_required(VERSION 3.25)?

Alternatively, given that "The default installation of the latest version of MSVC 17.10.4 includes both CMake 3.28.3 and the vcpkg package manager", it may be easier to just require cmake >= 3.25 on MSVC and enforce the policy this way?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit in a2317e2: Maybe add a comment that this can be removed after cmake_minimum_required(VERSION 3.25)?

It seems redundant to me. Since policies are tied to specific versions, it's generally understood that one should review every if(CMAKE_VERSION...) and if(POLICY...) statement in the code when updating the minimum required version.

Alternatively, given that "The default installation of the latest version of MSVC 17.10.4 includes both CMake 3.28.3 and the vcpkg package manager", it may be easier to just require cmake >= 3.25 on MSVC and enforce the policy this way?

The build system becomes aware that it is building on MSVC only after the project() call, which makes such an implementation unfeasible given modern CMake requirements (here and here).

Besides, the current implementation is idiomatic for enforcing individual policies.

The other alternative is to specify cmake_minimum_required(VERSION 3.22...3.25), but this has a drawback, as the build system's behavior depends on the actual CMake version.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apparently, there is no need to enable the CMP0141 policy, as we have not used the CMAKE_MSVC_DEBUG_INFORMATION_FORMAT variable since hebasto#215.

Fixed in #30803.

if(MSVC)
try_append_cxx_flags("/WX /options:strict" VAR working_compiler_werror_flag SKIP_LINK)
else()
try_append_cxx_flags("-Werror" VAR working_compiler_werror_flag SKIP_LINK)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit in commit 4a0af29 ("cmake: Add TryAppendCXXFlags module"):

could it make sense to mention that IF_CHECK_FAILED is never applied to the linker flags, whereas IF_CHECK_PASSED seems to be? (IF_CHECK_FAILED is unused dead code, so nothing blocking)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The unused dead code has been deleted in #30803.

Comment thread src/CMakeLists.txt Resolved
Comment thread CMakeLists.txt
Comment on lines +124 to +125
# The ZeroMQ project has provided config files since v4.2.2.
# TODO: Switch to find_package(ZeroMQ) at some point in the future.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

d2fda82: 4.2.2 Looks ancient, so maybe it can be dropped already?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, packages in the main distros still lack CMake configuration files. For example, see https://packages.debian.org/sid/amd64/libzmq3-dev/filelist.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Makes sense. I guess the comment should be adjusted to say that this won't be possible to do any time soon, or at all, for the reason given by you?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in #30803.

Comment thread CMakeLists.txt

option(BUILD_BENCH "Build bench_bitcoin executable." OFF)
option(BUILD_FUZZ_BINARY "Build fuzz binary." OFF)
cmake_dependent_option(BUILD_FOR_FUZZING "Build for fuzzing. Enabling this will disable all other targets and override BUILD_FUZZ_BINARY." OFF "NOT MSVC" OFF)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3d85379: Why does this need special handling for msvc? What problem is this trying to solve?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lack of fuzzing engines for MSVC?

@maflcko maflcko Sep 3, 2024 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BUILD_FOR_FUZZING has nothing to do with fuzz engines. According to the documentation it is an option to "Build for fuzzing. Enabling this will disable all other targets and override BUILD_FUZZ_BINARY."

Can you explain this a bit better? Maybe with an example output of a msvc compile error or otherwise steps to reproduce?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BUILD_FOR_FUZZING has nothing to do with fuzz engines. According to the documentation it is an option to "Build for fuzzing. Enabling this will disable all other targets and override BUILD_FUZZ_BINARY."

You're right, there's no need for a special case for MSVC.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in #30803.

Comment thread cmake/module/Maintenance.cmake Resolved
Comment thread doc/build-windows.md Resolved
Comment thread doc/build-windows.md
gmake -C depends HOST=x86_64-w64-mingw32 # Use "-j N" for N parallel jobs.
cmake -B build --toolchain depends/x86_64-w64-mingw32/toolchain.cmake
cmake --build build # Use "-j N" for N parallel jobs.
ctest --test-dir build # Use "-j N" for N parallel tests. Some tests are disabled if Python 3 is not available.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can't run exe on WSL without wine, can you?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't recall installing Wine.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regardless of WSL, given these are the general instructions for any Linux, wine should be added as a requirement, otherwise the current instructions will fail:

gmake -C depends HOST=x86_64-w64-mingw32 -j12 NO_QT=1 NO_WALLET=1 NO_ZMQ=1 NO_NATPMP=1 NO_UPNP=1
cmake -B build --toolchain depends/x86_64-w64-mingw32/toolchain.cmake
cmake --build build
bitcoin# wine
bash: wine: command not found
bitcoin# wine64
bash: wine64: command not found
<snip>
ctest --test-dir build
0% tests passed, 117 tests failed out of 117

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for testing @fanquake. Just to clarify: I left the review comment, because running the tests was added as part of this pull request, even though it wasn't present previously with autotools.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather not include Wine installation instructions in this build doc. It is not a trivial process and could create an additional maintenance burden in the future.

it wasn't present previously with autotools.

Let's just remove it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure! Here are the steps to cross-compile for Windows and run unit test on a fresh Ubuntu 24.04 installation:

$ sudo apt install binfmt-support cmake curl git g++-mingw-w64-x86-64-posix make pkg-config wine-binfmt
$ sudo update-binfmts --enable wine
$ git clone https://github.com/bitcoin/bitcoin.git
$ cd bitcoin
$ gmake -C depends -j $(nproc) HOST=x86_64-w64-mingw32 NO_QT=1
$ cmake -B build --toolchain depends/x86_64-w64-mingw32/toolchain.cmake
$ cmake --build build -j $(nproc)
$ ctest --test-dir build -j $(nproc)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. That is what I did; so I'll have to retest. Although, above you said we'd have to do wine configuration etc, and that was a reason to not add it to the docs. However I don't see anything like that here?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although, above you said we'd have to do wine configuration etc, and that was a reason to not add it to the docs. However I don't see anything like that here?

$ sudo update-binfmts --enable wine

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was under the impression that happened automatically on wine installation. Will also re-test.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was under the impression that happened automatically on wine installation.

I'm not sure, but it can depend on a distro.

Comment thread src/test/CMakeLists.txt
)
if(test_suite_name)
add_test(NAME ${test_suite_name}
COMMAND test_bitcoin --run_test=${test_suite_name} --catch_system_error=no

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ab2e99b: This is missing -l test_suite and DEBUG_LOG_OUT

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in #30803.

Comment thread CMakeLists.txt
message("\n")
message("Configure summary")
message("=================")
message("C++ compiler .......................... ${CMAKE_CXX_COMPILER_ID} ${CMAKE_CXX_COMPILER_VERSION}, ${CMAKE_CXX_COMPILER}")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in commit a2317e2: This will only print the compiler name, not the flags.

E.g. -DCMAKE_CXX_COMPILER='clang++;-stdlib=libc++;-m32' will print the flags nowhere. (This is used in the ci/test/00_setup_env_i686_multiprocess.sh task)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in #30803.

@ryanofsky

ryanofsky commented Sep 10, 2024 •

Copy link
Copy Markdown
Contributor

Are there release notes for cmake transition? Seems like the autoconf -> cmake mapping table https://gist.github.com/hebasto/2ef97d3a726bfce08ded9df07f7dab5e would be especially helpful to have in release notes to help users and packagers convert previous configurations to new ones.

fanquake added a commit that referenced this pull request Sep 12, 2024
1cc93fe build: Delete dead code that implements `IF_CHECK_FAILED` option (Hennadii Stepanov)
341ad23 build: Delete MSVC special case for `BUILD_FOR_FUZZING` option (Hennadii Stepanov)
fdad128 build: Stop enabling CMake's CMP0141 policy (Hennadii Stepanov)
b2a6f54 doc: Drop `ctest` command from Windows cross-compiling instructions (Hennadii Stepanov)
73b6185 build: Print `CMAKE_CXX_COMPILER_ARG1` in summary (Hennadii Stepanov)
f03c942 build, test: Add missed log options (Hennadii Stepanov)
6f2cb0e doc: Amend comment about ZeroMQ config files (Hennadii Stepanov)

Pull request description:

  This PR addresses the following comments:
  - #30454 (comment)
  - #30454 (comment)
  - #30454 (comment)
  - #30454 (comment)
  - #30454 (comment)
  - #30454 (comment)
  - #30454 (comment)

ACKs for top commit:
  sipsorcery:
    tACK 1cc93fe (win11 msvc).
  maflcko:
    re-ACK 1cc93fe

Tree-SHA512: a390797bb4d3b7eb9163653b6c9c324e7a01090f6cdda74df7349a24a5c4a2084e5912878747f56561315afc70cae9adb1c363f47ceb0af96004ea591d25171b
@hebasto hebasto mentioned this pull request Oct 15, 2024
16 tasks done
@glozow

glozow commented Mar 3, 2025

Copy link
Copy Markdown
Contributor

Unsure if a release note was written, apologies if I missed it. @hebasto @theuni @fanquake could you add to https://github.com/bitcoin-core/bitcoin-devwiki/wiki/29.0-Release-Notes-draft?

@hebasto

hebasto commented Mar 7, 2025

Copy link
Copy Markdown
Member Author

Unsure if a release note was written, apologies if I missed it. @hebasto @theuni @fanquake could you add to https://github.com/bitcoin-core/bitcoin-devwiki/wiki/29.0-Release-Notes-draft?

Added.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

No open projects
Status: 🚧 CMake

Development

Successfully merging this pull request may close these issues.