Repository navigation
build: Introduce CMake-based build system - #30454
Conversation
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code CoverageFor detailed information about the code coverage, see the test coverage report. ReviewsSee the guideline for information on the review process.
If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
I'd say the section can be removed (or moved to a separate comment), because when this will be merged, I presume many more differences will accumulate. Even looking at the outstanding ports (https://github.com/bitcoin/bitcoin/issues?q=label%3A%22Needs+CMake+port%22+is%3Aclosed) right now, there are some. |
Missing link/reference? |
Thanks! Added. |
| // Distributed under the MIT software license, see the accompanying | ||
| // file COPYING or http://www.opensource.org/licenses/mit-license.php. | ||
|
|
||
| #include <config/bitcoin-config.h> // IWYU pragma: keep |
There was a problem hiding this comment.
In commit ff36846
Could you remind me what's going on here, please? I assume these aren't needed because we're meant to be adding compile definitions to specific libs. But from what I can tell, we're not actually adding them to sha256.cpp? So... is this currently using an un-optimized sha2?
Generally I'd feel more comfortable turning these into opt-outs, I think, to avoid accidentally building without optims.
For the sake of review/sanity, I think for this commit it's worth documenting (in the commit message) exactly which defines are affected and where in CMake they've been added.
There was a problem hiding this comment.
But from what I can tell, we're not actually adding them to sha256.cpp? So... is this currently using an un-optimized sha2?
Please note the PUBLIC keyword, which propagates the definition as a usage requirement:
bitcoin/src/crypto/CMakeLists.txt
Line 125 in 2b7c0f4
and
bitcoin/src/crypto/CMakeLists.txt
Line 128 in 2b7c0f4
where that usage requirement is actually applied.
The same approach works for other definitions used in src/crypto/sha256.cpp.
For the sake of review/sanity, I think for this commit it's worth documenting (in the commit message) exactly which defines are affected and where in CMake they've been added.
Sure thing! I'll update the commit message during the next branch update.
There was a problem hiding this comment.
One can see that they are indeed disabled now in https://corecheck.dev/bitcoin/bitcoin/pulls/30454 (coverage report + benchmarks).
Shouldn't autotools be nuked in this pull request, assuming that cmake and autotools are apparently incompatible, as designed now?
There was a problem hiding this comment.
Hmm https://corecheck.dev/bitcoin/bitcoin/pulls/30454 is down again
There was a problem hiding this comment.
Hmm https://corecheck.dev/bitcoin/bitcoin/pulls/30454 is down again
It is up now :)
There was a problem hiding this comment.
Hmm https://corecheck.dev/bitcoin/bitcoin/pulls/30454 is down again
It is up now :)
Nice. The output there looks better now. But I haven't yet tried locally.
Could add the 29.0 milestone? |
Done. |
e78551b doc: Fix typo in `build-unix.md` (Hennadii Stepanov) Pull request description: Addresses #30454 (comment). ACKs for top commit: davidgumberg: ACK e78551b l0rinc: ACK e78551b tdb3: ACK e78551b Tree-SHA512: a312a7718ac3fe4a9a204518f1306f4d4622b9c11e027f26960d15ac2e2380682604890a2ae4b874a28e0798f243f8226475cda3a8f468a1aa98df7ab26eea5f
|
ci msan fuzz looks broken, see #30760 |
ismaelsadeeq
left a comment
There was a problem hiding this comment.
Post-merge Tested ACK 4105129
Tested on M2 Pro, Sonoma 14.1.1
Clang version 18.1.7
CMake version 3.29.5
Configuration and compilation completed smoothly without any issues.
Started bitcoind without any issue and tested a few RPCs.
I also ran unit tests and benchmarks, all ran smoothly.
maflcko
left a comment
There was a problem hiding this comment.
Submitting my review comments, now that this has been merged.
| if(POLICY CMP0141) | ||
| # MSVC debug information format flags are selected by an abstraction. | ||
| # We want to use the CMAKE_MSVC_DEBUG_INFORMATION_FORMAT variable | ||
| # to select the MSVC debug information format. | ||
| cmake_policy(SET CMP0141 NEW) | ||
| endif() |
There was a problem hiding this comment.
nit in a2317e2: Maybe add a comment that this can be removed after cmake_minimum_required(VERSION 3.25)?
Alternatively, given that "The default installation of the latest version of MSVC 17.10.4 includes both CMake 3.28.3 and the vcpkg package manager", it may be easier to just require cmake >= 3.25 on MSVC and enforce the policy this way?
There was a problem hiding this comment.
nit in a2317e2: Maybe add a comment that this can be removed after
cmake_minimum_required(VERSION 3.25)?
It seems redundant to me. Since policies are tied to specific versions, it's generally understood that one should review every if(CMAKE_VERSION...) and if(POLICY...) statement in the code when updating the minimum required version.
Alternatively, given that "The default installation of the latest version of MSVC 17.10.4 includes both CMake 3.28.3 and the vcpkg package manager", it may be easier to just require cmake >= 3.25 on MSVC and enforce the policy this way?
The build system becomes aware that it is building on MSVC only after the project() call, which makes such an implementation unfeasible given modern CMake requirements (here and here).
Besides, the current implementation is idiomatic for enforcing individual policies.
The other alternative is to specify cmake_minimum_required(VERSION 3.22...3.25), but this has a drawback, as the build system's behavior depends on the actual CMake version.
There was a problem hiding this comment.
Apparently, there is no need to enable the CMP0141 policy, as we have not used the CMAKE_MSVC_DEBUG_INFORMATION_FORMAT variable since hebasto#215.
Fixed in #30803.
| if(MSVC) | ||
| try_append_cxx_flags("/WX /options:strict" VAR working_compiler_werror_flag SKIP_LINK) | ||
| else() | ||
| try_append_cxx_flags("-Werror" VAR working_compiler_werror_flag SKIP_LINK) |
There was a problem hiding this comment.
nit in commit 4a0af29 ("cmake: Add TryAppendCXXFlags module"):
could it make sense to mention that IF_CHECK_FAILED is never applied to the linker flags, whereas IF_CHECK_PASSED seems to be? (IF_CHECK_FAILED is unused dead code, so nothing blocking)
There was a problem hiding this comment.
The unused dead code has been deleted in #30803.
| # The ZeroMQ project has provided config files since v4.2.2. | ||
| # TODO: Switch to find_package(ZeroMQ) at some point in the future. |
There was a problem hiding this comment.
d2fda82: 4.2.2 Looks ancient, so maybe it can be dropped already?
There was a problem hiding this comment.
Unfortunately, packages in the main distros still lack CMake configuration files. For example, see https://packages.debian.org/sid/amd64/libzmq3-dev/filelist.
There was a problem hiding this comment.
Thanks. Makes sense. I guess the comment should be adjusted to say that this won't be possible to do any time soon, or at all, for the reason given by you?
|
|
||
| option(BUILD_BENCH "Build bench_bitcoin executable." OFF) | ||
| option(BUILD_FUZZ_BINARY "Build fuzz binary." OFF) | ||
| cmake_dependent_option(BUILD_FOR_FUZZING "Build for fuzzing. Enabling this will disable all other targets and override BUILD_FUZZ_BINARY." OFF "NOT MSVC" OFF) |
There was a problem hiding this comment.
3d85379: Why does this need special handling for msvc? What problem is this trying to solve?
There was a problem hiding this comment.
Lack of fuzzing engines for MSVC?
There was a problem hiding this comment.
BUILD_FOR_FUZZING has nothing to do with fuzz engines. According to the documentation it is an option to "Build for fuzzing. Enabling this will disable all other targets and override BUILD_FUZZ_BINARY."
Can you explain this a bit better? Maybe with an example output of a msvc compile error or otherwise steps to reproduce?
There was a problem hiding this comment.
BUILD_FOR_FUZZINGhas nothing to do with fuzz engines. According to the documentation it is an option to "Build for fuzzing. Enabling this will disable all other targets and override BUILD_FUZZ_BINARY."
You're right, there's no need for a special case for MSVC.
| gmake -C depends HOST=x86_64-w64-mingw32 # Use "-j N" for N parallel jobs. | ||
| cmake -B build --toolchain depends/x86_64-w64-mingw32/toolchain.cmake | ||
| cmake --build build # Use "-j N" for N parallel jobs. | ||
| ctest --test-dir build # Use "-j N" for N parallel tests. Some tests are disabled if Python 3 is not available. |
There was a problem hiding this comment.
You can't run exe on WSL without wine, can you?
There was a problem hiding this comment.
I don't recall installing Wine.
There was a problem hiding this comment.
Regardless of WSL, given these are the general instructions for any Linux, wine should be added as a requirement, otherwise the current instructions will fail:
gmake -C depends HOST=x86_64-w64-mingw32 -j12 NO_QT=1 NO_WALLET=1 NO_ZMQ=1 NO_NATPMP=1 NO_UPNP=1
cmake -B build --toolchain depends/x86_64-w64-mingw32/toolchain.cmake
cmake --build build
bitcoin# wine
bash: wine: command not found
bitcoin# wine64
bash: wine64: command not found
<snip>
ctest --test-dir build
0% tests passed, 117 tests failed out of 117There was a problem hiding this comment.
Thanks for testing @fanquake. Just to clarify: I left the review comment, because running the tests was added as part of this pull request, even though it wasn't present previously with autotools.
There was a problem hiding this comment.
I'd rather not include Wine installation instructions in this build doc. It is not a trivial process and could create an additional maintenance burden in the future.
it wasn't present previously with autotools.
Let's just remove it.
There was a problem hiding this comment.
Sure! Here are the steps to cross-compile for Windows and run unit test on a fresh Ubuntu 24.04 installation:
$ sudo apt install binfmt-support cmake curl git g++-mingw-w64-x86-64-posix make pkg-config wine-binfmt
$ sudo update-binfmts --enable wine
$ git clone https://github.com/bitcoin/bitcoin.git
$ cd bitcoin
$ gmake -C depends -j $(nproc) HOST=x86_64-w64-mingw32 NO_QT=1
$ cmake -B build --toolchain depends/x86_64-w64-mingw32/toolchain.cmake
$ cmake --build build -j $(nproc)
$ ctest --test-dir build -j $(nproc)
There was a problem hiding this comment.
Ok. That is what I did; so I'll have to retest. Although, above you said we'd have to do wine configuration etc, and that was a reason to not add it to the docs. However I don't see anything like that here?
There was a problem hiding this comment.
Although, above you said we'd have to do wine configuration etc, and that was a reason to not add it to the docs. However I don't see anything like that here?
$ sudo update-binfmts --enable wine
There was a problem hiding this comment.
I was under the impression that happened automatically on wine installation. Will also re-test.
There was a problem hiding this comment.
I was under the impression that happened automatically on wine installation.
I'm not sure, but it can depend on a distro.
| ) | ||
| if(test_suite_name) | ||
| add_test(NAME ${test_suite_name} | ||
| COMMAND test_bitcoin --run_test=${test_suite_name} --catch_system_error=no |
There was a problem hiding this comment.
ab2e99b: This is missing -l test_suite and DEBUG_LOG_OUT
| message("\n") | ||
| message("Configure summary") | ||
| message("=================") | ||
| message("C++ compiler .......................... ${CMAKE_CXX_COMPILER_ID} ${CMAKE_CXX_COMPILER_VERSION}, ${CMAKE_CXX_COMPILER}") |
There was a problem hiding this comment.
in commit a2317e2: This will only print the compiler name, not the flags.
E.g. -DCMAKE_CXX_COMPILER='clang++;-stdlib=libc++;-m32' will print the flags nowhere. (This is used in the ci/test/00_setup_env_i686_multiprocess.sh task)
|
Are there release notes for cmake transition? Seems like the autoconf -> cmake mapping table https://gist.github.com/hebasto/2ef97d3a726bfce08ded9df07f7dab5e would be especially helpful to have in release notes to help users and packagers convert previous configurations to new ones. |
1cc93fe build: Delete dead code that implements `IF_CHECK_FAILED` option (Hennadii Stepanov) 341ad23 build: Delete MSVC special case for `BUILD_FOR_FUZZING` option (Hennadii Stepanov) fdad128 build: Stop enabling CMake's CMP0141 policy (Hennadii Stepanov) b2a6f54 doc: Drop `ctest` command from Windows cross-compiling instructions (Hennadii Stepanov) 73b6185 build: Print `CMAKE_CXX_COMPILER_ARG1` in summary (Hennadii Stepanov) f03c942 build, test: Add missed log options (Hennadii Stepanov) 6f2cb0e doc: Amend comment about ZeroMQ config files (Hennadii Stepanov) Pull request description: This PR addresses the following comments: - #30454 (comment) - #30454 (comment) - #30454 (comment) - #30454 (comment) - #30454 (comment) - #30454 (comment) - #30454 (comment) ACKs for top commit: sipsorcery: tACK 1cc93fe (win11 msvc). maflcko: re-ACK 1cc93fe Tree-SHA512: a390797bb4d3b7eb9163653b6c9c324e7a01090f6cdda74df7349a24a5c4a2084e5912878747f56561315afc70cae9adb1c363f47ceb0af96004ea591d25171b
|
Unsure if a release note was written, apologies if I missed it. @hebasto @theuni @fanquake could you add to https://github.com/bitcoin-core/bitcoin-devwiki/wiki/29.0-Release-Notes-draft? |
Added. |
This PR introduces a new CMake-based build system, which is a drop-in replacement for the current Autotools-based build system.
ML announcement: https://groups.google.com/g/bitcoindev/c/hgKkfQWzrTo
As discussed during the recent CoreDev meetup in April, the switch from Autotools to CMake is intended to happen as soon as possible after branching 28.x off, which means that 29.0 will be built using CMake.
This PR branch is essentially the staging branch, with every change reviewed and tested by a group of contributors, including (in alphabetical order):
Reviewing in a separate staging repo was suggested in #27060 (comment).
The accompanying changes to the OSS-Fuzz project are available in hebasto/oss-fuzz#8.
Please refer to the build options parity table. The "auto" value is no longer available; non-default values must be specified explicitly. Additionally, the new default values have been chosen to suit the everyday build experience for the majority of developers.
System requirements for using the CMake-based build system:
A note for Windows users: The default installation of the latest version of MSVC 17.10.4 includes both CMake 3.28.3 and the vcpkg package manager).
We, the build system developers, kindly ask reviewers to refrain from making suggestions that are not directly related to the migration process or can be implemented separately. Bugs in the scripts and errors in the updated documentation should be the focus of this PR. Please be advised that comments not aligned with this PR's goal may be ignored.
Thank you all for your understanding.