New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bootstrappable Builds #8929
base: master
Are you sure you want to change the base?
Bootstrappable Builds #8929
Conversation
09e44ca
to
ee3964a
Compare
I cannot wait until we finalize and merge this. Thanks for your great work @tobtoht |
7ff7aca
to
008cf5e
Compare
e733f5f
to
a10037f
Compare
I don't expect to make major changes to this PR, it's ready for testing. Reviews on any of the sub-PRs would help move this forward. |
#!/bin/bash | ||
if [ "$1" != "-cc1" ]; then | ||
- `dirname $0`/clang{version} {flags} "$@" | ||
+ env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH -u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH -u LIBRARY_PATH `dirname $0`/clang{version} {flags} "$@" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need to unset these environment variables here because they refer to Guix profile packages. We want clang to only include headers from the NDK. We can't unset them in build.sh
because native_
package builds would fail to find the necessary include headers. Unsetting these variables does not affect non-Guix builds.
+ #if $(real-version) < "4.0.0" | ||
+ #{ | ||
+ # flags darwin.compile.c++ OPTIONS $(condition) : -fcoalesce-templates ; | ||
+ #} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Boost 1.64.0 doesn't recognize that we're building with Clang and passes a flags that results in an error. We don't support GCC < 4.0 at all, so commenting out the lines here is fine. Patch can be dropped when we update Boost.
@@ -0,0 +1,76 @@ | |||
Note that this has been modified from the original commit, to use __has_include |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
riscv64-linux-gnu
builds fail without this patch
@@ -0,0 +1,22 @@ | |||
Without ffile-prefix-map, the debug symbols will contain paths for the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Building on aarch64 should produce bit-for-bit identical release binaries.
gcc-toolchain-10 | ||
(list gcc-toolchain-10 "static") | ||
;; Scripting | ||
perl |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
perl
is needed to build OpenSSL.
[ -e /usr/bin/env ] || ln -s --no-dereference "$(command -v env)" /usr/bin/env | ||
[ -e /bin/bash ] || ln -s --no-dereference "$(command -v bash)" /bin/bash | ||
[ -e /bin/sh ] || ln -s --no-dereference "$(command -v sh)" /bin/sh | ||
[ -e /lib64/ld-linux-x86-64.so.2 ] || ln -s --no-dereference "${NATIVE_GCC}/lib/ld-linux-x86-64.so.2" /lib64/ld-linux-x86-64.so.2 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The Android NDK toolchain cannot (yet) be bootstrapped. The compiler binaries included in the NDK have their dynamic interpreter set to the standard x86_64 interpreter path, which does not exist in this location in the Guix environment.
The alternative was patchelf
-ing all binaries included in the NDK, but this is hacky and adds a dependency on patchelf
for non-Guix builders.
if [[ -n "$DEPENDS_ONLY" ]]; then | ||
exit 0 | ||
fi |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is useful when debugging reproducibility issues in depends packages. Skips ahead to the next target, so we don't spend time building Monero binaries.
contrib/gitian/docker/ | ||
contrib/gitian/sigs/ | ||
# guix | ||
/guix |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Finished builds are available in ./guix/guix-build-<COMMIT>/
├── guix-build-222ea1a30058
│ ├── build
│ │ └── distsrc-222ea1a30058-x86_64-linux-gnu # build directory
│ ├── logs
│ │ └── x86_64-linux-gnu # various logs for reproducibility debugging
│ │ ├── depends-gen_id-build.txt
│ │ ├── depends-gen_id-host.txt
│ │ ├── depends-hashes.txt
│ │ ├── depends-packages.txt
│ │ ├── guix-env.txt
│ │ ├── guix-hashes.txt
│ │ └── SHA256SUMS.part
│ ├── output
│ │ └── x86_64-linux-gnu
│ │ └── monero-x86_64-linux-gnu-222ea1a30058.tar.bz2 # binary tarball
│ └── var
│ ├── precious_dirs # directories to keep when running `./contrib/guix/guix-clean`
│ └── profiles
@@ -267,4 +275,4 @@ $(foreach package,$(all_packages),$(eval $(call int_config_attach_build_config,$ | |||
$(foreach package,$(all_packages),$(eval $(call int_add_cmds,$(package)))) | |||
|
|||
#special exception: if a toolchain package exists, all non-native packages depend on it | |||
$(foreach package,$(packages),$(eval $($(package)_unpacked): |$($($(host_arch)_$(host_os)_native_toolchain)_cached) )) | |||
$(foreach package,$(packages),$(eval $($(package)_extracted): |$($($(host_arch)_$(host_os)_native_toolchain)_cached) )) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
_unpacked
target doesn't exist.
gawk | ||
sed | ||
moreutils | ||
patchelf |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
patchelf
is unused. It is occasionally useful for debugging.
esac | ||
|
||
export GIT_DISCOVERY_ACROSS_FILESYSTEM=1 | ||
export USE_DEVICE_TREZOR_MANDATORY=1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Force Trezor support for release binaries.
$(1)_cc=$$($$($(1)_type)_CC) | ||
$(1)_cxx=$$($$($(1)_type)_CXX) | ||
$(1)_objc=$$($$($(1)_type)_OBJC) | ||
$(1)_objcxx=$$($$($(1)_type)_OBJCXX) | ||
$(1)_ar=$$($$($(1)_type)_AR) | ||
$(1)_ranlib=$$($$($(1)_type)_RANLIB) | ||
$(1)_libtool=$$($$($(1)_type)_LIBTOOL) | ||
$(1)_nm=$$($$($(1)_type)_NM) | ||
$(1)_cflags=$$($$($(1)_type)_CFLAGS) \ | ||
$$($$($(1)_type)_$$(release_type)_CFLAGS) | ||
$(1)_cxxflags=$$($$($(1)_type)_CXXFLAGS) \ | ||
$$($$($(1)_type)_$$(release_type)_CXXFLAGS) | ||
$(1)_arflags=$$($$($(1)_type)_ARFLAGS) \ | ||
$$($$($(1)_type)_$(release_type)_ARFLAGS) | ||
$(1)_ldflags=$$($$($(1)_type)_LDFLAGS) \ | ||
$$($$($(1)_type)_$$(release_type)_LDFLAGS) \ | ||
-L$$($($(1)_type)_prefix)/lib | ||
$(1)_cppflags=$$($$($(1)_type)_CPPFLAGS) \ | ||
$$($$($(1)_type)_$$(release_type)_CPPFLAGS) \ | ||
-I$$($$($(1)_type)_prefix)/include |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Delay expansion of package variables.
.github/workflows/guix.yml
Outdated
@@ -0,0 +1,63 @@ | |||
name: ci/gh-actions/guix | |||
|
|||
on: [push, pull_request] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I removed guix and depends package caching from the CI workflow because it uses too much space and causes evictions for other caches (10 GB limit). We only need to run it when there is a change to contrib/{depends,guix}
, which happens infrequently and doesn't need to complete quickly.
on: [push, pull_request]
is temporary for testing.
path: contrib/depends/sources | ||
key: sources-${{ hashFiles('contrib/depends/packages/*') }} | ||
- name: install dependencies | ||
run: sudo apt update; sudo apt -y install guix git ca-certificates |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It may be worth investigating if using the installer script is faster.
(("-rpath=") "-rpath-link=")) | ||
#t)))))))) | ||
|
||
(define-public glibc-2.27 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The minimum glibc version for all Linux targets is now 2.27 (Ubuntu 18.04, Debian 10).
For more context, see: #9171 (comment)
-DCMAKE_EXE_LINKER_FLAGS="${HOST_LDFLAGS}" \ | ||
-DCMAKE_SHARED_LINKER_FLAGS="${HOST_LDFLAGS}" \ | ||
-DCMAKE_SKIP_RPATH=ON \ | ||
-DMANUAL_SUBMODULES=1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can't actually check if submodules are checked out here because we're building in an extracted source archive.
The guix-build
script runs git submodule update --init --recursive --progress
before starting a build to make sure submodules are checked out.
886f037
to
4398fc7
Compare
@@ -6,5 +6,6 @@ $(package)_sha256_hash=df75d30ecafc429e905134333aeae56ac65fac67cb4182622398fd717 | |||
|
|||
define $(package)_stage_cmds | |||
mkdir -p $($(package)_staging_dir)/$(host_prefix)/native/SDK &&\ | |||
rm -rf usr/include/readline && \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Prevent clang from including readline headers from the SDK. We statically link our own version of readline.
SET(CMAKE_C_COMPILER @prefix@/native/bin/clang) | ||
SET(CMAKE_C_COMPILER @CC@) | ||
SET(CMAKE_C_COMPILER_TARGET ${CLANG_TARGET}) | ||
SET(CMAKE_C_FLAGS_INIT -B${_CMAKE_TOOLCHAIN_PREFIX}) | ||
SET(CMAKE_CXX_COMPILER @prefix@/native/bin/clang++ -stdlib=libc++) | ||
SET(CMAKE_CXX_COMPILER @CXX@ -stdlib=libc++) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We're now using the system / guix-provided clang for darwin builds.
SET(CMAKE_ASM_COMPILER clang) | ||
SET(CMAKE_ASM-ATT_COMPILER as) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CMake automatically reduces CMAKE_CXX_COMPILER
to the first command to derive CMAKE_ASM_COMPILER
. This happens to be env
instead of clang
, so we need to set it explicitly.
unset LIBRARY_PATH | ||
unset CPATH | ||
unset C_INCLUDE_PATH | ||
unset CPLUS_INCLUDE_PATH | ||
unset OBJC_INCLUDE_PATH | ||
unset OBJCPLUS_INCLUDE_PATH |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These are automatically set by Guix, but don't necessarily point to the correct paths. This is fixed below.
final_build_id_long+=$($(package)_build_id_long) | ||
final_build_id_long+=:[recipe]:$(1)-$($(1)_version)-$($(1)_recipe_hash)-$(release_type):[deps]$(foreach dep,$($(1)_build_id_deps),$(shell echo ":$(dep)")):[$($(1)_type)_id]:$($($(1)_type)_id_string): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Increase verbosity of final_build_id_long
to ease debugging reproducibility issues.
build_id_string:=$(realpath $(GUIX_ENVIRONMENT)) | ||
$(host_arch)_$(host_os)_id_string:=$(realpath $(GUIX_ENVIRONMENT)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any change to manifest.scm
invalidates the cache.
|
hashes
|
I recently designed a process for gitian-building entirely in RAM with only 12GB total space required, significantly less than:
Could Guix build results be made to match those generated by gitian? |
@kevcrumb The figures in the readme are a bit off, I'll update them.
If target specific build/cache directories are removed after each successfully built target, about 13.1 GB of storage is required at any time, not counting a base VM image with |
On a 16 GB system that would leave a mere 2.9 of RAM for the host's OS and the entire build process combined. Anyway, my question was rather if we could get matching checksums between gitian and Guix, so that both methods could be employed interchangeably. (dropping the term "reproducible builds" here and a mention of #1854, as I always struggle to find this issue) |
No, this is not possible. We can't recreate the Gitian build environment in Guix. Even if we could, it would be unmaintainable and too limiting. |
This PR proposes to replace Gitian with Guix to achieve bootstrappable builds for release binaries.
This is a monolithic PR for demo / testing purposes. To ease review, any required changes to
depends
will be submitted in separate PRs (unless they are not compatible with Gitian).What?
https://github.com/Sjors/nado-book/blob/master/attacks/guix.md
https://youtube.com/watch?v=I2iShmUTEl8
If you have 15 minutes, please consider watching the presentation by Carl Dong linked above (YouTube video)
as it covers the exact migration (Gitian -> Guix) this PR proposes to make.
Why?
How to test?
See README.md
Notes
d5ca4d4fd713a9f7e17e074a1e37dda99bbb09fc
(Dec 7, 2023)I removed guix and depends package caching from the
guix.yml
workflow because it uses too much space and causes evictions for other caches (10 GB limit). We only need to run it when there is a change tocontrib/{depends,guix}
, which happens infrequently and doesn't need to complete quickly.The guix timemachine URL is set to https://github.com/tobtoht/guix.git because the official repo on git.savannah.gnu.org often fails to fetch during CI runs for whatever reason. There used to be a mirror at guix-mirror/guix, but it is now out of service. Reviewers ought to confirm the pinned commit is a legit hash.
This PR removes
native_clang
because Guix provides us with a bootstrapped package and non-guix builds can use the system clang. This PR introduces backwards incompatible compiler flags, bumping the minimum Clang version to 10 (up from 9) for depends builds. Clang >=10 is available in the package managers of all supported distros (including Ubuntu 18.04).Sub-PRs
Follow-up PRs:
To-do
Notable changes
Where to start review?
contrib/guix/README.md
->
contrib/guix/guix-build
->
contrib/guix/manifest.scm
->
contrib/guix/libexec/build.sh
Status: Ready for testing