Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bootstrappable Builds #8929

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open

Bootstrappable Builds #8929

wants to merge 9 commits into from

Conversation

tobtoht
Copy link
Contributor

@tobtoht tobtoht commented Jul 2, 2023

This PR proposes to replace Gitian with Guix to achieve bootstrappable builds for release binaries.

This is a monolithic PR for demo / testing purposes. To ease review, any required changes to depends will be submitted in separate PRs (unless they are not compatible with Gitian).

What?

https://github.com/Sjors/nado-book/blob/master/attacks/guix.md
https://youtube.com/watch?v=I2iShmUTEl8

If you have 15 minutes, please consider watching the presentation by Carl Dong linked above (YouTube video)
as it covers the exact migration (Gitian -> Guix) this PR proposes to make.

Why?

  • Bootstrappability allows us to audit and reproduce our toolchain instead of blindly trusting binary downloads, thus providing better binary security guarantees.
  • Gitian is in maintenance mode.
  • Easier to set up and run than Gitian, because you only need to install Guix. This hopefully leads to more participants in the pre-release verified reproduction process.
  • Unlike Gitian, we are not limited to the package set of a particular Ubuntu version. Guix allows us to pick and choose our toolchains. This would e.g. make it easier to add a modern Rust compiler to our build environment. Packages that are not available in Guix can easily be defined in the manifest or upstreamed.
  • Making changes to the build system and debugging build issues takes less time because we have shell access to the build environment. The monero source code is bind mounted into the container, so edits to package definitions can be tested incrementally.

How to test?

See README.md

Notes

  • Guix timemachine: d5ca4d4fd713a9f7e17e074a1e37dda99bbb09fc (Dec 7, 2023)
  • Glibc: 2.27
  • GCC: 10.5.0
  • Clang: 11.1.0 (darwin and freebsd targets)
  • Linux headers: 6.1.67

I removed guix and depends package caching from the guix.yml workflow because it uses too much space and causes evictions for other caches (10 GB limit). We only need to run it when there is a change to contrib/{depends,guix}, which happens infrequently and doesn't need to complete quickly.

The guix timemachine URL is set to https://github.com/tobtoht/guix.git because the official repo on git.savannah.gnu.org often fails to fetch during CI runs for whatever reason. There used to be a mirror at guix-mirror/guix, but it is now out of service. Reviewers ought to confirm the pinned commit is a legit hash.

This PR removes native_clang because Guix provides us with a bootstrapped package and non-guix builds can use the system clang. This PR introduces backwards incompatible compiler flags, bumping the minimum Clang version to 10 (up from 9) for depends builds. Clang >=10 is available in the package managers of all supported distros (including Ubuntu 18.04).

Sub-PRs

Follow-up PRs:

To-do

  • Working builds for all targets
  • GitHub Actions
  • Minimize manifest
  • Fix release archive structure
  • Fix rpath
  • Build attestation
  • Fix reproducibility defects
  • All builds tested
  • Set up guix.sigs repo

Notable changes

  • The minimum glibc version for all Linux targets is now 2.27 (Ubuntu 18.04, Debian 10)
  • The source archive now includes all submodules.

Where to start review?

contrib/guix/README.md
->
contrib/guix/guix-build
->
contrib/guix/manifest.scm
->
contrib/guix/libexec/build.sh

Status: Ready for testing

Target Builds Tested
x86_64-linux-gnu 🮱
arm-linux-gnueabihf 🮱
aarch64-linux-gnu 🮱
riscv64-linux-gnu 🮱
i686-linux-gnu 🮱
i686-w64-mingw32 🮱
x86_64-w64-mingw32 🮱
x86_64-unknown-freebsd 🮱
x86_64-apple-darwin 🮱
aarch64-apple-darwin 🮱
arm-linux-androideabi 🮱
aarch64-linux-android 🮱

@0xFFFC0000
Copy link
Collaborator

I cannot wait until we finalize and merge this. Thanks for your great work @tobtoht

@tobtoht
Copy link
Contributor Author

tobtoht commented Feb 25, 2024

I don't expect to make major changes to this PR, it's ready for testing. Reviews on any of the sub-PRs would help move this forward.

contrib/depends/Makefile Outdated Show resolved Hide resolved
#!/bin/bash
if [ "$1" != "-cc1" ]; then
- `dirname $0`/clang{version} {flags} "$@"
+ env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH -u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH -u LIBRARY_PATH `dirname $0`/clang{version} {flags} "$@"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to unset these environment variables here because they refer to Guix profile packages. We want clang to only include headers from the NDK. We can't unset them in build.sh because native_ package builds would fail to find the necessary include headers. Unsetting these variables does not affect non-Guix builds.

Comment on lines +11 to +14
+ #if $(real-version) < "4.0.0"
+ #{
+ # flags darwin.compile.c++ OPTIONS $(condition) : -fcoalesce-templates ;
+ #}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Boost 1.64.0 doesn't recognize that we're building with Clang and passes a flags that results in an error. We don't support GCC < 4.0 at all, so commenting out the lines here is fine. Patch can be dropped when we update Boost.

@@ -0,0 +1,76 @@
Note that this has been modified from the original commit, to use __has_include
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

riscv64-linux-gnu builds fail without this patch

@@ -0,0 +1,22 @@
Without ffile-prefix-map, the debug symbols will contain paths for the
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Building on aarch64 should produce bit-for-bit identical release binaries.

gcc-toolchain-10
(list gcc-toolchain-10 "static")
;; Scripting
perl
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perl is needed to build OpenSSL.

[ -e /usr/bin/env ] || ln -s --no-dereference "$(command -v env)" /usr/bin/env
[ -e /bin/bash ] || ln -s --no-dereference "$(command -v bash)" /bin/bash
[ -e /bin/sh ] || ln -s --no-dereference "$(command -v sh)" /bin/sh
[ -e /lib64/ld-linux-x86-64.so.2 ] || ln -s --no-dereference "${NATIVE_GCC}/lib/ld-linux-x86-64.so.2" /lib64/ld-linux-x86-64.so.2
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Android NDK toolchain cannot (yet) be bootstrapped. The compiler binaries included in the NDK have their dynamic interpreter set to the standard x86_64 interpreter path, which does not exist in this location in the Guix environment.

The alternative was patchelf-ing all binaries included in the NDK, but this is hacky and adds a dependency on patchelf for non-Guix builders.

Comment on lines +259 to +260
if [[ -n "$DEPENDS_ONLY" ]]; then
exit 0
fi
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is useful when debugging reproducibility issues in depends packages. Skips ahead to the next target, so we don't spend time building Monero binaries.

contrib/gitian/docker/
contrib/gitian/sigs/
# guix
/guix
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finished builds are available in ./guix/guix-build-<COMMIT>/

├── guix-build-222ea1a30058
│   ├── build
│   │   └── distsrc-222ea1a30058-x86_64-linux-gnu  # build directory
│   ├── logs
│   │   └── x86_64-linux-gnu  # various logs for reproducibility debugging
│   │       ├── depends-gen_id-build.txt
│   │       ├── depends-gen_id-host.txt
│   │       ├── depends-hashes.txt
│   │       ├── depends-packages.txt
│   │       ├── guix-env.txt
│   │       ├── guix-hashes.txt
│   │       └── SHA256SUMS.part
│   ├── output
│   │   └── x86_64-linux-gnu
│   │       └── monero-x86_64-linux-gnu-222ea1a30058.tar.bz2  # binary tarball
│   └── var
│       ├── precious_dirs  # directories to keep when running `./contrib/guix/guix-clean`
│       └── profiles

@@ -267,4 +275,4 @@ $(foreach package,$(all_packages),$(eval $(call int_config_attach_build_config,$
$(foreach package,$(all_packages),$(eval $(call int_add_cmds,$(package))))

#special exception: if a toolchain package exists, all non-native packages depend on it
$(foreach package,$(packages),$(eval $($(package)_unpacked): |$($($(host_arch)_$(host_os)_native_toolchain)_cached) ))
$(foreach package,$(packages),$(eval $($(package)_extracted): |$($($(host_arch)_$(host_os)_native_toolchain)_cached) ))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_unpacked target doesn't exist.

gawk
sed
moreutils
patchelf
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

patchelf is unused. It is occasionally useful for debugging.

esac

export GIT_DISCOVERY_ACROSS_FILESYSTEM=1
export USE_DEVICE_TREZOR_MANDATORY=1
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Force Trezor support for release binaries.

Comment on lines +3 to +22
$(1)_cc=$$($$($(1)_type)_CC)
$(1)_cxx=$$($$($(1)_type)_CXX)
$(1)_objc=$$($$($(1)_type)_OBJC)
$(1)_objcxx=$$($$($(1)_type)_OBJCXX)
$(1)_ar=$$($$($(1)_type)_AR)
$(1)_ranlib=$$($$($(1)_type)_RANLIB)
$(1)_libtool=$$($$($(1)_type)_LIBTOOL)
$(1)_nm=$$($$($(1)_type)_NM)
$(1)_cflags=$$($$($(1)_type)_CFLAGS) \
$$($$($(1)_type)_$$(release_type)_CFLAGS)
$(1)_cxxflags=$$($$($(1)_type)_CXXFLAGS) \
$$($$($(1)_type)_$$(release_type)_CXXFLAGS)
$(1)_arflags=$$($$($(1)_type)_ARFLAGS) \
$$($$($(1)_type)_$(release_type)_ARFLAGS)
$(1)_ldflags=$$($$($(1)_type)_LDFLAGS) \
$$($$($(1)_type)_$$(release_type)_LDFLAGS) \
-L$$($($(1)_type)_prefix)/lib
$(1)_cppflags=$$($$($(1)_type)_CPPFLAGS) \
$$($$($(1)_type)_$$(release_type)_CPPFLAGS) \
-I$$($$($(1)_type)_prefix)/include
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delay expansion of package variables.

@@ -0,0 +1,63 @@
name: ci/gh-actions/guix

on: [push, pull_request]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed guix and depends package caching from the CI workflow because it uses too much space and causes evictions for other caches (10 GB limit). We only need to run it when there is a change to contrib/{depends,guix}, which happens infrequently and doesn't need to complete quickly.

on: [push, pull_request] is temporary for testing.

path: contrib/depends/sources
key: sources-${{ hashFiles('contrib/depends/packages/*') }}
- name: install dependencies
run: sudo apt update; sudo apt -y install guix git ca-certificates
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may be worth investigating if using the installer script is faster.

https://guix.gnu.org/manual/en/html_node/Installation.html

(("-rpath=") "-rpath-link="))
#t))))))))

(define-public glibc-2.27
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The minimum glibc version for all Linux targets is now 2.27 (Ubuntu 18.04, Debian 10).

For more context, see: #9171 (comment)

-DCMAKE_EXE_LINKER_FLAGS="${HOST_LDFLAGS}" \
-DCMAKE_SHARED_LINKER_FLAGS="${HOST_LDFLAGS}" \
-DCMAKE_SKIP_RPATH=ON \
-DMANUAL_SUBMODULES=1
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't actually check if submodules are checked out here because we're building in an extracted source archive.

The guix-build script runs git submodule update --init --recursive --progress before starting a build to make sure submodules are checked out.

@@ -6,5 +6,6 @@ $(package)_sha256_hash=df75d30ecafc429e905134333aeae56ac65fac67cb4182622398fd717

define $(package)_stage_cmds
mkdir -p $($(package)_staging_dir)/$(host_prefix)/native/SDK &&\
rm -rf usr/include/readline && \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prevent clang from including readline headers from the SDK. We statically link our own version of readline.

Comment on lines -81 to +84
SET(CMAKE_C_COMPILER @prefix@/native/bin/clang)
SET(CMAKE_C_COMPILER @CC@)
SET(CMAKE_C_COMPILER_TARGET ${CLANG_TARGET})
SET(CMAKE_C_FLAGS_INIT -B${_CMAKE_TOOLCHAIN_PREFIX})
SET(CMAKE_CXX_COMPILER @prefix@/native/bin/clang++ -stdlib=libc++)
SET(CMAKE_CXX_COMPILER @CXX@ -stdlib=libc++)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're now using the system / guix-provided clang for darwin builds.

Comment on lines +87 to +88
SET(CMAKE_ASM_COMPILER clang)
SET(CMAKE_ASM-ATT_COMPILER as)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CMake automatically reduces CMAKE_CXX_COMPILER to the first command to derive CMAKE_ASM_COMPILER. This happens to be env instead of clang, so we need to set it explicitly.

Comment on lines +81 to +86
unset LIBRARY_PATH
unset CPATH
unset C_INCLUDE_PATH
unset CPLUS_INCLUDE_PATH
unset OBJC_INCLUDE_PATH
unset OBJCPLUS_INCLUDE_PATH
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are automatically set by Guix, but don't necessarily point to the correct paths. This is fixed below.

Comment on lines -49 to +57
final_build_id_long+=$($(package)_build_id_long)
final_build_id_long+=:[recipe]:$(1)-$($(1)_version)-$($(1)_recipe_hash)-$(release_type):[deps]$(foreach dep,$($(1)_build_id_deps),$(shell echo ":$(dep)")):[$($(1)_type)_id]:$($($(1)_type)_id_string):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Increase verbosity of final_build_id_long to ease debugging reproducibility issues.

Comment on lines +103 to +104
build_id_string:=$(realpath $(GUIX_ENVIRONMENT))
$(host_arch)_$(host_os)_id_string:=$(realpath $(GUIX_ENVIRONMENT))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any change to manifest.scm invalidates the cache.

@tobtoht
Copy link
Contributor Author

tobtoht commented Feb 29, 2024

423dcee4a23f697669b74188588eb1fa023b0921de1ba8c31860c6c670d6cdc2  monero-aarch64-apple-darwin-9ea73c0ff089.tar.bz2
9c40bc1ff4267f25bc2a36649bd5fd06fed2ed0fe1aadfdfca2265248ca78106  monero-aarch64-linux-android-9ea73c0ff089.tar.bz2
d7c1099361828707d2a57e5fb5407f32323bc23c5009942b0c551f17b5fd7f5c  monero-aarch64-linux-gnu-9ea73c0ff089.tar.bz2
486e6282b26dc18c4dbe1f71821bf14bd87352b6eccf29b9753f17d92bad103a  monero-arm-linux-androideabi-9ea73c0ff089.tar.bz2
85c57bde9b139303655c6f57b1305be37e613f51e0f264361b4f6e4136c42dc5  monero-arm-linux-gnueabihf-9ea73c0ff089.tar.bz2
d24844c3396c2f4b3cd0ee0d74a02d26070480361e4fdcf7b3792ac77cbb63dd  monero-i686-linux-gnu-9ea73c0ff089.tar.bz2
14265f23ae8937d233861c3371be32a1a1a6e112c5b5e193ec24f819802f5dc4  monero-i686-w64-mingw32-9ea73c0ff089.zip
49133fe551ca06965b0ace3283fd87a1d1bd4863a5c97e29bf02f892e96feeaa  monero-riscv64-linux-gnu-9ea73c0ff089.tar.bz2
75e452f349dc0996792f3551ff452301c7107080628111f5fd2364fee69e04c6  monero-source-9ea73c0ff089.tar.gz
93c0bd491b284e920cdc042e750fad2b215896f0bcff8736b921769fe75d622f  monero-x86_64-apple-darwin-9ea73c0ff089.tar.bz2
fb9a3725c547d4313577b7937ea1c34a3c4db02eae30159dd1b57e4c447f0a8c  monero-x86_64-linux-gnu-9ea73c0ff089.tar.bz2
e5aca1771d83612844e7e85aeef1d9b7897f0575757cb207e6151ab092073f25  monero-x86_64-unknown-freebsd-9ea73c0ff089.tar.bz2
58d7586c7698e64f20c584d61a98bb4140239c4b6e2184a946a93508355f3650  monero-x86_64-w64-mingw32-9ea73c0ff089.zip

@plowsof
Copy link
Contributor

plowsof commented Feb 29, 2024

hashes

423dcee4a23f697669b74188588eb1fa023b0921de1ba8c31860c6c670d6cdc2  monero-aarch64-apple-darwin-9ea73c0ff089.tar.bz2
9c40bc1ff4267f25bc2a36649bd5fd06fed2ed0fe1aadfdfca2265248ca78106  monero-aarch64-linux-android-9ea73c0ff089.tar.bz2
d7c1099361828707d2a57e5fb5407f32323bc23c5009942b0c551f17b5fd7f5c  monero-aarch64-linux-gnu-9ea73c0ff089.tar.bz2
486e6282b26dc18c4dbe1f71821bf14bd87352b6eccf29b9753f17d92bad103a  monero-arm-linux-androideabi-9ea73c0ff089.tar.bz2
85c57bde9b139303655c6f57b1305be37e613f51e0f264361b4f6e4136c42dc5  monero-arm-linux-gnueabihf-9ea73c0ff089.tar.bz2
d24844c3396c2f4b3cd0ee0d74a02d26070480361e4fdcf7b3792ac77cbb63dd  monero-i686-linux-gnu-9ea73c0ff089.tar.bz2
14265f23ae8937d233861c3371be32a1a1a6e112c5b5e193ec24f819802f5dc4  monero-i686-w64-mingw32-9ea73c0ff089.zip
49133fe551ca06965b0ace3283fd87a1d1bd4863a5c97e29bf02f892e96feeaa  monero-riscv64-linux-gnu-9ea73c0ff089.tar.bz2
75e452f349dc0996792f3551ff452301c7107080628111f5fd2364fee69e04c6  monero-source-9ea73c0ff089.tar.gz
93c0bd491b284e920cdc042e750fad2b215896f0bcff8736b921769fe75d622f  monero-x86_64-apple-darwin-9ea73c0ff089.tar.bz2
fb9a3725c547d4313577b7937ea1c34a3c4db02eae30159dd1b57e4c447f0a8c  monero-x86_64-linux-gnu-9ea73c0ff089.tar.bz2
e5aca1771d83612844e7e85aeef1d9b7897f0575757cb207e6151ab092073f25  monero-x86_64-unknown-freebsd-9ea73c0ff089.tar.bz2
58d7586c7698e64f20c584d61a98bb4140239c4b6e2184a946a93508355f3650  monero-x86_64-w64-mingw32-9ea73c0ff089.zip

@kevcrumb
Copy link
Contributor

I recently designed a process for gitian-building entirely in RAM with only 12GB total space required, significantly less than:

  • 16GB of free disk space on the partition that /gnu/store will reside in
  • 8GB of free disk space per platform triple

Could Guix build results be made to match those generated by gitian?

@tobtoht
Copy link
Contributor Author

tobtoht commented Mar 18, 2024

@kevcrumb The figures in the readme are a bit off, I'll update them.

Description location All targets Max. per target
guix cache /gnu/store 8.0 GB -
git repository 0.8 GB -
depends sources cache SOURCES_PATH or contrib/depends/sources 0.9 GB -
depends package cache BASE_CACHE or contrib/depends/built 1.2 GB 290 MB
extracted depends packages contrib/depends/<target> 6.7 GB 1100 MB
build directories guix/guix-build-<commit>/build 9.6 GB 980 MB
outputs guix/guix-build-<commit>/output 1.0 GB -
total 28.2 GB 13.1 GB

If target specific build/cache directories are removed after each successfully built target, about 13.1 GB of storage is required at any time, not counting a base VM image with guix, git, make and curl installed. I could add an option that does that for storage constrained environments.

@kevcrumb
Copy link
Contributor

about 13.1 GB of storage is required at any time

On a 16 GB system that would leave a mere 2.9 of RAM for the host's OS and the entire build process combined.

Anyway, my question was rather if we could get matching checksums between gitian and Guix, so that both methods could be employed interchangeably.

(dropping the term "reproducible builds" here and a mention of #1854, as I always struggle to find this issue)

@tobtoht
Copy link
Contributor Author

tobtoht commented May 13, 2024

Anyway, my question was rather if we could get matching checksums between gitian and Guix

No, this is not possible. We can't recreate the Gitian build environment in Guix. Even if we could, it would be unmaintainable and too limiting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants