Add building and testing of PGO+LTO LLVM and Julia #345

Zentrik · 2024-03-26T11:10:57Z

I ended up just using the profile from Julia being built for PGO, locally it seems to work fine. If anyone has better ideas for what to profile, please do say.

I don't think we need to run the full test suite but it seems good to run at least some stuff to make sure nothing got messed up. The testing seems to finish a fair bit quicker than the gnuassertrr tests and not that far behind the other test pipelines, so this shouldn't slow down CI much.

Zentrik · 2024-03-26T11:22:29Z

I don't think I have permissions to fix ERROR: Pipeline '.buildkite/pipelines/main/launch_upload_jobs.yml' fails treehash signature check! You may need to re-run cryptic/bin/sign_treehashes!.

Zentrik · 2024-03-26T14:34:21Z

Build currently failing as the bump to patchelf JuliaLang/julia#53845 introduced a check that the compiler supports C++ 17. Not sure why that check is failing on CI as it does seem to be using clang 16.

We run /cache/build/builder-amdci4-3/julialang/julia-buildkite/deps/srccache/patchelf-0.18.0//configure --prefix=/cache/build/builder-amdci4-3/julialang/julia-buildkite/contrib/pgo-lto/stage1.build/usr --build=x86_64-unknown-linux-gnu --libdir=/cache/build/builder-amdci4-3/julialang/julia-buildkite/contrib/pgo-lto/stage1.build/usr/lib --bindir=/cache/build/builder-amdci4-3/julialang/julia-buildkite/contrib/pgo-lto/stage1.build/usr/tools LDFLAGS="-fuse-ld=lld -flto=thin -fprofile-generate=/cache/build/builder-amdci4-3/julialang/julia-buildkite/contrib/pgo-lto/profiles -Wl,-rpath,'\$\$ORIGIN' -Wl,-z,origin -Wl,-rpath-link,/cache/build/builder-amdci4-3/julialang/julia-buildkite/contrib/pgo-lto/stage1.build/usr/lib -Wl,--enable-new-dtags " F77="gfortran -m64" CC="/cache/build/builder-amdci4-3/julialang/julia-buildkite/contrib/pgo-lto/stage0.build/usr/tools/clang " CXX="/cache/build/builder-amdci4-3/julialang/julia-buildkite/contrib/pgo-lto/stage0.build/usr/tools/clang++ " LD="/cache/build/builder-amdci4-3/julialang/julia-buildkite/contrib/pgo-lto/stage0.build/usr/tools/ld.lld" LDFLAGS="" CPPFLAGS="" for the PGO build and the failure is checking whether /cache/build/builder-amdci4-3/julialang/julia-buildkite/contrib/pgo-lto/stage0.build/usr/tools/clang++ supports C++17 features with -std=c++17... no.
For the non PGO build we run /cache/build/tester-amdci5-9/julialang/julia-buildkite/deps/srccache/patchelf-0.18.0//configure --prefix=/cache/build/tester-amdci5-9/julialang/julia-buildkite/usr --build=x86_64-linux-gnu --libdir=/cache/build/tester-amdci5-9/julialang/julia-buildkite/usr/lib --bindir=/cache/build/tester-amdci5-9/julialang/julia-buildkite/usr/tools LDFLAGS=" -Wl,-rpath,'\$\$ORIGIN' -Wl,-z,origin -Wl,-rpath-link,/cache/build/tester-amdci5-9/julialang/julia-buildkite/usr/lib -Wl,--enable-new-dtags " F77="gfortran -m64" CC="gcc -m64 " CXX="g++ -m64 " LD="ld" LDFLAGS="" CPPFLAGS="" and checking whether g++ -m64 supports C++17 features with -std=c++17... yes is the relevant check.

The line that runs the check is https://github.com/NixOS/patchelf/blob/99c24238981b7b1084313aca8f5c493bb46f302c/configure.ac#L35 and the macro defined in https://github.com/NixOS/patchelf/blob/master/m4/ax_cxx_compile_stdcxx.m4.

The only obvious difference I see is the -m64 but I really have no clue why this is failing on CI.

Zentrik · 2024-03-26T19:39:35Z

Maybe clang is picking the wrong gcc?

Zentrik · 2024-03-26T20:36:47Z

Ok so the actual error is

configure:6320: /cache/build/builder-amdci4-6/julialang/julia-buildkite/contrib/pgo-lto/stage0.build/usr/tools/clang++ -m64  -std=c++17 -c -fprofile-generate=/cache/build/builder-amdci4-6/julialang/julia-buildkite/contrib/pgo-lto/profiles -Xclang -mllvm -Xclang -vp-counters-per-site=6  conftest.cpp >&5
conftest.cpp:437:10: fatal error: 'initializer_list' file not found
#include <initializer_list>

JuliaLang/julia#45641 (comment) mentions something about libstdc++ detection, so maybe something to do with that.
Clang does seem to be loading /usr/lib/gcc/x86_64-linux-gnu/10 the only gcc installation it finds whereas under 'Print software versions' we see gcc version 9.

haampie · 2024-03-26T21:13:03Z

Is that a partial gcc 10 install? No g++10 / libstdc++-10-dev?

You can force clang to use gcc 9 if necessary with --gcc-install-dir=/usr/lib/gcc/x86_64-linux-gnu/9

Notice that --gcc-toolchain is entirely useless when multiple GCCs are installed in the same prefix /usr.

Zentrik · 2024-03-26T21:31:53Z

I'm quite out of my depth here but I'm not sure gcc is installed in /usr/lib/gcc/x86_64-linux-gnu/9 because this was printed which seems to suggest it's installed elsewhere

COLLECT_LTO_WRAPPER=/usr/local/bin/../libexec/gcc/x86_64-linux-gnu/9.1.0/lto-wrapper
Target: x86_64-linux-gnu
Configured with: /workspace/srcdir/gcc-9.1.0/configure --prefix=/workspace/destdir --with-build-sysroot=/workspace/destdir/x86_64-linux-gnu --with-sysroot=/workspace/destdir/x86_64-linux-gnu --with-gxx-include-dir=/workspace/destdir/x86_64-linux-gnu/include/c++/9.1.0 --target=x86_64-linux-gnu --host=x86_64-linux-gnu --build=x86_64-linux-gnu --disable-multilib --disable-werror --disable-bootstrap --enable-threads=posix --program-prefix=x86_64-linux-gnu- --with-arch=x86-64 --enable-languages=c,c++

Zentrik · 2024-03-26T21:34:56Z

It does seem to find the correct gcc version now. Yes, it does seem to get through patchelf's configure phase but now fails at making it

/cache/build/builder-amdci4-0/julialang/julia-buildkite/contrib/pgo-lto/stage0.build/usr/tools/clang++ -DPACKAGE_NAME=\"patchelf\" -DPACKAGE_TARNAME=\"patchelf\" -DPACKAGE_VERSION=\"0.18.0\" -DPACKAGE_STRING=\"patchelf\ 0.18.0\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DPACKAGE=\"patchelf\" -DVERSION=\"0.18.0\" -DHAVE_CXX17=1 -I. -I/cache/build/builder-amdci4-0/julialang/julia-buildkite/deps/srccache/patchelf-0.18.0/src    -Wall -Wextra -Wcast-qual -std=c++17 -D_FILE_OFFSET_BITS=64     -fprofile-generate=/cache/build/builder-amdci4-0/julialang/julia-buildkite/contrib/pgo-lto/profiles -Xclang -mllvm -Xclang -vp-counters-per-site=6 -MT patchelf.o -MD -MP -MF .deps/patchelf.Tpo -c -o patchelf.o /cache/build/builder-amdci4-0/julialang/julia-buildkite/deps/srccache/patchelf-0.18.0/src/patchelf.cc
/cache/build/builder-amdci4-0/julialang/julia-buildkite/deps/srccache/patchelf-0.18.0/src/patchelf.cc:19:10: fatal error: 'algorithm' file not found
#include <algorithm>

haampie · 2024-03-26T21:35:40Z

The build environment likely contains just the C compiler part of GCC 10, and clang has poor detection & heuristics for the GCC install dir. If multiple are in the same prefix it picks the newest, and it will do so also when there are missing components like C++ stdlib header files. The --gcc-toolchain flag is useles cause it expects the prefix /usr, in which both GCC 9 and 10 live.

Use --gcc-install-dir=/usr/lib/gcc/x86_64-linux-gnu/9 or uninstall GCC 10.

Zentrik · 2024-03-26T21:44:59Z

/cache/build/builder-amdci5-3/julialang/julia-buildkite/contrib/pgo-lto/stage0.build/usr/tools/clang  -fprofile-generate=/cache/build/builder-amdci5-3/julialang/julia-buildkite/contrib/pgo-lto/profiles -Xclang -mllvm -Xclang -vp-counters-per-site=6 -O2 -m64 --gcc-install-dir=/usr/lib/gcc/x86_64-linux-gnu/9 -fPIC -std=c99 -Wsign-conversion -Wall -Wextra -pedantic -DUTF8PROC_EXPORTS  -c -o utf8proc.o utf8proc.c
clang-16: error: '/usr/lib/gcc/x86_64-linux-gnu/9' does not contain a GCC installation

On the previous commit I saw

Found candidate GCC installation: /usr/local/bin/../lib/gcc/x86_64-linux-gnu/9.1.0//../../../../lib/gcc/x86_64-linux-gnu/9.1.0
Selected GCC installation: /usr/local/bin/../lib/gcc/x86_64-linux-gnu/9.1.0//../../../../lib/gcc/x86_64-linux-gnu/9.1.0

So I presume the specified path was wrong.

Zentrik · 2024-03-26T22:03:54Z

Thanks, it seems to be working much better now. I just need to figure out why there's a hash mismatch, I had the same locally but I just refreshed it.

pipelines/main/platforms/build_linux.pgo_lto.arches

Now takes ~10 minutes. Precompiling OrdinaryDiffEq takes 367s and Plots 135s. Actually running the script takes 1-2 minutes.

Not really sure why this works, but it seems to.

I think 0f8bae0 might have slowed down the binary. At least locally this seems to be roughly the same as using pgo_script prior to 0f8bae0. I have 7 binaries locally and cannot work out why some are slower than others but I think this should be good enough.

Buildkite was complaining about non-alphanumeric character

Hopefully this fixes the error as I have no idea how to debug this

This seems to be what the form source builds use, though they seem to have been failing for the last year. Not sure if I still need to specify `--gcc-install-dir`.

This gets passed to patch elf's configure step

I don't think ROOTFS_IMAGE_NAME actually does anything on its own

Not sure why gcc doesn't.

This reverts commit c2ebde3.

This reverts commit a5d1274.

GCCFLAG should no longer be required now that we use the llvmpasses image

Zentrik · 2024-04-21T17:32:24Z

This looks good to me, testing was 15% faster. If we want -Werror I think we just need to merge or otherwise fix JuliaLang/julia#53897 and bump libwhich to vtjnash/libwhich@613b3f0.

pipelines/main/launch_unsigned_jobs.yml

giordano reviewed Mar 29, 2024

View reviewed changes

pipelines/main/platforms/build_linux.pgo_lto.arches Outdated Show resolved Hide resolved

Zentrik added 19 commits April 21, 2024 12:55

Add building and testing of PGO+LTO LLVM and Julia

410794c

Fix run pgo_script command

2b5ecb2

Make pgo script a bit quicker

3bcd21c

Now takes ~10 minutes. Precompiling OrdinaryDiffEq takes 367s and Plots 135s. Actually running the script takes 1-2 minutes.

Fix lto build directory

7165ba3

Not really sure why this works, but it seems to.

Fix upload name for lto binary

f364466

Replace + with - character in .arches file

ed33e3c

Buildkite was complaining about non-alphanumeric character

Add -m64 as a flag to clang

faf3add

Hopefully this fixes the error as I have no idea how to debug this

Print patchelf config log if make fails

91ad2aa

Fix typo

9235198

Try finding correct gcc - taken from CLANGSA_CXXFLAGS in julia Makefile

4985754

Specify gcc version exactly

3c662a4

Try different path

0d1c40f

Try different path + pass into flags used for building not just config

781c768

Specify CFLAGS and CXXFLAGS properly

adb596b

Remove hardcoded gcc install dir

ec56621

Upload profile data so builds can be reproduced

16e397a

Switch rootfs image to llvm_passes

da9d72e

This seems to be what the form source builds use, though they seem to have been failing for the last year. Not sure if I still need to specify `--gcc-install-dir`.

Add back gcc install dir for sanitize opts

cfdfa7e

This gets passed to patch elf's configure step

Zentrik added 6 commits April 21, 2024 12:55

Add back gc install dir everywhere

e7d05c3

Update rootfs_hash

e1e661e

I don't think ROOTFS_IMAGE_NAME actually does anything on its own

Stop clang throwing error on unused function in crc32

634dffd

Not sure why gcc doesn't.

Fix

4917177

Revert "Fix"

364a9a2

This reverts commit c2ebde3.

Revert "Stop clang throwing error on unused function in crc32"

a98a728

This reverts commit a5d1274.

Zentrik force-pushed the main branch 3 times, most recently from fc58388 to f2ffa42 Compare April 21, 2024 12:13

Zentrik added 2 commits April 21, 2024 13:16

Remove -Werror till JuliaLang/julia#53897 merges

37aa170

Clean up

5ebf7b5

GCCFLAG should no longer be required now that we use the llvmpasses image

Zentrik force-pushed the main branch from f2ffa42 to 5ebf7b5 Compare April 21, 2024 12:17

Try to fix downloading pgo binary when testing

4013901

Merge branch 'main' into main

ccbd20d

Zentrik commented Jul 13, 2024

View reviewed changes

pipelines/main/launch_unsigned_jobs.yml Show resolved Hide resolved

Add comment

4ced57b

Zentrik commented Jul 13, 2024

View reviewed changes

pipelines/main/launch_unsigned_jobs.yml Outdated Show resolved Hide resolved

Tabs to spaces

64500dc

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add building and testing of PGO+LTO LLVM and Julia #345

Add building and testing of PGO+LTO LLVM and Julia #345

Zentrik commented Mar 26, 2024 •

edited

Loading

Zentrik commented Mar 26, 2024

Zentrik commented Mar 26, 2024 •

edited

Loading

Zentrik commented Mar 26, 2024

Zentrik commented Mar 26, 2024 •

edited

Loading

haampie commented Mar 26, 2024 •

edited

Loading

Zentrik commented Mar 26, 2024

Zentrik commented Mar 26, 2024

haampie commented Mar 26, 2024 •

edited

Loading

Zentrik commented Mar 26, 2024

Zentrik commented Mar 26, 2024

Zentrik commented Apr 21, 2024

Add building and testing of PGO+LTO LLVM and Julia #345

Are you sure you want to change the base?

Add building and testing of PGO+LTO LLVM and Julia #345

Conversation

Zentrik commented Mar 26, 2024 • edited Loading

Zentrik commented Mar 26, 2024

Zentrik commented Mar 26, 2024 • edited Loading

Zentrik commented Mar 26, 2024

Zentrik commented Mar 26, 2024 • edited Loading

haampie commented Mar 26, 2024 • edited Loading

Zentrik commented Mar 26, 2024

Zentrik commented Mar 26, 2024

haampie commented Mar 26, 2024 • edited Loading

Zentrik commented Mar 26, 2024

Zentrik commented Mar 26, 2024

Zentrik commented Apr 21, 2024

Zentrik commented Mar 26, 2024 •

edited

Loading

Zentrik commented Mar 26, 2024 •

edited

Loading

Zentrik commented Mar 26, 2024 •

edited

Loading

haampie commented Mar 26, 2024 •

edited

Loading

haampie commented Mar 26, 2024 •

edited

Loading