Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Use BinaryBuilder for all available dependencies by default #31441

Merged
merged 20 commits into from Mar 30, 2019

Conversation

@staticfloat
Copy link
Member

commented Mar 22, 2019

The list of BB-enabled build dependencies is now:

  • LLVM
  • OpenBLAS
  • SuiteSparse
  • OpenLibm
  • GMP
  • MbedTLS
  • LibSSH2
  • MPFR
  • curl
  • LibGit2
  • PCRE
  • libunwind
  • libosxunwind

Furthermore, they default to being used on all systems, as long as it can accurately identify the host system triplet. Use make USE_BINARYBUILDER=0 to disable.

@StefanKarpinski

This comment has been minimized.

Copy link
Member

commented Mar 22, 2019

I’m very excited about this.

@staticfloat staticfloat changed the title Use BinaryBuilder for all available dependencies by default [WIP] Use BinaryBuilder for all available dependencies by default Mar 22, 2019

@staticfloat staticfloat reopened this Mar 22, 2019

@staticfloat

This comment has been minimized.

Copy link
Member Author

commented Mar 22, 2019

TODO:

  • We've got weird llvm-config problems on MacOS and FreeBSD where the BB environment's flags are making it out into llvm-config --cflags. I hypothesize this is because on those platforms llvm-config is being built by clang, and it keeps more flags than it otherwise would in that case, which we definitely don't want.

  • PCRE2 on Windows is calling itself libpcre2-8-0.dll instead of the expected libpcre2-8.dll. Fix that.

  • openblas_set_num_threads threw a fit on Aarch64

  • linux64's libgit2 didn't load its certificates properly; this is probably a problem with my libgit2 build, I'm missing a patch or something.

PATCHELF_VER = 0.9
MBEDTLS_VER = 2.16.0
CURL_VER = 7.56.0
MBEDTLS_BB_REL = v0.17.0

This comment has been minimized.

Copy link
@quinnj

quinnj Mar 22, 2019

Member

Just a thumbs up here on the right version here for mbedtls 👍 . That's about all I have to contribute in terms of review!

@ViralBShah

This comment has been minimized.

Copy link
Member

commented Mar 23, 2019

I tried make on this branch, but it started building openblas. Perhaps not ready to test just yet?

@ViralBShah ViralBShah added the build label Mar 23, 2019

@vtjnash

This comment has been minimized.

Copy link
Member

commented Mar 23, 2019

The BinaryBuilder files for Win32 SuiteSparse are built incorrectly (JuliaPackaging/Yggdrasil#7)

@ViralBShah

This comment has been minimized.

Copy link
Member

commented Mar 23, 2019

I guess we should get dsfmt into BB as well.

@StefanKarpinski

This comment has been minimized.

Copy link
Member

commented Mar 24, 2019

It doesn’t have to be all or nothing, much easier to get most deps using BB and then add more later.

@staticfloat

This comment has been minimized.

Copy link
Member Author

commented Mar 26, 2019

Lots of progress; many platforms building properly now; but MacOS has started giving this mysterious error during bootstrap, which is very difficult to track down:

InexactError(:check_top_bit, UInt32, 0xffffffd0)

As I insert println() statements to try and figure out where this is coming from, it seems to move around. My best guess is that it's due to a problem within PCRE; some kind of incompatibility, but I haven't been able to figure out where check_top_bit is being called from, much less why it's throwing an InexactError.

In other news, our fastest platform (linux64, running on nanosoldier) did the full build+test in less than 45 minutes, which makes me pretty happy.

@staticfloat staticfloat changed the title [WIP] Use BinaryBuilder for all available dependencies by default Use BinaryBuilder for all available dependencies by default Mar 27, 2019

@staticfloat

This comment has been minimized.

Copy link
Member Author

commented Mar 27, 2019

That's a lot of green. I'm satisfied with this now. I'm going to do a giant rebase to get to a reasonable number of commits, and then I think we're ready to merge.

@staticfloat staticfloat force-pushed the sf/the_great_bb8 branch from f8aa623 to f314d74 Mar 27, 2019

staticfloat added 11 commits Mar 21, 2019
Bump BinaryBuilder LLVM to include new patches, GCC ABI Multiversioni…
…ng, and FreeBSD support

Also fixes the `--sysroot` problem with `llvm-config --cflags` by
manually filtering them out of the LLVM build
Default to BB turned on for the projects we've precompiled so far,
but only enable BB if we can parse the triplet
Add cxx11 string ABI probing to BB tarball selection
Use this to automatically fail out when users try to use a `gcc4`
tarball with a system that compiles for `cxx11`.  As of the time
of this commit, this will apply to LLVM, OpenBLAS, OpenLibm and
SuiteSparse, although the only place we truly care aobut this is
LLVM, because it's the only one that passes around `std::string`s.

Also reduce the amount of work we're forcing `make` to do when
constructing BB triplets

@staticfloat staticfloat force-pushed the sf/the_great_bb8 branch 3 times, most recently from c561478 to 6593c53 Mar 27, 2019

@staticfloat

This comment has been minimized.

Copy link
Member Author

commented Mar 27, 2019

Okay actually it looks like the libuv build is causing a segfault on FreeBSD. I thought this might be because of some missing -fPIC flags, but it doesn't look like it.

@staticfloat staticfloat force-pushed the sf/the_great_bb8 branch 3 times, most recently from f23923e to a932d46 Mar 29, 2019

@staticfloat staticfloat force-pushed the sf/the_great_bb8 branch 2 times, most recently from a4dbf21 to ae21e66 Mar 29, 2019

@staticfloat staticfloat force-pushed the sf/the_great_bb8 branch from ae21e66 to 09d4e86 Mar 30, 2019

@staticfloat

This comment has been minimized.

Copy link
Member Author

commented Mar 30, 2019

Wow, green flush! I think this is ready to merge.

@staticfloat staticfloat merged commit 0557467 into master Mar 30, 2019

11 of 14 checks passed

buildbot/tester_freebsd64 Run complete
Details
continuous-integration/appveyor/pr AppVeyor build failed
Details
continuous-integration/travis-ci/pr The Travis CI build failed
Details
buildbot/package_freebsd64 Run complete
Details
buildbot/package_linux32 Run complete
Details
buildbot/package_linux64 Run complete
Details
buildbot/package_macos64 Run complete
Details
buildbot/package_win32 Run complete
Details
buildbot/package_win64 Run complete
Details
buildbot/tester_linux32 Run complete
Details
buildbot/tester_linux64 Run complete
Details
buildbot/tester_macos64 Run complete
Details
buildbot/tester_win32 Run complete
Details
buildbot/tester_win64 Run complete
Details

@fredrikekre fredrikekre deleted the sf/the_great_bb8 branch Mar 30, 2019

@fredrikekre

This comment has been minimized.

Copy link
Member

commented Mar 30, 2019

I get

error during bootstrap:
LoadError("sysimg.jl", 3, LoadError("Base.jl", 363, LoadError("version.jl", 50, InexactError(:check_top_bit, UInt32, 0xffffffd0))))
rec_backtrace at /Users/fredrik/julia-master/src/stackwalk.c:94
record_backtrace at /Users/fredrik/julia-master/src/task.c:210
jl_throw at /Users/fredrik/julia-master/src/task.c:417
jl_parse_eval_all at /Users/fredrik/julia-master/src/ast.c:895
jl_load at /Users/fredrik/julia-master/src/toplevel.c:861
exec_program at /Users/fredrik/julia-master/usr/bin/julia (unknown line)
true_main at /Users/fredrik/julia-master/usr/bin/julia (unknown line)
main at /Users/fredrik/julia-master/usr/bin/julia (unknown line)

*** This error might be fixed by running `make clean`. If the error persists, try `make cleanall`. ***
make[1]: *** [/Users/fredrik/julia-master/usr/lib/julia/sys.ji] Error 1
make: *** [julia-sysimg-ji] Error 2

on a fresh clone on macos.

@KristofferC

This comment has been minimized.

Copy link
Contributor

commented Mar 30, 2019

Works for me on Windows.

@ViralBShah

This comment has been minimized.

Copy link
Member

commented Mar 31, 2019

I now note that on mac, everything except objconv and dsfmt is coming from BB. It seems that yggdrasil does have objconv and JuliaMath has a dsfmt builder.

@staticfloat I assume you have this coming in and a new issue isn't necessary.

@Gnimuc

This comment has been minimized.

Copy link
Contributor

commented Apr 1, 2019

Will the next official Julia minor release use BB2-built binaries as well?

@staticfloat

This comment has been minimized.

Copy link
Member Author

commented Apr 1, 2019

That's the way things look, yes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.