ARROW-8222: [C++] Use bcp to make a slim boost for bundled build #6734
Conversation
|
Hmm, thrift appears to need more of boost. Will see if I can figure out what it needs and add that. |
|
Looks like Gandiva may require a little more, and grepping through the files, some tests import other boost headers (which we may by chance already have included, IDK). I'll explore some more tomorrow (unless someone comments here why this is a bad idea); fallback could be to change which URL to download boost from depending on which cmake args are set (i.e. if gandiva is being built, and/or if tests are being built, get the full boost). |
|
@github-actions crossbow submit macos-r-autobrew |
|
Revision: ad657cc6b6c50e525d6f31186096b3a389a4d24a Submitted crossbow builds: ursa-labs/crossbow @ actions-49
|
|
After poking at this some more, I haven't been able to pare back the heavier components at all. Whether for regex or locale, we need I'll work on the crossbow job for updating this bundle and then consider this done, pending review/approval. Will also want to run the all the nightlies against this. |
|
I'm going to need some help again with |
|
@github-actions crossbow submit -g r |
|
Revision: 4869bf999412e91e1b77289c08f4f9ac219d3d2a Submitted crossbow builds: ursa-labs/crossbow @ actions-51
|
$ python3 -m pip install -r dev/archery/requirements-lint.txt
$ python3 run-cmake-format.pywill work. |
|
I don't understand the Ruby failure, or at least don't see how it's related: https://github.com/apache/arrow/pull/6734/checks?check_run_id=543791775#step:6:1181 |
|
Oh, Meson 0.54.0 released today and it requires Ninja 1.7 or later https://mesonbuild.com/Release-notes-for-0-54-0.html#ninja-version-requirement-bumped-to-17 but Ubuntu 16.04 ships Ninja 1.5.1. |
|
@github-actions crossbow submit -g r |
|
Revision: dec6be8 Submitted crossbow builds: ursa-labs/crossbow @ actions-52
|
Because Meson 0.54.0 requires Ninja 1.7 but Ubuntu 16.04 provides Ninja 1.5.1 by default.
|
@github-actions crossbow submit -g r |
|
Revision: 3277c77 Submitted crossbow builds: ursa-labs/crossbow @ actions-53
|
|
Okay, now this is done IMO, for real this time. Review appreciated. Note that this will also fix the widespread failures on master due to whatever is up with bintray.com/boostorg. |
wesm
left a comment
wesm
left a comment
There was a problem hiding this comment.
+1. Looks good here. Removing a bunch of Boost a good idea
| # to upload is at ${BOOST_FILE}/${BOOST_FILE}.tar.gz | ||
| # | ||
| # Also, you must have a bintray account on the "ursalabs" organization and | ||
| # set the BINTRAY_USER and BINTRAY_APIKEY env vars. |
There was a problem hiding this comment.
What do you think about hosting these tarballs as downloads from tagged GitHub releases (based on date or something)? Can do after this. Seems GitHub will be more reliable than Bintray
There was a problem hiding this comment.
I'm fine with whatever works; it was just quickest for me to use bintray because it's familiar. If someone wants to modify the script to put a tarball somewhere else, that's cool.
They're not mutually exclusive, even: we can have multiple URLs as backups in the external project definition. So if you're worried about reliability, we can add redundancy.
|
Merging this. Thanks @nealrichardson for leading the charge on this |
| "https://dl.bintray.com/ursalabs/arrow-boost/boost_${ARROW_BOOST_BUILD_VERSION_UNDERSCORES}.tar.gz" | ||
| ) | ||
| if(NOT CMAKE_VERSION VERSION_LESS 3.7) | ||
| # Append as a backup URL the full source from boostorg |
4 participants
This patch switches our bundled boost ep to use a slimmer version of boost, which was built with the script added at
cpp/build_support/trim-boost.sh. It uses the official boostorg big tarball as a fallback URL if for some reason ours is unavailable (as we've seen with other ep's, sometimes the download hosts are subject to rate limiting or other downtime, so having redundancy would more generally be good).The resulting tarball is 10mb, much less than 113mb of the full boost but larger than the 800k I suggested in the ticket. This is because in order to build regex and filesystem, we need
config build boost_install headers, which add some weight. Boost.Build also seems to requirelog, andpredefis needed by log apparently.In addition to slimming the boost tarball, this patch also refines the cmake logic that determines when boost is required. Due to recent-ish efforts to reduce usage of boost, we only need boost for tests and Gandiva, and for Parquet but only on gcc < 4.9. When building thrift_ep, we also need boost. By narrowing the condition involving Parquet to just gcc < 4.9, we are able to remove boost entirely from the R macOS and Windows packages.
Outstanding questions/to-dos that I'm aware of:
Add a crossbow job to build the boost bundle: to update the boost version or change what's included in the bundle, edit cpp/build_support/trim-boost.sh and run an on-demand build via PR comment.On further reflection, crossbow isn't appropriate because it would mean that anyone could edit the script and overwrite the bundle that gets pulled into all source builds. Instead, I added a script that one can run locally, just requiring a bintray user and token in the ursalabs organization (available to any committer on request).