Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] Upgrade Boost to version 1.77 #2095

Merged
merged 6 commits into from Nov 3, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
10 changes: 5 additions & 5 deletions 3rd_party/3rd_party.sh
Expand Up @@ -47,7 +47,7 @@ case `uname` in
else
BOOST_ARCH=a64
fi
BOOST_EXTENSION=mt-${BOOST_ARCH}-1_71.dylib
BOOST_EXTENSION=mt-${BOOST_ARCH}-1_77.dylib
BOOST_LIBRARIES='atomic chrono date_time filesystem iostreams log log_setup program_options regex system thread'
XML_LOCATION=
GCC_RT_LOCATION=
Expand All @@ -72,7 +72,7 @@ case `uname` in
MKL_PREFIX=libmkl_
MKL_LIBRARIES=`cd "$MKL_LOCATION" && ls $MKL_PREFIX*$MKL_EXTENSION`
fi
BOOST_EXTENSION=mt-${BOOST_ARCH}-1_71.so.1.71.0
BOOST_EXTENSION=mt-${BOOST_ARCH}-1_77.so.1.77.0
BOOST_LIBRARIES='atomic chrono date_time filesystem iostreams log log_setup program_options regex system thread'
XML_LOCATION=/usr/local/gcc103/lib
XML_EXTENSION=.so.2
Expand All @@ -91,7 +91,7 @@ case `uname` in
SYSROOT=/usr/local/sysroot-x86_64-apple-macosx10.14
BOOST_LOCATION=$SYSROOT/usr/local/lib
BOOST_COMPILER=clang
BOOST_EXTENSION=mt-x64-1_71.dylib
BOOST_EXTENSION=mt-x64-1_77.dylib
BOOST_LIBRARIES='atomic chrono date_time filesystem iostreams log log_setup program_options regex system thread'
XML_LOCATION=
GCC_RT_LOCATION=
Expand All @@ -111,7 +111,7 @@ case `uname` in
echo "Cannot cross compile to $CPP_CROSS_COMPILE"
exit 3
fi
BOOST_EXTENSION=mt-${BOOST_ARCH}-1_71.so.1.71.0
BOOST_EXTENSION=mt-${BOOST_ARCH}-1_77.so.1.77.0
BOOST_LIBRARIES='atomic chrono date_time filesystem iostreams log log_setup program_options regex system thread'
XML_LOCATION=$SYSROOT/usr/local/gcc103/lib
XML_EXTENSION=.so.2
Expand All @@ -137,7 +137,7 @@ case `uname` in
# server is currently set up
BOOST_LOCATION=/$LOCAL_DRIVE/usr/local/lib
BOOST_COMPILER=vc
BOOST_EXTENSION=mt-x64-1_71.dll
BOOST_EXTENSION=mt-x64-1_77.dll
BOOST_LIBRARIES='chrono date_time filesystem iostreams log log_setup program_options regex system thread'
XML_LOCATION=/$LOCAL_DRIVE/usr/local/bin
XML_EXTENSION=.dll
Expand Down
2 changes: 1 addition & 1 deletion 3rd_party/licenses/boost-INFO.csv
@@ -1,2 +1,2 @@
name,version,revision,url,license,copyright,sourceURL
Boost C++ Libraries,1.71.0,,http://www.boost.org,BSL-1.0,,
Boost C++ Libraries,1.77.0,,http://www.boost.org,BSL-1.0,,
8 changes: 4 additions & 4 deletions build-setup/linux.md
Expand Up @@ -171,17 +171,17 @@ sudo make install

to install.

### Boost 1.71.0
### Boost 1.77.0

Download version 1.71.0 of Boost from <https://boostorg.jfrog.io/artifactory/main/release/1.71.0/source/boost_1_71_0.tar.bz2>. You must get this exact version, as the Machine Learning Makefiles expect it.
Download version 1.77.0 of Boost from <https://boostorg.jfrog.io/artifactory/main/release/1.77.0/source/boost_1_77_0.tar.bz2>. You must get this exact version, as the Machine Learning Makefiles expect it.

Assuming you chose the `.bz2` version, extract it to a temporary directory:

```
bzip2 -cd boost_1_71_0.tar.bz2 | tar xvf -
bzip2 -cd boost_1_77_0.tar.bz2 | tar xvf -
```

In the resulting `boost_1_71_0` directory, run:
In the resulting `boost_1_77_0` directory, run:

```
./bootstrap.sh --without-libraries=context --without-libraries=coroutine --without-libraries=graph_parallel --without-libraries=mpi --without-libraries=python --without-icu
Expand Down
39 changes: 6 additions & 33 deletions build-setup/macos.md
Expand Up @@ -54,7 +54,8 @@ For C++17 Xcode 10 is required, and this requires macOS High Sierra or above. Th

- If you are using High Sierra, you must install Xcode 10.1.x
- If you are using Mojave, you must install Xcode 11.3.x
- If you are using Catalina or Big Sur, you must install Xcode 12.3.x or above
- If you are using Catalina, you must install Xcode 12.4.x
- If you are using Big Sur or Monterey, you must install Xcode 13.1.x or above

Xcode is distributed as a `.xip` file; simply double click the `.xip` file to expand it, then drag `Xcode.app` to your `/Applications` directory.
(Older versions of Xcode can be downloaded from [here](https://developer.apple.com/download/more/), provided you are signed in with your Apple ID.)
Expand All @@ -67,17 +68,17 @@ xcode-select --install

at the command prompt.

### Boost 1.71.0
### Boost 1.77.0

Download version 1.71.0 of Boost from <https://boostorg.jfrog.io/artifactory/main/release/1.71.0/source/boost_1_71_0.tar.bz2>. You must get this exact version, as the Machine Learning Makefiles expect it.
Download version 1.77.0 of Boost from <https://boostorg.jfrog.io/artifactory/main/release/1.77.0/source/boost_1_77_0.tar.bz2>. You must get this exact version, as the Machine Learning Makefiles expect it.

Assuming you chose the `.bz2` version, extract it to a temporary directory:

```
bzip2 -cd boost_1_71_0.tar.bz2 | tar xvf -
bzip2 -cd boost_1_77_0.tar.bz2 | tar xvf -
```

In the resulting `boost_1_71_0` directory, run:
In the resulting `boost_1_77_0` directory, run:

```
./bootstrap.sh --with-toolset=clang --without-libraries=context --without-libraries=coroutine --without-libraries=graph_parallel --without-libraries=mpi --without-libraries=python --without-icu
Expand All @@ -97,34 +98,6 @@ to:
(3ul)(17ul)(29ul)(37ul)(53ul)(67ul)(79ul) \
```

Then edit `tools/build/src/tools/darwin.jam` and change:

```
case arm :
{
if $(instruction-set) {
options = -arch$(_)$(instruction-set) ;
} else {
options = -arch arm ;
}
}
```

to:

```
case arm :
{
if $(instruction-set) {
options = -arch$(_)$(instruction-set) ;
} else if $(address-model) = 64 {
options = -arch arm64 ;
} else {
options = -arch arm ;
}
}
```

To complete the build, type:

```
Expand Down
8 changes: 4 additions & 4 deletions build-setup/windows.md
Expand Up @@ -133,15 +133,15 @@ nmake
nmake install
```

### Boost 1.71.0
### Boost 1.77.0

Download version 1.71.0 of Boost from <https://boostorg.jfrog.io/artifactory/main/release/1.71.0/source/boost_1_71_0.tar.bz2> . You must get this exact version, as the Machine Learning Makefiles expect it.
Download version 1.77.0 of Boost from <https://boostorg.jfrog.io/artifactory/main/release/1.77.0/source/boost_1_77_0.tar.bz2> . You must get this exact version, as the Machine Learning Makefiles expect it.

Assuming you chose the `.bz2` version, extract it in a Git bash shell using the GNU tar that comes with Git for Windows, e.g.:

```
cd /c/tools
tar jxvf /z/cpp_src/boost_1_71_0.tar.bz2
tar jxvf /z/cpp_src/boost_1_77_0.tar.bz2
```

Edit `boost/unordered/detail/implementation.hpp` and change line 287 from:
Expand All @@ -159,7 +159,7 @@ to:
Start a command prompt using Start Menu -&gt; Apps -&gt; Visual Studio 2019 -&gt; x64 Native Tools Command Prompt for VS 2019, then in it type:

```
cd \tools\boost_1_71_0
cd \tools\boost_1_77_0
bootstrap.bat
b2 -j6 --layout=versioned --disable-icu --toolset=msvc-14.2 cxxflags="-std:c++17" linkflags="-std:c++17" --build-type=complete -sZLIB_INCLUDE="C:\tools\zlib-1.2.11" -sZLIB_LIBPATH="C:\tools\zlib-1.2.11" -sZLIB_NAME=zdll --without-context --without-coroutine --without-graph_parallel --without-mpi --without-python architecture=x86 address-model=64 optimization=speed inlining=full define=BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS define=BOOST_LOG_WITHOUT_DEBUG_OUTPUT define=BOOST_LOG_WITHOUT_EVENT_LOG define=BOOST_LOG_WITHOUT_SYSLOG define=BOOST_LOG_WITHOUT_IPC define=_WIN32_WINNT=0x0601
b2 install --prefix=C:\usr\local --layout=versioned --disable-icu --toolset=msvc-14.2 cxxflags="-std:c++17" linkflags="-std:c++17" --build-type=complete -sZLIB_INCLUDE="C:\tools\zlib-1.2.11" -sZLIB_LIBPATH="C:\tools\zlib-1.2.11" -sZLIB_NAME=zdll --without-context --without-coroutine --without-graph_parallel --without-mpi --without-python architecture=x86 address-model=64 optimization=speed inlining=full define=BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS define=BOOST_LOG_WITHOUT_DEBUG_OUTPUT define=BOOST_LOG_WITHOUT_EVENT_LOG define=BOOST_LOG_WITHOUT_SYSLOG define=BOOST_LOG_WITHOUT_IPC define=_WIN32_WINNT=0x0601
Expand Down
2 changes: 1 addition & 1 deletion dev-tools/docker/build_linux_aarch64_cross_build_image.sh
Expand Up @@ -22,7 +22,7 @@
HOST=docker.elastic.co
ACCOUNT=ml-dev
REPOSITORY=ml-linux-aarch64-cross-build
VERSION=5
VERSION=6

set -e

Expand Down
2 changes: 1 addition & 1 deletion dev-tools/docker/build_linux_aarch64_native_build_image.sh
Expand Up @@ -34,7 +34,7 @@ sleep 5
HOST=docker.elastic.co
ACCOUNT=ml-dev
REPOSITORY=ml-linux-aarch64-native-build
VERSION=5
VERSION=6

set -e

Expand Down
2 changes: 1 addition & 1 deletion dev-tools/docker/build_linux_build_image.sh
Expand Up @@ -34,7 +34,7 @@ sleep 5
HOST=docker.elastic.co
ACCOUNT=ml-dev
REPOSITORY=ml-linux-build
VERSION=19
VERSION=20

set -e

Expand Down
2 changes: 1 addition & 1 deletion dev-tools/docker/build_macosx_build_image.sh
Expand Up @@ -22,7 +22,7 @@
HOST=docker.elastic.co
ACCOUNT=ml-dev
REPOSITORY=ml-macosx-build
VERSION=12
VERSION=13

set -e

Expand Down
2 changes: 1 addition & 1 deletion dev-tools/docker/linux_aarch64_cross_builder/Dockerfile
Expand Up @@ -10,7 +10,7 @@
#

# Increment the version here when a new tools/3rd party components image is built
FROM docker.elastic.co/ml-dev/ml-linux-aarch64-cross-build:5
FROM docker.elastic.co/ml-dev/ml-linux-aarch64-cross-build:6

MAINTAINER David Roberts <dave.roberts@elastic.co>

Expand Down
2 changes: 1 addition & 1 deletion dev-tools/docker/linux_aarch64_cross_image/Dockerfile
Expand Up @@ -24,7 +24,7 @@ RUN \
RUN \
mkdir -p /usr/local/sysroot-aarch64-linux-gnu/usr && \
cd /usr/local/sysroot-aarch64-linux-gnu/usr && \
wget --quiet -O - https://s3-eu-west-1.amazonaws.com/prelert-artifacts/dependencies/usr-aarch64-linux-gnu-5.tar.bz2 | tar jxf - && \
wget --quiet -O - https://s3-eu-west-1.amazonaws.com/prelert-artifacts/dependencies/usr-aarch64-linux-gnu-6.tar.bz2 | tar jxf - && \
cd .. && \
ln -s usr/lib lib && \
ln -s usr/lib64 lib64
Expand Down
2 changes: 1 addition & 1 deletion dev-tools/docker/linux_aarch64_native_builder/Dockerfile
Expand Up @@ -10,7 +10,7 @@
#

# Increment the version here when a new tools/3rd party components image is built
FROM docker.elastic.co/ml-dev/ml-linux-aarch64-native-build:5
FROM docker.elastic.co/ml-dev/ml-linux-aarch64-native-build:6

MAINTAINER David Roberts <dave.roberts@elastic.co>

Expand Down
6 changes: 3 additions & 3 deletions dev-tools/docker/linux_aarch64_native_image/Dockerfile
Expand Up @@ -74,14 +74,14 @@ RUN \
# Build Boost
RUN \
cd ${build_dir} && \
wget --quiet -O - https://boostorg.jfrog.io/artifactory/main/release/1.71.0/source/boost_1_71_0.tar.bz2 | tar jxf - && \
cd boost_1_71_0 && \
wget --quiet -O - https://boostorg.jfrog.io/artifactory/main/release/1.77.0/source/boost_1_77_0.tar.bz2 | tar jxf - && \
cd boost_1_77_0 && \
./bootstrap.sh --without-libraries=context --without-libraries=coroutine --without-libraries=graph_parallel --without-libraries=mpi --without-libraries=python --without-icu && \
sed -i -e 's/(17ul)(29ul)(37ul)(53ul)(67ul)(79ul) \\/(3ul)(17ul)(29ul)(37ul)(53ul)(67ul)(79ul) \\/' boost/unordered/detail/implementation.hpp && \
./b2 -j`nproc` --layout=versioned --disable-icu pch=off optimization=speed inlining=full define=BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS define=BOOST_LOG_WITHOUT_DEBUG_OUTPUT define=BOOST_LOG_WITHOUT_EVENT_LOG define=BOOST_LOG_WITHOUT_SYSLOG define=BOOST_LOG_WITHOUT_IPC define=_FORTIFY_SOURCE=2 cxxflags='-std=gnu++17 -fstack-protector -march=armv8-a+crc+crypto' linkflags='-std=gnu++17 -Wl,-z,relro -Wl,-z,now' && \
./b2 install --prefix=/usr/local/gcc103 --layout=versioned --disable-icu pch=off optimization=speed inlining=full define=BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS define=BOOST_LOG_WITHOUT_DEBUG_OUTPUT define=BOOST_LOG_WITHOUT_EVENT_LOG define=BOOST_LOG_WITHOUT_SYSLOG define=BOOST_LOG_WITHOUT_IPC define=_FORTIFY_SOURCE=2 cxxflags='-std=gnu++17 -fstack-protector -march=armv8-a+crc+crypto' linkflags='-std=gnu++17 -Wl,-z,relro -Wl,-z,now' && \
cd .. && \
rm -rf boost_1_71_0
rm -rf boost_1_77_0

# Build patchelf
RUN \
Expand Down
2 changes: 1 addition & 1 deletion dev-tools/docker/linux_aarch64_native_tester/Dockerfile
Expand Up @@ -10,7 +10,7 @@
#

# Increment the version here when a new tools/3rd party components image is built
FROM docker.elastic.co/ml-dev/ml-linux-aarch64-native-build:5
FROM docker.elastic.co/ml-dev/ml-linux-aarch64-native-build:6

MAINTAINER David Roberts <dave.roberts@elastic.co>

Expand Down
2 changes: 1 addition & 1 deletion dev-tools/docker/linux_builder/Dockerfile
Expand Up @@ -10,7 +10,7 @@
#

# Increment the version here when a new tools/3rd party components image is built
FROM docker.elastic.co/ml-dev/ml-linux-build:19
FROM docker.elastic.co/ml-dev/ml-linux-build:20

MAINTAINER David Roberts <dave.roberts@elastic.co>

Expand Down
6 changes: 3 additions & 3 deletions dev-tools/docker/linux_image/Dockerfile
Expand Up @@ -75,14 +75,14 @@ RUN \
# Build Boost
RUN \
cd ${build_dir} && \
wget --quiet -O - https://boostorg.jfrog.io/artifactory/main/release/1.71.0/source/boost_1_71_0.tar.bz2 | tar jxf - && \
cd boost_1_71_0 && \
wget --quiet -O - https://boostorg.jfrog.io/artifactory/main/release/1.77.0/source/boost_1_77_0.tar.bz2 | tar jxf - && \
cd boost_1_77_0 && \
./bootstrap.sh --without-libraries=context --without-libraries=coroutine --without-libraries=graph_parallel --without-libraries=mpi --without-libraries=python --without-icu && \
sed -i -e 's/(17ul)(29ul)(37ul)(53ul)(67ul)(79ul) \\/(3ul)(17ul)(29ul)(37ul)(53ul)(67ul)(79ul) \\/' boost/unordered/detail/implementation.hpp && \
./b2 -j`nproc` --layout=versioned --disable-icu pch=off optimization=speed inlining=full define=BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS define=BOOST_LOG_WITHOUT_DEBUG_OUTPUT define=BOOST_LOG_WITHOUT_EVENT_LOG define=BOOST_LOG_WITHOUT_SYSLOG define=BOOST_LOG_WITHOUT_IPC define=_FORTIFY_SOURCE=2 cxxflags='-std=gnu++17 -fstack-protector -msse4.2 -mfpmath=sse' linkflags='-std=gnu++17 -Wl,-z,relro -Wl,-z,now' && \
./b2 install --prefix=/usr/local/gcc103 --layout=versioned --disable-icu pch=off optimization=speed inlining=full define=BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS define=BOOST_LOG_WITHOUT_DEBUG_OUTPUT define=BOOST_LOG_WITHOUT_EVENT_LOG define=BOOST_LOG_WITHOUT_SYSLOG define=BOOST_LOG_WITHOUT_IPC define=_FORTIFY_SOURCE=2 cxxflags='-std=gnu++17 -fstack-protector -msse4.2 -mfpmath=sse' linkflags='-std=gnu++17 -Wl,-z,relro -Wl,-z,now' && \
cd .. && \
rm -rf boost_1_71_0
rm -rf boost_1_77_0

# Build patchelf
RUN \
Expand Down
2 changes: 1 addition & 1 deletion dev-tools/docker/linux_tester/Dockerfile
Expand Up @@ -10,7 +10,7 @@
#

# Increment the version here when a new tools/3rd party components image is built
FROM docker.elastic.co/ml-dev/ml-linux-build:19
FROM docker.elastic.co/ml-dev/ml-linux-build:20

MAINTAINER David Roberts <dave.roberts@elastic.co>

Expand Down
2 changes: 1 addition & 1 deletion dev-tools/docker/macosx_builder/Dockerfile
Expand Up @@ -10,7 +10,7 @@
#

# Increment the version here when a new tools/3rd party components image is built
FROM docker.elastic.co/ml-dev/ml-macosx-build:12
FROM docker.elastic.co/ml-dev/ml-macosx-build:13

MAINTAINER David Roberts <dave.roberts@elastic.co>

Expand Down
2 changes: 1 addition & 1 deletion dev-tools/docker/macosx_image/Dockerfile
Expand Up @@ -31,7 +31,7 @@ RUN \
RUN \
mkdir -p /usr/local/sysroot-x86_64-apple-macosx10.14/usr && \
cd /usr/local/sysroot-x86_64-apple-macosx10.14/usr && \
wget --quiet -O - https://s3-eu-west-1.amazonaws.com/prelert-artifacts/dependencies/usr-x86_64-apple-macosx10.14-4.tar.bz2 | tar jxf - && \
wget --quiet -O - https://s3-eu-west-1.amazonaws.com/prelert-artifacts/dependencies/usr-x86_64-apple-macosx10.14-5.tar.bz2 | tar jxf - && \
wget --quiet -O - https://s3-eu-west-1.amazonaws.com/prelert-artifacts/dependencies/xcode-x86_64-apple-macosx10.14-1.tar.bz2 | tar jxf - && \
wget --quiet -O - https://s3-eu-west-1.amazonaws.com/prelert-artifacts/dependencies/sdk-x86_64-apple-macosx10.14-1.tar.bz2 | tar jxf -

Expand Down
2 changes: 1 addition & 1 deletion dev-tools/download_macos_deps.sh
Expand Up @@ -25,7 +25,7 @@ DEST=/usr
case `uname -m` in

arm64)
ARCHIVE=local-arm64-apple-macosx11.1-4.tar.bz2
ARCHIVE=local-arm64-apple-macosx11.1-5.tar.bz2
;;

*)
Expand Down
4 changes: 2 additions & 2 deletions dev-tools/download_windows_deps.ps1
Expand Up @@ -9,9 +9,9 @@
# limitation.
#
$ErrorActionPreference="Stop"
$Archive="usr-x86_64-windows-2016-4.zip"
$Archive="usr-x86_64-windows-2016-5.zip"
$Destination="C:\"
if (!(Test-Path "$Destination\usr\local\include\pytorch\torch\csrc\jit\passes\frozen_ops_to_mkldnn.h")) {
if (!(Test-Path "$Destination\usr\local\lib\boost_system-vc142-mt-x64-1_77.dll")) {
Remove-Item "$Destination\usr" -Recurse -Force -ErrorAction Ignore
$ZipSource="https://s3-eu-west-1.amazonaws.com/prelert-artifacts/dependencies/$Archive"
$ZipDestination="$Env:TEMP\$Archive"
Expand Down
2 changes: 1 addition & 1 deletion dev-tools/jenkins_ci.sh
Expand Up @@ -106,7 +106,7 @@ case `uname` in
# need to hardcode the version that was used to build Boost for that
# version of Elasticsearch.
if [ "$HARDWARE_ARCH" = aarch64 ] ; then
export BOOSTCLANGVER=120
export BOOSTCLANGVER=13
fi
(cd .. && ./gradlew --info -Dbuild.version_qualifier=$VERSION_QUALIFIER -Dbuild.snapshot=$BUILD_SNAPSHOT -Dbuild.ml_debug=$ML_DEBUG $TASKS)
;;
Expand Down
6 changes: 6 additions & 0 deletions docs/CHANGELOG.asciidoc
Expand Up @@ -28,6 +28,12 @@

//=== Regressions

== {es} version 8.1.0

=== Enhancements

* Upgrade Boost libraries to version 1.77. (See {ml-pull}2095[#2095].)

== {es} version 8.0.0

=== Bug Fixes
Expand Down
2 changes: 1 addition & 1 deletion lib/core/CNamedPipeFactory.cc
Expand Up @@ -60,7 +60,7 @@ const bool SIGPIPE_IGNORED{ignoreSigPipe()};
//! be a fatal error (see https://svn.boost.org/trac10/ticket/4913).
//!
//! This class is a reimplementation of the Sink concept (see
//! http://www.boost.org/doc/libs/1_71_0/libs/iostreams/doc/concepts/sink.html)
//! http://www.boost.org/doc/libs/1_77_0/libs/iostreams/doc/concepts/sink.html)
//! that will retry writes that get interrupted.
//!
class CRetryingFileDescriptorSink : private boost::iostreams::file_descriptor {
Expand Down