From 444dd0cd90f060b5c1a1f8239e16b20cd219fee0 Mon Sep 17 00:00:00 2001 From: David Roberts Date: Thu, 22 Feb 2018 15:23:46 +0000 Subject: [PATCH 1/5] Upgrade the compiler used on Linux We have already upgraded the macOS and Windows compilers on the master branch. This change upgrades the Linux compiler. The new compiler is gcc 7.3, upgraded from gcc 6.2. The OS used to build releases is also upgraded, to CentOS 6 from Fedora 10. gcc 7.3 is built from source with the --enable-vtable-verify option so that the -fvtable-verify=std option could possibly be used in the future. Finally, the -std flag has been changed to allow C++14 features to be used. This matches macOS and Windows. However, for ease of backporting it is best that we don't use C++14 features until version 6.x is nearing end-of-life. --- 3rd_party/3rd_party.sh | 14 +- build-setup/linux.md | 47 ++-- dev-tools/docker/build_fedora10_base_image.sh | 260 ------------------ .../docker/build_linux-musl_build_image.sh | 2 +- dev-tools/docker/build_linux_build_image.sh | 4 +- dev-tools/docker/linux-musl_image/Dockerfile | 6 +- dev-tools/docker/linux_builder/Dockerfile | 2 +- dev-tools/docker/linux_image/Dockerfile | 50 ++-- dev-tools/vagrant/linux/provision.sh | 63 ++--- include/core/CMemory.h | 26 -- lib/core/unittest/CMemoryUsageTest.cc | 6 - mk/linux-musl.mk | 2 +- mk/linux.mk | 12 +- set_env.sh | 4 +- 14 files changed, 100 insertions(+), 398 deletions(-) delete mode 100755 dev-tools/docker/build_fedora10_base_image.sh diff --git a/3rd_party/3rd_party.sh b/3rd_party/3rd_party.sh index b76491fde8..efaaede194 100755 --- a/3rd_party/3rd_party.sh +++ b/3rd_party/3rd_party.sh @@ -63,22 +63,22 @@ case `uname` in if [ -z "$CPP_CROSS_COMPILE" ] ; then ldd --version 2>&1 | grep musl > /dev/null if [ $? -ne 0 ] ; then - APR_LOCATION=/usr/local/gcc62/lib + APR_LOCATION=/usr/local/gcc73/lib APR_EXTENSION=1.so.0 - BOOST_LOCATION=/usr/local/gcc62/lib + BOOST_LOCATION=/usr/local/gcc73/lib BOOST_COMPILER=gcc BOOST_EXTENSION=mt-1_65_1.so.1.65.1 BOOST_LIBRARIES='date_time filesystem iostreams program_options regex system thread' - LOG4CXX_LOCATION=/usr/local/gcc62/lib + LOG4CXX_LOCATION=/usr/local/gcc73/lib LOG4CXX_EXTENSION=.so.10 - XML_LOCATION=/usr/local/gcc62/lib + XML_LOCATION=/usr/local/gcc73/lib XML_EXTENSION=.so.2 # Ship the version of expat that came with the apr-util library - EXPAT_LOCATION=/usr/local/gcc62/lib + EXPAT_LOCATION=/usr/local/gcc73/lib EXPAT_EXTENSION=.so.0 - GCC_RT_LOCATION=/usr/local/gcc62/lib64 + GCC_RT_LOCATION=/usr/local/gcc73/lib64 GCC_RT_EXTENSION=.so.1 - STL_LOCATION=/usr/local/gcc62/lib64 + STL_LOCATION=/usr/local/gcc73/lib64 STL_PREFIX=libstdc++ STL_EXTENSION=.so.6 ATOMIC_LOCATION= diff --git a/build-setup/linux.md b/build-setup/linux.md index 110c5a278c..2072e0a6b3 100644 --- a/build-setup/linux.md +++ b/build-setup/linux.md @@ -6,16 +6,16 @@ You will need the following environment variables to be defined: - `JAVA_HOME` - Should point to the JDK you want to use to run Gradle. - `CPP_SRC_HOME` - Only required if building the C++ code directly using `make`, as Gradle sets it automatically. -- `PATH` - Must have `/usr/local/gcc62/bin` before `/usr/bin` and `/bin`. -- `LD_LIBRARY_PATH` - Must have `/usr/local/gcc62/lib64` and `/usr/local/gcc62/lib` before `/usr/lib` and `/lib`. +- `PATH` - Must have `/usr/local/gcc73/bin` before `/usr/bin` and `/bin`. +- `LD_LIBRARY_PATH` - Must have `/usr/local/gcc73/lib64` and `/usr/local/gcc73/lib` before `/usr/lib` and `/lib`. For example, you might create a `.bashrc` file in your home directory containing something like this: ``` umask 0002 export JAVA_HOME=/usr/local/jdk1.8.0_121 -export LD_LIBRARY_PATH=/usr/local/gcc62/lib64:/usr/local/gcc62/lib:/usr/lib:/lib -export PATH=$JAVA_HOME/bin:/usr/local/gcc62/bin:/usr/bin:/bin:/usr/sbin:/sbin:/home/vagrant/bin +export LD_LIBRARY_PATH=/usr/local/gcc73/lib64:/usr/local/gcc73/lib:/usr/lib:/lib +export PATH=$JAVA_HOME/bin:/usr/local/gcc73/bin:/usr/bin:/bin:/usr/sbin:/sbin:/home/vagrant/bin # Only required if building the C++ code directly using make - adjust depending on the location of your Git clone export CPP_SRC_HOME=$HOME/ml-cpp ``` @@ -38,7 +38,7 @@ On other Linux distributions the package names are generally the same and you ju Most of the tools are built via a GNU "configure" script. There are some environment variables that affect the behaviour of this. Therefore, when building ANY tool on Linux, set the following environment variable: ``` -export CXX='g++ -std=gnu++0x' +export CXX='g++ -std=gnu++14' unset LIBRARY_PATH ``` @@ -48,22 +48,27 @@ The `CXX` environment variable only needs to be set when building tools on Linux We have to build on old Linux versions to enable our software to run on the older versions of Linux that users have. However, this means the default compiler on our Linux build servers is also very old. To enable use of more modern C++ features, we use the default compiler to build a newer version of gcc and then use that to build all our other dependencies. -Download `gcc-6.2.0.tar.bz2` from . +Download `gcc-7.3.0.tar.bz2` from . Unlike most automake-based tools, gcc must be built in a directory adjacent to the directory containing its source code, so build and install it like this: ``` -tar jxvf gcc-6.2.0.tar.bz2 -cd gcc-6.2.0 +tar jxvf gcc-7.3.0.tar.bz2 +cd gcc-7.3.0 contrib/download_prerequisites cd .. -mkdir gcc-6.2.0-build -cd gcc-6.2.0-build -../gcc-6.2.0/configure --prefix=/usr/local/gcc62 --enable-languages=c,c++ --with-system-zlib --disable-multilib +mkdir gcc-7.3.0-build +cd gcc-7.3.0-build +unset CXX +unset LD_LIBRARY_PATH +export PATH=/usr/bin:/bin:/usr/sbin:/sbin +../gcc-7.3.0/configure --prefix=/usr/local/gcc73 --enable-languages=c,c++ --enable-vtable-verify --with-system-zlib --disable-multilib make -j 6 sudo make install ``` +It's important that gcc itself is built using the system compiler in C++98 mode, hence the adjustment to `PATH` and unsetting of `CXX` and `LD_LIBRARY_PATH`. + ### Git Modern versions of Linux will come with Git in their package repositories, and (since we're not redistributing it so don't really care about the exact version used) this is the easiest way to install it. The command will be: @@ -107,7 +112,7 @@ Anonymous FTP to ftp.xmlsoft.org, change directory to libxml2, switch to binary Uncompress and untar the resulting file. Then run: ``` -./configure --prefix=/usr/local/gcc62 --without-python --without-readline +./configure --prefix=/usr/local/gcc73 --without-python --without-readline ``` This should build an appropriate Makefile. Assuming it does, type: @@ -132,7 +137,7 @@ tar xvf apr-1.5.2.tar Build using: ``` -./configure --prefix=/usr/local/gcc62 +./configure --prefix=/usr/local/gcc73 make sudo make install ``` @@ -150,7 +155,7 @@ tar xvf apr-util-1.5.4.tar Build using: ``` -./configure --prefix=/usr/local/gcc62 --with-apr=/usr/local/gcc62/bin/apr-1-config --with-expat=builtin +./configure --prefix=/usr/local/gcc73 --with-apr=/usr/local/gcc73/bin/apr-1-config --with-expat=builtin make sudo make install ``` @@ -179,7 +184,7 @@ to: ``` #if LOG4CXX_HELGRIND -#define _LOG4CXX_OBJECTPTR_INIT(x) : ObjectPtrBase() { exchange(x); +#define _LOG4CXX_OBJECTPTR_INIT(x) : ObjectPtrBase() { exchange(x); #else #define _LOG4CXX_OBJECTPTR_INIT(x) : ObjectPtrBase(), p(x) { #endif @@ -239,7 +244,7 @@ In `src/test/cpp/xml/domtestcase.cpp` replace `0x` with `(char)0x` on lines 193 Once all the changes are made, configure using: ``` -./configure --prefix=/usr/local/gcc62 --with-charset=utf-8 --with-logchar=utf-8 --with-apr=/usr/local/gcc62 --with-apr-util=/usr/local/gcc62 +./configure --prefix=/usr/local/gcc73 --with-charset=utf-8 --with-logchar=utf-8 --with-apr=/usr/local/gcc73 --with-apr-util=/usr/local/gcc73 ``` This should build an appropriate Makefile. Assuming it does, type: @@ -264,12 +269,12 @@ bzip2 -cd boost_1_65_1.tar.bz2 | tar xvf - In the resulting `boost_1_65_1` directory, run: ``` -./bootstrap.sh cxxflags=-std=gnu++0x --without-libraries=context --without-libraries=coroutine --without-libraries=graph_parallel --without-libraries=log --without-libraries=mpi --without-libraries=python --without-libraries=test --without-icu +./bootstrap.sh cxxflags=-std=gnu++14 --without-libraries=context --without-libraries=coroutine --without-libraries=graph_parallel --without-libraries=log --without-libraries=mpi --without-libraries=python --without-libraries=test --without-icu ``` This should build the `b2` program, which in turn is used to build Boost. -Edit boost/unordered/detail/implementation.hpp and change line 270 from: +Edit `boost/unordered/detail/implementation.hpp` and change line 270 from: ``` (17ul)(29ul)(37ul)(53ul)(67ul)(79ul) \ @@ -297,7 +302,7 @@ Finally, run: ``` ./b2 -j6 --layout=versioned --disable-icu pch=off optimization=speed inlining=full define=BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS -sudo env PATH="$PATH" ./b2 install --prefix=/usr/local/gcc62 --layout=versioned --disable-icu pch=off optimization=speed inlining=full define=BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS +sudo env PATH="$PATH" ./b2 install --prefix=/usr/local/gcc73 --layout=versioned --disable-icu pch=off optimization=speed inlining=full define=BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS ``` to install the Boost headers and libraries. (Note the `env PATH="$PATH"` bit in the install command - this is because `sudo` usually resets `PATH` and that will cause Boost to rebuild everything again with the default compiler as part of the install!) @@ -309,7 +314,7 @@ Download the latest version of cppunit from $WORK_DIR/etc/sysconfig/network < diff --git a/dev-tools/docker/linux_image/Dockerfile b/dev-tools/docker/linux_image/Dockerfile index 9481369745..70b6b44a94 100644 --- a/dev-tools/docker/linux_image/Dockerfile +++ b/dev-tools/docker/linux_image/Dockerfile @@ -13,7 +13,7 @@ # strictly prohibited. # -FROM droberts195/fedora:10 +FROM centos:6 # This is basically automating the setup instructions in build-setup/linux.md @@ -22,34 +22,34 @@ MAINTAINER David Roberts # Make sure OS packages are up to date and required packages are installed RUN \ rm /var/lib/rpm/__db.* && \ - yum install -y coreutils curl diffutils file gcc gcc-c++ git libselinux rpm sed tar unzip which zip zlib-devel + yum install -y gcc gcc-c++ git unzip wget zip zlib-devel -# For compiling in C++11 mode -ENV CXX "g++ -std=gnu++0x" - -# Build gcc 6.2 +# Build gcc 7.3 RUN \ - wget --quiet -O - http://ftpmirror.gnu.org/gcc/gcc-6.2.0/gcc-6.2.0.tar.gz | tar zxf - && \ - cd gcc-6.2.0 && \ + wget --quiet -O - http://ftpmirror.gnu.org/gcc/gcc-7.3.0/gcc-7.3.0.tar.gz | tar zxf - && \ + cd gcc-7.3.0 && \ contrib/download_prerequisites && \ cd .. && \ - mkdir gcc-6.2.0-build && \ - cd gcc-6.2.0-build && \ - ../gcc-6.2.0/configure --prefix=/usr/local/gcc62 --enable-languages=c,c++ --with-system-zlib --disable-multilib && \ + mkdir gcc-7.3.0-build && \ + cd gcc-7.3.0-build && \ + ../gcc-7.3.0/configure --prefix=/usr/local/gcc73 --enable-languages=c,c++ --enable-vtable-verify --with-system-zlib --disable-multilib && \ make -j`grep -c '^processor' /proc/cpuinfo` && \ make install && \ cd .. && \ - rm -rf gcc-6.2.0 gcc-6.2.0-build + rm -rf gcc-7.3.0 gcc-7.3.0-build # Update paths to use the newly built compiler -ENV LD_LIBRARY_PATH /usr/local/gcc62/lib64:/usr/local/gcc62/lib:/usr/lib:/lib -ENV PATH /usr/local/gcc62/bin:/usr/bin:/bin:/usr/sbin:/sbin +ENV LD_LIBRARY_PATH /usr/local/gcc73/lib64:/usr/local/gcc73/lib:/usr/lib:/lib +ENV PATH /usr/local/gcc73/bin:/usr/bin:/bin:/usr/sbin:/sbin + +# For compiling in C++14 mode +ENV CXX "g++ -std=gnu++14" # Build libxml2 RUN \ wget --quiet -O - ftp://anonymous@ftp.xmlsoft.org/libxml2/libxml2-2.9.4.tar.gz | tar zxf - && \ cd libxml2-2.9.4 && \ - ./configure --prefix=/usr/local/gcc62 --without-python --without-readline && \ + ./configure --prefix=/usr/local/gcc73 --without-python --without-readline && \ make -j`grep -c '^processor' /proc/cpuinfo` && \ make install && \ cd .. && \ @@ -59,7 +59,7 @@ RUN \ RUN \ wget --quiet -O - http://archive.apache.org/dist/apr/apr-1.5.2.tar.bz2 | tar jxf - && \ cd apr-1.5.2 && \ - ./configure --prefix=/usr/local/gcc62 && \ + ./configure --prefix=/usr/local/gcc73 && \ make -j`grep -c '^processor' /proc/cpuinfo` && \ make install && \ cd .. && \ @@ -69,7 +69,7 @@ RUN \ RUN \ wget --quiet -O - http://archive.apache.org/dist/apr/apr-util-1.5.4.tar.bz2 | tar jxf - && \ cd apr-util-1.5.4 && \ - ./configure --prefix=/usr/local/gcc62 --with-apr=/usr/local/gcc62/bin/apr-1-config --with-expat=builtin && \ + ./configure --prefix=/usr/local/gcc73 --with-apr=/usr/local/gcc73/bin/apr-1-config --with-expat=builtin && \ make -j`grep -c '^processor' /proc/cpuinfo` && \ make install && \ cd .. && \ @@ -89,7 +89,7 @@ RUN \ sed -i -e '39s/0x/(char)0x/g' src/main/cpp/objectoutputstream.cpp && \ sed -i -e '84,92s/0x/(char)0x/g' src/main/cpp/objectoutputstream.cpp && \ sed -i -e '193,214s/0x/(char)0x/g' src/test/cpp/xml/domtestcase.cpp && \ - ./configure --prefix=/usr/local/gcc62 --with-charset=utf-8 --with-logchar=utf-8 --with-apr=/usr/local/gcc62 --with-apr-util=/usr/local/gcc62 && \ + ./configure --prefix=/usr/local/gcc73 --with-charset=utf-8 --with-logchar=utf-8 --with-apr=/usr/local/gcc73 --with-apr-util=/usr/local/gcc73 && \ make -j`grep -c '^processor' /proc/cpuinfo` && \ make install && \ cd .. && \ @@ -99,11 +99,11 @@ RUN \ RUN \ wget --quiet -O - http://sourceforge.mirrorservice.org/b/bo/boost/boost/1.65.1/boost_1_65_1.tar.bz2 | tar jxf - && \ cd boost_1_65_1 && \ - ./bootstrap.sh cxxflags=-std=gnu++0x --without-libraries=context --without-libraries=coroutine --without-libraries=graph_parallel --without-libraries=log --without-libraries=mpi --without-libraries=python --without-libraries=test --without-icu && \ + ./bootstrap.sh cxxflags=-std=gnu++14 --without-libraries=context --without-libraries=coroutine --without-libraries=graph_parallel --without-libraries=log --without-libraries=mpi --without-libraries=python --without-libraries=test --without-icu && \ sed -i -e 's/(17ul)(29ul)(37ul)(53ul)(67ul)(79ul) \\/(3ul)(17ul)(29ul)(37ul)(53ul)(67ul)(79ul) \\/' boost/unordered/detail/implementation.hpp && \ sed -i -e 's%#if ((defined(__linux__) \&\& !defined(__UCLIBC__) \&\& !defined(BOOST_MATH_HAVE_FIXED_GLIBC)) || defined(__QNX__) || defined(__IBMCPP__)) \&\& !defined(BOOST_NO_FENV_H)%#if ((!defined(BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS) \&\& defined(__linux__) \&\& !defined(__UCLIBC__) \&\& !defined(BOOST_MATH_HAVE_FIXED_GLIBC)) || defined(__QNX__) || defined(__IBMCPP__)) \&\& !defined(BOOST_NO_FENV_H)%g' boost/math/tools/config.hpp && \ ./b2 -j`grep -c '^processor' /proc/cpuinfo` --layout=versioned --disable-icu pch=off optimization=speed inlining=full define=BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS && \ - ./b2 install --prefix=/usr/local/gcc62 --layout=versioned --disable-icu pch=off optimization=speed inlining=full define=BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS && \ + ./b2 install --prefix=/usr/local/gcc73 --layout=versioned --disable-icu pch=off optimization=speed inlining=full define=BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS && \ cd .. && \ rm -rf boost_1_65_1 @@ -111,7 +111,7 @@ RUN \ RUN \ wget --quiet -O - http://dev-www.libreoffice.org/src/cppunit-1.13.2.tar.gz | tar zxf - && \ cd cppunit-1.13.2 && \ - ./configure --prefix=/usr/local/gcc62 && \ + ./configure --prefix=/usr/local/gcc73 && \ make -j`grep -c '^processor' /proc/cpuinfo` && \ make install && \ cd .. && \ @@ -119,11 +119,11 @@ RUN \ # Build patchelf RUN \ - wget --quiet -O - http://nixos.org/releases/patchelf/patchelf-0.8/patchelf-0.8.tar.gz | tar zxf - && \ - cd patchelf-0.8 && \ - ./configure --prefix=/usr/local/gcc62 && \ + wget --quiet -O - http://nixos.org/releases/patchelf/patchelf-0.9/patchelf-0.9.tar.gz | tar zxf - && \ + cd patchelf-0.9 && \ + ./configure --prefix=/usr/local/gcc73 && \ make -j`grep -c '^processor' /proc/cpuinfo` && \ make install && \ cd .. && \ - rm -rf patchelf-0.8 + rm -rf patchelf-0.9 diff --git a/dev-tools/vagrant/linux/provision.sh b/dev-tools/vagrant/linux/provision.sh index ecd9a41ba4..1497faaca3 100644 --- a/dev-tools/vagrant/linux/provision.sh +++ b/dev-tools/vagrant/linux/provision.sh @@ -31,48 +31,38 @@ if [ ! -f java.state ]; then touch java.state fi -# Install Gradle -if [ ! -f gradle.state ]; then - echo "Installing Gradle..." - echo " Downloading Gradle..." - wget --quiet -O gradle.zip https://services.gradle.org/distributions/gradle-3.3-all.zip - unzip -qq gradle.zip -d gradle - rm gradle.zip - touch gradle.state -fi - -# Various env variables -echo "Setting env variables..." -export CXX='g++ -std=gnu++0x' - -# ----------------- Compile gcc 6.2 ------------------------- +# ----------------- Compile gcc 7.3 ------------------------- if [ ! -f gcc.state ]; then - echo "Compiling GCC 6.2..." + echo "Compiling GCC 7.3..." echo " Downloading..." - wget --quiet http://ftpmirror.gnu.org/gcc/gcc-6.2.0/gcc-6.2.0.tar.gz + wget --quiet http://ftpmirror.gnu.org/gcc/gcc-7.3.0/gcc-7.3.0.tar.gz # gcc needs different source and build directories mkdir gcc-source - tar xfz gcc-6.2.0.tar.gz -C gcc-source --strip-components=1 + tar xfz gcc-7.3.0.tar.gz -C gcc-source --strip-components=1 cd gcc-source contrib/download_prerequisites cd .. mkdir gcc-build cd gcc-build echo " Configuring..." - ../gcc-source/configure --prefix=/usr/local/gcc62 --enable-languages=c,c++ --with-system-zlib --disable-multilib > configure.log 2>&1 + ../gcc-source/configure --prefix=/usr/local/gcc73 --enable-languages=c,c++ --enable-vtable-verify --with-system-zlib --disable-multilib > configure.log 2>&1 echo " Making..." make -j$NUMCPUS --load-average=$NUMCPUS > make.log 2>&1 make install > make_install.log 2>&1 cd .. - rm gcc-6.2.0.tar.gz + rm gcc-7.3.0.tar.gz touch gcc.state fi # Update paths to use the newly built compiler -export LD_LIBRARY_PATH=/usr/local/gcc62/lib64:/usr/local/gcc62/lib:/usr/lib:/lib -export PATH=/usr/local/gcc62/bin:/usr/bin:/bin:/usr/sbin:/sbin:/home/vagrant/bin +export LD_LIBRARY_PATH=/usr/local/gcc73/lib64:/usr/local/gcc73/lib:/usr/lib:/lib +export PATH=/usr/local/gcc73/bin:/usr/bin:/bin:/usr/sbin:/sbin:/home/vagrant/bin + +# Various env variables +echo "Setting env variables..." +export CXX='g++ -std=gnu++14' -# ----------------- Compile libxml ------------------------- +# ----------------- Compile libxml2 ------------------------- if [ ! -f libxml2.state ]; then echo "Compiling libxml..." echo " Downloading..." @@ -82,7 +72,7 @@ if [ ! -f libxml2.state ]; then cd libxml sed -i -e 's/-O2/-O3/g' configure echo " Configuring..." - ./configure --prefix=/usr/local/gcc62 --without-python --without-readline > configure.log 2>&1 + ./configure --prefix=/usr/local/gcc73 --without-python --without-readline > configure.log 2>&1 echo " Making..." make -j$NUMCPUS --load-average=$NUMCPUS > make.log 2>&1 make install > make_install.log 2>&1 @@ -100,7 +90,7 @@ if [ ! -f apr.state ]; then tar xfz apr-1.5.2.tar.gz -C apr --strip-components=1 cd apr echo " Configuring..." - ./configure --prefix=/usr/local/gcc62 > configure.log 2>&1 + ./configure --prefix=/usr/local/gcc73 > configure.log 2>&1 echo " Making..." make -j$NUMCPUS --load-average=$NUMCPUS > make.log 2>&1 make install > make_install.log 2>&1 @@ -118,7 +108,7 @@ if [ ! -f apr-util.state ]; then tar xfz apr-util-1.5.4.tar.gz -C apr-util --strip-components=1 cd apr-util echo " Configuring..." - ./configure --prefix=/usr/local/gcc62 --with-apr=/usr/local/gcc62/bin/apr-1-config --with-expat=builtin > configure.log 2>&1 + ./configure --prefix=/usr/local/gcc73 --with-apr=/usr/local/gcc73/bin/apr-1-config --with-expat=builtin > configure.log 2>&1 echo " Making..." make -j$NUMCPUS --load-average=$NUMCPUS > make.log 2>&1 make install > make_install.log 2>&1 @@ -151,7 +141,7 @@ if [ ! -f log4cxx.state ]; then sed -i -e '84,92s/0x/(char)0x/g' src/main/cpp/objectoutputstream.cpp sed -i -e '193,214s/0x/(char)0x/g' src/test/cpp/xml/domtestcase.cpp - ./configure --prefix=/usr/local/gcc62 --with-charset=utf-8 --with-logchar=utf-8 --with-apr=/usr/local/gcc62 --with-apr-util=/usr/local/gcc62 + ./configure --prefix=/usr/local/gcc73 --with-charset=utf-8 --with-logchar=utf-8 --with-apr=/usr/local/gcc73 --with-apr-util=/usr/local/gcc73 echo " Making..." make -j$NUMCPUS --load-average=$NUMCPUS > make.log 2>&1 @@ -171,7 +161,7 @@ if [ ! -f boost.state ]; then cd boost echo " Bootstrapping..." - ./bootstrap.sh cxxflags=-std=gnu++0x --without-libraries=context --without-libraries=coroutine --without-libraries=graph_parallel --without-libraries=log --without-libraries=mpi --without-libraries=python --without-libraries=test --without-icu > bootstrap.log 2>&1 + ./bootstrap.sh cxxflags=-std=gnu++14 --without-libraries=context --without-libraries=coroutine --without-libraries=graph_parallel --without-libraries=log --without-libraries=mpi --without-libraries=python --without-libraries=test --without-icu > bootstrap.log 2>&1 echo " Configuring..." @@ -180,7 +170,7 @@ if [ ! -f boost.state ]; then echo " Building..." ./b2 -j$NUMCPUS --layout=versioned --disable-icu pch=off optimization=speed inlining=full define=BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS > b2_make.log 2>&1 - ./b2 install --prefix=/usr/local/gcc62 --layout=versioned --disable-icu pch=off optimization=speed inlining=full define=BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS > b2_make_install.log 2>&1 + ./b2 install --prefix=/usr/local/gcc73 --layout=versioned --disable-icu pch=off optimization=speed inlining=full define=BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS > b2_make_install.log 2>&1 cd .. rm boost_1_65_1.tar.gz @@ -197,7 +187,7 @@ if [ ! -f cppunit.state ]; then tar xfz cppunit-1.13.2.tar.gz -C cppunit --strip-components=1 cd cppunit echo " Configuring..." - ./configure --prefix=/usr/local/gcc62 > configure.log 2>&1 + ./configure --prefix=/usr/local/gcc73 > configure.log 2>&1 echo " Making..." make -j$NUMCPUS --load-average=$NUMCPUS > make.log 2>&1 make install > make_install.log 2>&1 @@ -210,17 +200,17 @@ fi if [ ! -f patchelf.state ]; then echo "Compiling patchelf..." echo " Downloading..." - wget --quiet http://nixos.org/releases/patchelf/patchelf-0.8/patchelf-0.8.tar.gz + wget --quiet http://nixos.org/releases/patchelf/patchelf-0.9/patchelf-0.9.tar.gz mkdir patchelf - tar xfz patchelf-0.8.tar.gz -C patchelf --strip-components=1 + tar xfz patchelf-0.9.tar.gz -C patchelf --strip-components=1 cd patchelf echo " Configuring..." - ./configure --prefix=/usr/local/gcc62 > configure.log 2>&1 + ./configure --prefix=/usr/local/gcc73 > configure.log 2>&1 echo " Making..." make -j$NUMCPUS --load-average=$NUMCPUS > make.log 2>&1 make install > make_install.log 2>&1 cd .. - rm patchelf-0.8.tar.gz + rm patchelf-0.9.tar.gz touch patchelf.state fi @@ -244,9 +234,8 @@ umask 0002 export ML_SRC_HOME=/home/vagrant/ml/src export JAVA_HOME=/usr/lib/jvm/java-8-oracle unset JAVA_ROOT -export GRADLE_HOME=/home/vagrant/gradle/gradle-3.3 -export LD_LIBRARY_PATH=/usr/local/gcc62/lib64:/usr/local/gcc62/lib:/usr/lib:/lib -export PATH=$JAVA_HOME/bin:$GRADLE_HOME/bin:/usr/local/node-v6.9.0-linux-x64/bin:/usr/local/gcc62/bin:/usr/bin:/bin:/usr/sbin:/sbin:/home/vagrant/bin +export LD_LIBRARY_PATH=/usr/local/gcc73/lib64:/usr/local/gcc73/lib:/usr/lib:/lib +export PATH=$JAVA_HOME/bin:/bin:/usr/local/gcc73/bin:/usr/bin:/bin:/usr/sbin:/sbin:/home/vagrant/bin ulimit -c unlimited cd /home/vagrant/ml/src diff --git a/include/core/CMemory.h b/include/core/CMemory.h index 0c3b5e57d4..7f80671221 100644 --- a/include/core/CMemory.h +++ b/include/core/CMemory.h @@ -425,19 +425,6 @@ class CORE_EXPORT CMemory : private CNonInstantiatable } return capacity + 1; -#elif defined(Linux) && (!defined(_GLIBCXX_USE_CXX11_ABI) || _GLIBCXX_USE_CXX11_ABI == 0) - // All sizes > 0 use the heap, and the string structure is - // 1 pointer + 2 sizes + 1 null terminator - // We don't handle the reference counting, so may overestimate - // Even some 0 length strings may use the heap - see - // http://info.prelert.com/blog/clearing-strings - if (capacity == 0 && - t.data() == EMPTY_STRING.data()) - { - return 0; - } - return capacity + sizeof(void *) + (2 * sizeof(std::size_t)) + 1; - #else // Linux with C++11 ABI and Windows // For lengths up to 15 bytes there is no allocation if (capacity <= 15) @@ -948,19 +935,6 @@ class CORE_EXPORT CMemoryDebug : private CNonInstantiatable capacity = 0; } -#elif defined(Linux) && (!defined(_GLIBCXX_USE_CXX11_ABI) || _GLIBCXX_USE_CXX11_ABI == 0) - // All sizes > 0 use the heap, and the string structure is - // 1 pointer + 2 sizes + 1 null terminator - // We don't handle the reference counting, so may overestimate - // Even some 0 length strings may use the heap - see - // http://info.prelert.com/blog/clearing-strings - if (capacity > 0 || - t.data() != EMPTY_STRING.data()) - { - unused = capacity - length; - capacity += sizeof(void *) + (2 * sizeof(std::size_t)) + 1; - } - #else // Linux with C++11 ABI and Windows // For lengths up to 15 bytes there is no allocation if (capacity > 15) diff --git a/lib/core/unittest/CMemoryUsageTest.cc b/lib/core/unittest/CMemoryUsageTest.cc index b6f63fe6c3..e1922e0d51 100644 --- a/lib/core/unittest/CMemoryUsageTest.cc +++ b/lib/core/unittest/CMemoryUsageTest.cc @@ -1137,13 +1137,7 @@ void CMemoryUsageTest::testStringClear(void) // If the following assertion fails after a standard library upgrade then // the logic in include/core/CMemory.h needs changing as well as this test -#if defined(Linux) && (!defined(_GLIBCXX_USE_CXX11_ABI) || _GLIBCXX_USE_CXX11_ABI == 0) - // This seemingly strange result is due to the phenonenon described in - // http://info.prelert.com/blog/clearing-strings - CPPUNIT_ASSERT(TAllocator::usage() > usage3Copies); -#else CPPUNIT_ASSERT_EQUAL(usage3Copies, TAllocator::usage()); -#endif } void CMemoryUsageTest::testSharedPointer(void) diff --git a/mk/linux-musl.mk b/mk/linux-musl.mk index 12efc789fe..94c5771280 100644 --- a/mk/linux-musl.mk +++ b/mk/linux-musl.mk @@ -18,7 +18,7 @@ OS=Linux CPP_PLATFORM_HOME=$(CPP_DISTRIBUTION_HOME)/platform/linux-musl-x86_64 CC=gcc -CXX=g++ -std=gnu++0x +CXX=g++ -std=gnu++14 ifndef ML_DEBUG OPTCFLAGS=-O3 -Wdisabled-optimization diff --git a/mk/linux.mk b/mk/linux.mk index ff4a8aa94e..ab7e553755 100644 --- a/mk/linux.mk +++ b/mk/linux.mk @@ -18,7 +18,7 @@ OS=Linux CPP_PLATFORM_HOME=$(CPP_DISTRIBUTION_HOME)/platform/linux-x86_64 CC=gcc -CXX=g++ -std=gnu++0x +CXX=g++ -std=gnu++14 ifndef ML_DEBUG OPTCFLAGS=-O3 -Wdisabled-optimization @@ -36,7 +36,7 @@ PLATPICFLAGS=-fPIC PLATPIEFLAGS=-fPIE CFLAGS=-g $(OPTCFLAGS) -msse4.2 -mfpmath=sse -fstack-protector -fno-math-errno -fno-permissive -Wall -Wcast-align -Wconversion -Wextra -Winit-self -Wparentheses -Wpointer-arith -Wswitch-enum $(COVERAGE) CXXFLAGS=$(CFLAGS) -Wno-ctor-dtor-privacy -Wno-deprecated-declarations -Wold-style-cast -fvisibility-inlines-hidden -CPPFLAGS=-isystem $(CPP_SRC_HOME)/3rd_party/include -isystem /usr/local/gcc62/include -D$(OS) -DLINUX=2 -D_REENTRANT $(OPTCPPFLAGS) +CPPFLAGS=-isystem $(CPP_SRC_HOME)/3rd_party/include -isystem /usr/local/gcc73/include -D$(OS) -DLINUX=2 -D_REENTRANT $(OPTCPPFLAGS) CDEPFLAGS=-MM COMP_OUT_FLAG=-o LINK_OUT_FLAG=-o @@ -46,7 +46,7 @@ NETLIBS=-lnsl BOOSTVER=1_65_1 BOOSTGCCVER:=$(shell $(CXX) -dumpversion | awk -F. '{ print $$1$$2; }') # Use -isystem instead of -I for Boost headers to suppress warnings from Boost -BOOSTINCLUDES=-isystem /usr/local/gcc62/include/boost-$(BOOSTVER) +BOOSTINCLUDES=-isystem /usr/local/gcc73/include/boost-$(BOOSTVER) BOOSTCPPFLAGS=-DBOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS BOOSTREGEXLIBS=-lboost_regex-gcc$(BOOSTGCCVER)-mt-$(BOOSTVER) BOOSTIOSTREAMSLIBS=-lboost_iostreams-gcc$(BOOSTGCCVER)-mt-$(BOOSTVER) @@ -57,8 +57,8 @@ BOOSTDATETIMELIBS=-lboost_date_time-gcc$(BOOSTGCCVER)-mt-$(BOOSTVER) RAPIDJSONINCLUDES=-isystem $(CPP_SRC_HOME)/3rd_party/rapidjson/include RAPIDJSONCPPFLAGS=-DRAPIDJSON_HAS_STDSTRING -DRAPIDJSON_SSE42 EIGENCPPFLAGS=-DEIGEN_MPL2_ONLY -XMLINCLUDES=`/usr/local/gcc62/bin/xml2-config --cflags` -XMLLIBS=`/usr/local/gcc62/bin/xml2-config --libs` +XMLINCLUDES=`/usr/local/gcc73/bin/xml2-config --cflags` +XMLLIBS=`/usr/local/gcc73/bin/xml2-config --libs` DYNAMICLIBLDFLAGS=$(PLATPICFLAGS) -shared -Wl,--as-needed -L$(CPP_PLATFORM_HOME)/lib $(COVERAGE) -Wl,-z,relro -Wl,-z,now -Wl,-rpath,'$$ORIGIN/.' JAVANATIVEINCLUDES=-I$(JAVA_HOME)/include JAVANATIVELDFLAGS=-L$(JAVA_HOME)/jre/lib/server @@ -83,7 +83,7 @@ LIB_ML_CONFIG=-lMlConfig LIB_ML_MODEL=-lMlModel LIB_ML_TEST=-lMlTest -LIB_PATH+=-L/usr/local/gcc62/lib +LIB_PATH+=-L/usr/local/gcc73/lib # Using cp instead of install here, to avoid every file being given execute # permissions diff --git a/set_env.sh b/set_env.sh index 58a5885fad..8b074d0da3 100755 --- a/set_env.sh +++ b/set_env.sh @@ -105,7 +105,7 @@ fi case $SIMPLE_PLATFORM in linux) - PATH=/usr/local/gcc62/bin:/usr/bin:/bin:/usr/local/gcc62/sbin:/usr/sbin:/sbin + PATH=/usr/local/gcc73/bin:/usr/bin:/bin:/usr/local/gcc73/sbin:/usr/sbin:/sbin ;; linux-musl) @@ -134,7 +134,7 @@ fi case $SIMPLE_PLATFORM in linux) - export LD_LIBRARY_PATH=/usr/local/gcc62/lib64:/usr/local/gcc62/lib:/usr/lib:/lib + export LD_LIBRARY_PATH=/usr/local/gcc73/lib64:/usr/local/gcc73/lib:/usr/lib:/lib ;; linux-musl) From d5b4d7078200d029007a6b76464b6906cd83b50a Mon Sep 17 00:00:00 2001 From: David Roberts Date: Thu, 8 Mar 2018 13:48:42 +0000 Subject: [PATCH 2/5] gcc source code no longer available in .bz2 format (Also removed a reference to YaST) --- build-setup/linux.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/build-setup/linux.md b/build-setup/linux.md index 2072e0a6b3..3b0b0b928a 100644 --- a/build-setup/linux.md +++ b/build-setup/linux.md @@ -48,12 +48,12 @@ The `CXX` environment variable only needs to be set when building tools on Linux We have to build on old Linux versions to enable our software to run on the older versions of Linux that users have. However, this means the default compiler on our Linux build servers is also very old. To enable use of more modern C++ features, we use the default compiler to build a newer version of gcc and then use that to build all our other dependencies. -Download `gcc-7.3.0.tar.bz2` from . +Download `gcc-7.3.0.tar.gz` from . Unlike most automake-based tools, gcc must be built in a directory adjacent to the directory containing its source code, so build and install it like this: ``` -tar jxvf gcc-7.3.0.tar.bz2 +tar zxvf gcc-7.3.0.tar.gz cd gcc-7.3.0 contrib/download_prerequisites cd .. @@ -77,7 +77,7 @@ Modern versions of Linux will come with Git in their package repositories, and ( sudo yum install git ``` -on RHEL clones, or for SLES use YaST. **However**, Jenkins requires at minimum version 1.7.9 of Git, so if the version that yum installs is older you'll still have to build it from scratch. In this case, you may need to uninstall the version that yum installed: +on RHEL clones. **However**, Jenkins requires at minimum version 1.7.9 of Git, so if the version that yum installs is older you'll still have to build it from scratch. In this case, you may need to uninstall the version that yum installed: ``` git --version From f779713c86831528f4b7317178bde89975a54f7f Mon Sep 17 00:00:00 2001 From: David Roberts Date: Fri, 9 Mar 2018 10:28:59 +0000 Subject: [PATCH 3/5] Correct download links for APR --- build-setup/linux.md | 8 ++++---- build-setup/windows.md | 12 ++++++------ dev-tools/vagrant/linux/provision.sh | 4 ++-- 3 files changed, 12 insertions(+), 12 deletions(-) diff --git a/build-setup/linux.md b/build-setup/linux.md index 3b0b0b928a..8af1bdc724 100644 --- a/build-setup/linux.md +++ b/build-setup/linux.md @@ -126,12 +126,12 @@ to install. ### APR -For Linux, before building log4cxx you must download the Apache Portable Runtime (APR), and its utilities from . +For Linux, before building log4cxx you must download the Apache Portable Runtime (APR) from . Extract the tarball to a temporary directory: ``` -tar xvf apr-1.5.2.tar +tar jxvf apr-1.5.2.tar.bz2 ``` Build using: @@ -144,12 +144,12 @@ sudo make install ### APR utilities -For Linux, before building log4cxx you must download the Apache Portable Runtime (APR) utilities from . +For Linux, before building log4cxx you must download the Apache Portable Runtime (APR) utilities from . Extract the tarball to a temporary directory: ``` -tar xvf apr-util-1.5.4.tar +tar jxvf apr-util-1.5.4.tar.bz2 ``` Build using: diff --git a/build-setup/windows.md b/build-setup/windows.md index bab1963428..289866c3ee 100644 --- a/build-setup/windows.md +++ b/build-setup/windows.md @@ -161,9 +161,9 @@ nmake install ### APR -Download the Windows source for version 1.5.2 of the Apache Portable Runtime (APR) from . +Download the Windows source for version 1.5.2 of the Apache Portable Runtime (APR) from . -The file should be called `apr-1.5.2-win32-src.zip`. Unzip the file into `C:\tools`. You should end up with a subdirectory called `apr-1.5.2`. Rename this to simply `apr`. +Unzip the file into `C:\tools`. You should end up with a subdirectory called `apr-1.5.2`. Rename this to simply `apr`. Within the files `C:\tools\apr\apr.mak` and `C:\tools\apr\libapr.mak`, globally replace: @@ -189,9 +189,9 @@ nmake -f Makefile.win PREFIX=C:\usr\local ARCH="x64 Release" install ### APR iconv -Download the Windows source for version 1.2.1 of the Apache Portable Runtime (APR) version of iconv from . +Download the Windows source for version 1.2.1 of the Apache Portable Runtime (APR) version of iconv from . -The file should be called `apr-iconv-1.2.1-win32-src.zip`. Unzip the file into `C:\tools`. You should end up with a subdirectory called `apr-iconv-1.2.1`. Rename this to simply `apr-iconv`. +Unzip the file into `C:\tools`. You should end up with a subdirectory called `apr-iconv-1.2.1`. Rename this to simply `apr-iconv`. Within the files `C:\tools\apr-iconv\apriconv.mak` and `C:\tools\apr-iconv\libapriconv.mak`, globally replace: @@ -229,9 +229,9 @@ Note that this does not install all the iconv utilities - this will be done when ### APR util -Download the Windows source for version 1.5.4 of the Apache Portable Runtime (APR) utilities from . +Download the Windows source for version 1.5.4 of the Apache Portable Runtime (APR) utilities from . -The file should be called `apr-util-1.5.4-win32-src.zip`. Unzip the file into `C:\tools`. You should end up with a subdirectory called `apr-util-1.5.4`. Rename this to simply `apr-util`. +Unzip the file into `C:\tools`. You should end up with a subdirectory called `apr-util-1.5.4`. Rename this to simply `apr-util`. Within the files `C:\tools\apr-util\aprutil.mak` and `C:\tools\apr-util\libaprutil.mak`, globally replace: diff --git a/dev-tools/vagrant/linux/provision.sh b/dev-tools/vagrant/linux/provision.sh index 1497faaca3..783eb1c637 100644 --- a/dev-tools/vagrant/linux/provision.sh +++ b/dev-tools/vagrant/linux/provision.sh @@ -85,7 +85,7 @@ fi if [ ! -f apr.state ]; then echo "Compiling APR..." echo " Downloading..." - wget --quiet http://mirrors.sonic.net/apache//apr/apr-1.5.2.tar.gz + wget --quiet http://archive.apache.org/dist/apr/apr-1.5.2.tar.gz mkdir apr tar xfz apr-1.5.2.tar.gz -C apr --strip-components=1 cd apr @@ -103,7 +103,7 @@ fi if [ ! -f apr-util.state ]; then echo "Compiling APR-Utilities..." echo " Downloading..." - wget --quiet http://mirrors.sonic.net/apache//apr/apr-util-1.5.4.tar.gz + wget --quiet http://archive.apache.org/dist/apr/apr-util-1.5.4.tar.gz mkdir apr-util tar xfz apr-util-1.5.4.tar.gz -C apr-util --strip-components=1 cd apr-util From 74611ddfda8bea4e07b066ee8df42e7d3e86b692 Mon Sep 17 00:00:00 2001 From: David Roberts Date: Fri, 9 Mar 2018 15:23:56 +0000 Subject: [PATCH 4/5] Add note about reseting CXX after building gcc --- build-setup/linux.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/build-setup/linux.md b/build-setup/linux.md index 8af1bdc724..106e306fe2 100644 --- a/build-setup/linux.md +++ b/build-setup/linux.md @@ -69,6 +69,12 @@ sudo make install It's important that gcc itself is built using the system compiler in C++98 mode, hence the adjustment to `PATH` and unsetting of `CXX` and `LD_LIBRARY_PATH`. +After the gcc build is complete, if you are going to work through the rest of these instructions in the same shell remember to reset the `CXX` environment variable so that the remaining C++ components get built with C++14: + +``` +export CXX='g++ -std=gnu++14' +``` + ### Git Modern versions of Linux will come with Git in their package repositories, and (since we're not redistributing it so don't really care about the exact version used) this is the easiest way to install it. The command will be: From a2e2a1362cd06e4395ff14531aa785c55fee2c03 Mon Sep 17 00:00:00 2001 From: David Roberts Date: Fri, 9 Mar 2018 16:07:03 +0000 Subject: [PATCH 5/5] Add a note about checking the g++ version --- build-setup/linux.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/build-setup/linux.md b/build-setup/linux.md index 106e306fe2..b2d5b39db9 100644 --- a/build-setup/linux.md +++ b/build-setup/linux.md @@ -75,6 +75,20 @@ After the gcc build is complete, if you are going to work through the rest of th export CXX='g++ -std=gnu++14' ``` +To confirm that everything works correctly run: + +``` +g++ --version +``` + +It should print: + +``` +g++ (GCC) 7.3.0 +``` + +in the first line of the output. If it doesn't then double check that `/usr/local/gcc73/bin` is near the beginning of your `PATH`. + ### Git Modern versions of Linux will come with Git in their package repositories, and (since we're not redistributing it so don't really care about the exact version used) this is the easiest way to install it. The command will be: