Skip to content

Building HPCC

Gavin Halliday edited this page Apr 15, 2019 · 111 revisions

Assuming you already have your git repository set up (see Git-step-by-step), you can build and install HPCC.

Prerequisites

The build process requires some additional packages: NodeJS (this is used to package ECL Watch and related web pages) - recommended version 8.x.x LTS:

https://nodejs.org/en/download/package-manager/

On Linux (Ubuntu 18.04):

sudo apt-get install cmake bison flex build-essential binutils-dev libldap2-dev libcppunit-dev libicu-dev libxslt1-dev zlib1g-dev libboost-regex-dev libarchive-dev python-dev libv8-dev default-jdk libapr1-dev libaprutil1-dev libiberty-dev libhiredis-dev libtbb-dev libxalan-c-dev libnuma-dev nodejs libevent-dev libatlas-base-dev libblas-dev python3-dev default-libmysqlclient-dev libsqlite3-dev r-base-dev r-cran-rcpp r-cran-rinside r-cran-inline libmemcached-dev libcurl4-openssl-dev pkg-config libtool autotools-dev automake libssl-dev

On Linux (Ubuntu 17.10):

sudo apt-get install cmake bison flex build-essential binutils-dev libldap2-dev libcppunit-dev libicu-dev libxslt1-dev zlib1g-dev libboost-regex-dev libssl-dev libarchive-dev python-dev libv8-dev default-jdk libapr1-dev libaprutil1-dev libiberty-dev libhiredis-dev libtbb-dev libxalan-c-dev libnuma-dev nodejs libevent-dev libatlas-base-dev libblas-dev python3-dev default-libmysqlclient-dev libsqlite3-dev r-base-dev r-cran-rcpp r-cran-rinside r-cran-inline libmemcached-dev libcurl4-openssl-dev pkg-config libtool autotools-dev automake

On Linux (Ubuntu 16.04):

sudo apt-get install cmake bison flex build-essential binutils-dev libldap2-dev libcppunit-dev libicu-dev libxslt1-dev zlib1g-dev libboost-regex-dev libssl-dev libarchive-dev python-dev libv8-dev default-jdk libapr1-dev libaprutil1-dev libiberty-dev libhiredis-dev libtbb-dev libxalan-c-dev libnuma-dev libevent-dev libatlas-base-dev libblas-dev libatlas-dev python3-dev libcurl4-openssl-dev libtool autotools-dev automake

On Linux (Ubuntu 14.04):

sudo apt-get install g++ gcc make cmake bison flex binutils-dev libldap2-dev libcppunit-dev libicu-dev libxslt1-dev zlib1g-dev libboost-regex-dev libssl-dev libarchive-dev python2.7-dev libv8-dev openjdk-7-jdk libapr1-dev libaprutil1-dev libiberty-dev libhiredis-dev libtbb-dev libxalan-c-dev libnuma-dev libevent-dev libatlas-base-dev python3-dev libcurl4-openssl-dev libtool autotools-dev automake

Additional information for building versions prior to 7.0 on Ubuntu 18.04

get openssl-1.0.2o.tar.gz from https://www.openssl.org/source/

unpack, build, and install:

./config -fPIC shared
make
make install

This installs to /usr/local/ssl by default.

Build the platform with the following additional CMake options:

-DOPENSSL_LIBRARIES=/usr/local/ssl/lib/libssl.so -DOPENSSL_SSL_LIBRARY=/usr/local/ssl/lib/libssl.so

Centos Builds

Regardless of which version of Centos you will be building on, it is suggested that you enable the EPEL repository

sudo yum install -y epel-release

CentOS 6.4:

sudo yum install gcc-c++ gcc make bison flex binutils-devel openldap-devel libicu-devel libxslt-devel libarchive-devel boost-devel openssl-devel apr-devel apr-util-devel hiredis-devel numactl-devel libmysqlclient-dev libevent-devel tbb-devel atlas-devel python34 R-core-devel R-Rcpp-devel R-inline R-RInside-devel nodejs libcurl-devel

CentOS 7:

sudo yum install gcc-c++ gcc make bison flex binutils-devel openldap-devel libicu-devel libxslt-devel libarchive-devel boost-devel openssl-devel apr-devel apr-util-devel hiredis-devel numactl-devel mariadb-devel libevent-devel tbb-devel atlas-devel python34 libmemcached-devel sqlite-devel v8-devel python-devel python34-devel java-1.8.0-openjdk-devel R-core-devel R-Rcpp-devel R-inline R-RInside-devel nodejs cmake3 rpm-build libcurl-devel

The version of cmake that is in the Centos 6 and 7 repositories is too old, so you will need to install cmake3 and use that instead. Alternatively you can download the source from www.cmake.org and build and install from source - this is quite straightforward.

Cmake build and installation example
wget https://cmake.org/files/v3.7/cmake-3.7.2.tar.gz
tar xzf cmake-3.7.2.tar.gz
cd cmake-3.7.2
./bootstrap
make -j2
sudo make install
Other Linux Distributions

On Linux (Fedora 19):

sudo yum install gcc-c++ gcc make fedora-packager cmake bison flex binutils-devel openldap-devel libicu-devel  xerces-c-devel xalan-c-devel libarchive-devel boost-devel openssl-devel apr-devel apr-util-devel

On Linux (Fedora 23):

sudo dnf install gcc-c++ gcc make fedora-packager cmake bison flex binutils-devel openldap-devel libicu-devel  xerces-c-devel xalan-c-devel libarchive-devel boost-devel openssl-devel apr-devel apr-util-devel numactl-devel tbb-devel libxslt-devel nodejs

On Mac (Snow Leopard):

sudo port install bison flex binutils openldap icu xalanc zlib boost openssl libarchive

You can disable some functionality in order to reduce the list of required components, if necessary. Optional components, such as the plugins for interfacing to external languages, will be disabled automatically if the required libraries and headers are not found at build time.

Optional dependencies:

To build with support for all the plugins for third party embedded code, additional dependencies may be required. If not found, the system will default to skipping those components.

Building with R Support:

First insure that the R language is installed on your system. For Ubuntu use sudo apt-get install r-base-dev. For centos distributions use sudo yum install -y R-core-devel.

To install the prerequisites for building R support, use the following for all distros:

wget https://cran.r-project.org/src/contrib/00Archive/Rcpp/Rcpp_0.12.1.tar.gz
wget https://cran.r-project.org/src/contrib/00Archive/RInside/RInside_0.2.12.tar.gz
wget http://cran.r-project.org/src/contrib/inline_0.3.14.tar.gz
sudo R CMD INSTALL Rcpp_0.12.1.tar.gz RInside_0.2.12.tar.gz inline_0.3.14.tar.gz
Building CouchBase plugin:

To install the prerequisites for libcouchbase, go to http://developer.couchbase.com/documentation/server/current/sdk/c/start-using-sdk.html and install the libcouchbase repo for your distribution. Then add the libcouchbase libraries and headers to your system.

Building with Spark support:

The following Cmake flags are useful for developers building 7.0.0+:

  • SPARK=ON enables building of the spark plugin package
  • INCLUDE_SPARK=ON includes the spark plugin in a platform build
  • SUPPRESS_SPARK=ON suppresses the building of a spark plugin in an INCLUDE_PLUGINS=ON build with the platform
  • SPARK_URL select a specific version of spark to build against
  • SPARK_HPCC_VERSION and WSCLIENT_VERSION select a specific version of spark-hpcc to fetch and build against
  • SPARK_HPCC_JAR and WSCLIENT_JAR absolute file paths for using local jars instead of downloading from maven central

Building on windows

From 7.4 the build system has been changed to make it easier to build on windows. It is dependent on two other projects chocolatey and vcpkg for installing the dependencies:

Install chocolatey:

https://chocolatey.org/install

Use that to install bison/flex:

choco install winflexbison3

Install vcpkg:

git clone https://github.com/Microsoft/vcpkg
cd vcpkg
bootstrap-vcpkg

Use vcpkg to install various packages:

vcpkg install zlib
vcpkg install boost
vcpkg install icu
vcpkg install libxslt
vcpkg install tbb
vcpkg install cppunit
vcpkg install libarchive
vcpkg install apr
vcpkg install apr-util

You may need to force vcpkg to build the correct version, e.g. for 64bit:

vcpkg install zlib zlib:x64-windows

Get Latest Sources

(See Git-step-by-step for full instructions)

git fetch upstream
git checkout master
git pull
git submodule update --init --recursive

Alternatively, for a quick build, you can simply do..

git clone [-b <branch name>] --recurse-submodules https://github.com/hpcc-systems/HPCC-Platform.git

Where [ ] denotes an optional argument.

CMake

The minimum version of Cmake required to build the HPCC Platform is 3.3.2 on Linux. If your system does not supply an adequate version of cmake in its repositories, you may download it here at cmake.org.

When building with CMake, it is recommended to create a separate directory for the built files. You can create it either inside the source tree or outside, depends much on personal preference. I'll assume you're creating it outside. If your source is under, say, ~/hpcc/src, create a build directory outside of the tree at ~/hpcc/build.

Now you need to run CMake to populate your build directory with Makefiles and special configuration to build HPCC, packages, tests, etc.

For release builds, do:

cd ~/hpcc/build
cmake ../src

For Plugins, do:

Make sure the build directory is cleared, target the normal platform
# cmake –D<Plugin Name>=ON ../HPCC-Platform
# make –j6 package
*these are the allowable names for plugin names;
CASSANDRAEMBED
REMBED
V8EMBED
MEMCACHED
PYEMBED
REDIS
JAVAEMBED
KAFKA
SQLITE3EMBED
MYSQLEMBED

If testing during development and you want to include plugins (except R) in the package, do: cd ~/hpcc/build cmake -DTEST_PLUGINS=ON ../src

If you're running a debug build, do:

cd ~/hpcc/build
cmake -DCMAKE_BUILD_TYPE:STRING=Debug ../src

To build the client tools only, do:

cd ~/hpcc/build
cmake -DCLIENTTOOLS_ONLY=1 ../src

To enable signing of the ecl standard library, insure you have a gpg private key loaded into your gpg keychain and do:

cd ~/hpcc/build
# Add -DSIGN_MODULES_KEYID and -DSIGN_MODULES_PASSPHRASE if applicable
cmake -DSIGN_MODULES=ON ../src

In some cases, users have found that when packaging for ubuntu, the dpkg-shlibdeps portion of the packaging adds an exceptional amount of time to the build process. To turn this off (and to create a package without dynamic dependency generation) do:

cd ~/hpcc/build
cmake -DUSE_SHLIBDEPS=OFF ../src

CMake will check for necessary dependencies like binutils, boost regex, cppunit, pthreads, etc. If everything is correct, it'll create the necessary Makefiles and you're ready to build HPCC.

####NOTE: We default to using libxslt in place of Xalan for xslt support. Should you prefer to use libxalan, you can specify -DUSE_LIBXALAN on the cmake command line.

Building

It doesn't take too long to compile HPCC, but it's highly recommended that you use all your CPUs to do so. When building complex projects (where I/O plays an important part), it's often recommended to use 50% more threads than the number of CPUs you have.

So, on a quad-core machine, the command you should type is:

make -j6

Alternatively you can call a build system agnostic variant (works with make, ninja, XCode, Visual Studio etc.)

cmake --build .

That will make all binaries, libraries and scripts necessary to create the package.

Creating a package

The recommended way to install HPCC in your machine (even for testing) is to use distro packages. CMake has already detected your system, so it know whether to generate TGZ files, DEP or RPM packages.

Just type:

make package

Alternatively you can use the build system agnostic variant:

cmake --build . --target package

and it'll create the appropriate package for you to install. The package will be inside the build directory.

In my system (Ubuntu 11.04), it created a package called hpccsystems-platform-community_3.3.0-1trunk-Debugnatty_amd64.deb.

Installing the package

The HPCC package has dependencies built in, and your system might not have them all (you can have the source package in order to build, but not the library package installed by your package manager, for example).

Install the package:

sudo dpkg -i hpccsystems-platform-community_6.0.0-trunk0trusty_amd64.deb

(note that the name of the package you just built will depend on the branch you checked out, the distro, and other options).

Now fix any missing dependencies:

sudo apt-get -f install

(see here for Ubuntu based installation).

Starting the services

If your installation succeeded, you should see the service in /etc/init.d (or /etc/rc.d/init.d on RPM based distros) called hpcc-init. This service follows the standard Linux service interface, where you can use commands like:

sudo /etc/init.d/hpcc-init status
sudo /etc/init.d/hpcc-init start
sudo /etc/init.d/hpcc-init stop

Use hpcc-init start and make sure there are no error messages. If everything went ok, check that the cluster is up and running by visiting the admin page on your browser.

To create a development installation on Linux

Building a development installation for quick startup and debugging is very similar to the normal build process. The only major difference are the flags passed to the invocation of cmake to build your make files, and instead of doing a make package you run make install and make configure. There are two key cmake variables that must be set, and some optional variables depending on the environment you're building in.

DESTDIR=/home/user/path
RUNTIME_USER=<my_user>
RUNTIME_GROUP=<my_alternate_group> (optional)
HOME_DIR=/path/to/alternate/home/directory (optional)

Destdir is used for the new root installation path of your build. For instance if you set DESTDIR=/home/user/runtime, then everything will be laid down there as though it was root. Then in order to start the platform, you simply need to call the hpcc-init script directly from /home/user/runtime/etc/init.d/hpcc-init. It is important that you do not set the DESTDIR environment variable in your shell. Allow cmake to handle DESTDIR, not the shell environment.

Runtime User is generally your user name. This will allow you to run the entire platform as your user instead of as root. Running as root will mess up your runtime environment as certain files will be owned by root on creation instead of ${USER}. Never run as root when doing a development build.

Runtime group is optional and is for environments like OSx where your Runtime User and Runtime Group aren't both set to your user name.

The Home directory is a useful variable if you have the home path for your environment set to a non-standard location such as /mnt/disk1/home/my_user. It is important to remember that the DESTDIR should most likely match this path in most situations. If the HOME_DIR is standard, this will be done by default for you. Otherwise look to manipulate this variable.

Example:

cmake -DDESTDIR=$HOME/runtime -DRUNTIME_USER=$USER ../HPCC-Platform/
make install #instead of make package, will also include configure target

TODO

  • How to cross compile to other architectures
  • More Windows details
You can’t perform that action at this time.