Skip to content

Building Elasticsearch

Jump to bottom
linuxonz edited this page Jan 18, 2024 · 61 revisions

Building Elasticsearch

Below versions of Elasticsearch are available in respective distributions at the time of creation of these build instructions.

The instructions provided below specify the steps to build Elasticsearch 8.11.1 on Linux on IBM Z for following distributions:

  • RHEL (7.8, 7.9, 8.6, 8.8, 8.9, 9.0, 9.2, 9.3)
  • SLES (12 SP5, 15 SP5)
  • Ubuntu (20.04, 22.04)

Prerequisites:

  • Docker packages are provided for RHEL, SLES and Ubuntu in their respective repositories. More information about Docker CE can be found here

  • Ensure the current user belongs to group docker:

    Use the below command to add group docker if it does not exist:

    sudo groupadd docker

    Use the below command to add current user to group docker if it has not been done:

    sudo usermod -aG docker $USER && newgrp docker

  • Docker Compose is required for some elasticsearch integration tests. Docker Compose should be installed on all platforms where it is available if the integration tests are going to be run.

General Notes

  • When following the steps below please use a standard permission user unless otherwise specified.
  • A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.

Build and Install Elasticsearch

1) Build using script

If you want to build Elasticsearch using manual steps, go to STEP 2.

Use the following commands to build Elasticsearch using the build script. Please make sure you have wget installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Elasticsearch/8.11.1/build_elasticsearch.sh

# Build Elasticsearch
bash build_elasticsearch.sh  [Provide -t option for executing build with tests]

If the build completes successfully, go to STEP 9. In case of error, check logs for more details or go to STEP 2 to follow manual build steps.

2) Install build dependencies

export SOURCE_ROOT=/<source_root>/
export GRADLE_USER_HOME=$SOURCE_ROOT/.gradle

  • RHEL (7.8, 7.9)

    sudo yum install -y curl git gzip tar wget patch make gcc gcc-c++ rh-git227 fontconfig urw-fonts
source /opt/rh/rh-git227/enable

  • RHEL (8.6, 8.8, 8.9, 9.0, 9.2, 9.3)

    sudo yum install -y curl git gzip tar wget patch make gcc gcc-c++

  • SLES 12 SP5

    sudo zypper install -y curl libnghttp2-devel git gzip tar wget patch make gcc gcc-c++

  • SLES 15 SP5

    sudo zypper install -y curl git gzip tar wget patch make gcc gcc-c++

  • Ubuntu (20.04, 22.04)

    sudo apt-get update
sudo apt-get install -y curl git gzip tar wget patch locales make gcc g++
sudo locale-gen en_US.UTF-8

3) Install Java

  • With Eclipse Adoptium Temurin Runtime (previously known as AdoptOpenJDK hotspot)

    • Download and install Eclipse Adoptium Temurin Runtime (Java 17) from here.

  • With OpenJDK 17

    • RHEL (8.6, 8.8, 8.9, 9.0, 9.2, 9.3) 
      sudo yum install -y java-17-openjdk-devel
    • SLES 15 SP5 
      sudo zypper install -y java-17-openjdk java-17-openjdk-devel
    • Ubuntu (20.04, 22.04) 
      sudo apt-get install -y openjdk-17-jdk

    If running tests with OpenJDK 17, create a symbolic link to the java home directory to work around a gradle problem where gradle does not recognize the distro provided OpenJDK

    sudo mkdir -p /opt/java/
sudo ln -s <Path to JDK> /opt/java/jdk

Note: At the time of creation of these build instructions, Elasticsearch was verified with Eclipse Adoptium Temurin Runtime (Java 17) version (build 17.0.7+9) and OpenJDK 17.

4) Set the environment variables

export LANG="en_US.UTF-8"
export JAVA_HOME=<Path to JDK>
export ES_JAVA_HOME=/<Path to JDK>/
export JAVA17_HOME=/<Path to JDK>/
export PATH=$ES_JAVA_HOME/bin:$PATH

Note: Ensure system locale is set up correctly for Elasticsearch to build without encoding errors.

5) Build Prerequite Projects

5.1) JANSI v2.4.0

cd $SOURCE_ROOT
git clone https://github.com/fusesource/jansi.git
cd jansi
git checkout jansi-2.4.0
make clean-native native OS_NAME=Linux OS_ARCH=s390x

mkdir -p $SOURCE_ROOT/jansi-jar
cd $SOURCE_ROOT/jansi-jar
wget https://repo1.maven.org/maven2/org/fusesource/jansi/jansi/2.4.0/jansi-2.4.0.jar
jar xvf jansi-2.4.0.jar
cd org/fusesource/jansi/internal/native/Linux
mkdir s390x
cp $SOURCE_ROOT/jansi/target/native-Linux-s390x/libjansi.so s390x/
cd $SOURCE_ROOT/jansi-jar
jar cvf jansi-2.4.0.jar .

mkdir -p $SOURCE_ROOT/.gradle/caches/modules-2/files-2.1/org.fusesource.jansi/jansi/2.4.0/321c614f85f1dea6bb08c1817c60d53b7f3552fd/
cp jansi-2.4.0.jar $SOURCE_ROOT/.gradle/caches/modules-2/files-2.1/org.fusesource.jansi/jansi/2.4.0/321c614f85f1dea6bb08c1817c60d53b7f3552fd/
export sha256=$(sha256sum jansi-2.4.0.jar | awk '{print $1}')

5.2) Docker Images for osixia/openldap

cd $SOURCE_ROOT
export PATCH_URL="https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Elasticsearch/8.11.1/patch"
git clone https://github.com/osixia/docker-light-baseimage.git
cd docker-light-baseimage/
git checkout v1.2.0
curl -sSL "${PATCH_URL}/docker-light-baseimage.patch" | git apply -
make build
cd $SOURCE_ROOT
git clone https://github.com/osixia/docker-openldap.git
cd docker-openldap/
git checkout v1.4.0
curl -sSL "${PATCH_URL}/docker-openldap.patch" | git apply -
make build

6) Download Elasticsearch and apply patches

cd $SOURCE_ROOT
git clone https://github.com/elastic/elasticsearch
cd elasticsearch
git checkout v8.11.1
  • Apply gradle patches to create s390x distribution
export PATCH_URL="https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Elasticsearch/8.11.1/patch/elasticsearch.patch"
curl -o elasticsearch.patch $PATCH_URL
git apply elasticsearch.patch
sed -i 's|6cd91991323dd7b2fb28ca93d7ac12af5a86a2f53279e2b35827b30313fd0b9f|'"${sha256}"'|g' $SOURCE_ROOT/elasticsearch/gradle/verification-metadata.xml

Since the latest Elasticsearch version has added docker-export files in their repo, we also need to do the same for s390x. So create the following directories -

mkdir -p $SOURCE_ROOT/elasticsearch/distribution/docker/ubi-docker-s390x-export/
mkdir -p $SOURCE_ROOT/elasticsearch/distribution/docker/cloud-docker-s390x-export/
mkdir -p $SOURCE_ROOT/elasticsearch/distribution/docker/cloud-ess-docker-s390x-export/
mkdir -p $SOURCE_ROOT/elasticsearch/distribution/docker/docker-s390x-export/
mkdir -p $SOURCE_ROOT/elasticsearch/distribution/docker/ironbank-docker-s390x-export/

Inside each directory create a build.gradle file with the following content -

// This file is intentionally blank. All configuration of the
// export is done in the parent project.

7) Build

cd $SOURCE_ROOT/elasticsearch
CPU_NUM="$(grep -c ^processor /proc/cpuinfo)"
export GRADLE_USER_HOME=$SOURCE_ROOT/.gradle
./gradlew :distribution:archives:linux-s390x-tar:assemble --max-workers="$CPU_NUM"  --parallel

8) Testing (Optional)

cd $SOURCE_ROOT/elasticsearch
export JAVA_TOOL_OPTIONS="-Dfile.encoding=UTF8"
export JAVA_HOME=/<Path to JDK>/
export RUNTIME_JAVA_HOME=/<Path to JDK>/ # Only with Temurin 17
export RUNTIME_JAVA_HOME=/opt/java/jdk   # Only with OpenJDK 17
./gradlew --continue test internalClusterTest -Dtests.haltonfailure=false -Dtests.jvm.argline="-Xss2m"

Notes:

  • Environment Variables needed for rerunning tests have been added to $HOME/setenv.sh. To set the Environment Variables needed to rerun tests, please run: source $HOME/setenv.sh
  • On RHEL 8 and SLES with OpenJDK, you may need to change the system crypto policy with a command like: 'sudo update-crypto-policies --set LEGACY' in order for all security tests to pass.
  • On RHEL 9 with OpenJDK, some security tests will fail even with the LEGACY crypto policy.
  • If there is an stack overflow error, increase the -Xss arg value in the above command.
  • Some X-Pack test cases will fail as X-Pack plugins are not supported on s390x, such as Machine Learning features.
  • The node.processors setting is now bounded by the number of available processors. Some X-Pack Test case may fail for this reason. To fix this, ensure the value of node.processors setting does not exceed the number of available processors.
  • For more information regarding Elasticsearch testing, please refer to their testing documentation.
  • User can also create distributions as deb, rpm and docker using below commands.
./gradlew :distribution:packages:s390x-deb:assemble
./gradlew :distribution:packages:s390x-rpm:assemble
./gradlew :distribution:docker:docker-s390x-export:assemble

9) Install Elasticsearch

cd $SOURCE_ROOT/elasticsearch
sudo mkdir /usr/share/elasticsearch
sudo tar -xzf distribution/archives/linux-s390x-tar/build/distributions/elasticsearch-8.11.1-SNAPSHOT-linux-s390x.tar.gz -C /usr/share/elasticsearch --strip-components 1
sudo ln -sf /usr/share/elasticsearch/bin/* /usr/bin/

sudo /usr/sbin/groupadd elastic
sudo chown [username]:elastic -R /usr/share/elasticsearch/
  • Update configurations to disable unsupported xpack.ml
sudo echo 'xpack.ml.enabled: false' >> /usr/share/elasticsearch/config/elasticsearch.yml

10) Verify Elasticsearch Server

> elasticsearch --version
Version: 8.11.1-SNAPSHOT, Build: tar/6f9ff581fbcde658e6f69d6ce03050f060d1fd0c/2024-01-03T14:34:33.852328359Z, JVM: 17.0.9

11) Start Elasticsearch Server

elasticsearch &

Use curl --cacert /usr/share/elasticsearch/config/certs/http_ca.crt -u elastic https://localhost:9200 when running Elasticsearch. The output should be similar to this:

{
  "name" : "es01.example.com",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "2eVHcuOrSOeNIgLjuD4z4w",
  "version" : {
    "number" : "8.11.1-SNAPSHOT",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "6f9ff581fbcde658e6f69d6ce03050f060d1fd0c",
    "build_date" : "2024-01-03T14:34:33.852328359Z",
    "build_snapshot" : true,
    "lucene_version" : "9.8.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

Notes:

  • The password for the elastic user can be found in the output of the elasticsearch & command above.
  • On RHEL 9 with OpenJDK 17, it might be necessary to add the command line options --tls-max 1.1 --ciphers DEFAULT@SECLEVEL=0 to the curl command above.

References:

The information provided in this article is accurate at the time of writing, but on-going development in the open-source projects involved may make the information incorrect or obsolete. Please open issue or contact us on IBM Z Community if you have any questions or feedback.

Clone this wiki locally