Building Beats
The instructions provided below specify the steps to build Beats 8.12.2 on Linux on IBM Z for the following distributions:
- RHEL (7.8, 7.9, 8.6, 8.8, 8.9, 9.0, 9.2, 9.3)
- SLES (12 SP5, 15 SP5)
- Ubuntu (20.04, 22.04)
General Notes:
-
When following the steps below please use a standard permission user unless otherwise specified.
-
A directory
/<source_root>/
will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.
If you want to build Beats using manual steps, go to STEP 2.
Use the following commands to build Beats using the build script. Please make sure you have wget installed.
wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Beats/8.12.2/build_beats.sh
# Build Beats
bash build_beats.sh [Provide -t option for executing build with tests]
If the build completes successfully, go to STEP 4. In case of error, check logs
for more details or go to STEP 2 to follow manual build steps.
export SOURCE_ROOT=/<source_root>/
-
RHEL (7.8, 7.9)
sudo subscription-manager repos --enable=rhel-7-server-for-system-z-rhscl-rpms sudo yum install -y git curl make wget tar devtoolset-7-gcc-c++ devtoolset-7-gcc libpcap libpcap-devel which acl zlib-devel patch systemd-devel libjpeg-develbzip2-devel gcc gcc-c++ gdbm-devel libdb-devel libffi-devel libuuid-devel ncurses-devel readline-devel sqlite-devel tk-devel xz xz-devel source /opt/rh/devtoolset-7/enable sudo ln -f -s /opt/rh/devtoolset-7/root/usr/bin/s390x-redhat-linux-gcc /opt/rh/devtoolset-7/root/usr/bin/s390x-linux-gnu-gcc sudo ln -f -s /opt/rh/devtoolset-7/root/usr/bin/s390x-redhat-linux-c++ /opt/rh/devtoolset-7/root/usr/bin/s390x-linux-gnu-c++ sudo ln -f -s /opt/rh/devtoolset-7/root/usr/bin/s390x-redhat-linux-g++ /opt/rh/devtoolset-7/root/usr/bin/s390x-linux-gnu-g++
-
RHEL (8.6, 8.8, 8.9)
sudo yum install -y git curl make wget tar gcc gcc-c++ libpcap-devel openssl openssl-devel which acl zlib-devel patch systemd-devel libjpeg-devel python3.11 python3.11-devel bzip2-devel gdbm-devel libdb-devel libffi-devel libuuid-devel ncurses-devel readline-devel sqlite-devel tk-devel xz xz-devel
-
RHEL (9.0, 9.2, 9.3)
sudo yum install -y git curl make wget tar gcc gcc-c++ libpcap-devel openssl openssl-devel which acl zlib-devel patch systemd-devel libjpeg-devel python3.11 python3.11-devel bzip2-devel gdbm-devel libdb-devel libffi-devel libuuid-devel ncurses-devel readline-devel sqlite-devel tk-devel xz xz-devel
-
SLES 12 SP5
sudo zypper install -y git curl gawk make wget tar gcc7 gcc7-c++ libpcap1 libpcap-devel libffi48-devel acl patch libsystemd0 systemd-devel libjpeg62-devel gdbm-devel libbz2-devel libdb-4_8-devel libuuid-devel ncurses-devel readline-devel sqlite3-devel tk-devel xz-devel zlib-devel gzip libnghttp2-devel sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-7 100 sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-7 100 sudo update-alternatives --install /usr/bin/cpp cpp /usr/bin/cpp-7 100 sudo ln -f -s /usr/bin/gcc /usr/bin/cc sudo ln -f -s /usr/bin/g++ /usr/bin/c++
-
SLES 15 SP5
sudo zypper install -y git curl gawk make wget tar gcc gcc-c++ libpcap libpcap-devel acl patch libsystemd0 systemd-devel libjpeg62-devel openssl libopenssl-devel zlib-devel gzip gdbm-devel libbz2-devel libdb-4_8-devel libffi-devel libnsl-devel libuuid-devel ncurses-devel readline-devel sqlite3-devel tk xz-devel timezone
-
Ubuntu (20.04, 22.04)
sudo apt-get update sudo DEBIAN_FRONTEND=noninteractive apt-get install -y git curl make wget tar gcc g++ libcap-dev libpcap0.8-dev openssl libssh-dev acl rsync tzdata patch fdclone libsystemd-dev libjpeg-dev libffi-dev libbz2-dev libdb-dev libgdbm-dev liblzma-dev libncurses-dev libreadline-dev libsqlite3-dev libssl-dev tk-dev uuid-dev xz-utils zlib1g-dev
OPENSSL_VERSION="1.1.1s"
wget -q --no-check-certificate https://www.openssl.org/source/openssl-${OPENSSL_VERSION}.tar.gz
tar -xzf openssl-${OPENSSL_VERSION}.tar.gz
cd openssl-${OPENSSL_VERSION}
./config --prefix=/usr/local --openssldir=/usr/local
make
sudo make install
sudo ldconfig /usr/local/lib64
export PATH=/usr/local/bin:$PATH
export LDFLAGS="-L/usr/local/lib/ -L/usr/local/lib64/"
export LD_LIBRARY_PATH="/usr/local/lib/:/usr/local/lib64/"
export CPPFLAGS="-I/usr/local/include/ -I/usr/local/include/openssl"
export PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig:$PKG_CONFIG_PATH
-
Install
Python 3.11.4
Python >= 3.7 is required for running tests. Install
Python 3.11.4
using the following script:wget -q https://www.python.org/ftp/python/3.11.4/Python-3.11.4.tgz tar -xzf Python-3.11.4.tgz cd Python-3.11.4 ./configure --prefix=/usr/local --exec-prefix=/usr/local make sudo make install export PATH=/usr/local/bin:$PATH
Set the newly installed Python as default.
sudo ln -fs /usr/bin/python3 /usr/bin/python # for RHEL 7.x sudo update-alternatives --install /usr/bin/python python /usr/local/bin/python3.11 10 # except for RHEL 7.x and RHEL 8.x sudo update-alternatives --install /usr/local/bin/python3 python3 /usr/bin/python3.11 10 # for RHEL 9.x sudo update-alternatives --install /usr/bin/python3 python3 /usr/local/bin/python3.11 10 # except for RHEL 9.x sudo update-alternatives --display python3 python3 -V
-
Install
pip
(for Ubuntu, RHEL 8.x, RHEL 9.x and SLES)wget --no-check-certificate https://bootstrap.pypa.io/get-pip.py python3 get-pip.py rm get-pip.py
-
Install below dependencies
pip3 install wheel -v pip3 install "cython<3.0.0" pyyaml==5.4.1 --no-build-isolation -v
-
Install
rustc
cd $SOURCE_ROOT wget --no-check-certificate -O rustup-init.sh https://sh.rustup.rs bash rustup-init.sh -y export PATH=$PATH:$HOME/.cargo/bin rustup toolchain install 1.76.0 rustup default 1.76.0
Note: Upgrading Python on RHEL (7.8, 7.9) breaks system tools like yum. So we will switch back to original Python after running tests.
cd $SOURCE_ROOT
wget --no-check-certificate https://go.dev/dl/go1.22.0.linux-s390x.tar.gz
chmod ugo+r go1.22.0.linux-s390x.tar.gz
sudo tar -C /usr/local -xzf go1.22.0.linux-s390x.tar.gz
export PATH=$PATH:/usr/local/go/bin
sudo ln /usr/bin/gcc /usr/bin/s390x-linux-gnu-gcc # For RHEL & SLES
go version
export GOPATH=$(go env GOPATH)
mkdir -p $GOPATH
export PATH=$GOPATH/bin:$PATH
The default permissions of files in the GOPATH
directory should not have write access for group
or others
. Check default permissions using the below commands
cd $GOPATH
touch test && ls -la test && rm test
If group
or others
have write access to this test file, then we have to change the default permissions using ACL.
sudo setfacl -dm u::rwx,g::r,o::r $GOPATH
cd $GOPATH
touch test && ls -la test && rm test
If you get an error like setfacl: Operation not supported
. ACL might be disabled on your filesystem. To enable ACL refer the below links:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-access_control_lists (RHEL)
https://help.ubuntu.com/community/FilePermissionsACLs (Ubuntu)
https://documentation.suse.com/sles/12-SP5/html/SLES-all/cha-security-acls.html (SLES)
Please note that ACL cannot be enabled or modified from NFS client.
export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=true
export PYTHON_EXE=python3
export PYTHON_ENV=/tmp/venv3
export PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig:$PKG_CONFIG_PATH # for RHEL 7.x and SLES 12.5
mkdir -p $GOPATH/src/github.com/elastic
cd $GOPATH/src/github.com/elastic
git clone -b v8.12.2 https://github.com/elastic/beats.git
cd beats
curl -sSL https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Beats/8.12.2/patch/metricbeat.patch | git apply
Currently Beats available for linux are: filebeat, packetbeat, metricbeat, heartbeat and auditbeat.
Notes: Not all distros listed above are supported by each Beat, see https://www.elastic.co/support/matrix.
3.2.1) To compile, test and generate configuration files for a particular Beat use the commands as shown below
- For
filebeat
,packetbeat
,heartbeat
andauditbeat
cd $GOPATH/src/github.com/elastic/beats/<beat_name>
make
make update
make fmt
make unit
make system-tests
- For
metricbeat
cd $GOPATH/src/github.com/elastic/beats/metricbeat
mage build
mage update
mage fmt
mage test
- Adding to
/usr/bin
sudo cp "./<beat_name>" /usr/bin/
For example, for Packetbeat
:
cd $GOPATH/src/github.com/elastic/beats/packetbeat
make packetbeat
make update
make fmt
make unit
make system-tests
- Adding
Packetbeat
to/usr/bin
sudo cp "./packetbeat" /usr/bin/
Notes:
- We are skipping integration-test as it uses dockerfiles and docker images specific to x86.
sudo /usr/sbin/update-alternatives --remove python /usr/local/bin/python3.11
sudo /usr/sbin/update-alternatives --display python
sudo ln -fs /usr/bin/python2 /usr/bin/python
yum info python
To run a Beat use the command as shown below.
cd $GOPATH/src/github.com/elastic/beats/<beat_name>
sudo ./<beat_name> -e -c <beat_name>.yml -d "publish"
For example, for Packetbeat
:
cd $GOPATH/src/github.com/elastic/beats/packetbeat
sudo ./packetbeat -e -c packetbeat.yml -d "publish"
Notes:
If you get an error like CRIT Exiting: error loading config file: config file ("<beat_name>.yml") must be owned by the beat user (uid=0) or root
,either change the owner of the yml file to root user
sudo chown root <beat_name>.yml
sudo ./<beat_name> -e -c <beat_name>.yml -d "publish"
or disable strict permission and ownership check from commandline
sudo ./<beat_name> -strict.perms=false -e -c <beat_name>.yml -d "publish"
The information provided in this article is accurate at the time of writing, but on-going development in the open-source projects involved may make the information incorrect or obsolete. Please open issue or contact us on IBM Z Community if you have any questions or feedback.