Skip to content

Fluentd

Jump to bottom
Ashish Kumar edited this page Mar 23, 2016 · 17 revisions

Fluentd is data Collector for unified logging layer . Fluentd decouples data sources from backend systems by providing a unified logging layer in between. This layer allows developers and data analysts to utilize many types of logs as they are generated. Just as importantly, it mitigates the risk of "bad data" slowing down and misinforming your organization.A unified logging layer lets you and your organization make better use of data and iterate more quickly on your software.

Fluentd is written in Ruby for flexibility, with performance sensitive parts written in C

Fluentd is Apache 2.0 Licensed, fully open source software

Below are the steps to build fluentd on linux-on-power

Linux Distro : RHEL 7.1 PPC64LE
fluentd version : 0.12.20

a) Install below dependencies
sudo yum install git ruby gcc gcc-c++ kernel-devel ruby-devel openssl make tzdata
sudo yum install yum install libffi*

b) Build Ruby 2.3.0
https://github.com/conaku/linux-on-power/wiki/Ruby

c) Setup ruby and gem paths
export GEM_HOME=/tempdisk/software/ruby/ruby-2.3.0
export PATH=$PATH:$GEM_HOME/bin

d) Build and install latest fluentd gem and dependencies
$ gem install fluentd

e) validate the installation
list the pre-configured gems :
$ gem list fluentd

f) Build and install fluent-plugin-webhdfs
[root@sys-77402 fluentd]# gem install fluent-plugin-webhdfs
Building native extensions. This could take a while...
Successfully installed ffi-1.9.10
Fetching: bzip2-ffi-1.0.0.gem (100%)
Successfully installed bzip2-ffi-1.0.0
Fetching: fluent-plugin-webhdfs-0.5.1.gem (100%)
Successfully installed fluent-plugin-webhdfs-0.5.1
Parsing documentation for ffi-1.9.10
Installing ri documentation for ffi-1.9.10
Parsing documentation for bzip2-ffi-1.0.0
Installing ri documentation for bzip2-ffi-1.0.0
Parsing documentation for fluent-plugin-webhdfs-0.5.1
Installing ri documentation for fluent-plugin-webhdfs-0.5.1
Done installing documentation for ffi, bzip2-ffi, fluent-plugin-webhdfs after 10 seconds
3 gems installed
[root@sys-77402 fluentd]#

h) Build and install zookeeper gem

  • Initiate "gem install zookeeper"
    This gem install will fail and zookeeper gem will be checked out locally for investigation
    at location "/tempdisk/software/ruby/ruby-2.3.0/gems"

  • make below changes
    a) File : /tempdisk/software/ruby/ruby-2.3.0/gems/zookeeper-1.4.11/ext/zkc-3.4.5/c/config.guess
    Add below entries below ppc64 entries
    ppc64le:Linux:*:*)
    echo powerpc64le-unknown-linux-gnu
    exit ;;

  • build local zookeeper
    cd /tempdisk/software/ruby/ruby-2.3.0/gems/zookeeper-1.4.11/ext/zkc-3.4.5/c/src
    ./configure
    make
    make install

  • create a local tar of zookeeper build and create a ruby gem file and install local built gem
    cd /tempdisk/software/ruby/ruby-2.3.0/gems/zookeeper-1.4.11
    gem build zookeeper.gemspec
    gem install
    gem install zookeeper-1.4.11.gem

g) Build and install fluent-plugin-kafka
Initially the gem install fluent-plugin-kafka will fail with following error while installing zookeeper gem dependency . In order to resolve this dependency , we would need to fix the zookeeper build locally and install zookeeper gem .

Error
If the version you run (./config.guess) is already up to date, please
send the following data and any information you think might be
pertinent to config-patches@gnu.org in order to provide the needed
information to handle your system.
config.guess timestamp = 2005-07-08
uname -m = ppc64le
uname -r = 3.10.0-229.ael7b.ppc64le
uname -s = Linux
uname -v = #1 SMP Fri Jan 30 12:03:50 EST 2015

/usr/bin/uname -p = ppc64le
_/bin/uname -X = _
_hostinfo = _
_/bin/universe = _
_/usr/bin/arch -k = _
/bin/arch = ppc64le
_/usr/bin/oslevel = _
_/usr/convex/getsysinfo = _
UNAME_MACHINE = ppc64le
UNAME_RELEASE = 3.10.0-229.ael7b.ppc64le
UNAME_SYSTEM = Linux
UNAME_VERSION = #1 SMP Fri Jan 30 12:03:50 EST 2015
configure: error: cannot guess build type; you must specify one
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers. Check the mkmf.log file for more details. You may
need configuration options.

once the above zookeeper build is fixed locally and zookeeper gem is installed then reissue the gem install of fluent-plugin-kafka
[root@sys-77402 ~]# gem install fluent-plugin-kafka Fetching: zk-1.9.6.gem (100%) Successfully installed zk-1.9.6 Fetching: poseidon_cluster-0.3.3.gem (100%) Successfully installed poseidon_cluster-0.3.3 Fetching: fluent-plugin-kafka-0.1.3.gem (100%) Successfully installed fluent-plugin-kafka-0.1.3 Parsing documentation for zk-1.9.6 Installing ri documentation for zk-1.9.6 Parsing documentation for poseidon_cluster-0.3.3 Installing ri documentation for poseidon_cluster-0.3.3 Parsing documentation for fluent-plugin-kafka-0.1.3 Installing ri documentation for fluent-plugin-kafka-0.1.3 Done installing documentation for zk, poseidon_cluster, fluent-plugin-kafka after 1 seconds 3 gems installed

Clone this wiki locally