ES + Kibana + Logstash + FileBeat

[tmfdk333]

CentOS Linux release 7.4.1708, ELK 6.5.4, Ncloud(navercorp)

ES + Kibana + Logstash + FileBeat

[0] Download ES, Kibana, Logstash, Filebeat

• Java 1.8.0_131 or a later version

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.5.4.tar.gz
tar -xvzf elasticsearch-6.5.4.tar.gz
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.5.4-linux-x86_64.tar.gz
tar -xvzf kibana-6.5.4-linux-x86_64.tar.gz
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.5.4.tar.gz
tar -xvzf logstash-6.5.4.tar.gz
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.5.4-linux-x86_64.tar.gz
tar -xvzf filebeat-6.5.4-linux-x86_64.tar.gz

• Apache Web Log 실습 파일

wget https://s3.ap-northeast-2.amazonaws.com/kr.elastic.co/sample-data/weblog-sample.log.zip
unzip weblog-sample.log.zip

[1] Run Elasticsearch

cd elasticsearch-6.5.4

• Starting Elasticsearch: -d running as daemon, -p record the process id

echo 'bin/elasticsearch -d -p es.pid' > start.sh
echo 'kill `cat es.pid`' > stop.sh
chmod 755 start.sh stop.sh

• 스크립트 실행 및 프로세스 확인
  • 클러스터 구성 시 config/elasticsearch.yml에서 network.host 변경
  • By default, Elasticsearch binds to loopback addresses only

./start.sh
ps -ef | grep elasticsearch
curl localhost:9200

[2] Run Kibana

cd kibana-6.5.4-linux-x86_64
mkdir log

• Configuring Kibana
  • logging.dest specify a file where Kibana stores log output
  • pid.file specifies the path where Kibana creates the process ID file

# config/kibana.yml
pid.file: /home1/irteam/kibana-6.5.4-linux-x86_64/kibana.pid
logging.dest: /home1/irteam/kibana-6.5.4-linux-x86_64/log/kibana.log

• Starting and stopping Kibana

echo 'bin/kibana &' > start.sh
echo 'kill `cat kibana.pid`' > stop.sh
chmod 755 start.sh stop.sh

• 스크립트 실행 및 프로세스 확인
  • 웹 브라우저에서 http://localhost:5601 로 접속 가능
    • 외부에서 접속할 경우 config/kibana.yml에서 server.host 변경

./start.sh
ps -ef | grep `cat kibana.pid`
curl localhost:5601

[3] Run Logstash

cd logstash-6.5.4

• logstash.yml - Reloading the Config File

# config/logstash.yml
config.reload.automatic: true

• Output plugins
  • Stdout output plugin
  • Elasticsearch output plugin
• Filter plugins
  • Grok filter plugin(Logstash Grok Patterns) - parse unstructured log data into structured
  • Geoip filter plugin - adds information about the geographical location of IP addresses
  • Useragent filter plugin - Parse user agent strings into structured data
  • Mutate filter plugin - perform general mutations on fields
  • Date filter plugin

# Create weblog.conf
input {
    tcp {
        port => 9900
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    geoip {
        source => "clientip"
    }
    useragent {
        source => "agent"
        target => "useragent"
    }
    mutate {
        convert => {
            "bytes" => "integer"
        }
    }
    date {
        match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
}
output {
    # stdout { codec => "rubydebug" }
    elasticsearch { }
}

• Running Logstash from the Command Line - weblog.conf 파일을 설정으로 사용

bin/logstash -f weblog.conf &

• input/output 테스트 - netcat

# Apache Web Log 실습 데이터
echo '14.49.42.25 - - [12/Mar/2015:01:24:44 +0000] "GET /articles/ppp-over-ssh/ HTTP/1.1" 200 18586 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2b1) Gecko/20091014 Firefox/3.6b1 GTB5"' | nc localhost 9900

• Kibana(localhost:5601) > Dev Tools에서 GET logstash-*/_search로 입력된 데이터 확인

[4] Run Filebeat

cd filebeat-6.5.4-linux-x86_64

• Configure Filebeat: path Define the path (or paths) to your log files

# filebeat.yml
  enabled: true
  path: 
    - /home1/irteam/weblog-sample.log

• Start Filebeat: -e 콘솔에 메세지 출력, -c 다른 경로의 filebeat.yml 사용 가능

./filebeat &

• kibana(localhost:5601) > Dev Tools에서 GET filebeat-*/_search로 입력된 데이터 확인(total 30000)

[5] Filebeat → Logstash → Elasticsearch

• 파일 출력을 elasticsearch에서 logstash로 변경

# filebeat.yml
#output.elasticsearch:
#  hosts: ["localhost:9200"]

output.logstash:
  hosts: ["localhost:5044"] # 두칸 띄어쓰기

• 파일 입력을 tcp에서 beats로 변경

# weblog.conf 
input {
    #tcp { port => 9900 }
    beats {
        port => 5044
    }
}

• Run Filebeat(Filebeat는 한번 읽은 데이터는 다시 읽지 않기 때문에 data/registry 파일을 삭제해야 함)

rm data/registry
./filebeat &

• Kibana(localhost:5601) > Dev Tools에서 GET logstash-*/_search로 입력된 데이터 확인

[6] Reference

• Seoul Elasticsearch Community Meetup - Elastic Stack Hands On, Jongmin Kim