Skip to content
No description, website, or topics provided.
Shell
Branch: master
Clone or download

Latest commit

Fetching latest commit…
Cannot retrieve the latest commit at this time.

Files

Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
configuration
README.md
docker-compose.yml

README.md

The following is a model of setting up Cassandra logging for an existing ELK and Cassandra cluster.

provision cluster

Do the following from within the directory holding the docker-compose.yml file related to this repo:

docker-compose up -d

Do all of the following on each C* node:

install filebeat on each node

apt update
apt install curl -y
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.3-amd64.deb
dpkg -i filebeat-6.2.3-amd64.deb

for other operating systems, see: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation.html

create configuration file

edited from example at: https://github.com/thelastpickle/docker-cassandra-bootstrap/blob/master/cassandra/config/filebeat.yml

Put the following in /etc/filebeat/filebeat.yml.

output.elasticsearch:
    enabled: true
    hosts: ["172.16.238.31:9200"]
setup.kibana:
  host: "172.16.238.33:5601"
filebeat.prospectors:
    - input_type: log
      paths:
        - "/var/log/cassandra/system.log*"
      document_type: cassandra_system_logs
      exclude_files: ['\.zip$']
      multiline.pattern: '^TRACE|DEBUG|WARN|INFO|ERROR'
      multiline.negate: true
      multiline.match: after
    - input_type: log
      paths:
        - "/var/log/cassandra/gc.log.*.current"
      document_type: cassandra_gc_logs
      exclude_files: ['\.zip$']
      multiline.pattern: '^TRACE|DEBUG|WARN|INFO|ERROR'
      multiline.negate: true
      multiline.match: after

set up kibana dashboards

filebeat setup --dashboards

output: Loaded dashboards

start the beat

service filebeat start

Example output:

2018-04-12T20:43:03.798Z    INFO    instance/beat.go:468    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2018-04-12T20:43:03.799Z    INFO    instance/beat.go:475    Beat UUID: 2f43562f-985b-49fc-b229-83535149c52b
2018-04-12T20:43:03.800Z    INFO    instance/beat.go:213    Setup Beat: filebeat; Version: 6.2.3
2018-04-12T20:43:03.801Z    INFO    elasticsearch/client.go:145 Elasticsearch url: http://172.16.238.31:9200
2018-04-12T20:43:03.802Z    INFO    pipeline/module.go:76   Beat name: C1
Config OK

run some test data

cassandra-stress write n=20000 -pop seq=1..20000 -rate threads=4

view the graphs

Open this in a browser: http://localhost:5601

Sample queries:

  • message:WARN*
  • message:(ERROR* OR WARN*)
  • message:(ERROR* OR WARN*) AND beat.hostname:DC1C2

Can also filter display by choosing from available fields on left.

You can’t perform that action at this time.