Skip to content
Permalink
Branch: master
Find file Copy path
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
102 lines (73 sloc) 2.51 KB

Aggregate Apache Logs with fluentd plugin for MinIO Slack

In this recipe, we will learn how to use MinIO as log aggregator for fluentd using fluent-plugin-s3 plugin.

1. Prerequisites

  • Install MinIO Server from here.
  • Install mc from here

2. Installation

3. Recipe Steps

Step 1: Create a bucket.

This is the bucket where fluentd will aggregate semi-structured apache logs in real-time.

mc mb myminio/fluentd
Bucket created successfully ‘myminio/fluentd’.

Step 2: Modify the fluentd configuration to use MinIO as backend.

Replace with your own values for aws_key_id, aws_sec_key, s3_bucket, s3_endpoint.

Replace /etc/td-agent/td-agent.conf with:

<source>
  @type tail
  format apache2
  path /var/log/apache2/access.log
  pos_file /var/log/td-agent/apache2.access.log.pos
  tag s3.apache.access
</source>

<match>
  @type s3
   aws_key_id  `aws_key_id`
   aws_sec_key `aws_sec_key`
   s3_bucket `s3_bucket`
   s3_endpoint  `s3_endpoint`
   path logs/
   force_path_style true
   buffer_path /var/log/td-agent/s3
   time_slice_format %Y%m%d%H%M
   time_slice_wait 10m
   utc
   buffer_chunk_limit 256m
</match>

Step 3: Restart fluentd server.

sudo /etc/init.d/td-agent restart

Step 4: Check the fluentd logfile to confirm if everything is running.

tail -f /var/log/td-agent/td-agent.log
    path logs/
    force_path_style true
    buffer_path /var/log/td-agent/s3
    time_slice_format %Y%m%d%H%M
    time_slice_wait 10m
    utc
    buffer_chunk_limit 256m
  </match>
</ROOT>
2016-05-03 18:44:44 +0530 [info]: following tail of /var/log/apache2/access.log

Step 5: Test the configuration.

Ping the Apache server. This example uses the ab (Apache Bench) program.

ab -n 100 -c 10 http://localhost/

Step 6: Verify Aggregated Logs.

MinIO server's fluent bucket should show the aggregated logs.

mc ls myminio/fluentd/logs/
[2016-05-03 18:47:13 IST]   570B 201605031306_0.gz
[2016-05-03 18:58:14 IST]   501B 201605031317_0.gz

Note:

fleuntd should have access permission for your apache log file located at /var/log/apache2/access.log.

You can’t perform that action at this time.