Amazon ELB log input plugin
Ruby
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
lib/fluent/plugin Emit the time of the ELB access rather than the time that this plugin… Aug 8, 2018
test
.gitignore
.travis.yml add codeclimate setting into travis.yml Jul 5, 2017
Gemfile first commit Mar 7, 2014
LICENSE.txt
README.md
Rakefile Hide warning messages from aws-sdk Mar 29, 2017
fluent-plugin-elb-log.gemspec increment version number Aug 10, 2018
fluent.conf.sample change sample config Jun 20, 2015

README.md

Amazon ELB log input plugin for fluentd

Gem Version Build Status Code Climate Test Coverage

Overview

  • Amazon Web Services ELB log input plubin for fluentd

Requirements

fluent-plugin-elb-log fluentd ruby
>= 0.3.0 >= v0.14.0 >= 2.1
< 0.3.0 >= v0.12.0 >= 1.9

Installation

$ fluentd-gem fluent-plugin-elb-log

AWS ELB Settings

Different from version 0.4.x

  • Using version 3 of the AWS SDK for Ruby.

Support Application Load Balancer (ver 0.4.0 or later)

When SSL certification error

log:

SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

Do env setting follows:

SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt (If you using amazon linux)

Configuration

<source>
  @type elb_log

  # following attibutes are required
  region            <region name>
  s3_bucketname     <bucketname>
  s3_prefix         <elb log's prefix>
  timestamp_file    <proc last file timestamp record filename>
  buf_file          <buffer file path>
  refresh_interval  <interval number by second>
  tag               <tag name(default: elb.access)>
  delete            <boolean delete processed log files from S3(default: false)>

  # following attibutes are required if you don't use IAM Role
  access_key_id     <access_key>
  secret_access_key <secret_access_key>
</source>

Example setting

<source>
  @type elb_log
  region            us-east-1
  s3_bucketname     my-elblog-bucket
  s3_prefix         prodcution/web
  timestamp_file    /tmp/elb_last_at.dat
  buf_file          /tmp/fluentd-elblog.tmpfile
  refresh_interval  300
  tag               elb.access
  delete            false
  access_key_id     XXXXXXXXXXXXXXXXXXXX
  secret_access_key xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
</source>

<match **>
  @type stdout
</match>

json output example

{
    "account_id":"123456789012",
    "region":"ap-northeast-1",
    "logfile_date":"2015/06/15",
    "logfile_elb_name":"my-elb-name",
    "elb_ip_address":"52.0.0.0",
    "logfile_hash":"12squv5w",
    "elb_timestamp":"20150615T0400Z",
    "key":"TEST/AWSLogs/123456789012/elasticloadbalancing/ap-northeast-1/2015/06/15/123456789012_elasticloadbalancing_ap-northeast-1_my-elb-name_20150615T0400Z_52.68.215.138_69squv5w.log",
    "prefix":"TEST",
    "elb_timestamp_unixtime":1434340800,
    "time":"2015-06-15T03:47:12.728427+0000",
    "elb":"my-elb-name",
    "client":"54.1.1.1",
    "client_port":"43759",
    "backend":"10.0.0.1",
    "backend_port":"80",
    "request_processing_time":4.0e-05,
    "backend_processing_time":0.105048,
    "response_processing_time":2.4e-05,
    "elb_status_code":"200",
    "backend_status_code":"200",
    "received_bytes":0,
    "sent_bytes":4622,
    "request_method":"GET",
    "request_uri":"https://my-elb-test.example.com/",
    "request_protocol":"HTTP/1.1",
    "user_agent":"Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)",
    "ssl_cipher":"DHE-RSA-AES128-SHA",
    "ssl_protocol":"TLSv1.2",
    "type":"http",
    "target_group_arn":"arn:aws:elasticloadbalancing:ap-northeast-1:123456789012:targetgroup/lbgrp1/605122a4e4ee9f2d",
    "trace_id":"\"Root=1-xxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxx\""
    "option3":null
}