Skip to content
Amazon S3 output plugin for Fluentd event collector
Ruby
Pull request Compare This branch is 253 commits behind fluent:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
lib/fluent/plugin
test
AUTHORS
ChangeLog
Gemfile
README.rdoc
Rakefile
VERSION
fluent-plugin-s3.gemspec

README.rdoc

Amazon S3 output plugin for Fluent event collector

Overview

s3 output plugin buffers event logs in local file and upload it to S3 periodically.

This plugin splits files exactly by using the time of event logs (not the time when the logs are received). For example, a log '2011-01-02 message B' is reached, and then another log '2011-01-03 message B' is reached in this order, the former one is stored in “20110102.gz” file, and latter one in “20110103.gz” file.

Installation

Simply use RubyGems:

gem install fluent-plugin-s3

Configuration

<match pattern>
  type s3

  aws_key_id YOUR_AWS_KEY_ID
  aws_sec_key YOUR_AWS_SECRET/KEY
  s3_bucket YOUR_S3_BUCKET_NAME
  s3_endpoint s3-ap-northeast-1.amazonaws.com
  s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
  path logs/
  buffer_path /var/log/fluent/s3

  time_slice_format %Y%m%d-%H
  time_slice_wait 10m
  utc
</match>
aws_key_id (required)

AWS access key id.

aws_sec_key (required)

AWS secret key.

s3_bucket (required)

S3 bucket name.

s3_endpoint

s3 endpoint name. Example, Tokyo region is “s3-ap-northeast-1.amazonaws.com”.

s3_object_key_format

The format of S3 object keys. You can use several built-in variables:

  • %{path}

  • %{time_slice}

  • %{index}

  • %{file_extension}

to decide keys dynamically.

%{path} is exactly the value of path configured in the configuration file. E.g., “logs/” in the example configuration above. %{time_slice} is the time-slice in text that are formatted with time_slice_format. %{index} is the sequential number starts from 0, increments when multiple files are uploaded to S3 in the same time slice. %{file_extention} is always “gz” for now.

The default format is “%{path}%{time_slice}_%{index}.%{file_extension}”.

For instance, using the example configuration above, actual object keys on S3 will be something like:

"logs/20130111-22_0.gz"
"logs/20130111-23_0.gz"
"logs/20130111-23_1.gz"
"logs/20130112-00_0.gz"

With the configuration:

s3_object_key_format %{path}/events/ts=%{time_slice}/events_%{index}.%{file_extension}
path log
time_slice_format %Y%m%d-%H

You get:

"log/events/ts=20130111-22/events_0.gz"
"log/events/ts=20130111-23/events_0.gz"
"log/events/ts=20130111-23/events_1.gz"
"log/events/ts=20130112-00/events_0.gz"

The fluent-mixin-config-placeholders mixin is also incorporated, so additional variables such as %{hostname}, %{uuid}, etc. can be used in the s3_object_key_format. This could prove useful in preventing filename conflicts when writing from multiple servers.

s3_object_key_format %{path}/events/ts=%{time_slice}/events_%{index}-%{hostname}.%{file_extension}
path

path prefix of the files on S3. Default is “” (no prefix).

buffer_path (required)

path prefix of the files to buffer logs.

time_slice_format

Format of the time used as the file name. Default is '%Y%m%d'. Use '%Y%m%d%H' to split files hourly.

time_slice_wait

The time to wait old logs. Default is 10 minutes. Specify larger value if old logs may reache.

utc

Use UTC instead of local time.

Copyright

Copyright

Copyright © 2011 Sadayuki Furuhashi

License

Apache License, Version 2.0

Something went wrong with that request. Please try again.