Amazon DynamoDB output plugin for Fluent event collector
Ruby
Latest commit 1fb8f3a Jun 14, 2017 @gonsuke committed on GitHub Merge pull request #22 from cosmo0920/migrate-v0.14
Migrate to use v0.14 API
Permalink
Failed to load latest commit information.
lib/fluent/plugin
test Migrate to use v0.14 API Jun 12, 2017
AUTHORS added AUTHORS. Jun 11, 2012
ChangeLog bump version 0.1.8 Jul 10, 2012
Gemfile first commit. Jun 8, 2012
README.md Use multi worker feature instead of DetachMultiProcessMixin Jun 12, 2017
Rakefile
VERSION
fluent-plugin-dynamodb.gemspec Depends on FLuentd v0.14.15 or later Jun 12, 2017

README.md

Amazon DynamoDB output plugin for Fluentd event collector

##Installation

$ fluent-gem install fluent-plugin-dynamodb

##Configuration

###DynamoDB

First of all, you need to create a table in DynamoDB. It's easy to create via Management Console.

Specify table name, hash attribute name and throughput as you like. fluent-plugin-dynamodb will load your table schema and write event-stream out to your table.

Fluentd

<match dynamodb.**>
  @type dynamodb
  aws_key_id AWS_ACCESS_KEY
  aws_sec_key AWS_SECRET_ACCESS_KEY
  proxy_uri http://user:password@192.168.0.250:3128/
  dynamo_db_endpoint dynamodb.ap-northeast-1.amazonaws.com
  dynamo_db_table access_log
</match>
  • aws_key_id (optional) - AWS access key id. This parameter is required when your agent is not running on EC2 instance with an IAM Instance Profile.
  • aws_sec_key (optional) - AWS secret key. This parameter is required when your agent is not running on EC2 instance with an IAM Instance Profile.
  • proxy_uri (optional) - your proxy url.
  • dynamo_db_endpoint (required) - end point of dynamodb. see Regions and Endpoints
  • dynamo_db_table (required) - table name of dynamodb.

##TIPS

###retrieving data

fluent-plugin-dynamo will add time attribute and any other attributes of record automatically. For example if you read apache's access log via fluentd, structure of the table will have been like this.

id (Hash Key) time host path method referer code agent size
"a937f980-b304-11e1-bc96-c82a14fffef2" "2012-06-10T05:26:46Z" "192.168.0.3" "/index.html" "GET" "-" "200" "Mozilla/5.0" "4286"
"a87fc51e-b308-11e1-ba0f-5855caf50759" "2012-06-10T05:28:23Z" "192.168.0.4" "/sample.html" "GET" "-" "200" "Mozilla/5.0" "8933"

Item can be retrieved by the key, but fluent-plugin-dynamo uses UUID as a primary key. There is no simple way to retrieve logs you want. By the way, you can write scan-filter with AWS SDK like this, but Hive on EMR is the best practice I think.

###multiprocessing

If you need high throughput and if you have much provisioned throughput and abudant buffer, you can setup multiprocessing. fluent-plugin-dynamodb uses multi workers, so you can launch 6 workers as follows.

<match dynamodb.**>
  @type dynamodb
  aws_key_id AWS_ACCESS_KEY
  aws_sec_key AWS_SECRET_ACCESS_KEY
  proxy_uri http://user:password@192.168.0.250:3128/
  dynamo_db_endpoint dynamodb.ap-northeast-1.amazonaws.com
  dynamo_db_table access_log
</match>
<system>
  workers 6
</system>

###multi-region redundancy

As you know fluentd has copy output plugin. So you can easily setup multi-region redundancy as follows.

<match dynamo.**>
  @type copy
  <store>
    @type dynamodb
    aws_key_id AWS_ACCESS_KEY
    aws_sec_key AWS_SECRET_ACCESS_KEY
    dynamo_db_table test
    dynamo_db_endpoint dynamodb.ap-northeast-1.amazonaws.com
  </store>
  <store>
    @type dynamodb
    aws_key_id AWS_ACCESS_KEY
    aws_sec_key AWS_SECRET_ACCESS_KEY
    dynamo_db_table test
    dynamo_db_endpoint dynamodb.ap-southeast-1.amazonaws.com
  </store>
</match>

##TODO

  • auto-create table
  • tag_mapped

##Copyright

CopyrightCopyright (c) 2012- Takashi Matsuno
LicenseApache License, Version 2.0