Skip to content
Amazon DynamoDB output plugin for Fluent event collector
Ruby
Find file
Pull request Compare This branch is 2 commits ahead, 18 commits behind gonsuke:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
lib/fluent/plugin
test
AUTHORS
ChangeLog
Gemfile
README.md
Rakefile
VERSION
fluent-plugin-dynamodb.gemspec

README.md

Amazon DynamoDB output plugin for Fluent event collector

Installation

$ fluent-gem install fluent-plugin-dynamodb

Configuration

DynamoDB

First of all, you need to create a table in DynamoDB. It's easy to create via Management Console.

Specify table name, hash attribute name and throughput as you like. fluent-plugin-dynamodb will load your table schema and write event-stream out to your table.

currently supports only table with a primary key which has a string hash-key. (hash and range key is not supported.)

Fluentd

<match dynamodb.**>
  type dynamodb
  aws_key_id AWS_ACCESS_KEY
  aws_sec_key AWS_SECRET_ACCESS_KEY
  proxy_uri http://user:password@192.168.0.250:3128/
  dynamo_db_endpoint dynamodb.ap-northeast-1.amazonaws.com
  dynamo_db_table access_log
</match>
  • aws_key_id (required) - AWS access key id.
  • aws_sec_key (required) - AWS secret key.
  • proxy_uri (optional) - your proxy url.
  • dynamo_db_endpoint (required) - end point of dynamodb. see Regions and Endpoints
  • dynamo_db_table (required) - table name of dynamodb.

TIPS

retrieving data

fluent-plugin-dynamo will add time attribute and any other attributes of record automatically. For example if you read apache's access log via fluentd, structure of the table will have been like this.

id (Hash Key) time host path method referer code agent size
"a937f980-b304-11e1-bc96-c82a14fffef2" "2012-06-10T05:26:46Z" "192.168.0.3" "/index.html" "GET" "-" "200" "Mozilla/5.0" "4286"
"a87fc51e-b308-11e1-ba0f-5855caf50759" "2012-06-10T05:28:23Z" "192.168.0.4" "/sample.html" "GET" "-" "200" "Mozilla/5.0" "8933"

Item can be retrieved by the key, but fluent-plugin-dynamo uses UUID as a primary key. There is no simple way to retrieve logs you want. By the way, you can write scan-filter with AWS SDK like this, but Hive on EMR is the best practice I think.

multiprocessing

If you need high throughput and if you have much provisioned throughput and abudant buffer, you can setup multiprocessing. fluent-plugin-dynamo inherits DetachMultiProcessMixin, so you can launch 6 processes as follows.

<match dynamodb.**>
  type dynamodb
  aws_key_id AWS_ACCESS_KEY
  aws_sec_key AWS_SECRET_ACCESS_KEY
  proxy_uri http://user:password@192.168.0.250:3128/
  detach_process 6
  dynamo_db_endpoint dynamodb.ap-northeast-1.amazonaws.com
  dynamo_db_table access_log
</match>

multi-region redundancy

As you know fluentd has copy output plugin. So you can easily setup multi-region redundancy as follows.

<match dynamo.**>
  type copy
  <store>
    type dynamodb
    aws_key_id AWS_ACCESS_KEY
    aws_sec_key AWS_SECRET_ACCESS_KEY
    dynamo_db_table test
    dynamo_db_endpoint dynamodb.ap-northeast-1.amazonaws.com
  </store>
  <store>
    type dynamodb
    aws_key_id AWS_ACCESS_KEY
    aws_sec_key AWS_SECRET_ACCESS_KEY
    dynamo_db_table test
    dynamo_db_endpoint dynamodb.ap-southeast-1.amazonaws.com
  </store>
</match>

TODO

  • auto-create table
  • tag_mapped

Copyright

CopyrightCopyright (c) 2012- Takashi Matsuno
LicenseApache License, Version 2.0
Something went wrong with that request. Please try again.