Amazon DynamoDB output plugin for Fluent event collector
Ruby
Pull request Compare This branch is 18 commits ahead, 16 commits behind gonsuke:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
lib/fluent/plugin
test
AUTHORS
ChangeLog
Gemfile
README.md
Rakefile
VERSION
fluent-plugin-dynamodb-atomiccounter.gemspec

README.md

Amazon DynamoDB AtomicCounter output plugin

Installation

$ fluent-gem install fluent-plugin-dynamodb-atomiccounter

Configuration

DynamoDB

First of all, you need to create a table in DynamoDB. It's easy to create via Management Console.

Specify table name, hash attribute name and throughput as you like. fluent-plugin-dynamodb_atomiccounter will load your table schema and write event-stream out to your table.

currently supports only table with a primary key which has a string hash-key. (hash and range key is not supported.)

Fluentd

<match dynamodb.**>
  type dynamodb_atomiccounter
  aws_key_id AWS_ACCESS_KEY
  aws_sec_key AWS_SECRET_ACCESS_KEY
  proxy_uri http://user:password@192.168.0.250:3128/
  dynamo_db_endpoint dynamodb.ap-northeast-1.amazonaws.com
  dynamo_db_table access_log
  count_key path
</match>
  • aws_key_id (required) - AWS access key id.
  • aws_sec_key (required) - AWS secret key.
  • proxy_uri (optional) - your proxy url.
  • dynamo_db_endpoint (required) - end point of dynamodb. see Regions and Endpoints
  • dynamo_db_table (required) - table name of dynamodb.
  • count_key - key to be counted. default path.

TIPS

retrieving data

id (Hash Key) counts
"/" "1458"
"/sample.html" "847"
"/about/" "373"

multi-region redundancy

As you know fluentd has copy output plugin. So you can easily setup multi-region redundancy as follows.

<match dynamo.**>
  type copy
  <store>
    type dynamodb_atomiccounter
    aws_key_id AWS_ACCESS_KEY
    aws_sec_key AWS_SECRET_ACCESS_KEY
    dynamo_db_table test
    dynamo_db_endpoint dynamodb.ap-northeast-1.amazonaws.com
    tag dynamo.count
    count_key path
    flush_interval 10s
  </store>
  <store>
    type dynamodb_atomiccounter
    aws_key_id AWS_ACCESS_KEY
    aws_sec_key AWS_SECRET_ACCESS_KEY
    dynamo_db_table test
    dynamo_db_endpoint dynamodb.ap-southeast-1.amazonaws.com
    tag dynamo.count
    count_key path
    flush_interval 10s
  </store>
</match>

TODO

  • test

Copyright

CopyrightCopyright (c) 2013 - Keisuke Kadoyama
LicenseApache License, Version 2.0