MongoDB plugin for Fluentd event collector
Ruby
Pull request Compare This branch is 206 commits behind fluent:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
bin
lib/fluent/plugin
test
vendor
.gitignore
.gitmodules
.travis.yml
AUTHORS
Gemfile
README.rdoc
Rakefile
VERSION
fluent-plugin-mongo.gemspec

README.rdoc

MongoDB plugin for Fluent

Component

MongoOutput

Store Fluentd event as a MongoDB Document to MongoDB database.

MongoOutput set “time” field to a document by default. You set false to include_time_key parameter if you disable this behaivor.

tag mapped mode

Tag mapped to MongoDB collection automatically.

MongoReplSetOutput

Replica Set version of MongoOutput

MongoBackupOutput

Store fluent-event to local capped collection for backup.

MongoTailInput

Tail capped collection to input data.

Configuration

MongoOutput

<match mongo.**>
  type mongo
  database fluent
  collection test

  # Following attibutes are optional
  host fluenter
  port 10000

  # Set 'capped' if you want to use capped collection
  capped
  capped_size 100m

  # Other buffer configurations here
</match>

Tag mapped mode

Use tag_mapped parameter.

If tag name is “foo.bar”, auto create collection “foo.bar” and insert data.

<match forward.*>
  type mongo
  database fluent

  # Set 'tag_mapped' if you want to use tag mapped mode.
  tag_mapped

  # If tag is "forward.foo.bar", then prefix "forward." is removed.
  # Collection name to insert is "foo.bar".
  remove_tag_prefix forward.

  # This configuration is used if tag not found. Default is 'untagged'.
  collection misc

  # Other configurations here
</match>

NOTE

Broken data as a BSON

Fluentd event sometimes has an invalid record as a BSON. In such case, Mongo plugin marshals an invalid record using Marshal.dump and re-inserts its to same collection.

If passed following invalid record:

{"key1": "invalid value", "key2": "valid value", "time": ISODate("2012-01-15T21:09:53Z") }

then Mongo plugin converts this record to following format:

{"__broken_data": Marshal.dump result of {"key1": "invalid value", "key2": "valid value"}, "time": ISODate("2012-01-15T21:09:53Z") }

Mongo-Ruby-Driver cannot detect an invalid attribute, so Mongo plugin marshals all attributes excluding Fluentd keys(“tag_key” and “time_key”).

If you want to ignore an invalid record, set true to ignore_invalid_record parameter in match.

<match forward.*>
  ...

  # ignore invalid documents at write operation
  ignore_invalid_document true

  ...
</match>
Buffer size limitation

Mongo plugin has the limitation of buffer size. Because MongoDB and mongo-ruby-driver checks the total object size at each insertion. If total object size gets over the size limitation, then MongoDB returns error or mongo-ruby-driver raises an exception.

So, Mongo plugin resets buffer_chunk_limit if configurated value is larger than above limitation:

  • Before v1.8, max of buffer_chunk_limit is 2MB

  • After v1.8, max of buffer_chunk_limit is 10MB

MongoReplSetOutput

Use mongo_replset type in match.

<match mongo.**>
  type mongo_replset
  database fluent
  collection logs

  # each node separated by ','
  nodes localhost:27017,localhost:27018,localhost:27019

  # num_retries is threshold at failover, default is 60.
  # If retry count reached this threshold, mongo plugin raises an exception.
  num_retries 30

  # following optional parameters passed to ReplSetConnection of mongo-ruby-driver.
  # See mongo-ruby-driver docs for more detail.
  #name replset_name
  #read secondary
  #refresh_mode sync
  #refresh_interval 60
</match>

MongoBackupOutput

Use mongo_backup type in match. mongo_backup alwalys use capped collection.

<match ...>
  type mongo_backup
  capped_size 100m

  <store>
    type tcp
    host 192.168.0.13
    ...
  </store>
</match>

Tail capped collection

Use mongo_tail type in source.

<source>
  type mongo_tail
  database fluent
  collection capped_log

  tag app.mongo_log

  # Convert 'time'(BSON's time) to fluent time(Unix time).
  time_key time

  # You can store last ObjectId to tail over server's shutdown
  id_store_file /Users/repeatedly/devel/fluent-plugin-mongo/last_id
</source>

TODO

More configuration

  • Authentication

  • Select insert or update

  • etc

Tool

You can tail mongo capped collection.

mongo-tail -f

Copyright

Copyright

Copyright © 2011- Masahiro Nakagawa

License

Apache License, Version 2.0