Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
MongoDB plugin for Fluentd event collector
Ruby

This branch is 102 commits behind fluent:master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
bin
lib/fluent/plugin
test
vendor
.gitignore
.gitmodules
.travis.yml
AUTHORS
Gemfile
README.rdoc
Rakefile
VERSION
fluent-plugin-mongo.gemspec

README.rdoc

MongoDB plugin for Fluent

Component

MongoOutput

Store Fluentd event as a MongoDB Document to MongoDB database.

MongoOutput set “time” field to a document by default. You set false to include_time_key parameter if you disable this behaivor.

tag mapped mode

Tag mapped to MongoDB collection automatically.

MongoReplSetOutput

Replica Set version of MongoOutput

MongoBackupOutput

Store fluent-event to local capped collection for backup.

MongoTailInput

Tail capped collection to input data.

Configuration

MongoOutput

<match mongo.**>
  type mongo
  database fluent
  collection test

  # Following attibutes are optional
  host fluenter
  port 10000

  # Set 'capped' if you want to use capped collection
  capped
  capped_size 100m

  # Other buffer configurations here
</match>

Tag mapped mode

Use tag_mapped parameter.

If tag name is “foo.bar”, auto create collection “foo.bar” and insert data.

<match forward.*>
  type mongo
  database fluent

  # Set 'tag_mapped' if you want to use tag mapped mode.
  tag_mapped

  # If tag is "forward.foo.bar", then prefix "forward." is removed.
  # Collection name to insert is "foo.bar".
  remove_tag_prefix forward.

  # This configuration is used if tag not found. Default is 'untagged'.
  collection misc

  # Other configurations here
</match>

NOTE

Broken data as a BSON

Fluentd event sometimes has an invalid record as a BSON. In such case, Mongo plugin marshals an invalid record using Marshal.dump and re-inserts its to same collection.

If passed following invalid record:

{"key1": "invalid value", "key2": "valid value", "time": ISODate("2012-01-15T21:09:53Z") }

then Mongo plugin converts this record to following format:

{"__broken_data": Marshal.dump result of {"key1": "invalid value", "key2": "valid value"}, "time": ISODate("2012-01-15T21:09:53Z") }

Mongo-Ruby-Driver cannot detect an invalid attribute, so Mongo plugin marshals all attributes excluding Fluentd keys(“tag_key” and “time_key”).

If you want to ignore an invalid record, set true to ignore_invalid_record parameter in match.

<match forward.*>
  ...

  # ignore invalid documents at write operation
  ignore_invalid_document true

  ...
</match>
Buffer size limitation

Mongo plugin has the limitation of buffer size. Because MongoDB and mongo-ruby-driver checks the total object size at each insertion. If total object size gets over the size limitation, then MongoDB returns error or mongo-ruby-driver raises an exception.

So, Mongo plugin resets buffer_chunk_limit if configurated value is larger than above limitation:

  • Before v1.8, max of buffer_chunk_limit is 2MB

  • After v1.8, max of buffer_chunk_limit is 10MB

MongoReplSetOutput

Use mongo_replset type in match.

<match mongo.**>
  type mongo_replset
  database fluent
  collection logs

  # each node separated by ','
  nodes localhost:27017,localhost:27018,localhost:27019

  # num_retries is threshold at failover, default is 60.
  # If retry count reached this threshold, mongo plugin raises an exception.
  num_retries 30

  # following optional parameters passed to ReplSetConnection of mongo-ruby-driver.
  # See mongo-ruby-driver docs for more detail.
  #name replset_name
  #read secondary
  #refresh_mode sync
  #refresh_interval 60
</match>

MongoBackupOutput

Use mongo_backup type in match. mongo_backup alwalys use capped collection.

<match ...>
  type mongo_backup
  capped_size 100m

  <store>
    type tcp
    host 192.168.0.13
    ...
  </store>
</match>

Tail capped collection

Use mongo_tail type in source.

<source>
  type mongo_tail
  database fluent
  collection capped_log

  tag app.mongo_log

  # Convert 'time'(BSON's time) to fluent time(Unix time).
  time_key time

  # You can store last ObjectId to tail over server's shutdown
  id_store_file /Users/repeatedly/devel/fluent-plugin-mongo/last_id
</source>

TODO

More configuration

  • Authentication

  • Select insert or update

  • etc

Tool

You can tail mongo capped collection.

mongo-tail -f

Copyright

Copyright

Copyright © 2011- Masahiro Nakagawa

License

Apache License, Version 2.0

Something went wrong with that request. Please try again.