fluent-plugin-bufferize, a plugin for Fluentd
An adapter plugin which enables existing non-buffered plugins to resend messages easily in case of unexpected exceptions without creating duplicate messages.
Why
Buffered plugin accumulates many messages in buffer and sends all the messages at a same time. There are many APIs that does not support such reqeusts like bulk-insert. In that case, you have to use non-buffered output and implement resend meschanism yourself because non-buffered output lacks exception handling functionality.
To use this plugin, you just have to create non-buffered plugin without caring exception handling. If an exception happens in your plugin, the request is issued again automatically. With file buffer, none of messages are lost even on sudden fluentd process down.
Configuration
Just embrace existing configuration by directive.
If you use following configuration:
<match *>
type http
endpoint_url http://foo.bar.com/
http_method put
</match>
Modify it like this:
<match *>
type bufferize
buffer_type file
buffer_path /var/log/fluent/myapp.*.buffer
<config>
type http
endpoint_url http://foo.bar.com/
http_method put
</config>
</match>
This is a buffered output plugin. For more information about parameters, please refer official document.
Example of application
These plugins are good compatibility to fluent-plugin-bufferize.
Installation
Add this line to your application's Gemfile:
gem 'fluent-plugin-bufferize'
And then execute:
$ bundle
Or install it yourself as:
$ gem install fluent-plugin-bufferize
Contributing
- Fork it
- Create your feature branch (
git checkout -b my-new-feature) - Commit your changes (
git commit -am 'Add some feature') - Push to the branch (
git push origin my-new-feature) - Create new Pull Request
Copyright
| Author | Masahiro Sano |
| Copyright | Copyright (c) 2013- Masahiro Sano |
| License | MIT License |