This is a part of output_forward config:
<buffer>
@type file
flush_mode interval
flush_interval 10s
retry_type periodic
retry_wait 10s
retry_timeout 10d
total_limit_size 500Mb
path /var/lib/td-agent/buffers/output_forward
overflow_action drop_oldest_chunk
</buffer>
# du -sh /var/lib/td-agent/buffers/output_forward/
226M /var/lib/td-agent/buffers/output_forward/
# tail /var/log/td-agent/td-agent.log
2018-04-28 08:34:18 +0000 [warn]: #0 dropping oldest chunk to make space after buffer overflow chunk_id="V\xAEGA\x1E=\xF2\x15kXr\xA7`\x1FE5"
2018-04-28 08:34:24 +0000 [warn]: #0 failed to write data into buffer by buffer overflow action=:drop_oldest_chunk
2018-04-28 08:34:24 +0000 [warn]: #0 dropping oldest chunk to make space after buffer overflow chunk_id="V\xAEGZ8ht\xA9Z\x1D\x8E,\xE2\xD9C\xCB"
2018-04-28 08:34:28 +0000 [warn]: #0 failed to write data into buffer by buffer overflow action=:drop_oldest_chunk
2018-04-28 08:34:28 +0000 [warn]: #0 dropping oldest chunk to make space after buffer overflow chunk_id="V\xAEG_\xACV\xABO\x9CQ\xC1\xAF\xB17@_"
2018-04-28 08:34:28 +0000 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many
data" location="/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/plugin/buffer.rb:264:in `write'" tag="syslog.daemon.info"
2018-04-28 08:34:28 +0000 [warn]: #0 failed to write data into buffer by buffer overflow action=:drop_oldest_chunk
2018-04-28 08:34:28 +0000 [warn]: #0 suppressed same stacktrace
2018-04-28 08:34:28 +0000 [warn]: #0 dropping oldest chunk to make space after buffer overflow chunk_id="V\xAEG`]\x94J\xD0\x05\x93\xF7\x97\x96,\x00\xA
B"
# date
Sat Apr 28 08:34:37 UTC 2018
After I changed Mb to MB it started to work fine.
At the same time total_limit_size 60Gb in output_elasticsearch buffer config works as expected.
2018-04-19 17:47:37 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '2.4.0'
2018-04-19 17:47:37 +0000 [info]: gem 'fluent-plugin-flowcounter' version '1.3.0'
2018-04-19 17:47:37 +0000 [info]: gem 'fluent-plugin-kafka' version '0.6.5'
2018-04-19 17:47:37 +0000 [info]: gem 'fluent-plugin-multiprocess' version '0.2.2'
2018-04-19 17:47:37 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.0.1'
2018-04-19 17:47:37 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.0.1'
2018-04-19 17:47:37 +0000 [info]: gem 'fluent-plugin-s3' version '1.1.0'
2018-04-19 17:47:37 +0000 [info]: gem 'fluent-plugin-td' version '1.0.0'
2018-04-19 17:47:37 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.3'
2018-04-19 17:47:37 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.2.2'
2018-04-19 17:47:37 +0000 [info]: gem 'fluentd' version '1.0.2'
This is a part of output_forward config:
After I changed Mb to MB it started to work fine.
At the same time total_limit_size 60Gb in output_elasticsearch buffer config works as expected.