buffer overflow - buffer space has too many data #591
Description
Problem
Upgraded from td-agent-3.1.1-0.el7.x86_64 to td-agent-3.4.1-0.el7.x86_64 and got the following errors (sample below):
2019-05-21 17:24:13 +0100 [warn]: #0 failed to flush the buffer. retry_time=1 next_retry_seconds=2019-05-21 17:24:14 +0100 chunk="5896845a863dd9ce7f71ab19064d0571" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logsto Elasticsearch cluster ({:host=>\"10.10.16.165\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}, {:host=>\"10.10.16.164\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}, {:host=>\"10.10.16.169\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): end of file reached (EOFError)"
2019-05-21 17:24:44 +0100 [warn]: #0 failed to flush the buffer. retry_time=6 next_retry_seconds=2019-05-21 17:25:17 +0100 chunk="5896845a863dd9ce7f71ab19064d0571" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logsto Elasticsearch cluster ({:host=>\"10.10.16.165\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}, {:host=>\"10.10.16.164\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}, {:host=>\"10.10.16.169\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): Connection reset by peer (Errno::ECONNRESET)"
2019-05-21 18:32:51 +0100 [warn]: #0 failed to write data into buffer by buffer overflow action=:throw_exception
2019-05-21 18:32:51 +0100 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:298:in `write'" tag="host.messages"
It initially seemed the upgrade was OK as it appeared to be running OK but after a couple of hours the buffer hockey sticks from under a 1 MB to over 500MB:
Before the upgrade the buffer was mostly under 1 MB and never over 2MB.
Steps to replicate
Install td-agent-3.4.1-0.el7.x86_64 and configure with:
<match **>
@type copy
<store>
@type elasticsearch_dynamic
hosts 10.10.16.165:9200,10.10.16.164:9200,10.10.16.169:9200
scheme https
logstash_format true
logstash_prefix proj.${tag_parts[0]}.${tag_parts[1]}
logstash_dateformat %Y%m%d
include_tag_key true
tag_key @log_name
user "*********"
password "**********"
ca_file "/etc/pki/ca-trust/source/anchors/CA.crt"
flush_interval "5s"
</store>
</match>
Expected Behavior or What you need to ask
Investigating this issue but expected the buffer to remain the same and not overflow.
...
Using Fluentd and ES plugin versions
- OS version
- Redhat r7.6-4
- Bare Metal/VM/Container
- VM
- Fluentd v0.12 or v0.14/v1.0
- td-agent 1.4.2
- ES plugin 3.x.y/2.x.y or 1.x.y
2019-05-22 10:18:54 +0100 [info]: gem 'fluent-plugin-concat' version '2.2.0'
2019-05-22 10:18:54 +0100 [info]: gem 'fluent-plugin-elasticsearch' version '3.5.1'
2019-05-22 10:18:54 +0100 [info]: gem 'fluent-plugin-kafka' version '0.9.4'
2019-05-22 10:18:54 +0100 [info]: gem 'fluent-plugin-prometheus' version '1.4.0'
2019-05-22 10:18:54 +0100 [info]: gem 'fluent-plugin-record-modifier' version '2.0.1'
2019-05-22 10:18:54 +0100 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.2.0'
2019-05-22 10:18:54 +0100 [info]: gem 'fluent-plugin-s3' version '1.1.10'
2019-05-22 10:18:54 +0100 [info]: gem 'fluent-plugin-secure-forward' version '0.4.5'
2019-05-22 10:18:54 +0100 [info]: gem 'fluent-plugin-td' version '1.0.0'
2019-05-22 10:18:54 +0100 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.4'
2019-05-22 10:18:54 +0100 [info]: gem 'fluent-plugin-webhdfs' version '1.2.3'
2019-05-22 10:18:54 +0100 [info]: gem 'fluentd' version '1.4.2'
- ES version (optional)
6..6.1