retry_limit times problem

[loveshell]

hi ,i tested type forward retry_limit ,i found default is 17.but when i test it's 5~6 times.why

[kzk]

I believe the log says the time of next retry. Could you give me your last 100 lines of your logs?

[loveshell]

2012-12-19 10:49:44 +0800: fluent/engine.rb:58:configure: using configuration file:

type tail_ex format apache path /data/logs/weblog/**.log tag ff.file pos_file /tmp/pos_file type forward host 1.1.1.1 port 5666 retry_limit 9 send_timeout 60s recover_wait 10s heartbeat_interval 1s phi_threshold 8 hard_timeout 60s flush_interval 1 2012-12-19 10:49:44 +0800: fluent/engine.rb:67:block in configure: adding source type="tail_ex" 2012-12-19 10:49:44 +0800: fluent/plugin.rb:89:register_impl: registered input plugin 'tail_ex' 2012-12-19 10:49:44 +0800: fluent/engine.rb:83:block in configure: adding match pattern="**" type="forward" 2012-12-19 10:49:44 +0800: plugin/1.rb:49:configure: 'host' option in forward output is obsoleted. Use ' host xxx ' instead. 2012-12-19 10:49:44 +0800: plugin/1.rb:79:block in configure: adding forwarding server '1.1.1.1:5666' host="1.1.1.1" port=5666 weight=60 2012-12-19 10:49:44 +0800: plugin/1.rb:165:rebuild_weight_array: rebuilding weight array lost_weight=0 2012-12-19 10:49:44 +0800: plugin/in_tail.rb:279:initialize: following tail of /data/logs/weblog/access.log 2012-12-19 10:49:46 +0800: fluent/output.rb:336:rescue in try_flush: temporarily failed to flush the buffer, next retry will be at 2012-12-19 10:49:47 +0800. error="Connection refused - connect(2)" instance=10751060 2012-12-19 10:49:46 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:238:in `initialize' 2012-12-19 10:49:46 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:238:in`new' 2012-12-19 10:49:46 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:238:in `connect' 2012-12-19 10:49:46 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:200:in`send_data' 2012-12-19 10:49:46 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:136:in `block in write_objects' 2012-12-19 10:49:46 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:130:in`times' 2012-12-19 10:49:46 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:130:in `write_objects' 2012-12-19 10:49:46 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/output.rb:439:in`write' 2012-12-19 10:49:46 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/buffer.rb:279:in `write_chunk' 2012-12-19 10:49:46 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/buffer.rb:263:in`pop' 2012-12-19 10:49:46 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/output.rb:303:in `try_flush' 2012-12-19 10:49:46 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/output.rb:120:in`run' 2012-12-19 10:49:48 +0800: fluent/output.rb:336:rescue in try_flush: temporarily failed to flush the buffer, next retry will be at 2012-12-19 10:49:49 +0800. error="Connection refused - connect(2)" instance=10751060 2012-12-19 10:49:48 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:238:in `initialize' 2012-12-19 10:49:48 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:238:in`new' 2012-12-19 10:49:48 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:238:in `connect' 2012-12-19 10:49:48 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:200:in`send_data' 2012-12-19 10:49:48 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:136:in `block in write_objects' 2012-12-19 10:49:48 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:130:in`times' 2012-12-19 10:49:48 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:130:in `write_objects' 2012-12-19 10:49:48 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/output.rb:439:in`write' 2012-12-19 10:49:48 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/buffer.rb:279:in `write_chunk' 2012-12-19 10:49:48 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/buffer.rb:263:in`pop' 2012-12-19 10:49:48 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/output.rb:303:in `try_flush' 2012-12-19 10:49:48 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/output.rb:120:in`run' 2012-12-19 10:49:51 +0800: fluent/output.rb:336:rescue in try_flush: temporarily failed to flush the buffer, next retry will be at 2012-12-19 10:49:53 +0800. error="Connection refused - connect(2)" instance=10751060 2012-12-19 10:49:51 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:238:in `initialize' 2012-12-19 10:49:51 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:238:in`new' 2012-12-19 10:49:51 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:238:in `connect' 2012-12-19 10:49:51 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:200:in`send_data' 2012-12-19 10:49:51 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:136:in `block in write_objects' 2012-12-19 10:49:51 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:130:in`times' 2012-12-19 10:49:51 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:130:in `write_objects' 2012-12-19 10:49:51 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/output.rb:439:in`write' 2012-12-19 10:49:51 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/buffer.rb:279:in `write_chunk' 2012-12-19 10:49:51 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/buffer.rb:263:in`pop' 2012-12-19 10:49:51 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/output.rb:303:in `try_flush' 2012-12-19 10:49:51 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/output.rb:120:in`run' 2012-12-19 10:49:54 +0800: fluent/output.rb:336:rescue in try_flush: temporarily failed to flush the buffer, next retry will be at 2012-12-19 10:50:01 +0800. error="Connection refused - connect(2)" instance=10751060 2012-12-19 10:49:54 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:238:in `initialize' 2012-12-19 10:49:54 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:238:in`new' 2012-12-19 10:49:54 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:238:in `connect' 2012-12-19 10:49:54 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:200:in`send_data' 2012-12-19 10:49:54 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:136:in `block in write_objects' 2012-12-19 10:49:54 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:130:in`times' 2012-12-19 10:49:54 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:130:in `write_objects' 2012-12-19 10:49:54 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/output.rb:439:in`write' 2012-12-19 10:49:54 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/buffer.rb:279:in `write_chunk' 2012-12-19 10:49:54 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/buffer.rb:263:in`pop' 2012-12-19 10:49:54 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/output.rb:303:in `try_flush' 2012-12-19 10:49:54 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/output.rb:120:in`run' 2012-12-19 10:50:03 +0800: fluent/output.rb:336:rescue in try_flush: temporarily failed to flush the buffer, next retry will be at 2012-12-19 10:50:17 +0800. error="Connection refused - connect(2)" instance=10751060 2012-12-19 10:50:03 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:238:in `initialize' 2012-12-19 10:50:03 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:238:in`new' 2012-12-19 10:50:03 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:238:in `connect' 2012-12-19 10:50:03 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:200:in`send_data' 2012-12-19 10:50:03 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:136:in `block in write_objects' 2012-12-19 10:50:03 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:130:in`times' 2012-12-19 10:50:03 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:130:in `write_objects' 2012-12-19 10:50:03 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/output.rb:439:in`write' 2012-12-19 10:50:03 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/buffer.rb:279:in `write_chunk' 2012-12-19 10:50:03 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/buffer.rb:263:in`pop' 2012-12-19 10:50:03 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/output.rb:303:in `try_flush' 2012-12-19 10:50:03 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/output.rb:120:in`run' 2012-12-19 10:50:03 +0800: plugin/1.rb:377:tick: detached forwarding server '1.1.1.1:5666' host="1.1.1.1" port=5666 phi=8.339864547893125 2012-12-19 10:50:03 +0800: plugin/1.rb:165:rebuild_weight_array: rebuilding weight array lost_weight=60 2012-12-19 10:50:18 +0800: fluent/output.rb:336:rescue in try_flush: temporarily failed to flush the buffer, next retry will be at 2012-12-19 10:50:49 +0800. error="no nodes are available" instance=10751060 2012-12-19 10:50:18 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:148:in `write_objects' 2012-12-19 10:50:18 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/output.rb:439:in`write' 2012-12-19 10:50:18 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/buffer.rb:279:in `write_chunk' 2012-12-19 10:50:18 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/buffer.rb:263:in`pop' 2012-12-19 10:50:18 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/output.rb:303:in `try_flush' 2012-12-19 10:50:18 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/output.rb:120:in`run' 2012-12-19 10:50:50 +0800: fluent/output.rb:336:rescue in try_flush: temporarily failed to flush the buffer, next retry will be at 2012-12-19 10:51:53 +0800. error="no nodes are available" instance=10751060 2012-12-19 10:50:50 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.30/lib/fluent/plugin/1.rb:148:in `write_objects' 2012-12-19 10:50:50 +0800: fluent/output.rb:337:rescue in try_flush: /usr/lib64/fluent/r

[repeatedly]

In my environment, the number of error result is 9. It is same as your conf.
Could you show me the full logs and full conf?

[loveshell]

sorry，I was wrong。this issue can be close and very thx for your reply。

[repeatedly]

Okay. I closed this issue.